# UMIonLegs: Universal Manipulation Interface on Legged Robots with Sim and
Real Data

Huy Ha∗, Yihuai Gao∗, Zipeng Fu  Jie Tan  Shuran Song

# UMI-on-Legs: Manipulation-Centric Whole Body Control with Sim-and-Real Data

Huy Ha∗, Yihuai Gao∗, Zipeng Fu  Jie Tan  Shuran Song

# Follow-Your-Hands: Manipulation-Centric Whole Body Control for Cross-
Embodiment Mobile Manipulation

Huy Ha∗, Yihuai Gao∗, Zipeng Fu  Jie Tan  Shuran Song

# Follow-Your-Hand: Scaling up Cross-embodiment Mobile Manipulation with Sim-
and-Real Data

Huy Ha∗, Yihuai Gao∗, Zipeng Fu  Jie Tan  Shuran Song

# UMI-on-Legs: Scaling up Cross-embodiment Mobile Manipulation with Sim-and-
Real Data

Huy Ha∗, Yihuai Gao∗, Zipeng Fu  Jie Tan  Shuran Song

# UMI on Legs: Making Manipulation Policies Mobile with Manipulation-Centric
Whole-body Controllers

Huy Ha∗, Yihuai Gao∗, Zipeng Fu  Jie Tan  Shuran Song

###### Abstract

We introduce UMI-on-Legs, a new framework that combines real-world and
simulation data for quadruped manipulation systems. We scale task-centric data
collection in the real world using a hand-held gripper (UMI), providing a
cheap way to demonstrate task-relevant manipulation skills without a robot.
Simultaneously, we scale robot-centric data in simulation by training whole-
body controller for task-tracking without task simulation setups. The
interface between these two policies is end-effector trajectories in the task
frame, inferred by the manipulation policy and passed to the whole-body
controller for tracking. We evaluate UMI-on-Legs on prehensile, non-
prehensile, and dynamic manipulation tasks, and report over 70% success rate
on all tasks. Lastly, we demonstrate the zero-shot cross-embodiment deployment
of a pre-trained manipulation policy checkpoint from prior work, originally
intended for a fixed-base robot arm, on our quadruped system. We believe this
framework provides a scalable path towards learning expressive manipulation
skills on dynamic robot embodiments. Please checkout our website for robot
videos, code, and data: https://umi-on-legs.github.io/

> Keywords: Manipulation, Visuo-motor policy, Whole-body Control

††$*$ Indicates equal contribution Fig 1:  UMI-on-Legs. We achieve fully
autonomous, expressive, real-world manipulation skills on quadrupeds by
combining real-world demonstrations using hand-held grippers (left) and
simulation-trained whole-body controllers (right). Our framework allows the
porting of _existing_ “table-top” manipulation policies to mobile manipulation
while enhancing mobility and power from the quadruped’s legs.

##  1 Introduction

Collecting robot data in simulation versus in the real world are two distinct
pathways for scaling up robot learning approaches, each with its challenges.
Real-world robot teleoperation allows for direct human demonstration of new
tasks. However, it is often bottlenecked by the robot’s physical hardware – in
addition to the cost and safety concerns, the involvement of robot hardware
usually makes the collected data robot-specific (e.g., uses body commands) or
even irrelevant whenever the robot’s physical form changes. In contrast,
simulation provides safe exploration and infinite resets for any new robot
design. However, it is often bottlenecked by the task diversity – accurately
simulating diverse objects and their dynamics and defining all tasks and their
associated rewards remains a significant challenge.

In this work, we introduce UMI-on-Legs, a quadruped manipulation system that
demonstrates an alternative way to combine real-world and simulation data:

  * •

Scaling task-centric data in the real world without a robot: We propose using
a hand-held gripper (i.e., UMI [1]) to demonstrate task-centric data without
requiring robot hardware. We model this manipulation data using a diffusion
policy [2] that is agnostic to the robot embodiment beyond the end-effector.
By allowing human operators to collect the data directly, we can easily
capture a diverse set of challenging tasks, bypassing the need for real-world
robot trial-and-error.

  * •

Scaling robot-centric data in simulation without task simulations: We train a
whole-body controller (WBC) in massively parallelized simulation [3, 4, 5, 6,
7], whose sim2real transfer to a wide range of terrain conditions [7, 8, 9] is
a well-studied problem. By training control policies to track end-effector
trajectories instead of interacting with objects, we can bypass the difficulty
of setting up realistic manipulation assets, engineering task rewards, and
modeling object dynamics.

Fig 2:  In-the-wild Manipulation, on Legs. Featuring robust, low-latency
iPhone odometry and onboard power/ compute, our mobile manipulation system is
complete for in-the-wild mobile manipulation.

A critical design decision in the framework regards the interface between
these two policies – it must be simple enough for non-expert users to
demonstrate yet expressive enough for complex manipulation skills. Most prior
legged manipulation systems use body velocity commands with single-step body-
frame end-effector targets [8, 10, 11, 12], which are embodiment-specific
(requires robot during data collection) and insufficient to represent complex,
dynamic manipulation trajectories. In this work, we propose to use end-
effector trajectories in the task-frame as the interface. Here, a manipulation
policy infers an end-effector trajectory from visual inputs, then passes the
trajectory to a whole body controller that infers embodiment-specific controls
for execution. This interface enables cross-embodiment generalization beyond
the gripper while providing additional preview information to the WBC for
anticipating future motions.

To support system deployment, we present an accessible, real-time odometry
solution, based on the Apple’s ARKit. Instead of assuming a tightly integrated
and precisely calibrated/synchronized system [13, 14, 15], our odometry
approach takes the self-contained, compact form factor of an iPhone.

We tested UMI-on-Legs with complex tasks requiring prehensile actions (pick &
place), non-prehensile whole-body actions (weight pushing), and dynamic
actions (tossing) in challenging scenarios with on-board sensing and in-the-
wild generalization. Thanks to our interface design, our asynchronous bi-level
policy formulation robustly executes these tasks, achieving over 70% on all
tasks. In summary, the main contributions of this work are:

  * •

UMI-on-Legs, a framework for combining real-world and simulation data for
training cross-embodiment mobile manipulation systems, which provides a
scalable path towards learning expressive manipulation skills on dynamic robot
embodiments. Namely, real-world data collected without any robots [1] can be
distilled into manipulation skills for different mobile robot platforms using
robot-specific controllers.

  * •

Within this framework, we propose a  manipulation-centric whole-body
controller, using an end-effector trajectory interface. This simple interface
enables _zero-shot cross-embodiment deployment_ of existing manipulation
policies while being expressive enough to represent complex manipulation
skills.

  * •

A real-world deployment system featuring a real-time, robust, and accessible
odometry approach for in-the-wild task-space tracking, addressing a common
bottleneck in achieving fast and precise manipulation in mobile manipulation
systems.

Fig 3:  Method Overview. Our system takes as input RGB images from a GoPro and
infers a camera-frame end-effector trajectory using a diffusion policy (a),
trained using real-world UMI demonstrations. We transform this trajectory into
the task-space, and use it as the interface to the WBC. This controller (c)
outputs joint position targets at 50Hz, which PD controllers subsequently
tracks.

##  2 Related Work

Mobile Pick-and-Place Systems. Prior works have demonstrated that pick&place
quadrupeds trained in simulation could be directly deployed in real [12, 16]
with the aid of object detectors/segmentation models for visual domain
transfer. For wheeled-base systems, more challenging mobile manipulation
skills have been demonstrated when carefully engineered manipulation
primitives are present [17, 18, 19, 20, 21], which enables perception and
planning [17, 20, 22] using foundation models [23, 24, 25, 26], online-
learning [18, 19], or trajectory optimization [21]. To alleviate the manual
design of observation and action spaces, Mobile-ALOHA [27] proposed a wheeled-
bimanual platform for demonstration collection. However, all these systems
assume quasi-static manipulation and/or heavy bases with a low center-of-mass.

Learning-based Controllers for Robot-specific Teleoperation. From learning
dynamic locomotion skills [28, 6, 29, 30] to manipulation skills [31, 32, 33],
reinforcement learning [34] (RL) with massively parallel simulators [3, 7] is
the dominant paradigm for tackling challenging control problems in legged
robots. To deploy these controllers, there exists a body of work for domain
adaptation [9, 8, 7] to varying embodiments and environmental conditions.
Towards supporting mobile-manipulation on quadrupedal systems, prior works
have also explored training whole-body controllers [8, 10, 11]. However,
beyond simplified scenarios using April-Tags [35], these prior whole-body
control quadruped systems _require human users_ to perform manipulation
(teleoperation, demonstration replay). Although fully-autonomous execution is
theoretically feasible, there are _high practical costs_ in collecting enough
data for learning effective manipulation skills. These practical challenges
stem from the dependence on robot-specific commands (e.g., body velocity
commands) and/or the requirement on the physical presence of the robot during
data collection. Concurrent and similar to our work, He et al. combines a
behavior-cloned policy and an RL controller with a task-frame trajectory
interface. However, they use the controller to heuristically generate robot-
specific demonstrations in simulation. Their algorithmic design is limited in
terms of scalability (demonstrations are robot/controller specific), and
usability/expressivity (task-specific heuristic demonstration generation).
Further, their system design is constrained by a tethered setup with an
external camera, since they rely on April Tag tracking.

Cross-embodiment Manipulation. Towards more performant, generalizable, and
robust manipulation, a large body of work on behavior cloning has innovated on
policy architectures [37, 2, 38, 39, 40, 41], sensor placements [37], action
spaces [38], data quality [37, 42], data size [43, 44, 45, 46, 47], and data
cost [48, 49, 1, 27, 48, 38]. From this well-studied space, some core design
decisions have emerged as dominant options, including the usage of pre-trained
visual encoders [25, 1, 43], a diffusion-based action decoding process [2, 1,
41, 43, 27, 49, 47], and end-effector sequence prediction [38, 1, 41, 49, 43,
47]. Further, when used with only ego-centric/wrist-mounted visual
observations, Yang et al. and Chi et al. has demonstrated zero-shot cross
embodiment visuo-motor policy transfer. However, by predicting sequences of
target poses, they assume perfect lower-level controllers capable of precisely
tracking these targets.

Fig 4:  Task- v.s. body-frame tracking. Our whole-body controller (WBC) learns
to track the target trajectory in task-frame (a), effectively compensating
base perturbations and, therefore, frees up the manipulation policy to focus
on making task progress. In contrast, most of existing WBCs use body-frame
tracking (b)[8, 12, 11, 10], trained to follow base perturbations. In effect,
they defer task-space tracking responsibilities to the low-rate manipulation
policy and fail to react quickly to body perturbations.

##  3 Method: Universal Manipulation Interface on Legged Robots

UMI-on-Legs (Fig 3) comprises of two main components: (1) a high-level
diffusion-based manipulation policy [2] which takes as input wrist-mounted
camera views and outputs sequences of end-effector pose targets into the
future in the camera frame, and (2) a low-level whole-body controller which
tracks end-effector pose targets by outputting joint position targets for both
the legs and arm. We train the manipulation policy using data collected in the
real world using UMI [1], a hand-held gripper data collection device and train
the WBC entirely in simulation using massively parallelized simulators [3].

Our choice of using task-frame end-effector trajectories as the interface has
the following benefits:

  * •

Intuitive demonstration: Using end-effector trajectories instead of robot-
specific low-level actions, we allow intuitive task demonstration by non-
expert users using hand-held devices like UMI [1].

  * •

High-level intention from preview horizon: With the preview horizon of future
targets, the whole body controller can anticipate upcoming movements. For
instance, the robots should brace accordingly if a high-velocity toss is
coming up. Meanwhile, if the target is moving within the arm’s reach range,
the body should lean instead of taking a step, which risks shaking the end
effector.

  * •

Precise and stable manipulation in the task-frame: Unlike most of legged
manipulation systems that uses body-frame tracking, our controller tracks
action in the task-space (Fig 4) that is persistent regardless of base
movement, enabling precise and stable manipulation.

  * •

Asynchronous multi-frequency execution: The interface defines a natural
inference hierarchy that allows a low-frequency manipulation policy (1-5Hz) to
coordinate with a high-frequency low-level controller (50Hz) for handling
drastically different sensor and inference latencies.

  * •

Compatible with any trajectory-based manipulation policy: Our interface
supports the plug-and-play of any trajectory-based manipulation policy [38, 1,
41, 49, 43, 47]. With the rise of policies trained on diverse datasets [46,
43, 45, 44, 47], our manipulation-centric WBC can accelerate the porting of
existing “table-top” manipulation skills to “mobile” manipulation.

###  3.1 Manipulation Policy with Behavior Cloning

Following the default configuration from Chi et al., we use a U-Net [50]
architecture diffusion policy [2] (Fig 3a) with the DDIM scheduler [51] and a
pre-trained CLIP vision encoder [25]. We use a longer action horizon of 64 to
provide more information into the future for the low-level controller. For our
cup rearrangement task, we directly use UMI [1]’s cup rearrangement
checkpoint. For the pushing and tossing task, we collect data and train
diffusion policies from scratch.

###  3.2 Whole-body Controller with Reinforcement Learning

To track end-effector trajectories predicted from the manipulation policy, we
propose to train a whole body controller with reinforcement learning in
simulation to infer arm and leg joint targets. Notably, setting up a
simulation to track these manipulation end-effector trajectories does not
require setting up the manipulation task and environment. This design greatly
alleviates a key bottleneck in using simulation data.

Task-frame trajectory tracking of manipulation trajectories. Prior works [8,
12, 11, 10] typically sample target end-effector poses in body-frame to train
their WBCs, which simplifies policy optimization but does not train the
controller whole-body coordination skills required to compensate body
movements and perturbations (Fig 4b). This issue is amplified in scenarios
where the arm’s momentum during manipulation causes significant base movements
(i.e., light-weight base, or dynamic arm movements) In contrast, we train our
controller to track pose trajectories in the task-frame (Fig 3a). This
formulation teaches the arm maintain its end-effector pose in task-frame by
compensating and cancelling out body movements or shakes. To provide the
controller with relevant reference trajectories, we use trajectories collected
with UMI [1].

Observation Space. The observation space includes robots’ 18 joint positions
and velocities, the base orientation and angular velocity, the previous
action, and the end-effector trajectory inferred by the manipulation policy.
We represent end effector pose using a 3D vector for the position and the 6D
rotation representation [52]. We densely sample target poses from -60ms to
60ms at 20ms interval relative to the current time, which informs the
controller of current velocity and acceleration. In addition, we include the
target 1000ms into the future, which helps the controller prepare to take foot
steps if necessary.

Rewards. The task objective rewards the policy for minimizing the position
error $\epsilon_{\textrm{pos}}$ and orientation error
$\epsilon_{\textrm{orn}}$ to the target pose:
$\exp(-(\frac{\epsilon_{\textrm{pos}}}{\sigma_{\textrm{pos}}}+\frac{\epsilon_{\textrm{orn}}}{\sigma_{\textrm{orn}}}))$
where $\sigma$ are scaling terms according to precision requirement. In this
formulation, the term for position and orientation are entangled. We observe
that this leads to more ideal behavior than separate position and orientation
terms, which causes the policy to achieve high precision in only either
position or orientation. We also observed that a $\sigma$ curriculum for both
position and orientation is necessary to enable exploration during the early
stage of training while forcing the policy to achieve high precision in the
later stages. On top of the main task reward, we follow common conventions [7,
12, 8, 9] and include extra regularization and shaping terms, detailed in the
supp. material.

Policy network architecture. We train a multi-layer perceptron (MLP)
controller which maps from observations to target joint positions for both the
legs (12 DOF) and the arm (6 DOF) (Fig 3c). A forward pass for this controller
takes $\approx$0.06ms on the robot’s on-board CPU, and is called at 50Hz
during deployment. The joint position targets are tracked using separate PD
controllers for the legs and the arm.

###  3.3 System Integration

Robot system setup. The robot system (Fig. 2) consists of a 12-DOF Unitree Go2
quadruped robot and a 6-DOF ARX5 robot arm, both powered by the battery of
Go2. We customize the ARX5 arm with Finray grippers and a GoPro to match the
UMI gripper [1]. Our whole-body controller runs on the Go2’s Jetson while our
diffusion policy inference runs on a separate desktop’s RTX 4090 over an
internet connection. We mount an iPhone for pose estimation and connect it to
the Jetson through an Ethernet cable.

Sim2Real transfer. Following prior works, we add random push force on the
robot during training to achieve better robustness. We randomize joint
friction, damping, contact frictions, body and arm masses and center of
masses. We also observed that it was crucial to model 20ms of control latency
during training. To account for noises in the odometry system, we randomly
transport the robot every 20 seconds mid-way through the episode. More details
can be find in appendix.

Accessible real-time odometry. The lack of real-time onboard task-space
tracking is a common limitation in prior quadruped manipulation works. By
assuming external tracking with motion capture [31] and/or AprilTags [11, 8,
36], their systems can’t be deployed fully-autonomously in the wild. In our
system, we address this drawback with an iPhone, mounted on the robot’s base.
Our choice of the rear mounting site avoids adding extra weight on the robot
arm, prevents arm-phone collisions, and minimizes motion blur and visual
occlusion. In contrast to many existing robust, real-time odometry solutions
[13, 14, 15], our odometry solution has a self-contained, compact form-factor
and uses only ubiquitous consumer electronic devices.

##  4 Experiments

We designed a series of experiments in simulation and the real world to
validate our key design decisions, which aims to answer the following key
questions:

  * •

Capability. Is our system UMI-on-Legs able to learn complex and challenging
manipulation skills such as whole-body dynamics actions (e.g., tossing)?

  * •

Robustness. Can our whole-body controller handle unexpected external
perturbations and object dynamics, encountered during manipulation?

  * •

Scalability. Can we use cross-embodiment manipulation data collected entirely
absent of the robot?

Metrics. In simulation, we report average position error (cm), orientation
error (radians), survival (%), and electrical power usage (kW) over 500
episodes. In real-world experiments, we report average success rate over 20
episodes.

Comparisons. We ablate our design choices for the WBC in Table 1. Starting
with our approach, we remove preview (i.e., trajectory observations), task-
frame tracking formulation, and manipulation trajectory training trajectories.
Removing all three simultaneously leads to a baseline similar to DeepWBC [8].
To investigate the efficacy of our iPhone-based odometry system, we compare
against an OptiTrack motion capture system in all our real-world experiments,
which serves as the oracle odometry. For real-world tossing, we also show the
no-preview baseline’s behavior in real (Fig. 5). Finally, for robot videos for
all tasks, please check the supplementary materials.

Fig 5:  Dynamic tossing requires dynamics whole-body coordination. Our
controller (top row) discovers a strategy to toss reliably given its limited
arm strength and body inertia, which involves three stages. First, as the arm
accelerates forward, the back legs pops up, leading to a leap and toss motion
(a, green). To prevent falling forwards, the robot tucks and curls its arms
and legs inwards (b, orange), inducing a backwards torque. This backwards
torque helps shift the front legs’ contact points forwards in front of its
center-of-mass, enabling a soft landing. In contrast, the controller with no
preview information (bottom row) leaps in an attempt to follow the fast target
acceleration forwards, but doesn’t know where the target will go next, thus,
dropping the ball. Please checkout our website robot videos!

###  4.1 Capability: Whole-Body Dynamic Tossing

Our first experiment aims at validating the system’s capability, i.e., whether
our policy interface can capture complex manipulation skills. To answer this
question, we chose a particularly challenging task – dynamic tossing. The
robot must toss pre-grasped balls towards a target bin 90cm away, where a
trial is successful if it lands within 40cm of bin’s center.

Task difficulties. Most of existing tossing systems assume a carefully
designed primitive [53, 17]. Recently, Chi et al. [1] demonstrated that
learning from hand-held gripper demonstrations results in equally capable
tossing skills, but has only been shown to work with a fixed-based robot arm
with a precise controller. For legged systems, reliable tossing depends on the
body placement and the tossing motion. Thus, in order to achieve a successful
toss, the controller must discover the right stance, utilize power from all of
its degrees of freedom, and learn the dynamic whole-body coordination required
to achieve precise tracking throughout the dynamic tossing motion.

Approach | Pos Err | Orn Err | Survival | Power
---|---|---|---|---
Units | cm$\downarrow$ | deg$\downarrow$ | % $\uparrow$ | kW$\downarrow$
Ours | 2.12 | 3.35 | 98.4 | 3.82
(-) Preview | 3.02 | 4.23 | 93.0 | 3.95
(-) Task-space | 15.49 | 15.55 | 0.0 | 4.74
(-) UMI Traj | 2.48 | 15.67 | 97.4 | 3.69
DeepWBC [8] | 22.2 | 66.22 | 0.0 | 5.92
Table 1:  Tossing Evaluation in Sim. Averaged over 500 tossing episodes.

Dynamic tossing requires dynamic whole-body coordination. By learning to track
tossing trajectories, our whole-body controller discovered a whole-body
tossing strategy (Fig 5). The motion requires dynamic whole-body coordination,
involving power from all joints during the toss, momentarily balancing on one
foot/two feet, and effective usage of body mass inertia to regain balance.
Overall, our system achieves $75\%$ and $70\%$ when using motion capture and
iphone odometry, respectively.

Effects of the interface design. From our ablations and baseline comparisons
(Table 1), we observed that training on task-frame manipulation trajectories
was crucial - removing either task-frame tracking or manipulation trajectories
((-) Task-space and (-) UMI Traj) significantly hurts position and/or
orientation precision. Meanwhile, removing trajectory information from the
WBC’s observation prevents the controller from anticipating future motions,
resulting in dangerous reaching behaviors ($-5.4\%$ survival) and more jittery
actions ($+130W$) compared to our approach, as demonstrated in Fig 5.

Failure analysis. Despite the highly-dynamic nature of this motion, the
controller manages safe landings during all motion capture evaluation trials.
To further investigate whether tossing misses were due to imprecise WBCs or
noisy end-effector trajectories, we tested our controller with calibrated
tossing trajectory replay with motion capture, which effectively acts as an
“oracle” manipulation policy, and report a $+10\%$ improvement (Fig. 5). We
hypothesize that scaling up tossing data collection further could close this
gap.

Fig 6:  Robustness to Unexpected Dynamics. When faced with unexpected
resistance from the kettlebell’s friction, our controller continuously tries
to push forwards. This leads to a wiggling motion along with increased pushing
force until static friction breaks into a dynamic slide, at which point the
robot catches itself by dropping its front feet forwards. We observed this
push-and-catch strategy while the robot pushes both 10lbs and 20lbs (b),
despite the latter succumbing to static friction more often. We also report
comparable performance with the motion capture oracle (c).

###  4.2 Robustness: End-effector Reaching Leads to Robust Whole-body Pushing

Our second task stress tests the controller’s ability to handle unexpected and
unseen external dynamics in zero-shot fashion. To evaluate this capability we
studied kettlebell pushing (Fig 6), where goal of this task is to push a 10lbs
kettlebell, such that it slides upright into a goal region. Between each
episode, the kettlebell’s distance from its goal is varied within the range
[80cm,120cm].

Task Difficulties. Although the end effector motion involved may be simple,
the whole body controller need to adapt to significantly changed dynamic
models. In order to be successful, the WBC must be robust to feet slippage,
overcome static friction, and adapt to when static friction suddenly breaks
into a slide. In addition, small imprecision in the pushing contact point
could lead to the kettlebell toppling over.

Robustness to large disturbances. To our surprise, even with such a large
disturbance, our controller was able to maintain balance and finish the task,
without ever training on simulated forces or weights. Our controller completes
the task with 95% and 90% success rates for motion capture and iPhone
deployment respectively. We observed that as soon as the WBC observes a larger
position tracking error caused by the weight, it changes its strategy to lean
forward in order to exert more force - a strategy which would cause the robot
to fall forwards in absence of the kettlebell. By sending only end effector
trajectories without any base velocity, the controller can also move its legs
forward to reach faraway target poses. In addition, we also tested the policy
with a 20lbs kettlebell, and observed 4 successful episodes out of 5.

Failure analysis. The most common failure case in this task was due to the
kettlebell toppling sideways, after which we halt the system. Although more
human demonstrations could be collected to reorient the kettlebell, the weight
is too heavy for our arm to pick up again. Further, we observed that our
hardware system often overheats during this task’s evaluation, and requires
frequent cool-down breaks. Besides the obvious hardware improvement, we
hypothesize training with simulated forces [10] could lead to more elegant,
energy efficient pushing behavior.

###  4.3 Scalability: Plug-and-play Cross-Embodiment Manipulation Policies

Fig 7:  Making Existing Manipulation Policies Mobile. Although intended for
fixed-based arms with larger reach range and precise controllers (left, from
Chi et al.), the In-the-wild cup rearrangement policy from Chi et al. was able
to achieve 80-85% success rates zero-shot on our quadruped system with our
learned whole-body controller (center). We observed that our controller
learned a tilted stance, which minimizes front leg torque usage by delegates
more mass from the front-heavy system to the back legs.

In our third experiment, we want to validate the feasibility of directly
deploying a pre-trained manipulation policy from a prior work [1] with our WBC
without further fine-tuning, which demonstrates zero-shot generalization to an
unseen embodiment ***Here zero-shot is defined with respect to manipulation
policy, which is never trained with the target embodiment. To evaluate this
capability we studied the In-the-wild Cup Rearrangement Task proposed in Chi
et al.. The goal of this task is to place an espresso cup on its saucer with
its handle pointed to the left of the robot. In different episodes, the cup
and saucer is randomly positioned within a 20cm radius, with random cup
orientation. Following Chi et al.’s definition, the task is successful if the
cup is upright on the saucer with its handle $\pm 15^{\circ}$ from pointing
directly left. Notably, this task is always evaluated in an unseen environment
for both manipulation and whole-body controller.

Task difficulties. The biggest challenge for this task to effectively leverage
cross-embodiment manipulation data and policy. In the original UMI paper [1],
the policy was trained on data collected by humans, and intended for 6 DoF
industrial robot arms (e.g., UR5 and Franka) with a static base and precise
controllers. In this experiment, we are directly deploying this pretrained
policy on a drastically different embodiment – an 18 DoF legged manipulator.
Moreover, the intrinsic difficulty of the pushing and rearranging task also
tests the system ability to perform precise 6 DoF end-effector movements,
including prehensile and non-prehensile manipulation. This kind of precise and
gentle actions have been traditional difficult for low-cost legged
manipulation systems.

Results and Findings. Without further finetuning of the manipulation policy,
our system achieved 85% and 80% success rate using motion capture and iPhone
odometry respectively, despite the cross-embodiment gap. Since the
manipulation policy was originally intended for a UR5e system with a larger
kinematic reach range than our ARX5 arm, it often predicts action sequences
which span the large workspace. To support the manipulation policy, we observe
that our system dynamically tilts and leans its base in-order to increase
reach range and/or counterbalance the arm.

###  4.4 Limitations

Although we’ve demonstrated UMI-on-Legs’ capabilities on various manipulation
skills, we observed several limitations that could be addressed in future
works. First, our system only enables gripper-based manipulation, a limitation
inherited from UMI [1]. However, with the rise of low-cost and portable whole-
body manipulation demonstration devices and whole-body manipulation quadruped
works [54, 55, 31], an interesting future direction would be to extend our
interface to support scalable whole-body manipulation. Second, despite the
scalability of our embodiment-agnostic interface, incorporating embodiment-
specific constraints back up to the high-level manipulation policy is an
important next step. Further, towards a complete mobile manipulation platform,
our system could be extended with abilities for collision avoidance [56],
force feedback and force-based control [10].

##  5 Conclusion

We introduce UMI-on-Legs, a framework that combines real-world human
demonstrations and simulation-based reinforcement learning. With task-frame
end-effector trajectories as the interface, our formulation allows intuitive
demonstrations from hand-held devices yet is expressive enough to represent
complex manipulation skills such as dynamic tossing and pushing, and precise
pick and place.

Although, as community, we’ve yet to have a unanimous foundation manipulation
policy, we believe our framework is an important step towards enabling the
transfer of such policies to different robot platforms. With an expressive
embodiment-agnostic interface in place, we can separately develop ever more
general visuo-motor policies and robust WBCs, knowing that they can be
plugged-and-played together in the end.

##  6 Acknowledgements

We would like the thank Cheng Chi, Zhenjia Xu, and Chuer Pan for guidance with
UMI, and Haochen Shi, Antonio Loquercio, Haoran He, and Jiyuan Shi for tips
with real-world deployment of RL controllers. A special thanks to Zhenjia Xu,
for giving us the idea and starter code to try out iPhone ARKit camera pose
estimation. We thank Yifan Hou, Qi Wu, Shuo Yang, Tianyu Li, Xuxin Cheng, and
Kexin Shi for fruitful discussions about controller formulation,
implementation and deployment. We thank Jingyun Yang and Zi-ang Cao for
helpful instructions to set up the motion capture system. For paper writing
and figures, we thank Zeyi Liu, Austin Patel, Mengda Xu, Dominik Bauer,
Xiaomeng Xu, Mandi Zhao, and Samir Gadre for helpful feedback. Finally, we
thank Carl Vondrick for providing the compute. This work was supported in part
by the Google Research Award, NSF Award #2132519, #2143601, #2037101. We would
like to thank ARX for the ARX robot hardware. The views and conclusions
contained herein are those of the authors and should not be interpreted as
necessarily representing the official policies, either expressed or implied of
the sponsors.

## References

  * Chi et al. [2024] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song.  Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots.  In _Proceedings of Robotics: Science and Systems (RSS)_ , 2024.
  * Chi et al. [2023] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song.  Diffusion policy: Visuomotor policy learning via action diffusion.  In _Proceedings of Robotics: Science and Systems (RSS)_ , 2023.
  * Makoviychuk et al. [2021] V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, et al.  Isaac gym: High performance gpu-based physics simulation for robot learning.  _arXiv preprint arXiv:2108.10470_ , 2021.
  * Todorov et al. [2012] E. Todorov, T. Erez, and Y. Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ international conference on intelligent robots and systems_ , pages 5026–5033. IEEE, 2012.
  * Freeman et al. [2021] C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem.  Brax–a differentiable physics engine for large scale rigid body simulation.  _arXiv preprint arXiv:2106.13281_ , 2021.
  * Caluwaerts et al. [2023] K. Caluwaerts, A. Iscen, J. C. Kew, W. Yu, T. Zhang, D. Freeman, K.-H. Lee, L. Lee, S. Saliceti, V. Zhuang, et al.  Barkour: Benchmarking animal-level agility with quadruped robots.  _arXiv preprint arXiv:2305.14654_ , 2023.
  * Rudin et al. [2022] N. Rudin, D. Hoeller, P. Reist, and M. Hutter.  Learning to walk in minutes using massively parallel deep reinforcement learning.  In _Conference on Robot Learning_ , pages 91–100. PMLR, 2022.
  * Fu et al. [2023] Z. Fu, X. Cheng, and D. Pathak.  Deep whole-body control: Learning a unified policy for manipulation and locomotion.  In _Conference on Robot Learning_ , pages 138–149. PMLR, 2023.
  * Kumar et al. [2021] A. Kumar, Z. Fu, D. Pathak, and J. Malik.  Rma: Rapid motor adaptation for legged robots.  _arXiv preprint arXiv:2107.04034_ , 2021.
  * Portela et al. [2024] T. Portela, G. B. Margolis, Y. Ji, and P. Agrawal.  Learning force control for legged manipulation.  _arXiv preprint arXiv:2405.01402_ , 2024.
  * Pan et al. [2024] G. Pan, Q. Ben, Z. Yuan, G. Jiang, Y. Ji, J. Pang, H. Liu, and H. Xu.  Roboduet: A framework affording mobile-manipulation and cross-embodiment.  2024\.
  * Liu et al. [2024] M. Liu, Z. Chen, X. Cheng, Y. Ji, R. Yang, and X. Wang.  Visual whole-body control for legged loco-manipulation.  _arXiv preprint arXiv:2403.16967_ , 2024.
  * Xu et al. [2022] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang.  Fast-lio2: Fast direct lidar-inertial odometry.  _IEEE Transactions on Robotics_ , 38(4):2053–2073, 2022.
  * Wisth et al. [2022] D. Wisth, M. Camurri, and M. Fallon.  Vilens: Visual, inertial, lidar, and leg odometry for all-terrain legged robots.  _IEEE Transactions on Robotics_ , 39(1):309–326, 2022.
  * Hartley et al. [2018] R. Hartley, M. G. Jadidi, L. Gan, J.-K. Huang, J. W. Grizzle, and R. M. Eustice.  Hybrid contact preintegration for visual-inertial-contact state estimation using factor graphs.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 3783–3790. IEEE, 2018.
  * Yokoyama et al. [2023] N. Yokoyama, A. Clegg, J. Truong, E. Undersander, T.-Y. Yang, S. Arnaud, S. Ha, D. Batra, and A. Rai.  Asc: Adaptive skill coordination for robotic mobile manipulation.  _IEEE Robotics and Automation Letters_ , 9(1):779–786, 2023.
  * Wu et al. [2023] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser.  Tidybot: Personalized robot assistance with large language models.  _Autonomous Robots_ , 2023.
  * Xiong et al. [2024] H. Xiong, R. Mendonca, K. Shaw, and D. Pathak.  Adaptive mobile manipulation for articulated objects in the open world.  _arXiv preprint arXiv:2401.14403_ , 2024.
  * Sun et al. [2022] C. Sun, J. Orbik, C. M. Devin, B. H. Yang, A. Gupta, G. Berseth, and S. Levine.  Fully autonomous real-world reinforcement learning with applications to mobile manipulation.  In _Conference on Robot Learning_ , pages 308–319. PMLR, 2022.
  * Ahn et al. [2022] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Lew et al. [2023] T. Lew, S. Singh, M. Prats, J. Bingham, J. Weisz, B. Holson, X. Zhang, V. Sindhwani, Y. Lu, F. Xia, et al.  Robotic table wiping via reinforcement learning and whole-body trajectory optimization.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 7184–7190. IEEE, 2023.
  * Huang et al. [2022] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  In _International Conference on Machine Learning_ , pages 9118–9147. PMLR, 2022.
  * Gu et al. [2021] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  _arXiv preprint arXiv:2104.13921_ , 2021.
  * Kirillov et al. [2023] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al.  Segment anything.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4015–4026, 2023.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Fu et al. [2024] Z. Fu, T. Z. Zhao, and C. Finn.  Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.  In _arXiv_ , 2024.
  * Cheng et al. [2023] X. Cheng, K. Shi, A. Agarwal, and D. Pathak.  Extreme parkour with legged robots.  _arXiv preprint arXiv:2309.14341_ , 2023.
  * Zhuang et al. [2023] Z. Zhuang, Z. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao.  Robot parkour learning.  _arXiv preprint arXiv:2309.05665_ , 2023.
  * Margolis et al. [2024] G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal.  Rapid locomotion via reinforcement learning.  _The International Journal of Robotics Research_ , 43(4):572–587, 2024.
  * Arm et al. [2024] P. Arm, M. Mittal, H. Kolvenbach, and M. Hutter.  Pedipulate: Enabling manipulation skills using a quadruped robot’s leg.  _arXiv preprint arXiv:2402.10837_ , 2024.
  * Lin et al. [2024] C. Lin, X. Liu, Y. Yang, Y. Niu, W. Yu, T. Zhang, J. Tan, B. Boots, and D. Zhao.  Locoman: Advancing versatile quadrupedal dexterity with lightweight loco-manipulators.  _arXiv preprint arXiv:2403.18197_ , 2024.
  * Ji et al. [2023] Y. Ji, G. B. Margolis, and P. Agrawal.  Dribblebot: Dynamic legged manipulation in the wild.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 5155–5162. IEEE, 2023.
  * Schulman et al. [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Olson [2011] E. Olson.  Apriltag: A robust and flexible visual fiducial system.  In _2011 IEEE international conference on robotics and automation_ , pages 3400–3407. IEEE, 2011.
  * He et al. [2024] Z. He, K. Lei, Y. Ze, K. Sreenath, Z. Li, and H. Xu.  Learning visual quadrupedal loco-manipulation from demonstrations.  _arXiv preprint arXiv:2403.20328_ , 2024.
  * Mandlekar et al. [2021] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín.  What matters in learning from offline human demonstrations for robot manipulation.  _arXiv preprint arXiv:2108.03298_ , 2021.
  * Zhao et al. [2023] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn.  Learning fine-grained bimanual manipulation with low-cost hardware.  _arXiv preprint arXiv:2304.13705_ , 2023.
  * Shafiullah et al. [2022] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto.  Behavior transformers: Cloning $k$ modes with one stone.  _Advances in neural information processing systems_ , 35:22955–22968, 2022.
  * Brohan et al. [2022] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Team et al. [2024] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al.  Octo: An open-source generalist robot policy.  _arXiv preprint arXiv:2405.12213_ , 2024.
  * Belkhale et al. [2024] S. Belkhale, Y. Cui, and D. Sadigh.  Data quality in imitation learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Khazatsky et al. [2024] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al.  Droid: A large-scale in-the-wild robot manipulation dataset.  _arXiv preprint arXiv:2403.12945_ , 2024.
  * Ebert et al. [2021] F. Ebert, Y. Yang, K. Schmeckpeper, B. Bucher, G. Georgakis, K. Daniilidis, C. Finn, and S. Levine.  Bridge data: Boosting generalization of robotic skills with cross-domain datasets.  _arXiv preprint arXiv:2109.13396_ , 2021.
  * Walke et al. [2023] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, et al.  Bridgedata v2: A dataset for robot learning at scale.  In _Conference on Robot Learning_ , pages 1723–1736. PMLR, 2023.
  * Padalkar et al. [2023] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al.  Open x-embodiment: Robotic learning datasets and rt-x models.  _arXiv preprint arXiv:2310.08864_ , 2023.
  * Yang et al. [2024] J. Yang, C. Glossop, A. Bhorkar, D. Shah, Q. Vuong, C. Finn, D. Sadigh, and S. Levine.  Pushing the limits of cross-embodiment learning for manipulation and navigation.  _arXiv preprint arXiv:2402.19432_ , 2024.
  * Wu et al. [2023] P. Wu, Y. Shentu, Z. Yi, X. Lin, and P. Abbeel.  Gello: A general, low-cost, and intuitive teleoperation framework for robot manipulators.  _arXiv preprint arXiv:2309.13037_ , 2023.
  * Ha et al. [2023] H. Ha, P. Florence, and S. Song.  Scaling up and distilling down: Language-guided robot skill acquisition.  In _Conference on Robot Learning_ , pages 3766–3777. PMLR, 2023.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_ , pages 234–241. Springer, 2015.
  * Song et al. [2020] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Zhou et al. [2019] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 5745–5753, 2019.
  * Zeng et al. [2020] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser.  Tossingbot: Learning to throw arbitrary objects with residual physics.  _IEEE Transactions on Robotics_ , 36(4):1307–1319, 2020.
  * Jeon et al. [2023] S. Jeon, M. Jung, S. Choi, B. Kim, and J. Hwangbo.  Learning whole-body manipulation for quadrupedal robot.  _IEEE Robotics and Automation Letters_ , 9(1):699–706, 2023.
  * Cheng et al. [2023] X. Cheng, A. Kumar, and D. Pathak.  Legs as manipulator: Pushing quadrupedal agility beyond locomotion.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 5106–5112. IEEE, 2023.
  * Miki et al. [2024] T. Miki, J. Lee, L. Wellhausen, and M. Hutter.  Learning to walk in confined spaces using 3d representation.  _arXiv preprint arXiv:2403.00187_ , 2024.

UMI on Legs Appendix

##  7 Things that did not work

In this section, we discuss things we tried, but did not improve the
performance or caused other issues. We hope that through openly discussing our
unsuccessful attempts, the community could gain a more complete understanding
of the project, learn from our mistakes, and improve upon our attempts.

###  7.1 Privileged policy distillation and observation history.

We tried to include privileged information (kp, kd, friction, damping, ground
truth poses, base velocities, etc.) to train a privileged policy and distill
it through supervised learning and online regularization. However, it didn’t
provide performance boost and introduce instability when we include
observation history longer than 1 step. This could be because that the Python
ROS2 timer is imprecise, thus the observation history can easily be out of
distribution given the wrong history timestamps. This was less of an issue on
the Unitree A1 platform, which ships with a real-time kernel.

###  7.2 Precise grasping for tossing.

To train diffusion policies for grasping and tossing, we collected 500
episodes of human demonstration. However in the grasping phase, the small
shakes from the controller results in distributional drift in visuo-motor
manipulation policy. We hypothesize that robust, fully-autonomous grasping and
tossing could be achieved with more data.

###  7.3 System Reliability for Fully-untethered Deployment.

Although the system is capable of various tasks in the real world without any
external cables, the reliability is in general not high enough. We summarize
some hardware and system level challenges we encountered as follows. Given a
small base-to-arm weight ratio, the joints of the dog are likely to overheat
in 10 to 30 minutes according to the robot posture and then goes to an error
mode. Therefore, we needed to frequently cool down the motors and fine-tune
the policy to a more energy-efficient posture. On the power front, the battery
supplies different voltages depending on how full it is. When fully charged,
the voltage is too high for the arm, and the setup requires a voltage adaptor.
When closer to empty, the policy’s behavior is more dampened.

###  7.4 Velocity integration

We used the iOS ARKit to run VIO on an iPhone 15 Pro. However, the estimated
camera pose sometimes drift heavily under dynamic actions including tossing.
We also measure the delay of the pose estimator. Although the Ethernet
communication time (between the iPhone and the onboard Jetson) is negligible,
we found the the latency from the movement to the pose update is still roughly
140ms. This latency introduced a significant Sim2Real gap during deployment,
which manifests as low frequency oscillations which slowly diverges.

Using low-latency foot contacts, joint position/velocities, and IMU readings,
we can estimate the base velocity for each timestamp. Using base velocity
estimates from 140ms into the past, we can integrate every received iPhone
pose forward in time by 140ms. However, the shaking behavior due to the
latency is not addressed, which can only be mitigated by higher action rate
regularization fine-tuning. As the ARKit pose estimation runs at 60Hz, we
expect a better implementation could achieve shorter latency.

##  8 Deployment

In this section we elaborate important details in real-world deployment and
Sim2Real transfer.

###  8.1 iPhone placement

We mounted the iPhone on the back of the robot with following
considerations:1) Since the iPhone is facing back, the robot arm will not be
in the view during manipulation, so ARKit’s visual-inertial odometry can
better track the surrounding environment. 2) iphone is mounted at a fixed
angle of 60∘ to the back plane of the Go2, thus the camera is pointing upwards
even when Go2 is slightly bending down. This increases the number of visual
features in the iPhone camera, thus provides more robust tracking compared to
our original design, which was a 90∘ mount. 3) This position is kinematically
unreachable for the robot arm, so the arm will unlikely damage the iPhone.

###  8.2 Robot URDF

For our highly dynamic tasks, we observed that the domain randomization and
adaptive policies [9, 8] were insufficient to bridge the gap between a heavily
mis-specified model (provided by the manufacturer) and the real system, a gap
that is exacerbated when performing dynamic arm movements. We found that
disassembling the ARX5 arm, reweighing each component, and recomputing mass,
center of mass and inertia matrices in the URDF††† Download our code, data and
checkpoints from our website. were crucial for both simulation stability as
well as successfully real-world deployment.

###  8.3 Latency

While all observations and actions are perfectly synchronized in simulation,
the communication and code runtime of the real-world robot system introduce a
significant amount of latency in different aspects.

The most important latency comes from the robot joint state observation and
the joint command execution. This includes the motor encoder readout, ROS2
communication, whole-body controller inference, action sent back to motors and
being executed on motors. Since we were unable to exactly measure all the
latency sources, we swept the end-to-end latency from 0ms to 30ms with 5ms
interval in simulation and find that 20ms works the best in our real-world
system. We observed high frequency shaking if the latency is mismatched in
simulator and real-world robots.

Robot pose estimators (motion capture and iphone) also introduce latency. We
observed an 8ms latency from the motion capture system and 140ms latency from
the iphone ARKit. We simulated a 10ms pose latency in simulation to close the
sim2real gap. We also implemented inertial-legged velocity estimation to
integrate the latest pose for iPhone, but the performance didn’t improve
significantly. See 7.4 for detailed explanation.

We also observe latency in the Python ROS2 program. Due to the global
interpreter lock, all the Python ROS2 callback functions have to run
alternately, so one callback function will block the others if it takes too
long. We optimize the run time of all callback functions and detach the
callback functions that take too long to other ROS2 nodes. This allows the
joint observation update to run closer to 200Hz and policy inference closer to
50Hz. A C++ implementation with more precise timer and lower latency can
achieve better performance given the same checkpoint and evaluation setup.

###  8.4 Safety

We observed that directly deploying the whole-body controller checkpoints in
the simulation section led to some safety issues and addressed them by fine-
tuning the checkpoints with more conservative reward schemes:

  * •

Shaking: To reduce the shaking and oscillation behavior of the robot, we
increased the action rate regularization. The controller will get less reward
if the predicted action is farther away from the previous action. We also
disabled the on board lidar that introduces shaking behavior.

  * •

Overheat Shutdowns: The calf joint of the robot uses a linkage configuration
rather than directly applies the output torque. Therefore, the required output
torque of the motor is higher than the other joints, leading to frequent
overheating and emergency shutdown. We observed that increased torque
regularization during training allows the controller to run up to 30 minutes
continuously‡‡‡Please check out evaluation videos on our website, which shows
our controller running continuously for 20-30 minutes.

  * •

Unsafe Configurations: In the simulator, the controller is likely to twist the
legs to achieve higher precision with less movements, but this makes it unsafe
to deploy in the real world. We added reward terms to regularize the body to a
more balanced pose, so that the center of mass can roughly stay in the center.

##  9 Training

###  9.1 Manipulation Policy

In this section, we report hyperparameters used for training our manipulation
diffusion policy. Visual observation and proprioception horizon means how many
history image and robot states (with 0.1s interval) are used as input to the
policy. Output Steps means the action length of the diffusion policy output.
Execution Steps and Execution Frequency indicate how many steps of the
diffusion policy output is actually executed in the real-world robot. To
enable more stable end-effector trajectory updates, we add linear
interpolation between the executed trajectory and the newly-updated trajectory
for a duration of “Trajectory Update Smoothing Time”.

###  9.2 Whole-Body Dynamic Tossing

We increased the inference and execution steps for a smoother toss, since the
trajectory is less likely to update during the dynamic toss. We also increased
the execution frequency to get farther toss. Please refer to Table 2 for
detailed parameters.

Hyperparameters | Values
---|---
Training Set Trajectory Number | 500
Diffusion Policy Visual Observation Horizon | 2
Diffusion Policy Proprioception Horizon | 4
Diffusion Policy Output Steps | 64
Diffusion Policy Execution Steps | 40
Diffusion Policy Execution Frequency | 12Hz
Trajectory Update Smoothing Time | 0.1s
Table 2:  Hyper parameter set of the dynamic tossing task.

###  9.3 End-effector Reaching Leads to Robust Whole-body Pushing

Since the trajectory motion is much simpler, we collected fewer human
demonstrations for the diffusion policy. We increased the “Trajectory Update
Smoothing Time” to prevent heavy shakes due to the trajectory update. Please
refer to Table 2 for detailed parameters.

Hyperparameters | Values
---|---
Training Set Trajectory Number | 25
Diffusion Policy Visual Observation Horizon | 2
Diffusion Policy Proprioception Horizon | 2
Diffusion Policy Output Steps | 32
Diffusion Policy Execution Steps | 10
Diffusion Policy Execution Frequency | 10Hz
Trajectory Update Smoothing Time | 0.3s
Table 3:  Hyper parameter set of the whole-body pushing task.

###  9.4 Plug-and-play Cross-Embodiment Manipulation Policies

As the UMI cup rearrange task requires much higher precision but is quasi-
static, we decreased the execution frequency for better stability. As
mentioned in the main text, we directly use the publicly released checkpoint
from Chi et al.. We did not collect any extra data for this task.

Hyperparameters | Values
---|---
Training Set Trajectory Number | 1400
Diffusion Policy Visual Observation Horizon | 1
Diffusion Policy Proprioception Horizon | 2
Diffusion Policy Output Steps | 16
Diffusion Policy Execution Steps | 8
Diffusion Policy Execution Frequency | 5Hz
Trajectory Update Smoothing Time | 0.1s
Table 4:  Hyper parameter set of the UMI cup rearrangement task.

###  9.5 In-the-wild Cup Rearrangement Evaluation

We evaluated the whole-body controller on the cup rearrangement task in the
wild. We tested it in some challenging scenarios including unstable terrain
(grass, dirt, collapsible table), direct sunlight which made the robot easily
overheated, and unseen tables and cups to show the generalizability of the
diffusion policy. Please checkout the supplementary video for more details.

###  9.6 Reward Terms

During RL training of the whole body controller, we include the following
reward terms:

  * •

Joint Limit, Joint Acceleration, Joint Torque, Root Height, Collision, Action
Rate: We use the same definition as prior works [8, 7].

  * •

Body-EE Alignment: We regularize joint 0 and joint 3 of the arm (the joints
that mostly affect the yaw angle of the gripper) to stay close to their
initial pose. This allows the arm to be aligned with the body.

  * •

Even Mass Distribution: We regularize the standard deviation of the force of
the four feet to be lower. The motors are less likely to overheat if the body
mass is distributed more evenly on the four legs.

  * •

Feet Under Hips: We regularize the feet’s planar position to be close to that
of their respective hips, for all four legs. This regularization improves the
stability of the standing pose.

  * •

Pose Reaching: We minimize the position error $\epsilon_{\textrm{pos}}$ and
orientation error $\epsilon_{\textrm{orn}}$ to the target pose using an
unified reward function:
$\exp(-(\frac{\epsilon_{\textrm{pos}}}{\sigma_{\textrm{pos}}}+\frac{\epsilon_{\textrm{orn}}}{\sigma_{\textrm{orn}}}))$.
We apply $\sigma$ curriculum to the reward functions: as the error get
smaller, we decrease $\sigma$ for a more peaky reward function, which
encourages the controller to further reduce reaching error.
$\sigma_{\textrm{pos}}$ is set to $[2,0.1,0.5,0.1,0.05,0.01,0.005]$ when the
position error is smaller than $[100,1.0,0.8,0.5,0.4,0.2,0.1]$ respectively;
$\sigma_{\textrm{orn}}$ is set to $[8.0,4.0,2.0,1.0,0.5]$ when the orientation
error is smaller than $[100.0,1.0,0.8,0.6,0.2]$ respectively.

Name | Weight
---|---
Joint Limit | -10
Joint Acceleration | -2.5e-7
Joint Torque | -1e-4
Root Height | -1
Collision | -1
Action Rate | -0.01
Body-EE Alignment | -1
Even Mass Distribution | -1
Feet Under Hips | -1
Pose Reaching | 4
Table 5:  Reward terms.

##  10 Evaluation

###  10.1 Real World Tossing

To achieve a pre-grasped state during tossing evaluations, we handed the
policy the tennis ball. We count episodes where the robot falls during the
toss as a failure. Finally, we measure the distance from the location where
the ball first lands to the center of the bin, and report a success if it
lands within 40cm of the bin’s center.

###  10.2 Simulation Ablations

In each episode, a random grasp and toss end-effector trajectory is sampled
from our dataset (§ 9.2), the robot is uniformly randomly initialized in the
range of [-5.0cm,5.0cm] for x and y position, and [-0.1rad,0.1rad] for z
orientation, and its initial joint positions are sampled uniformly within 5%
of that joints’ full range, centered around the default joint configuration.
Position and orientation errors are averaged over all timesteps, while
survival is 1 only if the policy lasts for 17 seconds without a terminal
collision. Here, a terminal collision is defined as any robot part contact
that is not the robot’s feet or gripper.

Similar to training, evaluation includes a pose latency of 10ms and the full
range of domain randomization reported in Table 6.

Hyperparameters | Values
---|---
Init XY Position | [-0.1m,0.1m]
Init Z Orientation | [-0.05rad,0.05rad]
Joint Damping | [0.01,0.5]
Joint Friction | [0.0,0.05]
Geometry Friction | [0.1,8.0]
Mass Randomization | [-0.25,0.25]
Center of Mass Randomization | [-0.1m,0.1m]
Table 6:  Domain Randomization Hyperparameters.

All approaches were trained for 4000 iterations, and the performance at
checkpoint 4000 is reported. Power usage report is electrical power usage
based on real hardware’s voltages, manufacturers’ reported torque constants,
and the simulation’s motor torques.

##  11 Misc.

Cost of the system. We report the cost of our entire robot system based on the
market price in Table 7, which is roughly 1/4 of other quadruped manipulation
systems in [12, 10]

Item | Cost($)
---|---
Unitree Go2 Edu Plus | 12,500.00
ARX5 Robot Arm | 10,000.00
GoPro Hero9 | 210.99
GoPro Media Mod | 79.99
GoPro Max Lens Mod | 68.69
iPhone 15 Pro | 999.00
Elgato Capture Card | 147.34
Total | 24,006.01
Table 7:  System Cost.