# AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation
System

Yuzhe Qin1, Wei Yang2, Binghao Huang1, Karl Van Wyk2
Hao Su1, Xiaolong Wang1, Yu-Wei Chao2, Dieter Fox2
1UC San Diego 2NVIDIA
https://yzqin.github.io/anyteleop/

###### Abstract

Vision-based teleoperation offers the possibility to endow robots with human-
level intelligence to physically interact with the environment, while only
requiring low-cost camera sensors. However, current vision-based teleoperation
systems are designed and engineered towards a particular robot model and
deploy environment, which scales poorly as the pool of the robot models
expands and the variety of the operating environment increases. In this paper,
we propose AnyTeleop, a unified and general teleoperation system to support
multiple different arms, hands, realities, and camera configurations within a
single system. Although being designed to provide great flexibility to the
choice of simulators and real hardware, our system can still achieve great
performance. For real-world experiments, AnyTeleop can outperform a previous
system that was designed for a specific robot hardware with a higher success
rate, using the same robot. For teleoperation in simulation, AnyTeleop leads
to better imitation learning performance, compared with a previous system that
is particularly designed for that simulator.

figureWe present AnyTeleop, a vision-based teleoperation system for a variety
of scenarios to solve a wide range of manipulation tasks. AnyTeleop can be
used for various robot arms with different robot hands. It also supports
teleoperation within different realities, such as IsaacGym (top row), and
SAPIEN simulator (middle row), and real world (bottom rows).

00footnotetext: Yuzhe Qin was an intern at NVIDIA during the project.

##  I Introduction

A grand goal of robotics is to endow robots with human-level intelligence to
physically interact with the environment. Teleoperation [39], as a direct
means to acquire human demonstrations for teaching robots, has been a powerful
paradigm to approach this goal [22, 11, 67, 17, 34, 6, 19, 5, 38, 53, 63].
Compared to gripper-based manipulators, teleoperating dexterous hand-arm
systems poses unprecedented challenges and often requires specialized
apparatus that comes with high costs and setup efforts, such as Virtual
Reality (VR) devices [4, 17, 15], wearable gloves [29, 30], handheld
controller [47, 48, 20], haptic sensors [12, 23, 52, 55], or motion capture
trackers [68]. Fortunately, recent developments in vision-based teleoperation
[2, 24, 16, 26, 43, 27, 21, 22, 3] have provided a low-cost and more
generalizable alternative for teleoperating dexterous robot systems.

Despite the progress, the current paradigm of vision-based teleoperation
systems still falls short when it comes to scaling up data collection for
robot teaching. First, prior systems are often designed and engineered towards
a particular robot model or deployment environment. For example, some systems
rely on vision-based hand tracking models trained on datasets collected in the
deployed studio [24, 16], and some rely on human-robot retargeting models [65,
13] or collision avoidance models [54] trained for the particular robot at
use. These systems will scale poorly as the pool of robot models expands and
the variety of operating environments increases. Second, each system is
created and coupled with one specific “reality”, either only in the real world
or with a particular choice of simulators. For example, the HAPTIX [23] motion
capture system is only developed for teleoperation in MuJoCo-based
environments [58]. To facilitate large-scale data collection with simulation
as well as closing sim-to-real gaps, we need teleoperation systems to operate
both in virtual (with arbitrary choices of simulators) and in the real world.
Finally, existing teleoperation systems are often tailored for single-operator
and single-robot settings. To teach robots how to collaborate with other robot
agents as well as with human agents, a teleoperation system should be designed
to support multiple pilot-robot partners where the robots can physically
interact with each other in a shared environment.

In this paper, we aim to set the foundation for scaling up data collection
with vision-based dexterous teleoperation, by filling in the aforementioned
gaps. To this end, we propose AnyTeleop, a unified and general teleoperation
system (Fig. AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand
Teleoperation System), which can be used for:

  * •

Diverse robot arm and dexterous hand models;

  * •

Diverse realities, i.e. different choices of simulators or the real world;

  * •

Teleoperation from diverse geographic locations, via a browser-based web
visualizer developed for remote visual feedback;

  * •

Diverse camera configurations, e.g. RGB camera with or without depth, single
or multiple cameras;

  * •

Diverse operator-robot partnerships, e.g. two operators separately piloting
two robots to collaboratively solve a manipulation task.

To achieve this goal, we first develop a general and high-performance motion
retargeting library to translate human motion to robot motion in real time
without learned models. Our collision avoidance module is also learning-free
and powered by CUDA-based geometry queries. They can adapt to new robots given
only the kinematic model, i.e., URDF files. Second, we develop a web-based
viewer compatible with standard browsers, to achieve simulator-agnostic
visualization and enable remote teleoperation across the internet. Third, we
define a general software interface for visual-based teleoperation, which
standardizes and decouples each module inside the teleoperation system. It
enables smooth deployment on different simulators or real hardware.

While being very general to support many settings with a single system, our
system can still achieve great performance in the experiments. For real-world
teleoperation, AnyTeleop can outperform a previous system [54] designed for
specific robot hardware with higher success rates on 8 out of 10 tasks
proposed in their paper, using the same robot as [54]. For simulated
environment teleportation, the smoother and collision-free demonstrations
collected by AnyTeleop can bring better imitation learning results with higher
success rates on 5 out of 6 tasks proposed in their paper, compared with a
previous system [43] specifically designed for that simulator. Finally, we
demonstrate that AnyTeleop can be extended to support collaborative
manipulation, which to our best knowledge has neither been achieved in the
literature of vision-based teleoperation nor on dexterous hands.

Our system is also packaged to be easily-deployable. The containerized design
makes installation easy and frees users from handling software dependencies.
We are committed to open-sourcing the system and benefiting the community.

##  II Related Work

Figure 1: Paradigms of vision-based teleoperation systems in independent and
collaborative settings. The system should support any arm-hand models, existed
in either virtual or the real world, can operate with flexible camera
configurations, provide visual feedback for both local or remote presence, and
support multiple robots piloted in a shared space.

Vision-based Robot Teleoperation. Recent years have witnessed an increasing
interest in teleoperation of dexterous robot hands by human hands. It relies
on accurate tracking of human hand motions and finger articulations. Compared
to the costly wearable hand tracking solutions, such as gloves [29, 30, 37],
marker-based motion capture systems [68, 31], inertia sensors [66] or VR
headsets [4, 28, 51, 41], vision-based hand tracking is particularly favorable
due to its low cost and low intrusion to the human operator. Early research in
vision-based teleoperation focused on improving the performance of hand
tracking [57, 60, 1] and mapping human hand pose to robot hand pose [24, 2, 5,
35]. Recent works have expanded the scope of teleoperating a single robot hand
to complete arm-hand systems [16, 25, 67]. However, these systems are designed
and engineered towards a particular robot model (e.g., Kuka arm with Allegro
hand in [16], and PR2 arm with Shadow hand in [26]), and rely on retargeting
or collision detection models trained for specific robot hardware (e.g.,
Allegro hand and XArm6) [16, 54] , making them difficult to transfer to new
arm-hand systems and new environments. In contrast, our system is highly
modularized with a versatile hand-tracking solution compatible with an
arbitrary number of cameras, and configurable robot hand retargeting and
motion generation modules for easy adaption to various robot arms and robot
hand choices. This allows our system to achieve better performance compared to
prior systems on various tasks while generalizing to a set of robot arm-hand
systems and multiple environments.

Teleoperation in Different Reality. Manipulation with a dexterous robot hand
is challenging due to its high degree of freedom. In recent years, dexterous
robot teleoperation has been actively studied and shown promising progress in
controlling a multi-fingered hand to perform manipulation tasks in the real
world [4, 16, 36] by leveraging the morphological similarity between the
dexterous hand and the human hand.

With the advancement of data-driven approaches for robot manipulation [12,
18], there is a growing need to collect human demonstrations in robotics. To
enable easy and scalable data collection, teleoperation has also gained
attention in simulated environments [62, 32, 58, 14, 8]. This provides a
scalable solution to data collection by eliminating the need for real
hardware, while maintaining access to oracle world information. For example,
Mandlekar et al. [33] developed a crowd-sourcing platform to teleoperate
robots via mobile devices as controllers. Tung et al. [59] further extended
this framework to allow multi-arm collaborative teleoperation. The above
frameworks rely on inertial sensors for control signals and thus are limited
to parallel-gripper and simple tasks such as pick-and-place. Our system offers
the ability to perform a wide range of dexterous tasks with robots of
different morphologies by utilizing state-of-the-art techniques in perception,
optimization, and control. In addition, AnyTeleop is designed to support
teleoperation in both virtual and the real world with a unified framework.

| Sensor Requirements | Robot-related Support | Use Case
---|---|---|---
| Calibration | Contact | Depth | Multiple | Multiple | Reality | Collision | Remote | Collaborative
| Free | Free | Free | Arms | Hands | Free | Teleop | Teleop
DexPilot [16] |  2613 | ✓ |  2613 |  2613 |  2613 | Real | ✓ |  2613 |  2613
Holo-Dex [4] | ✓ | ✓ | ✓ | No Arm |  2613 | Real |  2613 | ✓ |  2613
DIME [5] |  2613 | ✓ | ✓ | No Arm |  2613 | Real |  2613 | ✓ |  2613
TeachNet [24] | ✓ | ✓ |  2613 | No Arm |  2613 | Sim&Real |  2613 |  2613 |  2613
Telekinesis [54] | ✓ |  2613 | ✓ |  2613 |  2613 | Real | ✓ | ✓ |  2613
Qin et al. [43] | ✓ | ✓ |  2613 | No Arm | ✓ | Sim |  2613 | ✓ |  2613
MVP-Real [45] |  2613 |  2613 | ✓ | No Arm |  2613 | Real |  2613 | ✓ |  2613
Transteleop [26] |  2613 |  2613 |  2613 |  2613 |  2613 | Real | ✓ | ✓ |  2613
Mosbach et al. [37] |  2613 |  2613 | ✓ |  2613 |  2613 | Sim |  2613 | ✓ |  2613
AnyTeleop | ✓ | ✓ | ✓ | ✓ | ✓ | Sim&Real | ✓ | ✓ | ✓
TABLE I: Comparison of Vision-Based Teleoperation System. We compare
AnyTeleop’s capabilities with related visual teleoperation system for multi-
fingered dexterous robots. “No Arm” in the column of “Multiple Arms” means
this system can only control hand motion but not arm-hand systems.

##  III System Overview

Fig. 1 illustrates our proposed paradigms of vision-based teleoperation
systems. Below we introduce the features and designs of our system which
realize the paradigms.

###  III-A System Features

  1. 1.

Any arm-hand. As shown in Fig. AnyTeleop: A General Vision-Based Dexterous
Robot Arm-Hand Teleoperation System, AnyTeleop is designed for arbitrary
dexterous arm-hand systems that are not limited to any specific robot type.

  2. 2.

Any reality. AnyTeleop is decoupled from specific hardware drivers or physics
simulators. It can support different realities as visualized in Fig.
AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation
System.

  3. 3.

Anywhere remote teleoperation. AnyTeleop provides a web-based visualizer to
monitor the teleoperation and simulation in standard web browsers, e.g.
Chrome.

  4. 4.

Any camera configuration. AnyTeleop can consume data from both RGB and RGB-D
cameras, and from either single or multiple cameras. Most importantly, it does
not require extrinsic calibration as in most previous systems. This allows
more flexible camera configurations and lower deployment overhead.

  5. 5.

Any number of operator-robot partnerships. AnyTeleop supports collaborative
settings where operators separately pilot two robots to collaboratively solve
a manipulation task.

  6. 6.

Simple deployment. AnyTeleop and all libraries are encapsulated as a docker
image that can be downloaded and deployed on any Linux machine, which frees
users from handling troublesome dependencies.

Table I compares AnyTeleop with other vision-based dexterous teleoperation
systems. We compare the systems in three dimensions: (i) sensor requirements;
(ii) robot-related support; (iii) afforded use cases. Among all these
teleoperation systems, AnyTeleop is the only one which can support different
robot arms and enable collaborative teleoperation. It is also one of the only
two systems that can support different dexterous hands.

###  III-B System Design

The architecture of the teleoperation system is shown in Fig. 2. The
teleoperation server (Section IV) receives the camera stream from the driver,
detects the hand pose, and then converts it to joint control commands. The
client receives these commands via network communication and uses them to
control a simulated or real robot. The system is designed with three key
principles: modularity, communication-focused, and containerization.
Modularity is achieved by implementing well-defined input-output interfaces
for each sub-component, allowing for wide applicability to different robot
arms, dexterous hands, cameras, and realities. Communication-focused design
allows for remote and collaborative teleoperation and reduces computation
requirements on the operator’s side by deploying heavy computations on a
powerful server. Finally, the containerized design makes installation and
deployment easier compared to other robotics systems with heavy software
dependencies.

Figure 2: System Architecture. AnyTeleop is composed of four components: (i)
camera driver, which captures the human hand pose in RGB or RGB-D format; (ii)
teleportation server, the core component in our system, which performs hand
pose detection and converts detection results to robot control commands; (iii)
teleoperated robot, which is either a real robot or a simulated robot in a
virtual environment; (iv) web visualizer, which enables remote visualization
across the internet.

##  IV Teleoperation Server

The teleoperation server, outlined in Section III, utilizes the RGB or RGB-D
data from one or multiple cameras and generates smooth and collision-free
control commands for the robot arm and dexterous hand. It consists of four
modules: (i) the hand pose detection module, which predicts hand wrist and
finger poses from the camera stream, (ii) the detection fusion module, which
integrates the results from multiple cameras, (iii) the hand pose retargeting
module, which maps human hand poses to the dexterous robot hand, and (iv) the
motion generation module, which produces high-frequency control signals for
the robot arm. A standardized software interface is defined for all four
modules to facilitate flexibility and generalizability in AnyTeleop .

###  IV-A Hand Pose Detection

The hand pose detection module offers a unique feature to utilize input from
various camera configurations, including RGB or RGB-D cameras, and single or
multiple cameras. The design principle is to leverage more information, such
as depth, and additional cameras, to improve performance when available. But
it can also perform the task with minimal input, i.e. a single RGB camera. The
detection module has two outputs: local finger keypoint positions in the wrist
frame and global 6D wrist pose in the camera frame. The finger keypoint
detection only requires RGB data while the wrist pose detection can optionally
use depth information to achieve better results.

Finger Keypoint Detection. Our finger keypoint detection utilizes MediaPipe
[64], a lightweight, RGB-based hand detection tool that can operate in real-
time on a CPU. The MediaPipe detector can accurately locate 3D keypoints of 21
hand-knuckle coordinates in the wrist frame and 2D keypoints on the image.

Wrist Pose Detection from RGB-D. We use the pixel positions of the detected
keypoints to retrieve the corresponding depth values from the depth image.
Then, utilizing known intrinsic camera parameters, we compute the 3D positions
of the keypoints in the camera frame. The alignment of the RGB and depth
images is handled by the camera driver. With the 3D keypoint positions in both
the local wrist frame and global camera frame, we can estimate the wrist pose
using the Perspective-n-Point (PnP) algorithm.

Wrist Pose Detection from RGB only. The orientation of the hand can be
computed analytically from the local positions of the detected keypoints.
However, determining the wrist position in the camera frame can be challenging
without explicit 3D information. To enhance MediaPipe for global wrist pose
estimation, we adopt the approach used in FrankMocap [50] by incorporating an
additional neural network that predicts the weak perspective transformation
scale of the hand. The weak perspective transformation approximates the
original perspective camera model by assuming that the observed object is
farther from the camera than its size. Together with intrinsic parameters,
this scale factor can be used to approximate the 3D position of the hand. The
wrist position computed this way has a larger error than depth camera, but it
is still sufficient for many downstream teleoperation tasks.

###  IV-B Detection Fusion

The detection fusion module integrates multiple camera detection results.
Self-occlusion can be a problem when performing hand pose detection,
especially when the hand is perpendicular to the camera plane. Using multiple
cameras can alleviate this problem by providing additional views. However,
there are two main challenges in fusing multiple detection results: (i) each
camera can only estimate the hand pose in its own frame and (ii) there is no
straightforward metric to quantify the confidence of each detection result.

To overcome the first challenge, we perform an auto-calibration process using
the human hand as a natural marker. We use the first $N$ frames of hand
detection results from multiple cameras to calculate the relative rotation
between each camera, expressed in $SO(3)$. We find that although the absolute
position of detected hand pose is not so accurate in RGB-only setting, the
relative motion between consecutive frames is more robust. With orientation
between each camera, we can transform the detected relative motion from
different cameras into a single frame.

To address the second challenge, we use the SMPL-X [40] hand shape parameters
predicted from the detection module, as inspired by Qin et al. [43]. During
teleoperation, the true shape parameters should remain constant for a given
operator, but the predicted values can contain errors during self-occlusion.
We observe that larger shape parameter prediction errors often correspond to
larger pose errors. To approximate the confidence score, we take the mean of
the estimated shape parameters in the first $N$ frames as a reference and
compute the error between the predicted shape parameters and the reference.
Implementation-wise, we require the operator to spread their fingers during
the first $N$ frames to ensure an accurate reference value of shape
parameters. The fusion module then selects the relative motion captured by the
camera with the highest confidence score and forwards it to the next module.
In implementation, we choose $N=50$.

Figure 3: Real Robot Teleoperation Tasks. We replicate the ten manipulation
tasks proposed in Sivakumar et al. [54] using same or similar objects. Top
row, left to right: Pickup Box Object, Pickup Fabric Toy, Box Rotation,
Scissor Pickup, Cup Stack. Bottom row, left to right: Two Cup Stacking,
Pouring Cubes onto Plate, Cup Into Plate, Open Drawer and Open Drawer and
Pickup Cup.

###  IV-C Hand Pose Retargeting

The hand pose retargeting module maps the human hand pose data obtained from
perception algorithms into joint positions of the teleoperated robot hand.
This process is often formulated as an optimization problem [42, 16], where
the difference between the keypoint vectors of the human and robot hand is
minimized. The optimization can be defined as follows:

$\begin{split}\min_{q_{t}}\sum_{i=0}^{N}||\alpha
v_{t}^{i}-f_{i}(q_{t})&||^{2}+\beta||q_{t}-q_{t-1}||^{2}\\\ \mathrm{s.t.}\quad
q_{l}\leq q_{t}&\leq q_{u},\end{split}$ (1)

where $q_{t}$ represents the joint positions of the robot hand at time step
$t$, $v^{i}_{t}$ is the $i$-th keypoint vector for human hand computed from
the detected finger keypoints, $f_{i}(q_{t})$ is the $i$-th forward kinematics
function which takes the robot hand joint positions $q_{t}$ as input and
computes the $i$-th keypoint vector for the robot hand, $q_{l}$ and $q_{u}$
are the lower and upper limits of the joint position, $\alpha$ is a scaling
factor to account for hand size difference. An additional penalty term with
weight $\beta$ is included to improve temporal smoothness. When retargeting to
a different morphology, such as a Dclaw in Figure AnyTeleop: A General Vision-
Based Dexterous Robot Arm-Hand Teleoperation System, we need to specify the
keypoint vectors mapping between the robot and human fingers manually. It is
worth noting that this module only considers the robot hand.

###  IV-D Motion Generation

Given the detected wrist and hand pose, our goal is to generate smooth and
collision-free motion of robot arm to reach the target Cartesian end-effector
pose. Real-time motion generation methods are required to have a smooth
teleoperation experience. In the prior work of [16], the robot motion is
driven by Riemannian Motion Policies (RMPs) [49, 7] that can calculate
acceleration fields in real-time. However, accelerations towards a particular
end-effector pose do not guarantee natural trajectories. In this work, we
adopt CuRobo [56], a highly parallelized collision-free robot motion
generation library accelerated by GPUs, to generate natural and reactive robot
motion in real-time. In AnyTeleop, the motion generation module receives the
Cartesian pose of the end-effector at a low frequency ($25$ Hz) from the hand
detection and retargeting modules, and generates collision-free joint-space
trajectories within joint limits at a higher frequency ($120$ Hz). The
generated trajectories are ready for safe execution by impedance controllers
on either a simulated or real robot.

##  V Web-based Teleoperation Viewer

To better support the teleoperation tasks, we implement a web-based
visualization module to facilitate remote and collaborative teleoperation,
especially for teleoperation in simulated environments. It has the following
features: (i) browser-based viewer, which makes it easily accessible remotely;
(ii) synchronized visualization, i.e. two operators working on the same
collaborative task should see the same scene synchronously from their own
local view ports. The viewer is developed based upon the meshcat [10] library
and utilize Three.js [9] for rendering. The visualization server ports the
simulation results onto the browser after each simulation iteration. Operators
can get visual feedback from the browser window and move their hands to
control the corresponding robot. More details about the implementation of our
viewer can be found in the supplementary materials.

##  VI System Evaluation

HardWare |  | Desktop | Laptop
---|---|---|---
GPU | RTX 3090 | RTX 2070
CPU | i9-10980XE | i7-8750
Profiling | Modules | Time (ms) | Time (ms)
Hand Pose (RGB) | $26\pm 5$ | $34\pm 5$
Hand Pose (RGB-D) | $27\pm 5$ | $35\pm 5$
Fusion |  $1\pm 0$ |  $1\pm 0$
Retargeting |  $9\pm 7$ | $10\pm 9$
Motion |  $8\pm 3$ | $11\pm 5$
TABLE II: Profiling Results. We profile different modules inside teleoperation
server on both desktop and laptop. The time is measured when all teleoperation
modules are run on the same computer simultaneously.

Manipulation Task | RL | Baseline [43] | Ours
---|---|---|---
Floating-Hand | Relocate | $36.3\pm 15.3$ | $49.7\pm 18.3$ | $\textbf{53.7}\pm\textbf{12.2}$
Flip Mug | $33.7\pm 15.0$ | $\textbf{51.3}\pm\textbf{34.7}$ | $47.3\pm 28.3$
Open Door | $69.3\pm 38.0$ | $64.7\pm 14.7$ | $\textbf{73.3}\pm~{}\textbf{9.0}$
Arm-Hand | Relocate | $33.7\pm 29.3$ | $40.3\pm 36.7$ | $\textbf{70.0}\pm~{}\textbf{9.8}$
Flip Mug | $31.0\pm 28.7$ | $36.0\pm 32.4$ | $\textbf{53.7}\pm\textbf{24.0}$
Open Door | $34.7\pm 31.7$ | $51.3\pm 30.7$ | $\textbf{79.7}\pm\textbf{15.5}$

tableImitation Learning Experiments in SAPIEN Environments. The left figure
visualizes the three tasks we use for both teleoperation data collection and
imitation learning. The transparent object represents the goal of the task
while the black arrow represents the steps of the task. The right table shows
the success rate of the evaluated methods. We use $\pm$ to represent the mean
and standard deviation over three random seeds. The success rate is computed
from 100 trials.

###  VI-A Profiling Analysis

We perform profiling on modules mentioned in Section IV on a desktop and a
laptop. As shown in Table II, the most time-consuming module is hand pose
detection, which runs on a GPU for real-time inference. The designed maximum
frequency for hand pose detection is 25Hz, so both the desktop and laptop can
meet the requirement. Both the retargeting module and the fusion module run at
the same frequency as the hand detection module due to the publisher and
subscriber logic. For best performance, the motion generation module should
run at 120Hz but can still work with a lower frequency. Notably, we found it
difficult to achieve this throughput when running all these modules on the
same computer. Luckily, with our communication-oriented design, we can run the
control modules on a separate machine to achieve the best performance.

Task | AnyTeleop | Telekinesis [54]
---|---|---
Pickup Box Object | 1.0 | 0.9
Pickup Fabric Toy | 1.0 | 0.9
Box Rotation | 0.6 | 0.6
Scissor Pickup | 0.8 | 0.7
Cup Stack | 0.9 | 0.6
Two Cup Stacking | 0.7 | 0.3
Pouring Cubes onto Plate | 0.7 | 0.7
Cup Into Plate | 1.0 | 0.8
Open Drawer | 1.0 | 0.9
Open Drawer and Pickup Object | 0.9 | 0.6
TABLE III: Real Robot Teleoperation Results. We replicate the experiment
settings and tasks in [54] and compare with [54]. For the baseline method, we
use the success rate reported in their paper [54]

###  VI-B Real Robot Teleoperation

In this section, we will test our AnyTeleop system across a wide range of
real-world tasks that covers diverse objects and manipulation skills. Besides,
we will compare our teleoperation performance of AnyTeleop with a similar
teleoperation system. A fair comparison of real-robot tasks is often very
challenging due to the difficulty in replicating the baseline methods
delicately. To ensure a more fair comparison, we replicate the ten
manipulation tasks proposed in Robotic Telekinesis [54] with the same XArm6
robot, Allegro hand, and similar objects. A trained operator attempt to solve
this tasks using AnyTeleop system. The ten tasks are visualized in Fig. 3.
Same as [54], we run each task ten times for AnyTeleop and use a single Intel
RealSense camera. For the baseline method, we directly use the results
reported in their paper.

As shown in Table III, AnyTeleop can get a higher success rate of 8/10 tasks
and the same success rate on 2/10 compared with the baseline. Although
AnyTeleop is designed to be more general, it can still outperform the baseline
system that was specifically designed for the XArm6-Allegro hardware. We find
that the major advantage of our system is the capability to handle objects
with thin-walled structures, such as the cup-stack, two-cup-stacking, and cup-
into-plate tasks. Our optimization-based retargeting module can close the
distance between finger tips, which makes grasping the cup more stable.
However, the network-based retargeting can hardly translate the fine-grained
precision grasp from human to robot, which leads to a lower success rate.

##  VII Applications

###  VII-A Imitation Learning

The most important application of the proposed system is imitation learning
from demonstration. We can first collect demonstrations on several dexterous
manipulation tasks and then use the data to train imitation learning
algorithms. In this experiment, we will show that the teleoperation data
collected using our AnyTeleop can better support downstream imitation learning
tasks. In the following subsection, we will first introduce the experiment
setting and baseline and then discuss the experimental results.

Baseline and Comparison. To fairly compare with previous teleoperation
systems, we need to align both the task setting and robot configuration
precisely. It is often challenging for real-robot hardware but much easier for
a simulated environment. Thus, we choose a recent vision-based teleoperation
work [43] that can be used for simulated robots as our baseline. It is worth
noting that we are comparing two teleoperation systems via the demonstration
data collected by each system. Thus, we compared with the baseline by training
the same learning algorithm on different demonstration data collected via the
baseline system and our teleoperation system. We follow [43] to choose Demo
Augmented Policy Gradient (DAPG) [46] as the imitation algorithm. We also
compare it with a pure reinforcement learning (RL) based algorithm from [44]
which does not utilize demonstrations. We provide the same dense reward for RL
training as previous work [43].

Figure 4: Collaborative Teleoperation for Handover Task. Operator $\\#1$ act
as the UR10-Schunk robot and operator $\\#2$ acts as the Kuka-Shadow robot. In
this task, the operator $\\#2$ needs to pick up an object on the table and
handover it to operator $\\#1$.

Manipulation Tasks. We directly use the manipulation tasks proposed by the
baseline work [43] for comparison, which include three tasks: (i) Relocate,
where the robot picks an object on the table and moves it to the target
position; (ii) Flip Mug, where the robot needs to rotate the mug for 90
degrees to flip it back; (iii) Open Door, where the robot needs to first
rotate the lever to unlock the door, and then pull it to open the door. These
tasks are visualized in the left figure of Table VI. The manipulated objects
in all three tasks are randomly initialized and the target position is also
randomized in _Relocate_. Each manipulation task has two variants: the
floating-hand variant and the arm-hand variant. The floating-hand is a
dexterous hand without a robot arm that can move freely in space. The arm-hand
means the hand is mounted on a robot arm with a fixed base, which is a more
realistic setting.

Demonstration Details. For the baseline teleoperation system [43], we directly
use the demonstration collected by the original authors with 50 demonstration
trajectories for each task. The baseline system only utilizes a single RGB-D
camera. For fairness, we also collect 50 trajectories for each task using the
single camera setup. The baseline system can only handle floating hands and
they propose a demonstration translation pipeline to convert the demonstration
with floating hands to demonstrations with arm-hand. For our AnyTeleop , we
collect demonstrations using the arm-hand setting and convert the
demonstration to floating hand so that the demonstration can be used by both
the floating-hand variant and arm-hand variant.

Figure 5: Collaborative Teleoperation System. Our system can be extended to
collaborative manipulation tasks even when operators are not in the same
physical location. Each operator can use a local computer with camera to
detect the hand pose and send the detection results to a central server.
Meanwhile, they can use the web browser to visualize the current simulation
environment, including the robot controlled by other operators.

Results and Discussion. For each method on each task, we train policies with
three different random seeds. For each policy, we evaluate it on 100 trials.
More details about the success metrics can be found in [43]. As shown in Table
VI, the imitation learning algorithm trained on demonstration collected by
AnyTeleop can outperform baseline and RL on most tasks with one exception.
Compared with the demonstration collected via the baseline system, our system
has two benefits that contribute to better performance in imitation learning:
(i) The collected trajectory is more smooth, which means that the state-action
pairs are more consistent and easier to be consumed by the network. (ii)
Different from the baseline, our system explicitly supports teleoperation with
arm-hand system and guarantees no self-collision. On the contrary, the
baseline system utilizes retargeting to generate joint trajectory for robot
arm, which may lead to several self-collision for robot arm. Thus we can
observe significant performance gain of our system for manipulation tasks with
arm-hand. For the flip mug task, the difficulty of collecting demonstration
with arm is much larger than with a floating hand, which influences the
demonstration quality.

###  VII-B Collaborative Manipulation

Collaborative manipulation is a key technology for the development of human-
robot systems [59]. Collecting demonstration data for collaborative
manipulation tasks has been a challenging task since it requires multiple
operators to work together seamlessly. With our modularized and extensible
system design and web-based visualization, our system enables convenient data
collection on collaborative tasks, even if operators are not in the same
physical location. In this section, we show that our teleoperation system can
be extended to a collaborative setting where multiple operators coordinate
together to perform manipulation tasks. We choose human-to-robot handover as
an example as shown in Fig. 4. In this setting, operator $\\#1$ control a
robot hand, and operator $\\#2$ control a human hand.

Collaborative Teleoperation System Design. Fig. 5 illustrates the system
architecture for multi-operator collaboration, which includes two components.
(i) Teleoperation Units: It is composed of a computer that is connected to at
least one camera and a human operator. In each teleoperation unit, the human
operator will watch the real-time visualization on a web browser and move the
hand accordingly to perform manipulation tasks. (ii) Central Server: it runs
the physical simulation and the web visualization server. The detection
results from multiple teleoperation units are sent to the server and converted
into robot control commands based on the pipeline in Section IV. Meanwhile,
the web visualizer server will keep synchronized with the simulated
environment and maintain the visualization resources as described in Section
V.

##  VIII Failure Modes

As illustrated on the project page, we have identified two failure modes: (i)
loss of tracking during fast human hand motion, which triggers a pause and re-
detection process; (ii) unreliable hand pose when the hand is in self-
occlusion. For mode (i), the workaround is to instruct the operator to slow
down their hand motion. The issue (ii) can be solved by incorporating multiple
cameras for tasks that require significant hand rotation.

##  IX Conclusion

In this paper, we introduced AnyTeleop, a versatile teleoperation system that
can be applied to diverse robots, assorted reality, and varied camera setup,
and can be operated by a flexible number of users from any geographic
locations. The experiments show that AnyTeleop outperforms previous systems in
both simulation and real-world scenarios while offering superior
generalizability and flexibility. Our commitment to an open-source approach
will facilitate further research in the field of teleoperation.

##  X Acknowledgement

We express our gratitude to Ankur Handa, Balakumar Sundaralingam, and Nick
Walker for their insightful discussions throughout the development process of
the motion control module in AnyTeleop. We would also like to extend our
thanks to Isabella Liu, An-Chieh Cheng, Ruihan Yang, Yang Fu, Linghao Chen,
Jiarui Xu, Xinyu Zhang, Xinyue Wei, Jiteng Mu, and Jianglong Ye for their
efforts in testing and evaluating the teleoperation system.

## References

  * Ajili et al. [2017] Insaf Ajili, Malik Mallem, and Jean-Yves Didier.  Gesture recognition for humanoid robot teleoperation.  In _2017 26Th IEEE international symposium on robot and human interactive communication (RO-MAN)_ , pages 1115–1120. IEEE, 2017.
  * Antotsiou et al. [2018] Dafni Antotsiou, Guillermo Garcia-Hernando, and Tae-Kyun Kim.  Task-oriented hand motion retargeting for dexterous manipulation imitation.  In _ECCV Workshops_ , 2018.
  * Aronson and Admoni [2022] Reuben M Aronson and Henny Admoni.  Gaze complements control input for goal prediction during assisted teleoperation.  In _Robotics science and systems_ , 2022.
  * Arunachalam et al. [2022a] Sridhar Pandian Arunachalam, Irmak Güzey, Soumith Chintala, and Lerrel Pinto.  Holo-dex: Teaching dexterity with immersive mixed reality.  _arXiv preprint arXiv:2210.06463_ , 2022a.
  * Arunachalam et al. [2022b] Sridhar Pandian Arunachalam, Sneha Silwal, Ben Evans, and Lerrel Pinto.  Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation.  _arXiv preprint arXiv:2203.13251_ , 2022b.
  * Chen et al. [2022] Sean Chen, Jensen Gao, Siddharth Reddy, Glen Berseth, Anca D Dragan, and Sergey Levine.  Asha: Assistive teleoperation via human-in-the-loop reinforcement learning.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 7505–7512. IEEE, 2022.
  * Cheng et al. [2020] Ching-An Cheng, Mustafa Mukadam, Jan Issac, Stan Birchfield, Dieter Fox, Byron Boots, and Nathan Ratliff.  Rmp flow: A computational graph for automatic motion policy generation.  In _Algorithmic Foundations of Robotics XIII: Proceedings of the 13th Workshop on the Algorithmic Foundations of Robotics 13_ , pages 441–457. Springer, 2020.
  * Coumans and Bai [2016] Erwin Coumans and Yunfei Bai.  PyBullet, a python module for physics simulation for games, robotics and machine learning.  _GitHub repository_ , 2016.
  * Danchilla and Danchilla [2012] Brian Danchilla and Brian Danchilla.  Three. js framework.  _Beginning WebGL for HTML5_ , pages 173–203, 2012.
  * Deits [2018] Robin Deits.  Meshcat.  https://github.com/rdeits/meshcat, 2018.
  * Du et al. [2012] Guanglong Du, Ping Zhang, Jianhua Mai, and Zeling Li.  Markerless kinect-based hand tracking for robot teleoperation.  _International Journal of Advanced Robotic Systems_ , 9(2):36, 2012.
  * Elsner et al. [2022] Jean Elsner, Gerhard Reinerth, Luis Figueredo, Abdeldjallil Naceri, Ulrich Walter, and Sami Haddadin.  Parti-a haptic virtual reality control station for model-mediated robotic applications.  _Frontiers in Virtual Reality_ , 3, 2022.
  * Fang et al. [2020] Bin Fang, Xiao Ma, Jiachun Wang, and Fuchun Sun.  Vision-based posture-consistent teleoperation of robotic arm using multi-stage deep neural network.  _Robotics and Autonomous Systems_ , 131:103592, 2020.
  * Gan et al. [2020] Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, et al.  Threedworld: A platform for interactive multi-modal physical simulation.  _arXiv preprint arXiv:2007.04954_ , 2020.
  * Gharaybeh et al. [2019] Zaid Gharaybeh, Howard Chizeck, and Andrew Stewart.  _Telerobotic control in virtual reality_.  IEEE, 2019.
  * Handa et al. [2020] Ankur Handa, Karl Van Wyk, Wei Yang, Jacky Liang, Yu-Wei Chao, Qian Wan, Stan Birchfield, Nathan Ratliff, and Dieter Fox.  Dexpilot: Vision-based teleoperation of dexterous robotic hand-arm system.  In _ICRA_ , 2020.
  * Hedayati et al. [2018] Hooman Hedayati, Michael Walker, and Daniel Szafir.  Improving collocated robot teleoperation with augmented reality.  In _International Conference on Human-Robot Interaction_ , 2018.
  * Huang et al. [2021] Zebin Huang, Ziwei Wang, Weibang Bai, Yanpei Huang, Lichao Sun, Bo Xiao, and Eric M Yeatman.  A novel training and collaboration integrated framework for human–agent teleoperation.  _Sensors_ , 21(24):8341, 2021.
  * Jang et al. [2022] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn.  Bc-z: Zero-shot task generalization with robotic imitation learning.  In _Conference on Robot Learning_ , pages 991–1002. PMLR, 2022.
  * Khadir et al. [2019] Bachir El Khadir, Jake Varley, and Vikas Sindhwani.  Teleoperator imitation with continuous-time safety.  _arXiv preprint arXiv:1905.09499_ , 2019.
  * Kofman et al. [2005] Jonathan Kofman, Xianghai Wu, Timothy J Luu, and Siddharth Verma.  Teleoperation of a robot manipulator using a vision-based human-robot interface.  _IEEE transactions on industrial electronics_ , 52(5):1206–1219, 2005.
  * Kofman et al. [2007] Jonathan Kofman, Siddharth Verma, and Xianghai Wu.  Robot-manipulator teleoperation by markerless vision-based hand-arm tracking.  _International Journal of Optomechatronics_ , 2007.
  * Kumar and Todorov [2015] Vikash Kumar and Emanuel Todorov.  Mujoco haptix: A virtual reality system for hand manipulation.  In _International Conference on Humanoid Robots (Humanoids)_ , 2015\.
  * Li et al. [2019] Shuang Li, Xiaojian Ma, Hongzhuo Liang, Michael Görner, Philipp Ruppel, Bin Fang, Fuchun Sun, and Jianwei Zhang.  Vision-based teleoperation of shadow dexterous hand using end-to-end deep neural network.  In _ICRA_ , 2019.
  * Li et al. [2020] Shuang Li, Jiaxi Jiang, Philipp Ruppel, Hongzhuo Liang, Xiaojian Ma, Norman Hendrich, Fuchun Sun, and Jianwei Zhang.  A mobile robot hand-arm teleoperation system by vision and IMU.  In _IROS_ , 2020.
  * Li et al. [2022] Shuang Li, Norman Hendrich, Hongzhuo Liang, Philipp Ruppel, Changshui Zhang, and Jianwei Zhang.  A dexterous hand-arm teleoperation system based on hand pose estimation and active vision.  _IEEE Transactions on Cybernetics_ , 2022.
  * Liang et al. [2020] Jacky Liang, Ankur Handa, Karl Van Wyk, Viktor Makoviychuk, Oliver Kroemer, and Dieter Fox.  In-hand object pose tracking via contact feedback and gpu-accelerated robotic simulation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6203–6209. IEEE, 2020.
  * Lipton et al. [2017] Jeffrey I Lipton, Aidan J Fay, and Daniela Rus.  Baxter’s homunculus: Virtual reality spaces for teleoperation in manufacturing.  _IEEE Robotics and Automation Letters_ , 3(1):179–186, 2017.
  * Liu et al. [2017] Hangxin Liu, Xu Xie, Matt Millar, Mark Edmonds, Feng Gao, Yixin Zhu, Veronica J Santos, Brandon Rothrock, and Song-Chun Zhu.  A glove-based system for studying hand-object manipulation via joint pose and force sensing.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6617–6624. IEEE, 2017.
  * Liu et al. [2019] Hangxin Liu, Zhenliang Zhang, Xu Xie, Yixin Zhu, Yue Liu, Yongtian Wang, and Song-Chun Zhu.  High-fidelity grasping in virtual reality using a glove-based system.  In _2019 international conference on robotics and automation (icra)_ , pages 5180–5186. IEEE, 2019.
  * Liu et al. [2021] Shaowei Liu, Hanwen Jiang, Jiarui Xu, Sifei Liu, and Xiaolong Wang.  Semi-supervised 3d hand-object poses estimation with interactions in time.  In _CVPR_ , 2021.
  * Makoviychuk et al. [2021] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al.  Isaac gym: High performance gpu-based physics simulation for robot learning.  _arXiv preprint arXiv:2108.10470_ , 2021.
  * Mandlekar et al. [2018] Ajay Mandlekar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, et al.  Roboturk: A crowdsourcing platform for robotic skill learning through imitation.  In _Conference on Robot Learning_ , pages 879–893. PMLR, 2018.
  * Mandlekar et al. [2020] Ajay Mandlekar, Danfei Xu, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese.  Human-in-the-loop imitation learning using remote teleoperation.  _arXiv preprint arXiv:2012.06733_ , 2020.
  * Meeker et al. [2020] Cassie Meeker, Maximilian Haas-Heger, and Matei Ciocarlie.  A continuous teleoperation subspace with empirical and algorithmic mapping algorithms for nonanthropomorphic hands.  _IEEE Transactions on Automation Science and Engineering_ , 19(1):373–386, 2020.
  * Mizera et al. [2019] C Mizera, T Delrieu, V Weistroffer, C Andriot, A Decatoire, and J-P Gazeau.  Evaluation of hand-tracking systems in teleoperation and virtual dexterous manipulation.  _IEEE Sensors Journal_ , 20(3):1642–1655, 2019\.
  * Mosbach et al. [2022] Malte Mosbach, Kara Moraw, and Sven Behnke.  Accelerating interactive human-like manipulation learning with GPU-based simulation and high-quality demonstrations.  In _Humanoids_ , 2022.
  * Muelling et al. [2015] Katharina Muelling, Arun Venkatraman, Jean-Sebastien Valois, John Downey, Jeffrey Weiss, Shervin Javdani, Martial Hebert, Andrew B Schwartz, Jennifer L Collinger, and J Andrew Bagnell.  Autonomy infused teleoperation with application to bci manipulation.  _arXiv preprint arXiv:1503.05451_ , 2015.
  * Niemeyer et al. [2016] Günter Niemeyer, Carsten Preusche, Stefano Stramigioli, and Dongjun Lee.  Telerobotics.  _Springer handbook of robotics_ , pages 1085–1108, 2016.
  * Pavlakos et al. [2019] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black.  Expressive body capture: 3d hands, face, and body from a single image.  In _CVPR_ , 2019.
  * Ponomareva et al. [2021] Polina Ponomareva, Daria Trinitatova, Aleksey Fedoseev, Ivan Kalinov, and Dzmitry Tsetserukou.  Grasplook: a vr-based telemanipulation system with r-cnn-driven augmentation of virtual environment.  In _2021 20th International Conference on Advanced Robotics (ICAR)_ , pages 166–171. IEEE, 2021.
  * Qin et al. [2021] Yuzhe Qin, Yueh-Hua Wu, Shaowei Liu, Hanwen Jiang, Ruihan Yang, Yang Fu, and Xiaolong Wang.  Dexmv: Imitation learning for dexterous manipulation from human videos.  _arXiv preprint arXiv:2108.05877_ , 2021.
  * Qin et al. [2022] Yuzhe Qin, Hao Su, and Xiaolong Wang.  From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation.  _RA-L_ , 7(4):10873–10881, 2022.
  * Qin et al. [2023] Yuzhe Qin, Binghao Huang, Zhao-Heng Yin, Hao Su, and Xiaolong Wang.  Dexpoint: Generalizable point cloud reinforcement learning for sim-to-real dexterous manipulation.  In _Conference on Robot Learning_ , pages 594–605. PMLR, 2023.
  * Radosavovic et al. [2022] Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell.  Real-world robot learning with masked visual pre-training.  _arXiv preprint arXiv:2210.03109_ , 2022.
  * Rajeswaran et al. [2018] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  In _RSS_ , 2018.
  * Rakita et al. [2017] Daniel Rakita, Bilge Mutlu, and Michael Gleicher.  A motion retargeting method for effective mimicry-based teleoperation of robot arms.  In _Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_ , pages 361–370, 2017.
  * Rakita et al. [2019] Daniel Rakita, Bilge Mutlu, and Michael Gleicher.  Remote telemanipulation with adapting viewpoints in visually complex environments.  _Robotics: Science and Systems XV_ , 2019.
  * Ratliff et al. [2018] Nathan D Ratliff, Jan Issac, Daniel Kappler, Stan Birchfield, and Dieter Fox.  Riemannian motion policies.  _arXiv preprint arXiv:1801.02854_ , 2018.
  * Rong et al. [2020] Yu Rong, Takaaki Shiratori, and Hanbyul Joo.  Frankmocap: Fast monocular 3d hand and body motion capture by regression and integration.  _arXiv preprint arXiv:2008.08324_ , 2020.
  * Rosen et al. [2020] Eric Rosen, David Whitney, Michael Fishman, Daniel Ullman, and Stefanie Tellex.  Mixed reality as a bidirectional communication interface for human-robot interaction.  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 11431–11438. IEEE, 2020.
  * Salvato et al. [2022] M Salvato, Negin Heravi, Allison M Okamura, and Jeannette Bohg.  Predicting hand-object interaction for improved haptic feedback in mixed reality.  _IEEE Robotics and Automation Letters_ , 7(2):3851–3857, 2022.
  * Shaw et al. [2022] Kenneth Shaw, Shikhar Bahl, and Deepak Pathak.  Videodex: Learning dexterity from internet videos.  _arXiv preprint arXiv:2212.04498_ , 2022.
  * Sivakumar et al. [2022] Aravind Sivakumar, Kenneth Shaw, and Deepak Pathak.  Robotic telekinesis: learning a robotic hand imitator by watching humans on youtube.  _arXiv preprint arXiv:2202.10448_ , 2022.
  * Son et al. [2013] Hyoung Il Son, Antonio Franchi, Lewis L Chuang, Junsuk Kim, Heinrich H Bulthoff, and Paolo Robuffo Giordano.  Human-centered design and evaluation of haptic cueing for teleoperation of multiple mobile robots.  _IEEE transactions on cybernetics_ , 43(2):597–609, 2013.
  * Sundaralingam et al. [2023] Balakumar Sundaralingam, Siva Hari, Adam Fishman, Caelan Garrett, Karl Van Wyk, Valts Blukis, Alexander Millane, Helen Oleynikova, Ankur Handa, Fabio Ramos, Nathan Ratliff, and Dieter Fox.  CuRobo: Parallelized collision-free robot motion generation.  In _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , June 2023.
  * Theobalt et al. [2004] Christian Theobalt, Irene Albrecht, Jörg Haber, Marcus Magnor, and Hans-Peter Seidel.  Pitching a baseball: tracking high-speed motion with multi-exposure images.  In _ACM SIGGRAPH 2004 Papers_ , pages 540–547. 2004.
  * Todorov et al. [2012] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _IROS_ , 2012.
  * Tung et al. [2021] Albert Tung, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Yuke Zhu, Li Fei-Fei, and Silvio Savarese.  Learning multi-arm manipulation through collaborative teleoperation.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9212–9219. IEEE, 2021.
  * Wang and Popović [2009] Robert Y Wang and Jovan Popović.  Real-time hand-tracking with a color glove.  _ACM transactions on graphics (TOG)_ , 28(3):1–8, 2009.
  * Wei et al. [2021] Dong Wei, Bidan Huang, and Qiang Li.  Multi-view merging for robot teleoperation with virtual reality.  _IEEE Robotics and Automation Letters_ , 6(4):8537–8544, 2021.
  * Xiang et al. [2020] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al.  Sapien: A simulated part-based interactive environment.  In _CVPR_ , 2020.
  * Ye et al. [2023] Jianglong Ye, Jiashun Wang, Binghao Huang, Yuzhe Qin, and Xiaolong Wang.  Learning continuous grasping function with a dexterous hand from human demonstrations.  _IEEE Robotics and Automation Letters_ , 8(5):2882–2889, 2023.
  * Zhang et al. [2020] Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann.  Mediapipe hands: On-device real-time hand tracking.  _arXiv preprint arXiv:2006.10214_ , 2020.
  * Zhang et al. [2021] Haodong Zhang, Weijie Li, Yuwei Liang, Zexi Chen, Yuxiang Cui, Yue Wang, and Rong Xiong.  Human-robot motion retargeting via neural latent optimization.  _CoRR_ , 2021.
  * Zhang et al. [2018a] Heng Zhang, Zeming Zhao, Yang Yu, Kai Gui, Xinjun Sheng, and Xiangyang Zhu.  A feasibility study on an intuitive teleoperation system combining imu with semg sensors.  In _Intelligent Robotics and Applications: 11th International Conference, ICIRA 2018, Newcastle, NSW, Australia, August 9–11, 2018, Proceedings, Part I 11_ , pages 465–474. Springer, 2018a.
  * Zhang et al. [2018b] Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg, and Pieter Abbeel.  Deep imitation learning for complex manipulation tasks from virtual reality teleoperation.  In _ICRA_ , 2018b.
  * Zhao et al. [2012] Wenping Zhao, Jinxiang Chai, and Ying-Qing Xu.  Combining marker-based mocap and rgb-d camera for acquiring high-fidelity hand motion data.  In _Proceedings of the ACM SIGGRAPH/eurographics symposium on computer animation_ , pages 33–42, 2012.

###  -A Supplementary Overview

This supplementary material provides more details, results and visualizations
accompanying the main paper, including

  * •

More details and visualization about teleoperation server, including detection
and retargeting modules;

  * •

More details about web-based teleoperation viewer;

  * •

Additional experimental results on system evaluation.

More visualization can be found at our project page: http://anyteleop.com.

###  -B Teleoperation Server

In this section, we will show more intermediate results from our system,
including visualization of hand pose detection results and the retargeting
results of various robot hands.

Visualization of Hand Pose Detection We visualize the hand pose detection
results in Figure 6. We showcase five typical cases, which include: (i) a hand
spreading out the fingers for teleoperation initialization, (ii) fingers
facing downwards in preparation for a top-down grasp, (iii) a precision grasp
using the thumb and index finger, (iv) a power grasp using all five fingers,
and (v) a failure case where the hand is positioned vertically relative to the
camera plane.

Visualization of Hand Pose Retargeting We demonstrate the results of hand pose
retargeting in Figure 9. The figure displays seven gestures being performed
using four different dexterous hands.

Figure 6: Hand Pose Detection Visualization. This figure visualizes the hand
detection results, with the white bounding box highlighting the predicted area
and red points marking the identified finger key points. The hand skeleton is
represented by the grey lines connecting the key points. Additionally, the
small grey points depict the 2D projection of 3D vertices from the SMPL-X hand
model. The figure showcases five diverse cases, from left to right: (i) a hand
spreading out the fingers to initiate teleoperation, (ii) fingers facing
downwards in preparation for a top-down grasp, (iii) a precision grasp using
the thumb and index finger, (iv) a power grasp using all five fingers, and (v)
a failure scenario where the hand is positioned vertically relative to the
camera plane.

###  -C Web-based Teleoperation Viewer

In this section, we demonstrate how the web-based visualizer provides
accessibility and multi-view support for teleoperation through its lightweight
rendering and capability to run in multiple browser windows. Figure 7 shows
screenshots of the web-based visualizer when it is used to visualize the five
IsaacGym tasks depicted in the Figure 1 in the main paper.

Lightweight Rendering vs High Visual Quality. The design of our web-based
viewer prioritizes accessibility and convenience, as it can be used on any
device with a browser and provides minimal but sufficient rendering
capabilities for teleoperation. Although the rendering quality may not be as
advanced as the original simulator viewer, simulation states can be saved for
offline rendering to produce high-quality visual data. For example, in visual
reinforcement learning tasks using RGB images as inputs, the rendered data can
be generated using a more powerful engine such as a ray tracer after
teleoperation is completed.

Multi-View Support for Teleoperation. In teleoperation, human operators often
require a clear understanding of the spatial relationships between objects and
robots to make informed decisions. This information can be provided through
multi-view rendering, which is a widely used technique in previous
teleoperation works [26, 43, 61]. Our web-based viewer offers multi-view
support to the operator by simply opening multiple browser windows. As shown
in Figure 8, an example of the operator using two views to perform a
manipulation task is displayed. The operator is able to open as many windows
as needed to enhance their teleoperation experience.

Figure 7: Web-based Viewer. The visualization of teleoperation process can be
perform in the web-based viewer, which features the five tasks from the
IsaacGym tasks shown in the Figure 1 of the main paper. The viewer utilizes
the three.js library for real-time rendering through a web browser. Figure 8:
Multi-view Support with More Browser Windows. The web-based viewer offers
multiple views to enhance the operator’s understanding of the 3D object
relationships. Additional views can be accessed by simply opening more browser
windows.

##  Appendix A System Evaluation on Camera Configurations

Camera Configuration | Completion Time | Error Percentage (%)
---|---|---
Single RGB | $109s$ | $28.1\%$
Single RGB-D | $87s$ | $21.8\%$
Two RGB-D |  $\mathbf{74}$s | $\mathbf{12.5\%}$
TABLE IV: Comparison of Camera Configurations. We evaluate the teleoperation
performance on the Play Piano task with different camera configurations.

In this section, we examine the impact of different camera configurations on
the teleoperation performance of our system, AnyTeleop , which is capable of
supporting diverse configurations including RGB, RGB-D, and single or multiple
cameras. Even with a minimal configuration, i.e. a single RGB camera, the
system can still perform effectively. Additionally, by adding more resources,
such as multiple cameras, our system can achieve better performance.

We use the Play Piano task implemented in IsaacGym [32] as the evaluation
scenario, which requires the robot hand to press piano keys in a specific
order. The task is shown in the Figure 1 in the main paper and the video in
the supplementary material. To quantify performance, we introduce two task
metrics: (i) completion time, i.e. the elapsed time from start to finish, and
(ii) the percentage of incorrect key presses, which measures the number of
incorrectly pressed keys relative to the total number of keys.

A trained operator performs the task ten times for each camera configuration.
As reported in Table IV, with additional information, such as depth, and
increasing number of cameras, the task can be completed faster and with fewer
errors, which demonstrates that our system allows users to easily trade-off
between efficiency and system cost based on their use case.

Figure 9: Visualization of Hand Pose Retargeting. The figure presents the
results of hand pose retargeting for seven gestures and four different
dexterous robot hands. The four hands are displayed in order from left to
right: (i) Schunk SVH hand; (ii) Shadow Hand; (iii) DLR Hand; (iv) Allegro
Hand.


# Reinforcement Learning Agents in Colonel Blotto

Joseph Christian G. Noel

###### Abstract

Models and games are simplified representations of the world. There are many
different kinds of models, all differing in complexity and which aspect of the
world they allow us to further our understanding of. In this paper we focus on
a specific instance of agent-based models, which uses reinforcement learning
(RL) to train the agent how to act in its environment. Reinforcement learning
agents are usually also Markov processes, which is another type of model that
can be used. We test this reinforcement learning agent in a Colonel Blotto
environment111Source code for the experiments can be found at
https://github.com/jinonoel/BlottoRL, and measure its performance against
Random agents as its opponent. We find that the RL agent handily beats a
single opponent, and still performs quite well when the number of opponents
are increased. We also analyze the RL agent and look at what strategies it has
arrived by looking at the actions that it has given the highest and lowest
$Q$-values. Interestingly, the optimal strategy for playing multiple opponents
is almost the complete opposite of the optimal strategy for playing a single
opponent.

##  1 Colonel Blotto

Colonel Blotto is a constant-sum game proposed by Émile Borel in 1921[1]. In
the game two or more players distribute resources (or coins) over several
fronts in a battlefield. A player wins a front if they have allocated more
resources to it than the other player. The objective of the game is to win
more fronts than your opponent.

A notable thing about Colonel Blotto is that there is no optimal best strategy
that will beat all other possible strategies. This means that the best
strategy can change in a given game depending on the opponent and what
strategy the opponent is using. One way to play Colonel Blotto would be to
update your strategy over time as you play the same opponent repeatedly. If
the opponent’s strategy is constant or doesn’t change much, it must be
possible to eventually ”learn” a strategy that would win against this opponent
more often than it loses. One way to do this would be to create an agent-based
model that would update its strategy as it plays an opponent multiple times.

##  2 Reinforcement Learning

Reinforcement learning is a form of agent-based modeling. In RL an agent
learns by performing actions which changes the state of an environment. After
each action the agent may also receive a ”reward” from the environment, whose
value should depend on how close the agent is to what we want it to achieve.
The goal for the agent is to maximize the cumulative reward that it receives
over time. After a series of actions, the agent eventually reaches a goal
state or terminal state, which signifies the end of an episode. The
environment is then reset and the agent starts again from an initial state and
the process repeats itself.

Formally, an RL model is a set of states $S$, a set of actions $A$, and
transition rules between states depending on the action taken. For state $s\in
S$ at time $t$, an agent performs an action $a\in A$, moves to a new state
$s^{\prime}$ and receives a reward $r_{t}$. The agent follows a policy $\pi$
for choosing which action to take, and $\pi(s,a)$ is the probability mapping
of an agent taking action $a$ while in state $s$. The goal of the agent is to
maximize the expected reward $Rt$,

$R_{t}=\sum_{k=0}^{\infty}\lambda^{k}r_{t+k}$ (1)

where $0\leq\gamma\leq 1$ is a discount factor for handling infinite horizons.

###  2.1 Markov Decision Process

Calculating the optimal policy in general is a hard problem, especially if we
need to keep track of the history of all states, actions, and rewards. To
simplify things, reinforcement learning problems are usually constructed as
Markov models, such that the information from the history of all states,
actions, and rewards before time $t$ is encapsulated in the current state at
time $t$. This is the Markov property, and control optimization tasks which
exhibit this property are called Markov decision processes (MDP). Formally,
the Markov property states that for transition probability function $Pr$,

$Pr(s_{t+1},r_{t+1}|s_{t},a_{t})=Pr(s_{t+1},r_{t+1}|s_{t},a_{t},r_{t},s_{t-1},a_{t-1},r_{t-1},...,s_{0},a_{0},r_{0})$
(2)

###  2.2 Value Function

The state-action value function $Q$ is the estimate of the expected rewards
the agent will receive by being at state $s$ at time $t$, taking action $a$,
and then following policy $\pi$ from $t+1$ onwards.

$Q^{\pi}(s,a)=\mathbb{E}_{\pi}(R_{t}|s_{t},a_{t})$ (3)

The reinforcement learning problem can be reduced to being able to calculate
$Q(s,a)$ accurately for all $s\in S$ and $a\in A$. The agent can then follow
the optimal policy $\pi^{*}$ by simply choosing the action $a$ that maximizes
$Q(s,a)$ for all $s\in S$.

$\pi^{*}(s)=\operatorname*{arg\,max}_{a}{Q(s,a)}$ (4)

###  2.3 Q-Learning

Q-Learning is one of the basic reinforcement learning methods, and takes full
advantage of the Markov property of the RL model. It approximates the state-
action value function $Q(s,a)$ in a recursive manner at time $t$ using a
weighted average of the current value and the new information retrieved from
applying the action $a_{t+1}$ and receiving subsequent reward $r_{t+1}$ from
the environment.

At each time step Q-Learning approximates the optimal policy by iteratively
updating $Q$ in the following manner, where $\alpha>0$ is the learning rate,
and $0\leq\gamma\leq 1$ is again the discount factor:

$Q(s_{t},a_{t})=Q(s_{t},a_{t})+\alpha[r_{t+1}+\gamma\max_{a}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t})]$
(5)

##  3 Experiments

We now show how a Q-Learning agent performs against a Random agent in Colonel
Blotto.

###  3.1 Experimental Setup

We setup a 2-player Colonel Blotto game with 3 fields and 10 coins per player.
This provides a total of 66 possible actions $a$ (ways to distribute the 10
coins on the 3 fields) that the players can choose from. The players aim to
distribute their 10 coins among the 3 fields in such a way that their coins
are greater than their opponent’s coins in each field. The winner of the game
is the player that wins more fields than their opponent, and receives a reward
of $r=1$ from the environment. The loser of the game receives a reward of
$r=-1$. A single game constitutes a single episode in reinforcement learning.
We count how many games each player has won over time and show the results.

Our reinforcement learning agent uses the Q-Learning algorithm for
approximating the optimal policy. We use $\alpha=0.1$ as the learning rate and
$\gamma=1$ as the discount factor. We also set $\epsilon=0.2$ as the
exploration rate for the agent. For the opponent, we use a Random agent that
will select one of the 66 possible actions at random with equal probability
for all of them. The experiments were run using the OpenSpiel[2] environment
simulator.

###  3.2 Results

We run 1 million games of Colonel Blotto to see the performance of the
reinforcement learning agent and show the results in Figure 1. While the RL
agent eventually wins significantly more games, it takes awhile before the RL
agent really begins to start defeating the Random agent regularly. In fact, if
we look at the just the first 200 games (Figure 2), both agents win games at
practically similar rates to start. However after 200 games the better win
rate of the RL agent can already be seen.

Figure 1: Performance of RL Agent vs Random Agent Figure 2: First 200 games of
RL Agent vs Random Agent

We also dive deeper and look at the $Q$-values that the agent has assigned to
each of the 66 possible actions, to better understand what strategy it has
arrived at. Below we show the top 4 and the bottom 4 actions based on their
$Q$-value scores. The 3 numbers for each action signify the number of coins
that the RL agent assigns for each of the 3 fields.

Top 4 Actions
---
[4, 0, 6]
[5, 0, 5]
[0, 4, 6]
[6, 1, 3]
Worst 4 Actions
---
[10, 0, 0]
[0, 10, 0]
[0, 8, 2]
[0, 0, 10]

As can be seen, the actions with the lowest $Q$-values are the ones where the
agent puts all their coins in just one front, guaranteeing a loss when their
opponent easily wins the other two fronts. On the opposite end, the best
actions are the ones where the agent splits their coins almost evenly among
just two fronts, with 0 coins allocated to the third front. By sacrificing one
of the fronts, they give themselves the best chance to win the other two
fronts they focused on, and have a high probability of winning the game in the
process. Against a Random agent, this results in the high win rates that were
shown in the earlier graphs.

###  3.3 Games with 3 or More Players

Next we analyze what happens when we extend Colonel Blotto to games of 3 or
more players. Colonel Blotto was originally envisioned as a 2-player game, and
when extending to more than 2 players we need to properly define first what
will be the winning conditions for the game and how that win is divided in
case there are multiple winners. For this paper we take the definition of the
implementation of Colonel Blotto in OpenSpiel222Implementation code can be
found in https://github.com/deepmind/open_spiel:

  1. 1.

Iterate over every front and get the players that have allocated the most
coins for each one. If there is a single winner for a front, give that player
1 point. However if there are multiple winners for a front, don’t assign any
points to those players.

  2. 2.

Find the players with the most numbers of points. If only one player has the
highest points count then there is only a single winner for the game.
Otherwise the players with the same highest points count are all considered
the winners.

  3. 3.

Each of the winners gets a score of $\frac{1}{num\\_winners}$

The returned score is also used as the environment reward for the winners in
reinforcement learning. For the losers of the game, they get a reward of
$\frac{-1}{num\\_losers}$.

Using this method for calculating winning games in Colonel Blotto of games
with 3 or more players, we play 1 million games while increasing the number of
Random agent opponents from 1 to 10 and show the results at each opponent
count. Figure 3 shows that there is a clear drop in games won for the RL agent
when the number of opponents is increased from 1 to 2. The wins then increase
when the number of opponents is set to 3, and then stays in that range without
much variation as the number of opponents keep increasing.

Figure 3: Won games by RL agent again different number of opponents Figure 4:
Average wins of a single Random opponent

It may seem that the RL agent isn’t doing well when more opponents are added
into the game. However one thing to remember is that although the number of
wins by the RL agent seems to be almost constant, the number of the opponents
increasing mean that the Random agents are getting fewer wins each. In Figure
4 we show the average count of won games of each Random agent as their number
increases. Since all the opponents share the same Random strategy, the
variance between each of them is very small and the average count shown here
are representative for all of them.

Finally we again look at the $Q$-values and see which actions the RL agent has
deemed best. For the table below we show the actions with the 4 highest and 4
lowest $Q$-values against 10 opponents.

Top 4 Actions
---
[10, 0, 0]]
[5, 0, 5]
[9, 1, 0]
[0, 10, 0]
Worst 4 Actions
---
[4, 4, 2]
[0, 5, 5]
[3, 3, 4]
[2, 5, 3]

We find that the previous worst strategies against 1 opponent, where the agent
sacrifices two fronts, are now the best strategies when playing against 10
opponents. The strategy for the RL agent now is to ensure that it wins one of
the fronts against all opponents. This is mainly due to the winning condition
detailed above where a player needs to win at least one front against all
opponents to have a chance at winning a game. This also shows how important it
is to properly set the rewards being returned by the environment in
reinforcement learning. By changing the conditions of the rewards in such a
way when we moved to 3 or more players, we have trained our RL agent to have
very different strategies compared to earlier.

##  4 Conclusion

We have shown how an RL agent using Q-Learning performs against a Random agent
in 1 million games of Colonel Blotto. It takes almost 200 episodes for the RL
agent to start differentiating itself to the Random agent, and after 1 million
episodes the RL agent has clearly shown its dominance.

We also investigated what strategies the RL agent has learned through its
repeated playing, and find that its preferred strategy is to sacrifice one of
the fronts and focus on the other 2 fronts, splitting the coins almost evenly
between them. This was shown empirically to be a a winning strategy against a
single Random agent.

Lastly, we also investigated what happens when we increase the number of
opponents from 1 to 10. We found that the number of games the RL agent wins is
significantly less when playing 2 or more opponents. However, the RL agent
still performs much better and wins more games compared to each single Random
agent. We also found that the optimal strategy when playing 10 opponents is
the complete opposite to playing just 1 opponent, mainly due to how the
winning conditions are designed and how rewards are being returned from the
environment.

## References

  * [1] E. Borel.  The theory of play and integral equations with skew symmetric kernels.  Econometrica, 21(1):97–100, 1953.
  * [2] M. Lanctot, E. Lockhart, J.-B. Lespiau, V. Zambaldi, S. Upadhyay, J. Pérolat, S. Srinivasan, F. Timbers, K. Tuyls, S. Omidshafiei, D. Hennes, D. Morrill, P. Muller, T. Ewalds, R. Faulkner, J. Kramár, B. D. Vylder, B. Saeta, J. Bradbury, D. Ding, S. Borgeaud, M. Lai, J. Schrittwieser, T. Anthony, E. Hughes, I. Danihelka, and J. Ryan-Davis.  OpenSpiel: A framework for reinforcement learning in games.  CoRR, abs/1908.09453, 2019.
  * [3] R. S. Sutton and A. G. Barto.  Reinforcement Learning: An Introduction.  The MIT Press, second edition, 2018.