# Empowering Embodied Manipulation: A Bimanual-Mobile Robot Manipulation
Dataset for Household Tasks

Tianle Zhang JD Explore Academy, China  Dongjiang Li Yihang Li JD Explore
Academy, China  Zecui Zeng JD Explore Academy, China  Lin Zhao JD Explore
Academy, China  Beijing Institute of Technology, China  Lei Sun JD Explore
Academy, China  Yue Chen JD Explore Academy, China  Xuelong Wei JD Explore
Academy, China  Yibing Zhan JD Explore Academy, China  Lusong Li JD Explore
Academy, China  Xiaodong He JD Explore Academy, China

###### Abstract

As Embodied AI advances, it increasingly enables robots to handle the
complexity of household manipulation tasks more effectively. However, the
application of robots in these settings remains limited due to the scarcity of
bimanual-mobile robot manipulation datasets. Existing datasets either focus
solely on simple grasping tasks using single-arm robots without mobility, or
collect sensor data limited to a narrow scope of sensory inputs. As a result,
these datasets often fail to encapsulate the intricate and dynamic nature of
real-world tasks that bimanual-mobile robots are expected to perform. To
address these limitations, we introduce BRMData, a Bimanual-mobile Robot
Manipulation Dataset designed specifically for household applications. The
dataset includes 10 diverse household tasks, ranging from simple single-arm
manipulation to more complex dual-arm and mobile manipulations. It is
collected using multi-view and depth-sensing data acquisition strategies.
Human-robot interactions and multi-object manipulations are integrated into
the task designs to closely simulate real-world household applications.
Moreover, we present a Manipulation Efficiency Score (MES) metric to evaluate
both the precision and efficiency of robot manipulation methods. BRMData aims
to drive the development of versatile robot manipulation technologies,
specifically focusing on advancing imitation learning methods from human
demonstrations. The dataset is now open-sourced and available at
https://embodiedrobot.github.io/, enhancing research and development efforts
in the field of Embodied Manipulation.

11footnotetext: Corresponding Author

##  1 Introduction

Recent advancements in large language models (LLMs) [1, 2, 3, 4] and
multimodal large language models (MLLMs) [5, 6, 7] have significantly advanced
the field of Embodied Manipulation, enhancing the capacity for robots to
interact with 3D physical world in a more human-like manner. This technology
aims to endow robots with the ability to perform manipulation tasks within
household environments, mirroring human dexterity and adaptability. Household
tasks present a substantial challenge due to their diversity and the complex
interactions they entail with various objects. Consequently, the development
of bimanual-mobile robots, which combine the manipulation capabilities of
humanoid robots with autonomous mobility, is crucial. These robots must learn
and adapt to a wide array of tasks, demanding sophisticated methods for
versatile robot manipulation.

Comprehensive robot manipulation datasets [8, 9, 10, 11] are critical for
advancing Embodied Manipulation. These datasets are essential for developing
and refining algorithms that enable robots to perform complex tasks with
human-like dexterity. However, existing datasets primarily focus on single-arm
robotic tasks, such as RT-1 [12], RH20T [13], and RT-X [8], which are limited
to basic object grasping and placement. This narrow focus fails to address the
complexities and coordination demands of bimanual-mobile manipulations.
Although some datasets include dual-arm manipulations, they often lack
mobility features [14] or are limited to single-sensor setups [11]. This
restriction hampers their ability to comprehensively represent the complex
tasks that bimanual-mobile robots must perform in diverse household
environments.

Figure 1: Illustration of our bimanual-mobile robot manipulation tasks.

To overcome these limitations, we introduce BRMData, a novel dataset
meticulously designed to advance the development and evaluation of bimanual-
mobile robots for household tasks. BRMData captures the intricacies of real-
world tasks through human demonstrations, offering a true reflection of the
dexterity and interaction complexity required. As depicted in Figure 1, this
dataset comprises 10 diverse manipulation tasks, including single-object pick-
and-place manipulation, dual-arm coordination, human-robot interactions, and
mobile manipulation. These tasks are specifically selected to incrementally
challenge and develop robot manipulation capabilities. Furthermore, we propose
a novel Manipulation Efficiency Score (MES), which considers both the success
rate and the efficiency of robot manipulation methods. This metric provides a
more comprehensive evaluation, ensuring that developed manipulation algorithms
not only complete tasks effectively but also in a timely manner. In summary,
we present the BRMData dataset with the following key features:

Task Diversity: BRMData encompasses a broad spectrum of manipulation tasks,
including single-arm and dual-arm tasks, as well as tabletop and mobile
manipulations. This diversity enhances the adaptability of robot learning
algorithms across various real-world scenarios.

Progressive Difficulty: The dataset features tasks with escalating complexity,
from single-arm object grasping to dual-arm coordination, human-robot
interactions, and multi-object pick-and-place tasks. This progression
facilitates diverse robot learning methods.

Comprehensive Sensory Data: Employing RGBD camera sensors with varied frame
rates, multi-view, and depth-sensing data, BRMData provides essential sensory
information for testing the robustness of robot learning methods.

##  2 Related Work

###  2.1 Robot Learning

In the realm of robot learning, the utilization of human demonstrations [15,
16, 17, 18, 12] as a means to imbue robots with manipulation capabilities has
garnered significant attention. Robomimic [19] conducts a comprehensive
examination of six offline learning algorithms across various simulated and
real-world manipulation tasks, delves into the nuances of learning from
offline human demonstrations for robot manipulation. DayDreamer [20] enables
robots to learn from minimal interaction by planning within a learned world
model, which significantly reduces the trial-and-error process typically
required in physical settings. CALVIN [21] offers environments for policy
learning under language instructions for long-horizon robotic manipulation
tasks, emphasizing the importance of comprehensive datasets in enhancing
robot-environment interactions. Similarly, DROID [10] provides extensive data
for robot manipulation, underscoring the necessity of testing and optimizing
algorithms for dynamic, real-world applications. Moreover, ARMBench [22] and
PLEX [23] have expanded the application scope by incorporating not only basic
manipulation tasks but also more complex scenarios and diverse operational
strategies.

###  2.2 Robot Manipulation Datasets

Robot manipulation datasets [24, 13, 25, 8] have seen significant evolution to
support imitation learning and policy generalization in complex tabletop tasks
and varied physical environments. RH20T [13] and RoboAgent [26] stand out as
examples, enabling robots to learn and execute tasks effectively in both
simulated and real settings. The MIME dataset [15] utilizes multiple
collection techniques, including kinesthetic teaching and visual
demonstrations, to gather data on human-robot interactions. Trained on a
substantial dataset of over 130k episodes spanning 17 months, RT-1 [12]
leverages transformer-based architectures to facilitate real-world control
tasks at scale. While BridgeData V2 [25] and RH20T [13] offer depth-sensing
and multi-view capabilities, they are limited in movable capabilities and
restricted to single-arm manipulation tasks. In contrast, our proposed
dataset, BRMData, integrates cross-sensor and depth-sensing features,
distinguishing it from similar datasets like ALOHA [14] and Mobile ALOHA [11]
lacking such capabilities.

##  3 BRMData Dataset

According to the comparison provided in Table 1, BRMData stands out due to its
comprehensive coverage of bimanual-mobile manipulations. Through the
integration of dual-arm coordination and cross-sensor data collection, BRMData
offers an enriched training environment unparalleled by existing
representative datasets. This section describes the methods employed in the
data collection and details the composition of manipulation tasks.

Table 1: Comparison to existing datasets for robot manipulation.

Dataset |  Human Demonstration | Movable | Dual-arm | Depth-sensing | Multi-view | Torque | Cross-sensor
---|---|---|---|---|---|---|---
MIME [15] | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$
RT-1 [12] | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
RH20T [13] | $\checkmark$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$
BridgeData V2 [25] | $\times$ | $\times$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$
ALOHA [14] | $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$
Mobile ALOHA [11] | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$
BRMData (Ours) | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$

###  3.1 System Setup

Figure 2: Illustration of hardware details.

As illustrated in Figure 2, the data collection hardware platform for BRMData
comprises 4 ARX-5 robot arms, each equipped with 7 joints and a parallel
gripper. The Follower-1 and Follower-2 robot arms, positioned at the front of
the platform, are specifically engineered for advanced bimanual manipulations.
Each of both arms features a movable RGBD camera sensor at the wrist. The
flexibility afforded by these cameras facilitates variable viewing angles and
provides detailed visual feedback, crucial for precision in manipulation
tasks. Positioned centrally between Follower-1 and Follower-2 is a fixed RGBD
camera sensor. This middle camera provides a consistent and broad visual
reference across the manipulation area. It serves as an integral component,
enhancing the coordination of the follower arms. By offering a complementary
perspective, it enriches the detailed imagery captured by the wrist-mounted
cameras.

The Leader-1 and Leader-2 robot arms, located at the rear of the platform, are
designated for teleoperation by human experts. This configuration facilitates
the synchronous replication of human-directed actions from the leader arms to
the follower arms, enhancing the platform’s capability in executing complex
manipulation sequences through a human-teaching data collection strategy.
Supporting the robotic components, the platform includes an Automated Guided
Vehicle (AGV) equipped with a two-wheel differential Tracer mobile base, which
provides essential mobility for executing mobile manipulation tasks within
indoor environment. The computational demands of the platform are managed by
an industrial computer equipped with an NVIDIA 4090 graphics card, an Intel
i7-13700 CPU, and 32GB of RAM. This computing setup ensures efficient
processing of the camera data and supports real-time algorithms necessary for
effective manipulation and interaction.

###  3.2 Data Collection and Processing

Figure 3: Illustration of the multi-view data collection of BRMData. (a) Left
Image: Captured from the left wrist-mounted camera, offering detailed visual
input from the left-arm’s side. (b) Right Image: Obtained from the right
wrist-mounted camera, providing critical visual information from the robot’s
right side. (c) Middle Image: Sourced from the centrally mounted fixed camera,
presenting a broad and central perspective of the operational area. (d) Human
Perspective View: Captured from a human viewpoint, not through the robot
platform, giving an external perspective on the scene.

The data collection strategy for the BRMData dataset is meticulously designed
to comprehensively capture a wide array of manipulation tasks. As shown in
Figure 3, both the left and right wrist-mounted RGBD cameras and the middle
camera record RGB and depth images at a resolution of $640\times 480$. The
integration of these three views provides a multi-perspective visual input,
essential for constructing a detailed 3D representation of the operational
environment. For specific tasks such as the Bottle Pick and Bottle Handoff,
image data are captured using Orbbec Dabai cameras operating at a frequency of
30Hz. For more dynamic manipulation tasks, Intel Realsense D435 cameras, which
captures data at a higher frequency of 60Hz, is employed to accommodate the
quick movements and frequent data updates required. Each manipulation task
within the dataset includes the collection of 50 action trajectories, gathered
at intervals that are specifically tailored to the complexity and specific
requirements of each task scenario. To enrich the diversity of the dataset,
each manipulation task is performed by three human experts. These experts use
the Leader-1 and Leader-2 robot arms to demonstrate the manipulation tasks,
with their actions being synchronously replicated on the Follower-1 and
Follower-2 arms. This structured approach ensures that every trajectory is
unique, capturing a broad spectrum of challenges and demands encountered in
real-world bimanual-mobile robot manipulation.

###  3.3 Properties of BRMData

BRMData is structured around diverse manipulation tasks. For detailed task
definitions, step-by-step descriptions, please refer to Appendix D.

Table 2: Comparative analysis of robot manipulation tasks across key
attributes

Type | Task | Movable |  Human-robot Interaction |  Multi-object Interaction |  Object Attribute | Steps
---|---|---|---|---|---|---
Tabletop Single-arm Manipulation | Bottle Pick | $\times$ | $\times$ | $\times$ | Rigid | 400
Single Fruit Pick | $\times$ | $\times$ | $\checkmark$ | Rigid | 400
Plate Place | $\times$ | $\times$ | $\times$ | Rigid | 600
Fruit Handover | $\times$ | $\checkmark$ | $\times$ | Rigid | 500
Tabletop Dual-arm Manipulation | Bottle Handoff | $\times$ | $\times$ | $\times$ | Rigid | 800
Cup Place | $\times$ | $\times$ | $\times$ | Rigid | 800
Multi Fruit Pick | $\times$ | $\times$ | $\checkmark$ | Rigid | 900
Wine Wipe | $\times$ | $\times$ | $\checkmark$ | Flexible | 1000
Mobile Manipulation | Garbage Recycle | $\checkmark$ | $\times$ | $\times$ | Rigid | 1200
High Five | $\checkmark$ | $\checkmark$ | $\times$ | Flexible | 500

####  3.3.1 Task Diversity

BRMData incorporates a broad spectrum of robotic manipulation tasks that
extend from simple tabletop single-arm manipulations to intricate bimanual and
mobile manipulations, as detailed in Table 2. The dataset provides a
foundation for investigating fundamental robotic interactions like Bottle
Pick, Plate Place, and Single Fruit Pick. These tasks encompass basic object
handling as well as advanced human-robot interactions. Tasks such as Bottle
Handoff and Wine Wipe further the complexity by necessitating coordinated
bimanual actions and managing multi-object interactions, representing advanced
manipulation challenges. The mobile manipulation tasks, including Garbage
Recycle and High Five, introduce navigation and dynamic interaction
capabilities, essential for operational efficacy in changing environments. The
diverse task range in BRMData enables comprehensive testing and refinement of
robotic manipulation strategies suitable for a variety of applications.

####  3.3.2 Progressive Difficulty

BRMData is carefully designed to present a series of tasks with increasing
levels of complexity and diversity, not only in the manipulation tasks but
also in the attributes and interaction with objects. This design facilitates a
gradual elevation in the skillset required from the robots, focusing on their
adaptability and precision in handling different object attributes and
interaction complexities.

Single vs. Multi-Object Manipulation:  Initially, tasks such as Bottle Pick
and Single Fruit Pick involve interactions with a single object, allowing for
focused assessments of basic manipulation capabilities. However, as the
complexity level rises, the tasks evolve to include multi-object interactions.
Tasks like Multi Fruit Pick and Wine Wipe require the robot to manage multiple
objects either sequentially or simultaneously. This shift from single to
multi-object manipulation tests the robot’s capability in task planning,
object prioritization, and execution of complex sequences involving various
objects.

Handling Rigid and Flexible Objects: The BRMData dataset differentiates
between rigid and flexible objects. Initial tasks like Bottle Pick, Single
Fruit Pick, and Plate Place involve rigid objects, which are generally simpler
to manipulate due to their stable and predictable nature. These tasks serve as
fundamental tests of a robot’s capabilities in gripping, object recognition,
and precise placement, laying the groundwork for more complex manipulations.
As the tasks progress in complexity, the dataset introduces flexible objects,
as seen in Wine Wipe and High Five tasks. The manipulation of flexible objects
poses additional challenges due to their deformable nature, which requires
more sophisticated sensory and feedback mechanisms to adjust grip strength and
manipulation strategies dynamically. For instance, in the Wine Wipe task, the
robot must not only lift a wine glass but also clean a spill with a cloth,
involving precise movements and pressure control to avoid slippage or damage.
The High Five task exemplifies the complexity of interacting with human hands,
which introduces unpredictability in object behavior and demands a nuanced
understanding of dynamic material properties. This progression from rigid to
flexible object manipulation significantly raises the difficulty level,
challenging the robot’s capacity to handle a diverse range of material
properties and interaction scenarios.

####  3.3.3 Variety of Collection Strategies

Static and Dynamic Imaging:  BRMData utilizes a combination of two mobile RGBD
cameras mounted on the robot arms and a centrally positioned fixed camera to
ensure an exhaustive visual representation of the manipulation environment.
The mobile cameras provide changing perspectives that track the motion of the
robot arms, capturing detailed data critical for analyzing interactions in
real-time. The fixed camera maintains a consistent view of the entire scene,
enabling the capture of invariant spatial references. This multi-view
arrangement significantly enhances the dataset by supplying comprehensive
spatial and temporal insights, essential for the complex tasks being
performed.

Depth-sensing: We utilize RGBD cameras to acquire spatial information of the
physical environment. These sensors facilitate the 3D environmental perception
by providing depth information that complements the RGB imagery. This dual
modality is essential for precise object detection, spatial reasoning, and
execution of complex manipulative tasks, particularly in environments where
object depth and positioning are critical for successful interaction.

Frame Rate Variability: The dataset incorporates cameras operating at
different frame rates to tailor data acquisition to the specific dynamics of
each task. For tasks that involve rapid movements, high-frame-rate cameras are
utilized to capture fine temporal details, which are crucial for analyzing
swift interactions. Conversely, tasksis characterized by slower or more static
actions are adequately documented using cameras with standard frame rates.
This approach ensures that the temporal resolution of the captured data is
precisely aligned with the specific requirements of each task.

###  3.4 Evaluation Metrics

Robot manipulation tasks are often evaluated using the success rate (SR).
While SR provides a measure of task completion success, it does not account
for the efficiency of task execution. High SR may still be accompanied by long
execution times, which is impractical for real-world applications where time
efficiency is crucial. Therefore, we introduce a novel metric: the
Manipulation Efficiency Score (MES). It is defined as
$\text{MES}=100\frac{\text{SR}}{T}$, where $T$ denotes the average execution
time. This metric captures both the effectiveness and efficiency of robot
manipulation methods, rewarding those that achieve high success rates with
minimal execution time, thereby highlighting both the quality and speed of
task execution.

##  4 Experiments

Rather than delving into an exhaustive analysis of different policy network
design choices, the goal of our experiments is to explore the extent to which
our dataset enhances the application of robot learning methods in executing
household tasks. Our experiments are designed to address the following key
questions: (a) Can BRMData effectively facilitate the learning of various
robot manipulation methods? (b) Can BRMData effectively reflect a progressive
increase in task difficulty? (c) Have we demonstrated fair and challenging
testing methodologies to advance the research in embodied manipulation?

###  4.1 Experimental Setup

To investigate the performance and adaptability of various robot learning
methods on BRMData, we conduct repetitive experiments using both single-task
and multi-task models. The methods used in the single-task setting include
Action Chunking with Transformers (ACT) [14] and Diffusion Policy (DP) [27].
The methods used in the multi-task experiments include MT-ACT [28], MT-ACT-
EB3, and MT-ACT-R3M. Additional details can be found in Appendix E and F.

###  4.2 Experimental Results

####  4.2.1 Can BRMData facilitate the learning of various robot manipulation
methods?

Table 3: Testing results of different robot learning models.

Method | Multiple Fruits Pick | Cup Place | Garbage Recycle
---|---|---|---
SR | MES | SR | MES | SR | MES
Single-task | ACT | 60 | 3.61 | 90 | 6.06 | 55 | 2.62
DP | 80 | 2.94 | 80 | 4.20 | 40 | 1.57
Multi-task | MT-ACT | 90 | 4.78 | 90 | 5.56 | 70 | 2.50
MT-ACT-EB3 | 70 | 2.08 | 100 | 3.57 | 40 | 1.03
MT-ACT-R3M | 50 | 2.79 | 90 | 5.82 | - | -

To explore the performance of different robot learning methods, we select
three complex robot manipulation tasks for testing, as shown in Table 3. In
single-task experiments, the ACT method generally outperforms the DP method in
terms of execution efficiency. Despite a lower SR in the Multiple Fruits Pick
task, ACT achieves a superior MES of 3.61, indicating better efficiency. In
the Cup Place task, ACT excels with a 90% SR and the highest MES of 6.06,
highlighting its precision and speed in task execution. In multi-task
experiments, MT-ACT proves to be highly adaptable and efficient, achieving a
90% SR and an MES of 4.78 in the Multiple Fruits Pick task, along with strong
performance across other tasks. The experimental results demonstrate the
effectiveness of BRMData in supporting the learning of various robot
manipulation methods.

####  4.2.2 Can BRMData effectively reflect a progressive increase in task
difficulty?

Figure 4: Performance comparison in single-arm and dual-arm robot manipulation
tasks.

Impact of Dual-Arm Coordination on Task Difficulty:  As illustrated in Figure
4, the experimental results of single-task methods demonstrate varying degrees
of success and efficiency, indicating the diverse difficulty levels in robot
manipulation tasks. In the Bottle Pick task, both ACT and DP methods achieve
high performance, with DP showing a SR of 80% and a MES of 7.84. However, when
the task complexity increases to the dual-arm Bottle Handoff task, both SR and
MES decline significantly for both methods. ACT’s SR drops to 75% and MES to
3.48, while DP’s SR falls to 60% and MES to 2.67. A similar trend is observed
in fruit manipulation tasks. For Single Fruit Pick, ACT achieves an SR of 90%
and a MES of 5.58, but in the dual-arm Multiple Fruits Pick task, ACT’s SR
reduces to 60% and MES to 3.61, underscoring the difficulty in managing
multiple items simultaneously. In placement tasks, Plate Place involves
simpler, single-arm action, where DP maintains a high SR of 100% and a MES of
8.70. In contrast, the dual-arm Cup Place task results in lower performance,
with DP’s SR at 80% and MES at 4.20. These comparisons clearly demonstrate
that dual-arm tasks introduce significant complexity, leading to reduced
success rates and efficiency.

Table 4: Testing results in robot manipulation tasks with human-robot interaction. Method |  | Single Fruit Pick |  | Fruit Handover |  | High Five
---|---|---|---|---|---|---
| SR | MES |  | SR | MES |  | SR | MES
ACT |  | 90 | 5.58 |  | 85 | 8.92 |  | 100 | 10.28
DP |  | 50 | 2.49 |  | 40 | 3.56 |  | 80 | 4.86

Impact of Human-Robot Interaction on Task Complexity:  As illustrated in Table
4, human-robot interaction significantly increases task complexity. In the
Fruit Handover task, DP’s SR drops to 40% compared to 50% in the Single Fruit
Pick task. The coordination required for transferring objects between the
robot and human demands precise timing and spatial alignment. Conversely, the
High Five task, despite involving human-robot interaction, has a simpler
dynamic without object manipulation, allowing for higher SR values. These
observations highlight that the complexity of human-robot interaction depends
heavily on the nature of the task, particularly whether it involves object
manipulation.

Table 5: Performance comparison in stationary versus mobile robot manipulation tasks. Method | Wine Wipe |  | Garbage Recycle
---|---|---|---
SR | MES |  | SR | MES
ACT | 100 | 5.06 |  | 55 | 2.62
DP | 80 | 3.38 |  | 40 | 1.57

Impact of Mobility on Task Complexity:  The experimental results in Table 5
demonstrate that tasks requiring mobility are significantly more complex than
stationary tasks. In the Wine Wipe task, which involves stationary
manipulation, ACT achieves a perfect SR of 100% and an MES of 5.06, while DP
achieves an SR of 80% and an MES of 3.38. However, when the task requires
mobility, as in Garbage Recycle, performance drops notably: ACT’s SR decreases
to 55% and MES to 2.62, and DP’s SR decreases to 40% and MES to 1.57. The dual
requirement for seamless integration of navigation and manipulation in mobile
tasks presents additional challenges, such as maintaining stability while
moving and accurately reaching designated locations. These results underscore
the challenges in maintaining high performance in tasks that involve mobility,
thus demonstrating the robustness and utility of BRMData in evaluating the
capabilities of robot learning methods under varied and complex conditions.

####  4.2.3 Have we demonstrated fair and challenging testing methodologies to
advance the research in embodied manipulation?

Table 6: Testing results of different single-task models.

|  Method |  Bottle Pick |  Bottle Handoff |  Single Fruit Pick |  Multiple Fruits Pick |  Plate Place |  Cup Place |  Fruit Handover |  High Five |  Wine Wipe |  Garbage Recycle
---|---|---|---|---|---|---|---|---|---|---|---
SR (%) |  ACT |  100 |  75 |  90 |  60 |  100 |  90 |  85 |  100 |  100 |  55
|  DP |  80 |  60 |  50 |  80 |  100 |  80 |  40 |  80 |  80 |  40
MES |  ACT |  11.04 |  3.48 |  5.58 |  3.61 |  10.11 |  6.06 |  8.92 |  10.28 |  5.06 |  2.62
|  DP |  7.84 |  2.67 |  2.49 |  2.94 |  8.70 |  4.20 |  3.56 |  4.86 |  3.38 |  1.57

Fairness and Effectiveness of MES: Table 6 shows that the MES metric offers a
comprehensive and fair measure of performance. By incorporating both SR and
execution time, MES effectively penalizes methods that take longer to complete
tasks, ensuring a balanced evaluation. For example, in the Multiple Fruits
Pick task, despite ACT having a lower SR (60%) compared to DP (80%), ACT’s MES
of 3.61 is higher than DP’s 2.94. This highlights ACT’s greater efficiency,
demonstrating that MES can fairly assess the trade-offs between success rates
and execution efficiency. Such a balance is crucial for real-world
applications where both effectiveness and efficiency are essential. The MES
also excels in highlighting the trade-offs between different approaches. For
instance, in the High Five task, ACT achieves a high MES of 10.28, indicating
both a high success rate and efficient execution. Conversely, DP achieves a
lower MES of 4.86, pointing to longer execution times despite a reasonable SR
of 80%. This shows that MES can effectively differentiate between methods
based on both their accuracy and operational efficiency. Furthermore, MES
helps to identify inefficiencies in more complex tasks, such as Garbage
Recycle, where the need for both manipulation and mobility challenges the
methods, resulting in lower MES values for both ACT (2.62) and DP (1.57). To
further assess the robustness of imitation learning methods under varying
conditions, additional experiments are conducted as detailed in Appendix G.

##  5 Conclusion

In summary, the BRMData dataset represents a significant advancement in the
field of Embodied AI by addressing the critical shortage of comprehensive
datasets for dual-arm mobile robot manipulation. By encompassing a range of
bimanual and mobile manipulations, BRMData facilitates the exploration of
complex interactions within dynamic environments that closely mimic real-world
scenarios. This dataset not only supports the deployment of humanoid robots
with enhanced capabilities but also provides a benchmark for testing and
improving state-of-the-art models in robot manipulation. The diverse task
configurations included in BRMData have proven instrumental in evaluating the
adaptability and effectiveness of various robotic models. This has allowed for
a detailed assessment of how different approaches perform under complex and
varied conditions, thus contributing valuable insights into the development of
more versatile and capable robotic systems. This work underscores the
importance of such datasets in advancing our understanding of robotic
capabilities and limitations, highlighting key areas for future exploration
and development in the realm of embodied robots.

## References

  * Achiam et al. [2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto.  Stanford alpaca: An instruction-following llama model, 2023.
  * Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  _See https://vicuna. lmsys. org (accessed 14 April 2023)_ , 2(3):6, 2023.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  In _International Conference on Machine Learning_ , pages 8469–8488. PMLR, 2023.
  * Zhu et al. [2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny.  Minigpt-4: Enhancing vision-language understanding with advanced large language models.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Liu et al. [2024] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Padalkar et al. [2023] Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al.  Open x-embodiment: Robotic learning datasets and rt-x models.  _arXiv preprint arXiv:2310.08864_ , 2023.
  * Iyer et al. [2024] Aadhithya Iyer, Zhuoran Peng, Yinlong Dai, Irmak Guzey, Siddhant Haldar, Soumith Chintala, and Lerrel Pinto.  Open teach: A versatile teleoperation system for robotic manipulation.  _arXiv preprint arXiv:2403.07870_ , 2024.
  * Khazatsky et al. [2024] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al.  Droid: A large-scale in-the-wild robot manipulation dataset.  _arXiv preprint arXiv:2403.12945_ , 2024.
  * Fu et al. [2024] Zipeng Fu, Tony Z Zhao, and Chelsea Finn.  Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body teleoperation.  _arXiv preprint arXiv:2401.02117_ , 2024.
  * Brohan et al. [2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Fang et al. [2023] Hao-Shu Fang, Hongjie Fang, Zhenyu Tang, Jirong Liu, Junbo Wang, Haoyi Zhu, and Cewu Lu.  Rh20t: A robotic dataset for learning diverse skills in one-shot.  _arXiv preprint arXiv:2307.00595_ , 2023.
  * Zhao et al. [2023] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn.  Learning fine-grained bimanual manipulation with low-cost hardware.  _arXiv preprint arXiv:2304.13705_ , 2023.
  * Sharma et al. [2018] Pratyusha Sharma, Lekha Mohan, Lerrel Pinto, and Abhinav Gupta.  Multiple interactions made easy (mime): Large scale demonstrations data for imitation.  In _Conference on robot learning_ , pages 906–915. PMLR, 2018.
  * Bahl et al. [2023] Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak.  Affordances from human videos as a versatile representation for robotics.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13778–13790, 2023.
  * Jang et al. [2022] Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn.  Bc-z: Zero-shot task generalization with robotic imitation learning.  In _Conference on Robot Learning_ , pages 991–1002. PMLR, 2022.
  * Lynch et al. [2023] Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence.  Interactive language: Talking to robots in real time.  _IEEE Robotics and Automation Letters_ , 2023.
  * Mandlekar et al. [2021] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martín.  What matters in learning from offline human demonstrations for robot manipulation.  _arXiv preprint arXiv:2108.03298_ , 2021.
  * Wu et al. [2023] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg.  Daydreamer: World models for physical robot learning.  In _Conference on Robot Learning_ , pages 2226–2240. PMLR, 2023.
  * Mees et al. [2022] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard.  Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks.  _IEEE Robotics and Automation Letters_ , 7(3):7327–7334, 2022.
  * Mitash et al. [2023] Chaitanya Mitash, Fan Wang, Shiyang Lu, Vikedo Terhuja, Tyler Garaas, Felipe Polido, and Manikantan Nambi.  Armbench: An object-centric benchmark dataset for robotic manipulation.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9132–9139. IEEE, 2023.
  * Thomas et al. [2023] Garrett Thomas, Ching-An Cheng, Ricky Loynd, Felipe Vieira Frujeri, Vibhav Vineet, Mihai Jalobeanu, and Andrey Kolobov.  Plex: Making the most of the available data for robotic manipulation pretraining.  In _Conference on Robot Learning_ , pages 2624–2641. PMLR, 2023.
  * Ebert et al. [2021] Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Daniilidis, Chelsea Finn, and Sergey Levine.  Bridge data: Boosting generalization of robotic skills with cross-domain datasets.  _arXiv preprint arXiv:2109.13396_ , 2021.
  * Walke et al. [2023] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen-Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al.  Bridgedata v2: A dataset for robot learning at scale.  In _Conference on Robot Learning_ , pages 1723–1736. PMLR, 2023.
  * Bharadhwaj et al. [2023] Homanga Bharadhwaj, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar.  Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking.  _arXiv preprint arXiv:2309.01918_ , 2023.
  * Chi et al. [2023] Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song.  Diffusion policy: Visuomotor policy learning via action diffusion.  _arXiv preprint arXiv:2303.04137_ , 2023.
  * Nair et al. [2023] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3m: A universal visual representation for robot manipulation.  In _Conference on Robot Learning_ , pages 892–909. PMLR, 2023.
  * Sohn et al. [2015] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  _Advances in neural information processing systems_ , 28, 2015.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Perez et al. [2018] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Tan and Le [2019] Mingxing Tan and Quoc Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pages 6105–6114. PMLR, 2019.
  * Nair et al. [2022] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta.  R3m: A universal visual representation for robot manipulation.  _arXiv preprint arXiv:2203.12601_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [N/A]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [Yes]

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [Yes]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes]

    2. (b)

Did you mention the license of the assets? [Yes]

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes]

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes]

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes]

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [N/A]

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [N/A]

##  Appendix A Licensing

BRMData is released under the open-source MIT license. For more details,
please visit https://embodiedrobot.github.io/. All personal identity
information involved in the dataset has been authorized.

##  Appendix B Dataset Accessing

  * •

Link to access BRMData:

http://box.jd.com/sharedInfo/1147DC284DDAEE91DC759E209F58DD60

  * •

Password for accessing BRMData: f2ss31

##  Appendix C Data Formats

⬇

[

{

"observations":

{

"images":

{

"cam_high": (480, 640, 3),

"cam_left_wrist": (480, 640, 3),

"cam_right_wrist": (480, 640, 3),

},

"images_depth":

{

"cam_high": (480, 640),

"cam_left_wrist": (480, 640),

"cam_right_wrist": (480, 640),

},

"qpos": (14), # 0-7:left arm, 7-14: right arm

"qvel": (14), # 0-7:left arm, 7-14: right arm

"effort": (14) # 0-7:left arm, 7-14: right arm

},

"action": (14), # 0-7:left arm, 7-14: right arm

"base_action": (2)

},

...

]

The dataset format for BRMData is structured to comprehensively capture both
visual observations and actions. It includes image data from three cameras,
each providing RGB images with a resolution of $480\times 640\times 3$. Depth
images from the same cameras are also included. Additionally, the dataset
records joint positions and efforts for 14 joints, divided equally between the
left and right follower arms. This format ensures a detailed representation of
the robot’s state and actions, facilitating advanced analysis and research in
robotic manipulation.

##  Appendix D Tasks of BRMData

###  D.1 Task Descriptions

Table 7: Description of robot manipulation tasks

Task |  Task Description
---|---
Bottle Pick |  A single robot arm picking up a cola bottle from position A on the table and placing it at position B.
Single Fruit Pick |  Pick up a fruit from a table and place it into a plate.
Plate Place |  A single robot arm picks up a randomly placed plate from the table and accurately positions it into a fixed dish rack.
Fruit Handover |  A single arm picks up a fruit from a plate and hands it over to a person.
Bottle Handoff |  Use two robot arms to transfer a cola bottle.
Cup Place |  Dual-arm coordination to pick up a cup initially upright on a table, invert it, and place it upside down in a dish rack.
Multi Fruit Pick |  Two robot arms collaboratively arrange a plate and sequentially place an orange and an apple into the plate on a table.
Wine Wipe |  The left arm lifts a wine glass off the table, then the right arm cleans the spilled wine with a cloth. After cleaning, both the wine glass and cloth are placed back on the table.
Garbage Recycle |  Pick up a waste item from the table, locate the garbage bin, navigate to it, and deposit the item into the bin.
High Five |  Lift a single arm to high-five an approaching person while moving.

Table 7 provides a comprehensive overview of the various robot manipulation
tasks included in our dataset. These tasks range from simple object
localization, such as picking and placing a bottle (Bottle Pick) and single
fruit handling (Single Fruit Pick), to more complex dual-arm coordination
tasks like Cup Place and Multi Fruit Pick. The tasks are designed to test the
robot’s ability to perform precise actions, handle objects collaboratively,
and interact with humans, such as in the Fruit Handover and High Five
scenarios.

###  D.2 Task Definitions

Figure 5: Task Definition of Bottle Pick.

1) As shown in Figure 5, the Bottle Pick task features a single robot arm
executing a precise sequence of movements, transferring a cola bottle from
position A to position B on a table on 400 defined steps. This task is
designed to assess the arm’s capabilities in object gripping and spatial
location perception, crucial for automated handling systems. By focusing on
the arm’s ability to accurately grasp a bottle and then navigate it between
two points, the task tests the precision of the arm’s motor skills and its
sensory feedback mechanisms. The challenge lies in ensuring the arm can
consistently and reliably handle objects while adapting to variations in
environmental conditions, such as different object orientations and positions,
which are key for real-world applications in logistics and manufacturing
automation.

Figure 6: Task Definition of Bottle Handoff.

2) As shown in Figure 6, the Bottle Handoff task demonstrates a coordinated
interaction between two robot arms, transferring a cola bottle across a
sequence of 800 steps. This task assesses the collaborative capabilities of
bimanual robot manipulation, emphasizing their synchronized movements and
precision in object handoffs. The core of this task is the fluid transfer of
the bottle from the left to the right robot arm, underscoring the critical
need for accurate timing and spatial coordination between the two arms. This
manipulation serves as a vital benchmark for the interoperability and
collaborative efficiency of bimanual-mobile robots, which are crucial for
improving operational harmony and enhancing task execution in complex
settings.

Figure 7: Task Definition of Single Fruit Pick.

3) As shown in Figure 7, the Single Fruit Pick task simulates human-like
decision-making by employing the nearest robot arm to execute the fruit
picking manipulation, a process encompassing 400 steps. This task involves an
orange initially placed on the one side of a dish, where the nearest robot arm
picks it up and places it inside the dish. This approach not only tests each
arm’s ability to perform precise pick-and-place actions independently, but
also emphasizes the efficiency of using spatially advantageous positions,
mimicking how humans would typically perform such a task. The task highlights
the robot arms’ precision, adaptability, and the capability to handle delicate
objects within confined spaces, showcasing their potential for applications
requiring nuanced and context-aware robotic actions.

Figure 8: Task Definition of Multi Fruit Pick.

4) As shown in Figure 8, the Multi Fruit Pick task demonstrates advanced
bimanual coordination on a table setup, where a dish and two fruits are
initially placed randomly. This 900-step task significantly advances the
complexity found in the Single Fruit Pick task by requiring sequential
manipulations between two robot arms. The right arm initiates by precisely
positioning the dish at the center of the table, setting the stage for
subsequent manipulations. The left arm then engages, placing the orange
followed by the apple into the dish. This sequence not only assesses each
arm’s precision in handling pick-and-place tasks, but also underscores the
challenge of harmonizing their actions within a shared workspace. The Multi
Fruit Pick task serves as a balanced evaluation of the robots’ bimanual
coordination, spatial awareness, and adaptability, vital for complex
manipulative tasks in real-world applications.

Figure 9: Task Definition of Plate Place.

5) As shown in Figure 9, the Plate Place task comprises 600 action steps,
offering a moderate level of complexity that focuses on the robot’s ability to
handle typical kitchenware with precision. The challenge lies in the robot’s
ability to detect the random position of the plate, accurately maneuver its
arm to grasp it, and then carefully place it on the narrow confines of a
draining rack. This tests not only the robot’s perception and localization
capabilities but also its end-effector control and fine motor skills.

Figure 10: Task Definition of Cup Place.

6) The Cup Place task is a sophisticated demonstration of bimanual
coordination, where a robot is tasked with placing a randomly positioned cup
into a specific slot on a dish rack already containing several cups. This
task, which includes 800 action steps, represents a notable escalation in
complexity compared to the simpler Plate Place task. The key challenge here
involves not only the synchronized movement of both arms but also the robot’s
ability to accurately identify empty slots within the partially filled rack
and manipulate the cup to fit snugly upside down in its designated space. This
process tests the robot’s cognitive capabilities to perceive its environment
and adapt its manipulation strategies accordingly.

Figure 11: Task Definition of Fruit Handover.

7) The Fruit Handover task demonstrates a robot interacting directly with a
human to transfer fruit, highlighting its potential for practical robotic
assistance in real-world scenarios. This task evaluates the collaborative
abilities of a single robot arm, focusing on its capacity to understand human
gestures and respond by delivering fruit safely and comfortably. The challenge
here is twofold: firstly, the robot must accurately detect and interpret the
human’s hand gesture as a cue to initiate the task. Secondly, the robot needs
to execute the task with a level of finesse that ensures the safety and
comfort of human interaction, which is critical for applications in settings
such as personal care or healthcare where robots assist directly with humans.

Figure 12: Task Definition of Wine Wipe.

8) The Wine Wipe task demonstrates a complex bimanual robot manipulation
across 1000 action steps, effectively mirroring a typical real-world cleaning
scenario. This task is designed to evaluate the robot’s finesse and the
coordination between its two arms, focusing on their ability to concurrently
execute multiple manipulation activities. The task primarily tests the robot’s
precision and endurance through the right arm’s repeated motions required to
thoroughly clean the spill, ensuring no residue remains. Additionally, the
task challenges the robot’s spatial cognition and memory by requiring it to
precisely replace the wine glass and cloth to their original positions. This
aspect is essential for roles that demand careful handling and reorganization
within dynamic and potentially cluttered environments. Thus, the Wine Wipe
task serves as a crucial benchmark for assessing advanced robot manipulation
skills necessary in service-oriented and domestic settings.

Figure 13: Task Definition of Gabbage Recycle.

9) The Gabbage Recycle task is a critical assessment of the bimanual-mobile
robot’s ability to integrate mobility with precise manipulative actions,
testing not just mechanical skills but also advanced cognitive functions. This
task specifically evaluates the robot’s proficiency in identifying and
interacting with a garbage bin within a spatially complex environment, a
significant step beyond static table-top manipulation tasks. The robot must
accurately detect the recycle bin’s location amidst other elements,
demonstrating effective environment mapping and object recognition. Designed
to challenge the robot with a sequence of 1200 steps, this task demands
sustained precision over an extended duration, simulating real-world scenarios
where robots are expected to perform prolonged tasks.

Figure 14: Task Definition of High Five.

10) The High Five task serves as a critical assessment of the robot’s
interactive capabilities, particularly in synchronizing its movements with
dynamic human actions. This task, involving a robot designed to extend its
right arm to high-five a human while simultaneously moving forward,
encapsulates 500 action steps. The task is designed to test the robot’s
precision in both timing and spatial judgment, a crucial skill set for
applications in interactive environments such as homes or offices. By
challenging the robot to coordinate its movement with an approaching person,
the task not only examines the robot’s ability to perform social gestures but
also highlights its adaptability and responsiveness to human presence.

##  Appendix E Experimental Setup

In the single-task experiments, we perform 20 trials per task to assess the
precision and consistency of each method in a controlled environment, with the
only variable being the initial positions of objects on the table. This setup
allows us to evaluate how well each method handles specific tasks with minimal
external interference, where each model is responsible for a single robot
manipulation task. In the multi-task experiments, we evaluate a single model’s
ability to manage and solve different robot manipulation tasks individually.
Each task undergoes 10 trials to explore how well the model can adapt to and
handle various tasks. This approach enables us to assess the model’s
flexibility and robustness across multiple tasks, reflecting its ability to
generalize and perform under varying conditions.

##  Appendix F Robot Learning Methods

###  F.1 Single-task Robot Learning Methods

Action Chunking with Transformers (ACT) [14]: The ACT method combines a
Conditional Variational Auto Encoder (CVAE) [29] with a Transformer
architecture [30] to predict sequences of actions, termed as action chunks,
rather than individual actions. This design is particularly suited for complex
and precision-demanding tasks. Initially, ACT employs a ResNet-18 [31]
backbone for image processing, maintaining a resolution of $640\times 480$
pixels for each view.

Diffusion Policy (DP) [27]: The DP method models robot behavior by
characterizing a robot’s visuomotor policy through a conditional denoising
diffusion process. This approach leverages the stochastic properties of
diffusion models to generate diverse and adaptable action sequences.

###  F.2 Multi-task Robot Learning Methods

MT-ACT [28]: The Multi-Task ACT (MT-ACT) model augments the ACT framework to
proficiently manage multiple tasks simultaneously. In our re-implementation
efforts, a FiLM [12, 32] network has been integrated into the ACT model to
concurrently execute several manipulation tasks, aligning each task with a
corresponding language instruction. These language commands facilitate FiLM-
enhanced, language-conditioned learning, enabling the MT-ACT model to
dynamically adapt to varying task demands. Detailed descriptions of the
specific language commands employed are provided in Table 8.

Table 8: Language Commands for Multi-task Methods Task |  Language Command
---|---
Gabbage Recycle |  grab the cola bottle on the table with the right arm, turn to the trash can, approach the trash can, and throw the cola bottle into the trash can
High Five |  approach the person, raise the right arm, and give he a high five
Multi Fruits Pick |  grab the plate with the right arm and place it in the center of the table, use the left arm to pick up the orange and put it on the plate, then pick up the peach and put it on the plate
Single Fruit Pick |  put an orange on the table into the plate
Cup Place |  pick up the gray cup on the table with the left arm, and then pass the cup to the right arm, place the cup on the shelf with the right arm
Plate Place |  pick up the plate on the table with the right arm and place it on the shelf
Fruit Handover |  grab the apple from the plate with the right arm and handover it to someone
Wine Wipe |  grab a red wine glass with the left arm and a cloth with the right arm to wipe the scattered red wine on the table

MT-ACT-EB3: In this variation of the MT-ACT model, the ResNet-18 image
backbone is substituted with an [33] model to capitalize on enhanced feature
extraction capabilities. Input images are resized to $300\times 300$ pixels
before processing.

MT-ACT-R3M: The MT-ACT-R3M variant employs the R3M model [34], which is
designed for robotic manipulation tasks with pre-trained visual
representations. This model adapts the image backbone for better suitability
in robotic applications, with input images resized to $224\times 224$ pixels.

##  Appendix G Robustness Testing for Robot Manipulation Tasks

Table 9: Robustness Test Results of Single-task Models Method | Task | SR (%) | MES
---|---|---|---
ACT | Plate Place | 40 | 4.29
Cup Place | 60 | 4.57
Wine Wipe | 30 | 1.27
Single Fruit Pick | 10 | 0.66
DP | Plate Place | 20 | 1.30
Cup Place | 40 | 2.11
Single Fruit Pick | 0 | 0
Wine Wipe | 50 | 2.22

The repetitive robot manipulation experiments focus on evaluating the model’s
inference execution and consistency, achieving high SRs due to the low-noise
task environment and minimal background changes. To assess the robustness of
robot learning methods, additional experiments are conducted by introducing
significant changes to the tabletop environment and adding distractors. These
changes aim to test how well the models maintain performance in the face of
unexpected variations and increased complexity. The results, shown in Table 9,
indicate a significant drop in both SR and MES compared to the initial
repetitive tests. For instance, the ACT method’s SR for Plate Place drops from
100% to 40%, with a corresponding decrease in MES to 4.29. Similarly, DP’s
performance in the Single Fruit Pick task falls to a 0% SR, indicating its
inability to handle the increased complexity introduced by background changes
and distractors. These findings underscore the critical importance of
robustness testing in advancing the research in embodied manipulation. High
performance in controlled, low-noise environments does not necessarily
translate to real-world efficacy. The robustness tests reveal the limitations
of the current models, particularly their struggles with environmental
variability and unforeseen obstacles. Such testing methodologies are essential
to understanding and improving the adaptability and resilience of robot
manipulation models. Ultimately, the goal is to develop models that are not
only precise and consistent in controlled settings but also resilient and
versatile in diverse and challenging real-world environments. This highlights
the ongoing need for refinement and enhancement of robot manipulation
algorithms to achieve reliable performance in practical applications.

##  Appendix H Limitations and Future Work

The proposed dataset mainly focuses on household scenes and is not
particularly abundant in quantity. In future work, we will collect data from
more fields to expand our BRMData, including logistics, medical fields, etc.
Moreover, we will always be committed to promoting the development of embodied
intelligence, including embodied methods and datasets.

##  Appendix I Implementation Details

In this paper, the hyperparameters of the evaluated methods for the proposed
dataset are shown in Table 10-11, respectively. Meanwhile, in the multi-task
robot learning methods, we adopt the CLIP [35] network to encode the language
commands. Besides, all the models are trained in a server equipped with 8
NVIDIA L40S graphics cards, an Intel Platinum 8468V CPU, and 1024GB of RAM.

Table 10: Hyperparameters of ACT learning rate | 4e-5
---|---
batch size | 48
# encoder layers | 4
# decoder layers | 7
feedforward dimension | 3200
hidden dimension | 512
# heads | 8
chunk size | 32
beta | 10
dropout | 0.1
backbone | pretrained ResNet18
Table 11: Hyperparameters of Diffusion Policy learning rate | 1e-4
---|---
batch size | 32
chunk size | 64
scheduler | DDIM
train and test diffusion steps | 50,10
ema power | 0.75
backbone | pretrained ResNet18
noise predictor | UNet
| RandomCrop (ratio=0.95) &
image augmentation | ColorJitter &
| RandomRotation