# Point Cloud Matters: Rethinking the Impact of
Different Observation Spaces on Robot Learning

Haoyi Zhu  Yating Wang  Di Huang  Weicai Ye  Wanli Ouyang  Tong He

###### Abstract

In this study, we explore the influence of different observation spaces on
robot learning, focusing on three predominant modalities: RGB, RGB-D, and
point cloud. Through extensive experimentation on over 17 varied contact-rich
manipulation tasks, conducted across two benchmarks and simulators, we have
observed a notable trend: point cloud-based methods, even those with the
simplest designs, frequently surpass their RGB and RGB-D counterparts in
performance. This remains consistent in both scenarios: training from scratch
and utilizing pretraining. Furthermore, our findings indicate that point cloud
observations lead to improved policy zero-shot generalization in relation to
various geometry and visual clues, including camera viewpoints, lighting
conditions, noise levels and background appearance. The outcomes suggest that
3D point cloud is a valuable observation modality for intricate robotic tasks.
We will open-source all our codes and checkpoints, hoping that our insights
can help design more generalizable and robust robotic models.

Machine Learning, ICML

##  1 Introduction

Figure 1: Point cloud has better zero-shot generalization ability on camera
view and visual changes. First row: Zero-shot camera view generalization of
different observations with training-from-scratch encoders (left) and with
SOTA PVRs (right) . Second row: Zero-shot generalization on different unseen
lighting conditions (left) and rendering noise levels (right). Third row:
Zero-shot results, with different background colors, of training-from-scratch
encoders (left) and PVRs (right). Detailed explanation and analysis can be
found in Sec. 6. Figure 2: Overview of this work. We examine the impact of
various observation spaces, specifically RGB, RGB-D, and point clouds, on
robot learning. For this evaluation, we select prevalent and state-of-the-art
methods, along with pretrained representations, and use them to train a action
chunking transformer-based policy network. This approach is assessed across 17
contact-rich tasks using two extensively utilized simulators and benchmarks.

The evolution of robot learning has been profoundly influenced by the
integration of visual observations, a cornerstone that enables robots to
perceive and interact with complex environments. A key challenge faced by
contemporary robot models lies in their limited generalization ability,
especially in dynamic and intricate settings.

Commonly, robotic vision has predominantly utilized 2D images (Radosavovic et
al., 2023; Parisi et al., 2022; Ma et al., 2022; Nair et al., 2023; Majumdar
et al., 2023; Brohan et al., 2022; Zhao et al., 2023; Chi et al., 2023) for
their simplicity and the considerable advancements in 2D foundation models (He
et al., 2020; Bao et al., 2021; Radford et al., 2021; He et al., 2022). For
instance, RT-1 (Brohan et al., 2022) has leveraged RGB images from extensive
real-world task datasets. Action Chunking Transformer (ACT) (Zhao et al.,
2023) employs multiple RGB cameras to inform its transformer policy. Despite
their ubiquity, RGB images often fall short of accurately capturing the three-
dimensional structure of environments, essential for precise action execution.
These methods also exhibit vulnerable generalization to changes such as
lighting conditions and camera viewpoints, owing to their reliance on
appearance.

Given that the actions of robot policies derived from observations are
typically three-dimensional in either Cartesian spaces or joint spaces,
integrating 3D information into observation spaces appears inherently
reasonable. For example, methods like CLIPort (Shridhar et al., 2022)
incorporate depth images for enhanced task execution, while others such as
Perveiver-Actor (Shridhar et al., 2023) and ACT3D (Gervet et al., 2023)
transform RGB-D images into point clouds for explicit spatial information.
Nevertheless, which modality is more suitable for robotic tasks remains
unexplored due to the absence of a unified comparative framework across these
modalities in the existing literature.

In this paper, we select three modalities that are widely used in robotic
tasks: RGB, RGB-D, and 3D point cloud. RGB images are readily available and
supported by robust pretrained visual representations (PVRs). RGB-D images
associate it with extra depth information. Point cloud, on the other hand,
provides precise spatial cues. To make a fair comparison, we avoid complex
designs and domain-specific techniques but instead opt for simple, commonly
accepted designs in each modality community. Our aim is to derive meaningful
insights from more challenging tasks, beyond the scope of easy settings often
explored in previous works. Therefore, rather than employing a basic 2- or
3-layer MLP as commonly done (Nair et al., 2023; Radosavovic et al., 2023;
Hansen et al., 2022; Majumdar et al., 2023), we utilize Action Chunking
Transformer (Zhao et al., 2023) as the controller to assess various
observation spaces across 17 contact-rich tasks in two well-known, realistic,
and challenging benchmarks and simulators: ManiSkill2 (Gu et al., 2023) and
RLBench (James et al., 2020).

We first evaluate each observation space under identical settings, differing
only in input modality and corresponding encoders. Recognizing the growing
prevalence and varied efficacy of pretrained visual representations (PVRs)
across different modalities, we also investigate the performance of state-of-
the-art PVRs within each observation space. Additionally, we also focus on the
zero-shot generalization capabilities regarding camera viewpoints, lighting
conditions, visual appearance, and sample efficiency.

To the best of our knowledge, our work is the first to undertake such an
extensive comparison of different observation spaces. Our key findings can be
summarized as follows:

  * •

Point cloud emerges as a promising observation modality for robot learning.
Simple point cloud baselines exhibit the highest mean success rate and mean
rank, both from scratch and with pretraining.

  * •

Point cloud demonstrates more robustness to camera view and visual changes,
sometimes generalizing effectively even under significant viewpoint
alterations.

##  2 Methods

Comparing different observation spaces convincingly is a non-trivial task.
Avoiding complex or niche tricks, we adopt the most recognized and state-of-
the-art (SOTA) methods across various observation domains. An overview of the
encoders and PVRs used is presented in Tab. 1.

###  2.1 Problem Formulation

The fundamental aim of robot learning is to develop a policy,
$\pi(\cdot|o_{t},\tau)$, deriving actions from visual observations. Here,
$o_{t}\in\mathcal{O}$ is the observation at time $t$, and $\tau$ represents an
optional task-specific target. The robot, guided by this policy, executes an
action $a_{t}\in\mathcal{A}$, generating new observations and receiving a
binary reward $r\in{0,1}$ at each episode’s end. Our goal is to maximize
expected rewards, considering task distribution, initial observations, and
transition dynamics.

Observation Space. We analyze in detail the observation spaces $\mathcal{O}$,
emphasizing that observations $o_{t}$ are projections of the real world
$w_{t}\in\mathcal{W}$ via different sensors
$h(\cdot):\mathcal{W}\rightarrow\mathcal{O}$. Our policy, therefore, is
$\pi_{h}(\cdot|o_{t}=h(w_{t}),\tau)$. We explore the diversity of $h$,
focusing on three observation spaces: $\mathcal{O}_{\mathrm{RGB}}$,
$\mathcal{O}_{\mathrm{RGB-D}}$, and $\mathcal{O}_{\mathrm{PCD}}$.

Representation Space. Visuo-motor policies often use an encoder,
$\mathcal{F}_{\theta}$, to transform $o_{t}$ into a state representation
$\hat{s}=\mathcal{F}_{\theta}(o_{t})$. The policy then becomes
$\pi(\cdot|\hat{s},\tau)$, with $\hat{s}=\mathcal{F}_{\theta}(o_{t})$ and
$o_{t}=h(w_{t})$. While previous studies (Nair et al., 2023; Radosavovic et
al., 2023; Hansen et al., 2022; Majumdar et al., 2023) have primarily focused
on pretrained RGB image encoders, our research evaluates various pretraining
methods across different observation types.

Behavior Cloning. We adopt behavior cloning (Pomerleau, 1988; Levine et al.,
2016; Zhang et al., 2018) for policy learning due to its simplicity and
universality. This method trains $\pi$ on a dataset $\mathcal{D}$ of
successful demonstrations. The objective is to align the robot’s actions with
these demonstrations by optimizing $\pi$ to minimize the negative log-
likelihood of the actions based on the observations and goal conditions.

Table 1: Overview of encoders and corresponding PVRs. #Params denotes number of model parameters while #Pretrained represents number of images or point clouds during pretraining. Obs. Space | Encoder | #Params | PVR | #Pretrained
---|---|---|---|---
RGB |  $\mathbf{\circ}$ ResNet50 | 23.5M |  $\bullet$ R3M | 5M
$\mathbf{\circ}$ ViT-B | 85.8M |  $\bullet$ VC-1 | 5.6M
RGB-D |  $\mathbf{\circ}$ MultiViT-B | 86.1M |  $\bullet$ MultiMAE | 1.28M
Point Cloud |  $\mathbf{\circ}$ SpUNet34 | 43.6M |  $\bullet$ PonderV2 | 4.5K

###  2.2 Encoders

RGB Observation. For RGB images, we utilize $\mathbf{\circ}$ ResNet50 (He et
al., 2016) and $\mathbf{\circ}$ ViT-B (Dosovitskiy et al., 2020), the two most
popular image encoders.

RGB-D Observation. $\mathbf{\circ}$ MultiViT-B (Bachmann et al., 2022), a
Vision Transformer variant, is used for RGB-D observations. This model is
designed for multi-modal inputs with distinct projection layers. It can
effectively integrate RGB data with depth information.

Point Cloud Observation. For point cloud, we select SparseUNet34
($\mathbf{\circ}$ SpUNet34) (Contributors, 2022), which is a sparse
convolutional network. It is widely adopted in the 3D vision community for
point cloud perception tasks.

Feature Extraction. Feature extraction approaches vary by encoder: ResNet uses
final layer features while ViT and MultiViT employ the [CLS] token. Regarding
SpUNet baselines, we simply use farthest point sampling (FPS) (Eldar et al.,
1997; Qi et al., 2017) and K-nearest neighborhood (KNN) (Duda et al., 1973;
Cover & Hart, 1967). Specifically, on the final 3D sparse convolutional
feature map, we first use FPS to select $S$ seed points, then employ KNN to
form $S$ clusters around the seeds. These clusters undergo a linear projection
and pooling layer, resulting in $S$ features as inputs for the policy network.
Our aim is to apply simple and common methods to facilitate a fair comparison.

###  2.3 Pretrained Visual Representations

In our study, we utilize publicly available model weights for the following
pretraining methods:

RGB PVR. $\bullet$ R3M (Nair et al., 2023) and $\bullet$ VC-1 (Majumdar et
al., 2023) are used for $\mathbf{\circ}$ ResNet50 and $\mathbf{\circ}$ ViT-B,
respectively. Both are acclaimed in the realm of Embodied AI. R3M (Nair et
al., 2023) mainly leverages time-contrastive learning and video-language
alignment to pretrain on Ego4D (Grauman et al., 2022) dataset. VC-1 pretrains
a ViT using a masked auto-encoding (MAE) (He et al., 2022) approach with a
broad training scope, encompassing a combination of multiple sources such as
Ego4D (Grauman et al., 2022) and ImageNet (Deng et al., 2009), _etc_.

RGB-D PVR. We selected Multi-modal Multi-task Masked Autoencoders ($\bullet$
MultiMAE) (Bachmann et al., 2022) as the PVR of $\mathbf{\circ}$ MultiViT for
RGB-D images. MultiMAE (Bachmann et al., 2022) is jointly pretrained on RGB,
depth, and semantic modalities using masked autoencoding (He et al., 2022) on
the pseudo-labeled ImageNet-1K dataset (Deng et al., 2009). We focus on its
RGB and depth components.

Point Cloud PVR. For point cloud data, we adopt $\bullet$ PonderV2 (Zhu et
al., 2023), the SOTA self-supervised learning method for point cloud. It
pretrains an $\mathbf{\circ}$ SpUNet via differentiable neural rendering on 3
point cloud datasets.

###  2.4 Policy Network

In this research, we explore non-trivial, contact-rich tasks to derive
meaningful insights. To avoid bottlenecks in policy learning, we diverge from
previous approaches (Parisi et al., 2022; Nair et al., 2023; Radosavovic et
al., 2023; Hansen et al., 2022; Majumdar et al., 2023) of using shallow 2- or
3-layer MLPs, common in 2D PVR studies. Instead, we employ the Action Chunking
Transformer (ACT) (Zhao et al., 2023) as our policy network. ACT (Zhao et al.,
2023) models behavior cloning using a conditional VAE (Sohn et al., 2015) and
has demonstrated remarkable success in a variety of fine-grained manipulation
tasks, both in simulated and real-world settings. See Appendix B for more
details.

##  3 Experimental Setup

Benchmarks and Simulators. Our study employs two well-known simulators and
benchmarks, ManiSkill2 (Gu et al., 2023) and RLBench (James et al., 2020), to
evaluate the impact of different observation spaces in robotic learning.
ManiSkill2 is recognized for its realistic physical simulation, while RLBench
offers a range of vision-guided, language-conditioned tasks. We selected 17
diverse, contact-rich tasks (8 from ManiSkill2 and 9 from RLBench), listed in
Tab. 2. The tasks are selected following prior works (Jia et al., 2023;
Shridhar et al., 2023) and we remove those tasks on which all methods have $0$
success rates. Detailed descriptions and examples of each task can be found in
Appendix A.

Evaluation Metrics. For each task, we employ success rate to evaluate the
performance of each method. In addition, we adopt the evaluation methodology
from VC-1 (Majumdar et al., 2023) encompassing two key metrics: Mean Success
and Mean Rank. Mean Success calculates the average success rate across all
tasks, providing an overall performance indicator. Mean Rank, on the other
hand, involves ranking each method based on their success rate for each task
and then averaging these rankings across all tasks. This metric offers
insights into the relative performance of methods across diverse tasks.

Table 2: Results on different observation spaces. M.S. denotes Mean Success,
while M.R. denots Mean Rank.

Tasks |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34
---|---|---|---|---
ManiSkill2 |  |  |  |
PickCube | 0.60 | 0.15 | 0.02 | 0.74
StackCube | 0.32 | 0.00 | 0.00 | 0.22
TurnFaucet | 0.49 | 0.27 | 0.35 | 0.39
Peg- Insertion- Side | Grasp | 0.73 | 0.36 | 0.16 | 0.81
Align | 0.18 | 0.02 | 0.01 | 0.28
Insert | 0.01 | 0.00 | 0.00 | 0.01
Excavate | 0.02 | 0.00 | 0.00 | 0.11
Hang | 0.86 | 0.80 | 0.78 | 0.80
Pour | 0.07 | 0.00 | 0.00 | 0.10
Fill | 0.79 | 0.30 | 0.76 | 0.66
RLBench |  |  |  |
open drawer | 0.00 | 0.16 | 0.20 | 0.44
sweep to dustpan of size | 0.72 | 0.80 | 0.68 | 0.90
meat off grill | 0.24 | 0.16 | 0.00 | 0.72
turn tap | 0.00 | 0.00 | 0.00 | 0.00
reach and drag | 0.32 | 0.28 | 0.04 | 0.20
put money in safe | 0.60 | 0.76 | 0.28 | 0.60
push buttons | 0.12 | 0.40 | 0.12 | 0.04
close jar | 0.04 | 0.00 | 0.00 | 0.04
place wine at rack location | 0.00 | 0.00 | 0.00 | 0.00
Mean Success Rate $\uparrow$ |  | 0.32 | 0.23 | 0.18 | 0.37
Mean Rank $\downarrow$ | 1.74 | 2.47 | 2.89 | 1.58

Training and Evalution Details. In ManiSkill2 (Gu et al., 2023) experiments,
each model is trained using the demonstrations provided by the official
ManiSkill2 challenge. For TurnFaucet-v0, we adhere to the protocol outlined by
Jia et al. (2023), utilizing 10 faucet models. For PegInsertionSide-v0, we
also report the results for each sub-stage (_i.e_. grasp, align, and insert),
following Jia et al. (2023). We take only the base camera with resolution of
$128\times 128$ for inputs. We evaluate each ManiSkill2 task using 400 fixed
random seeds. In the RLBench (James et al., 2020) experiments, following the
settings of PerAct (Shridhar et al., 2023), we train each task with 100
demonstrations and test them on 25 distinct, held-out scenes. Similarly, only
a front camera with $128\times 128$ resolution is used. Across all
experiments, we adopt an AdamW (Loshchilov & Hutter, 2017) optimizer with the
OneCycle (Smith & Topin, 2019) learning rate scheduler. Each method is trained
with learning rates of $1e-5$, $5e-5$, and $1e-4$. The checkpoint with lowest
validation loss is selected for evaluation. Further implementation details,
experimental procedures and hyperparameter settings are available in Appendix
C.

##  4 How do varying observation spaces influence robot learning performance?

Table 3: Results of PVRs on different observation spaces. Blue and red numbers
denote the relative performance changes compared to their corresponding
training-from-scratch encoders.

Tasks |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2
---|---|---|---|---
ManiSkill2 |  |  |  |
PickCube |  0.82↑0.22 |  0.77↑0.63 |  0.52↑0.49 | 0.87↑0.13
StackCube |  0.41↑0.09 |  0.06↑0.06 |  0.30↑0.30 | 0.35↑0.13
TurnFaucet | 0.46↓0.03 |  0.42↑0.15 |  0.37↑0.02 |  0.27↓0.12
Peg- Insertion- Side | Grasp | 0.84↑0.11 |  0.63↑0.27 |  0.71↑0.55 |  0.65↓0.17
Align |  0.23↑0.06 |  0.07↑0.05 |  0.15↑0.14 | 0.23↓0.05
Insert |  0.01↑0.01 |  0.00↑0.00 |  0.01↑0.01 |  0.02↑0.01
Excavate |  0.38↑0.36 |  0.20↑0.20 |  0.28↑0.28 | 0.38↑0.27
Hang |  0.84↓0.02 | 0.84↑0.04 |  0.77↓0.01 |  0.83↑0.04
Pour | 0.12↑0.06 |  0.04↑0.04 |  0.00↑0.00 |  0.11↑0.02
Fill | 0.88↑0.09 |  0.78↑0.48 |  0.68↓0.08 |  0.73↑0.17
RLBench |  |  |  |
open drawer |  0.00↑0.00 |  0.24↑0.08 |  0.36↑0.16 | 0.60↑0.16
sweep to dustpan of size |  0.52↓0.20 |  0.96↑0.16 | 1.00↑0.32 |  0.96↑0.06
meat off grill |  0.04↓0.20 |  0.12↓0.04 |  0.04↑0.04 | 0.72↑0.00
turn tap |  0.00↑0.00 |  0.04↑0.04 | 0.08↑0.08 |  0.00↑0.00
reach and drag |  0.04↓0.28 |  0.20↓0.08 |  0.16↑0.12 | 0.28↑0.08
put money in safe |  0.48↓0.12 | 0.72↓0.04 |  0.56↑0.28 |  0.64↑0.04
push buttons |  0.24↑0.12 |  0.44↑0.04 | 0.48↑0.36 |  0.16↑0.12
close jar |  0.08↑0.04 |  0.08↑0.08 |  0.00↑0.00 |  0.28↑0.24
place wine at rack location |  0.00↑0.00 |  0.00↑0.00 |  0.08↑0.08 |  0.16↑0.16
Mean Success Rate $\uparrow$ |  |  0.32↑0.00 |  0.34↑0.10 |  0.33↑0.15 | 0.43↑0.06
Mean Rank $\downarrow$ |  |  2.37↑0.63 |  2.63↑0.16 |  2.68↓0.21 | 1.89↑0.32

Figure 3: Performance distributions of different methods. The first row shows
the results of different encoders of different observation spaces trained from
scratch while the second row shows the performance of their corresponding
state-of-the-art pretrained visual representations. The tasks are sorted by
average success rate across all methods. Therefore, tasks on the right side
can be seen as more difficult. The dark cyan dashed line denotes the mean
success rate across all tasks. We can see that point cloud-based methods,
whether trained from scratch or with pretrained, have a higher mean success
rate and are better at difficult tasks. Figure 4: Overall rank distribution
per method. For every method, we compute the overall ranks it achieved on all
tasks. For example, $\bullet$ PonderV2 ranks 1st on 7 tasks.

Our initial investigation focuses on assessing the task performance across
different observation spaces. Each encoder is trained from scratch on
individual tasks, utilizing identical training data across all modalities
except for the input observation types. As shown in Tab. 2, $\mathbf{\circ}$
SpUNet34 ranks highest for 8 times, and $\mathbf{\circ}$ ResNet50 ranks
highest for 5 times. $\mathbf{\circ}$ MultiViT performs the worst (only 0.18
on average). Although depth images contain pseudo spatial clues, the explicit
representation of the 3D point cloud shows superiority on the majority of the
tasks. The results also reveal that no single observation modality
consistently outperforms others across all tasks. However, it is notable that
the point cloud encoder, $\mathbf{\circ}$ SpUNet34, achieves the highest mean
success rate (0.37) and mean rank (1.58) overall. This suggests that point
cloud, on average, possesses significant potential in the realm of robotic
learning tasks.

Moreover, both $\mathbf{\circ}$ ViT-B and $\mathbf{\circ}$ MultiViT-B show
notably lower performance (mean success rate of 0.23 and 0.18). This finding
aligns with existing literature in computer vision (Raghu et al., 2021;
d’Ascoli et al., 2021), suggesting that Vision Transformers may not be well-
suited for training from scratch on small-scale datasets.

For further analysis, we sort tasks based on their average success rate across
different methods, as depicted in the first row of Fig. 3. This visualization
of performance distributions offers insights into the relative difficulty of
each task and the efficacy of each encoder modality. Notably, the superiority
of $\mathbf{\circ}$ SpUNet34 becomes more apparent in tasks of moderate and
higher difficulty levels. This trend suggests that point cloud encoders may
offer distinct advantages in complex scenarios, potentially due to their
inherent capacity to capture detailed spatial information.

##  5 What is the performance impact of PVRs in these diverse observation
spaces?

A pivotal advancement in modern computer vision, particularly for RGB images,
is the development of pretrained visual representations (PVRs). These have
significantly improved performance in downstream tasks, including embodied AI
and robotics (Nair et al., 2023; Ma et al., 2022; Radosavovic et al., 2023;
Hansen et al., 2022; Majumdar et al., 2023). In our experiments, we utilize
publicly available weights for the corresponding encoders to assess the impact
of SOTA PVRs across different observation spaces.

Consistent with prior studies (Hansen et al., 2022; Majumdar et al., 2023),
our findings indicate that maintaining PVRs in a frozen state throughout
training tends to yield inferior results compared to encoders trained from
scratch, likely as a result of a significant domain gap from the training
data. However, contrary to some findings of Hansen et al. (2022), our
experiments indicate that straightforward end-to-end finetuning is effective
across most tasks without any specific tricks. We attribute this to the more
realistic nature of our environments, which presumably narrows the domain gap.

Results with finetuned PVRs are summarized in Tab. 3. While $\bullet$ R3M
enhances performance in many tasks, its performance drops on 6 tasks, thus
leading to an almost negligible increase in mean success rate. Notably, ViTs
exhibit significant improvement with appropriate pretraining. Both $\bullet$
VC-1 and $\bullet$ MultiMAE, leveraging the Masked Autoencoder (MAE)
pretraining paradigm, show the most substantial average improvements (0.10 and
0.15 respectively). $\bullet$ PonderV2, although demonstrating slightly lesser
average gains of 0.06, which may be caused by the limited pretraining data,
still maintains the best performance in terms of mean success rate and rank.
This underscores the efficacy of point cloud observation spaces and highlights
the potential for further advancements in point cloud PVRs.

To gain a broader perspective on the overall rank performance of all methods,
we combine the rankings of both train-from-scratch and pretrained encoders. As
illustrated in Fig. 4, RGB-D observations are unsatisfactory when trained from
scratch, unless with a strong pretraining. What’s more, PVRs are generally
critical, and point cloud encoders, especially when pretrained, stand out as
notably promising within the range of observation spaces.

##  6 Evaluating Zero-Shot Generalization Capabilities Across Observation
Spaces

A key objective in robot learning is to build an agent with strong zero-shot
generalization abilities. This section extensively evaluates the zero-shot
generalization capabilities of different observation spaces, focusing on
camera view (Sec. 6.1) and visual changes (Sec. 6.2). Note that the RGB color
is included in the feature of point clouds.

###  6.1 Zero-Shot Generalization to Camera View

Camera view generalization is crucial yet challenging for robot learning
models, as deploying a robot with the exact training camera positions is often
impractical. Camera view disturbance is extremely challenging since it can
lead to huge geometry changes. Despite its importance, this aspect is often
overlooked in prior works. Our study emphasizes the need for robust camera
view generalization. Given that the camera is fixed during training, we
conduct zero-shot evaluations across novel camera viewpoints, moving
vertically and horizontally by $5^{\circ}$ and $10^{\circ}$ respectively.

The mean success rates under these varied camera views are presented in Tab. 4
and visually represented in Fig. 1. All methods are significantly affected by
geometry changes caused by camera views, even with only $5$ degrees. The
results highlight a prevalent sensitivity to camera view alterations in
current robot models. However, leveraging point cloud data, both pretrained
and from scratch, shows notable resilience to these geometry changes.

This observation raises critical considerations. It appears that image-based
models are more likely to be overly attuned to specific training camera views
or geometries, suggesting a possible overfitting rather than a genuine
understanding of action space skills. Many algorithms such as Zhao et al.
(2023); Chi et al. (2023) are evaluated using identical camera setups as in
training, leaving their true robustness untested. The task of inferring 3D
positions or rotations from images without camera parameters is inherently
ill-posed to some extent. Thus, our findings suggest that further exploration
into point cloud representations could be pivotal in developing more
generalizable, camera view-robust robot models.

Table 4: Mean success rate on zero-shot camera view changes. Methods | Vertical | Horizontal | Average
---|---|---|---
| $5^{\circ}$ | $10^{\circ}$ | $5^{\circ}$ | $10^{\circ}$ |
$\mathbf{\circ}$ ResNet50 | 0.17 | 0.13 | 0.25 | 0.17 | 0.18
$\mathbf{\circ}$ ViT-B | 0.06 | 0.04 | 0.13 | 0.11 | 0.09
$\mathbf{\circ}$ MultiViT | 0.11 | 0.09 | 0.10 | 0.09 | 0.10
$\mathbf{\circ}$ SpUNet34 | 0.24 | 0.18 | 0.25 | 0.17 | 0.21
$\bullet$ R3M | 0.19 | 0.16 | 0.18 | 0.13 | 0.16
$\bullet$ VC-1 | 0.13 | 0.11 | 0.16 | 0.17 | 0.14
$\bullet$ MultiMAE | 0.21 | 0.15 | 0.23 | 0.18 | 0.19
$\bullet$ PonderV2 | 0.29 | 0.21 | 0.29 | 0.22 | 0.25

###  6.2 Zero-Shot Generalization to Visual Changes

Our previous analysis reveals that while point cloud methods generally excel,
they do not uniformly outperform in all tasks. Following our camera view
generalization study, a pertinent question arises: Do RGB and RGB-D approaches
truly excel in performance, or is their advantage claimed by a tendency to
overfit? To address this question, the StackCube task was selected for its
complexity and apparent preference for the RGB modality, as demonstrated by
its higher initial accuracy. As demonstrated in Fig. 1, we observe that
$\mathbf{\circ}$ SpUNet34, when compared in a zero-shot generalization
scenario involving diverse visual changes, is surpassed by the performance of
$\mathbf{\circ}$ ResNet50.

Figure 5: Examples of different lighting conditions. The light intensities are
$0.03,0.6,0.15,0.3$ from left to right respectively. Figure 6: Examples of
different noise levels. The ray tracing samples per pixel are $64,32,16,2$
from left to right respectively. Figure 7: Examples of different background
colors. The background color is denoted as ‘G0.2’, ‘G1.0’, ‘R0.2’, ‘R1.0’ from
left to right respectively, where ‘G’ and ‘R’ means green or red and the
number represents the value of green or red channel.

Zero-Shot Generalization to Lighting Conditions. A primary visual condition we
assess is lighting, a common variable in real-world applications. For
instance, a model trained in the morning might be deployed under evening
lighting conditions. We simulate such scenarios by varying lighting intensity,
testing it at levels of $0.03,0.05,0.15,0.6,1.8,3$, from the default intensity
of $0.3$. Visual illustrations of some lighting conditions are provided in
Fig. 5 and the results are depicted on the left side of the second row in Fig.
1. It becomes evident that RGB and RGB-D methods are significantly impacted by
lighting changes, whereas the point cloud approach demonstrates a higher
degree of robustness. Notably, darker lighting conditions pose a greater
challenge to robotic models. This observation underscores the vulnerability of
RGB and RGB-D methods to environmental lighting variations, highlighting the
resilience of point cloud methods in such scenarios.

Zero-Shot Generalization to Visual Noise. In real-world deployments, noise is
an unavoidable factor, often resulting from hardware inaccuracies or
environmental conditions. While simulators typically operate in noise-free
settings, we introduce visual noise to mimic real-world scenarios. This is
achieved by switching the rendering mode from default rasterization to ray
tracing, disabling the denoiser, and varying the number of ray tracing samples
per pixel, thus altering the noise level. We evaluate four distinct noise
levels, corresponding to $2,16,32,$, and $64$ ray tracing samples per pixel.
Visualizations of these noise levels are presented in Fig. 6. The outcomes of
these tests are displayed on the right side of the second row in Fig. 1. Our
research highlights a marked susceptibility of RGB and RGB-D methods to visual
noise, leading to a significant decline in their performance when subjected to
such conditions. In contrast, point cloud-based methods exhibited remarkable
robustness to noise. This resilience of point cloud methods is particularly
noteworthy, as it suggests their potential for more reliable performance in
noisy, real-world operational environments.

Zero-Shot Generalization to Background Colors. The ability to function across
diverse background conditions is essential for the deployment of robots in
real-world environments, where such conditions can change significantly. The
original setting features a gray floor, which we alter to red and green,
varying the ’R’ or ’G’ values to $0.2,0.6,1.0$ respectively. Visual examples
of these background color changes are illustrated in Fig. 7. The results,
shown on the right side of Fig. 1, align with our previous observations: point
cloud methods exhibit better robustness when it comes to adapting to
background color variations.

For comprehensive results on all our generalization experiments, we invite
readers to consult Tab. 6, Tab. 7, Tab. 8, Tab. 9 and Tab. 10 in Appendix D.
These findings underscore a significant insight: although RGB observations may
excel in some tasks, they lack robustness against various geometry (camera
view) and visual changes. In contrast, while point cloud observation spaces
may not universally outperform in all individual tasks, they demonstrate
markedly greater generalizability to a range of visual alterations and
geometric changes. This phenomenon can be attributed to the inherent nature of
the data representation. 2D images predominantly rely on planar color values,
whereas point clouds incorporate not only color but also explicit three-
dimensional spatial information. This multidimensional data structure reduces
overreliance on visual cues alone, a critical advantage in developing robust
robot models. Consequently, point cloud emerges as a promising observation
space for developing robot models with zero-shot adaptation to new
environments.

##  7 How do different observation spaces and PVRs influence the sample
efficiency?

Table 5: Sample efficiency results on RLBench tasks. #Training | 10 | 25 | 100
---|---|---|---
$\mathbf{\circ}$ ResNet-50 | 0.009 | 0.009 | 0.305
$\mathbf{\circ}$ ViT-B | 0.000 | 0.040 | 0.239
$\mathbf{\circ}$ MultiViT-B | 0.004 | 0.027 | 0.190
$\mathbf{\circ}$ SpUNet32 | 0.004 | 0.018 | 0.351
$\bullet$ R3M | 0.009 | 0.067 | 0.317
$\bullet$ VC-1 | 0.000 | 0.040 | 0.348
$\bullet$ MultiMAE | 0.000 | 0.031 | 0.334
$\bullet$ PonderV2 | 0.013 | 0.040 | 0.439

To understand the sample efficiency of various observation spaces, we also
conduct experiments with reduced training data on RLBench tasks. Specifically,
each method is trained using only $10$ and $25$ training samples. The outcomes
of these experiments are shown in Tab. 5 and Appendix E. Our analysis reveals
that point cloud observation spaces do not exhibit a pronounced advantage in
terms of sample efficiency compared to other modalities. Notably, our results
indicate that pretrained representations consistently bolster performance in
scenarios with limited training data. This suggests that leveraging pretrained
models can be particularly beneficial in few-shot learning contexts, where the
availability of extensive training datasets is constrained.

##  8 Related Work

Pretrained Visual Representations (PVRs). Self-supervised learning (SSL) in 2D
and 3D computer vision has recently gained significant interest, utilizing
various methods like contrastive learning (Chen et al., 2020; He et al., 2020;
Chen et al., 2021; Xie et al., 2020; Hou et al., 2021; Wu et al., 2023;
Radford et al., 2021), distillation-based technique (Caron et al., 2021;
Baevski et al., 2022; Oquab et al., 2023)s, reconstructive approaches (Bao et
al., 2021; He et al., 2022; Pang et al., 2022; Zhang et al., 2022; Yang et
al., 2023a), and differentiable rendering for pretraining (Huang et al., 2023;
Zhu et al., 2023; Yang et al., 2023b). This SSL progress has been integrated
into Embodied AI and robot learning (Parisi et al., 2022; Nair et al., 2023;
Radosavovic et al., 2023; Ma et al., 2022; Khandelwal et al., 2022; Yadav et
al., 2022; Majumdar et al., 2023; Fan et al., 2022), showing impressive
embodied task performance. However, most studies have focused on 2D RGB
spaces, neglecting varied observation spaces and 3D perceptual representations
(PVRs). Our research is pioneering in exploring the effects of different
observation spaces and their corresponding PVRs.

Observation Spaces in Robot Learning. In robot learning, estimating accurate
states from raw observations is challenging, with the predominant focus on 2D
RGB image-based action mapping. However, the relevance of 3D information is
increasingly recognized (Zeng et al., 2021; Gou et al., 2021; Shridhar et al.,
2022, 2023; Gervet et al., 2023; Yen-Chen et al., 2020; Fang et al., 2023b; Li
et al., 2022; Christen et al., 2023; Ze et al., 2023), as seen in methods like
CLIPort (Shridhar et al., 2022) and Perceiver-Actor (Shridhar et al., 2023).
Despite these advancements, there’s a lack of systematic comparison between 2D
and 3D methods, and some 3D techniques involve complex designs (Gervet et al.,
2023; Li et al., 2022; Christen et al., 2023) or rely on dense voxel
representations (Shridhar et al., 2023; Ze et al., 2023). Our work aims to
provide a more equitable comparison of different observation spaces using
simple and common techniques.

Robot Learning for Manipulation. Robot manipulation, often modeled as
(Partially-Observable) Markov Decision Processes (Kaelbling et al., 1998;
Kroemer et al., 2021), faces challenges in reinforcement learning (RL) (Sutton
& Barto, 1999; Finn & Levine, 2017; Gu et al., 2017; Schenck & Fox, 2018) due
to multi-objectiveness, non-stationarity, sim-to-real gaps, and poor sample
efficiency (Kroemer et al., 2021). Behavior cloning (Pomerleau, 1988; Levine
et al., 2016) has emerged as a successful alternative, with diverse approaches
like Perceiver-Actor’s (Shridhar et al., 2023), RT-1 (Brohan et al., 2022),
RT-2 (Brohan et al., 2023), VIMA (Jiang et al., 2023), _etc_. Additionally,
models like the Action Chunking Transformer (Zhao et al., 2023) and diffusion
policy (Chi et al., 2023) use generative models (conditional VAE (Sohn et al.,
2015) and diffusion model (Ho et al., 2020)) for modeling human
demonstrations. Recent foundational models like RT-X (Padalkar et al., 2023)
and Octo (Team et al., 2023) primarily use RGB images for observation. Our
research focuses on exploring the impact of different observation spaces,
aiming to provide insights for future robot learning advancements.

##  9 Conclusion

Our investigation into the influence of diverse observation spaces on robot
learning has led to several significant insights. We demonstrate that point
cloud-based methods, even with simplest designs, consistently outperform their
RGB and RGB-D counterparts in terms of both mean success rate and robustness
to various geometry or visual factors like camera viewpoints, lighting
conditions, noise levels and background colors. This superioity persists
regardless of whether the methods are trained from scratch or with
pretraining, underscoring the potential of point cloud as a promising
observation modality in robotics.

Limitations and Future Work. While point cloud methods show promise, they
exhibit limitations in sample efficiency. The burgeoning field of large-scale
3D datasets, such as RH20T (Fang et al., 2023a) and DL3DV-10K (Ling et al.,
2023), presents an opportunity to scale up 3D PVR methods using these
extensive datasets, which could potentially yield more robust and generalized
solutions for robotic learning.

Our study employed simple designs, such as FPS, which risks over-sampling or
full-sampling on background areas, potentially impacting policy learning.
Future work could explore dynamic sampling techniques like cross-attention or
object-centric representations to address this.

Additionally, this study focus on single modality scenarios. However,
integrating multiple observation spaces using advanced multi-modal techniques
could offer significant benefits. Extending beyond visual observation spaces
to include modalities like tactile sensing also presents an intriguing avenue
for research.

Finally, while our experiments are conducted on simulated benchmarks for
consistency and fairness, efforts to test and adapt these findings in real-
world scenarios are an essential step for future research in robot learning.

## References

  * Bachmann et al. (2022) Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A.  Multimae: Multi-modal multi-task masked autoencoders.  In _European Conference on Computer Vision_ , pp. 348–367. Springer, 2022.
  * Baevski et al. (2022) Baevski, A., Hsu, W.-N., Xu, Q., Babu, A., Gu, J., and Auli, M.  Data2vec: A general framework for self-supervised learning in speech, vision and language.  In _International Conference on Machine Learning_ , pp. 1298–1312. PMLR, 2022.
  * Bao et al. (2021) Bao, H., Dong, L., Piao, S., and Wei, F.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ , 2021.
  * Brohan et al. (2022) Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., Hsu, J., et al.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Brohan et al. (2023) Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023.
  * Caron et al. (2021) Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 9650–9660, 2021.
  * Chen et al. (2020) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Chen et al. (2021) Chen, X., Xie, S., and He, K.  An empirical study of training self-supervised vision transformers.  _arXiv preprint arXiv:2104.02057_ , 2021.
  * Chi et al. (2023) Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel, B., and Song, S.  Diffusion policy: Visuomotor policy learning via action diffusion.  _arXiv preprint arXiv:2303.04137_ , 2023.
  * Christen et al. (2023) Christen, S., Yang, W., Pérez-D’Arpino, C., Hilliges, O., Fox, D., and Chao, Y.-W.  Learning human-to-robot handovers from point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9654–9664, 2023.
  * Contributors (2022) Contributors, S.  Spconv: Spatially sparse convolution library.  https://github.com/traveller59/spconv, 2022.
  * Cover & Hart (1967) Cover, T. and Hart, P.  Nearest neighbor pattern classification.  _IEEE transactions on information theory_ , 13(1):21–27, 1967.
  * Damen et al. (2018) Damen, D., Doughty, H., Farinella, G. M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.  Scaling egocentric vision: The epic-kitchens dataset.  In _Proceedings of the European conference on computer vision (ECCV)_ , pp. 720–736, 2018.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255. Ieee, 2009.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Duda et al. (1973) Duda, R. O., Hart, P. E., et al.  _Pattern classification and scene analysis_ , volume 3.  Wiley New York, 1973.
  * d’Ascoli et al. (2021) d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., and Sagun, L.  Convit: Improving vision transformers with soft convolutional inductive biases.  In _International Conference on Machine Learning_ , pp. 2286–2296. PMLR, 2021.
  * Eldar et al. (1997) Eldar, Y., Lindenbaum, M., Porat, M., and Zeevi, Y. Y.  The farthest point strategy for progressive image sampling.  _IEEE Transactions on Image Processing_ , 6(9):1305–1315, 1997.
  * Fan et al. (2022) Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A.  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  _Advances in Neural Information Processing Systems_ , 35:18343–18362, 2022.
  * Fang et al. (2023a) Fang, H.-S., Fang, H., Tang, Z., Liu, J., Wang, J., Zhu, H., and Lu, C.  Rh20t: A robotic dataset for learning diverse skills in one-shot.  _arXiv preprint arXiv:2307.00595_ , 2023a.
  * Fang et al. (2023b) Fang, H.-S., Wang, C., Fang, H., Gou, M., Liu, J., Yan, H., Liu, W., Xie, Y., and Lu, C.  Anygrasp: Robust and efficient grasp perception in spatial and temporal domains.  _IEEE Transactions on Robotics_ , 2023b.
  * Finn & Levine (2017) Finn, C. and Levine, S.  Deep visual foresight for planning robot motion.  In _2017 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 2786–2793. IEEE, 2017.
  * Gervet et al. (2023) Gervet, T., Xian, Z., Gkanatsios, N., and Fragkiadaki, K.  Act3d: 3d feature field transformers for multi-task robotic manipulation.  In _Conference on Robot Learning_ , pp. 3949–3965. PMLR, 2023.
  * Gou et al. (2021) Gou, M., Fang, H.-S., Zhu, Z., Xu, S., Wang, C., and Lu, C.  Rgb matters: Learning 7-dof grasp poses on monocular rgbd images.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 13459–13466. IEEE, 2021.
  * Goyal et al. (2017) Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., et al.  The” something something” video database for learning and evaluating visual common sense.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 5842–5850, 2017.
  * Grauman et al. (2022) Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al.  Ego4d: Around the world in 3,000 hours of egocentric video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18995–19012, 2022.
  * Gu et al. (2023) Gu, J., Xiang, F., Li, X., Ling, Z., Liu, X., Mu, T., Tang, Y., Tao, S., Wei, X., Yao, Y., et al.  Maniskill2: A unified benchmark for generalizable manipulation skills.  _arXiv preprint arXiv:2302.04659_ , 2023.
  * Gu et al. (2017) Gu, S., Holly, E., Lillicrap, T., and Levine, S.  Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.  In _2017 IEEE international conference on robotics and automation (ICRA)_ , pp. 3389–3396. IEEE, 2017.
  * Hansen et al. (2022) Hansen, N., Yuan, Z., Ze, Y., Mu, T., Rajeswaran, A., Su, H., Xu, H., and Wang, X.  On pre-training for visuo-motor control: Revisiting a learning-from-scratch baseline.  _arXiv preprint arXiv:2212.05749_ , 2022.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * He et al. (2020) He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R.  Momentum contrast for unsupervised visual representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9729–9738, 2020.
  * He et al. (2022) He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 16000–16009, 2022.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hou et al. (2021) Hou, J., Graham, B., Nießner, M., and Xie, S.  Exploring data-efficient 3d scene understanding with contrastive scene contexts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15587–15597, 2021.
  * Huang et al. (2023) Huang, D., Peng, S., He, T., Yang, H., Zhou, X., and Ouyang, W.  Ponder: Point cloud pre-training via neural rendering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 16089–16098, 2023.
  * James et al. (2020) James, S., Ma, Z., Arrojo, D. R., and Davison, A. J.  Rlbench: The robot learning benchmark & learning environment.  _IEEE Robotics and Automation Letters_ , 5(2):3019–3026, 2020.
  * Jia et al. (2023) Jia, Z., Liu, F., Thumuluri, V., Chen, L., Huang, Z., and Su, H.  Chain-of-thought predictive control.  _arXiv preprint arXiv:2304.00776_ , 2023.
  * Jiang et al. (2023) Jiang, Y., Gupta, A., Zhang, Z., Wang, G., Dou, Y., Chen, Y., Fei-Fei, L., Anandkumar, A., Zhu, Y., and Fan, L.  Vima: General robot manipulation with multimodal prompts.  In _Fortieth International Conference on Machine Learning_ , 2023.
  * Kaelbling et al. (1998) Kaelbling, L. P., Littman, M. L., and Cassandra, A. R.  Planning and acting in partially observable stochastic domains.  _Artificial intelligence_ , 101(1-2):99–134, 1998.
  * Khandelwal et al. (2022) Khandelwal, A., Weihs, L., Mottaghi, R., and Kembhavi, A.  Simple but effective: Clip embeddings for embodied ai.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14829–14838, 2022.
  * Kroemer et al. (2021) Kroemer, O., Niekum, S., and Konidaris, G.  A review of robot learning for manipulation: Challenges, representations, and algorithms.  _The Journal of Machine Learning Research_ , 22(1):1395–1476, 2021.
  * Levine et al. (2016) Levine, S., Finn, C., Darrell, T., and Abbeel, P.  End-to-end training of deep visuomotor policies.  _The Journal of Machine Learning Research_ , 17(1):1334–1373, 2016.
  * Li et al. (2022) Li, Y., Li, S., Sitzmann, V., Agrawal, P., and Torralba, A.  3d neural scene representations for visuomotor control.  In _Conference on Robot Learning_ , pp. 112–123. PMLR, 2022.
  * Ling et al. (2023) Ling, L., Sheng, Y., Tu, Z., Zhao, W., Xin, C., Wan, K., Yu, L., Guo, Q., Yu, Z., Lu, Y., et al.  Dl3dv-10k: A large-scale scene dataset for deep learning-based 3d vision.  _arXiv preprint arXiv:2312.16256_ , 2023.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Ma et al. (2022) Ma, Y. J., Sodhani, S., Jayaraman, D., Bastani, O., Kumar, V., and Zhang, A.  Vip: Towards universal visual reward and representation via value-implicit pre-training.  _arXiv preprint arXiv:2210.00030_ , 2022.
  * Majumdar et al. (2023) Majumdar, A., Yadav, K., Arnaud, S., Ma, Y. J., Chen, C., Silwal, S., Jain, A., Berges, V.-P., Abbeel, P., Malik, J., et al.  Where are we in the search for an artificial visual cortex for embodied intelligence?  _arXiv preprint arXiv:2303.18240_ , 2023.
  * Marcel & Rodriguez (2010) Marcel, S. and Rodriguez, Y.  Torchvision the machine-vision package of torch.  In _Proceedings of the 18th ACM international conference on Multimedia_ , pp. 1485–1488, 2010.
  * Nair et al. (2023) Nair, S., Rajeswaran, A., Kumar, V., Finn, C., and Gupta, A.  R3m: A universal visual representation for robot manipulation.  In _Conference on Robot Learning_ , pp. 892–909. PMLR, 2023.
  * Oquab et al. (2023) Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Padalkar et al. (2023) Padalkar, A., Pooley, A., Jain, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Singh, A., Brohan, A., et al.  Open x-embodiment: Robotic learning datasets and rt-x models.  _arXiv preprint arXiv:2310.08864_ , 2023.
  * Pang et al. (2022) Pang, Y., Wang, W., Tay, F. E., Liu, W., Tian, Y., and Yuan, L.  Masked autoencoders for point cloud self-supervised learning.  In _European conference on computer vision_ , pp. 604–621. Springer, 2022.
  * Parisi et al. (2022) Parisi, S., Rajeswaran, A., Purushwalkam, S., and Gupta, A.  The unsurprising effectiveness of pre-trained vision models for control.  In _International Conference on Machine Learning_ , pp. 17359–17371. PMLR, 2022.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Pomerleau (1988) Pomerleau, D. A.  Alvinn: An autonomous land vehicle in a neural network.  _Advances in neural information processing systems_ , 1, 1988.
  * Qi et al. (2017) Qi, C. R., Su, H., Mo, K., and Guibas, L. J.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 652–660, 2017.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Radosavovic et al. (2023) Radosavovic, I., Xiao, T., James, S., Abbeel, P., Malik, J., and Darrell, T.  Real-world robot learning with masked visual pre-training.  In _Conference on Robot Learning_ , pp. 416–426. PMLR, 2023.
  * Raghu et al. (2021) Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A.  Do vision transformers see like convolutional neural networks?  _Advances in Neural Information Processing Systems_ , 34:12116–12128, 2021.
  * Schenck & Fox (2018) Schenck, C. and Fox, D.  Spnets: Differentiable fluid dynamics for deep neural networks.  In _Conference on Robot Learning_ , pp. 317–335. PMLR, 2018.
  * Shan et al. (2020) Shan, D., Geng, J., Shu, M., and Fouhey, D. F.  Understanding human hands in contact at internet scale.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 9869–9878, 2020.
  * Shridhar et al. (2022) Shridhar, M., Manuelli, L., and Fox, D.  Cliport: What and where pathways for robotic manipulation.  In _Conference on Robot Learning_ , pp. 894–906. PMLR, 2022.
  * Shridhar et al. (2023) Shridhar, M., Manuelli, L., and Fox, D.  Perceiver-actor: A multi-task transformer for robotic manipulation.  In _Conference on Robot Learning_ , pp. 785–799. PMLR, 2023.
  * Smith & Topin (2019) Smith, L. N. and Topin, N.  Super-convergence: Very fast training of neural networks using large learning rates.  In _Artificial intelligence and machine learning for multi-domain operations applications_ , volume 11006, pp. 369–386. SPIE, 2019.
  * Sohn et al. (2015) Sohn, K., Lee, H., and Yan, X.  Learning structured output representation using deep conditional generative models.  _Advances in neural information processing systems_ , 28, 2015.
  * Sutton & Barto (1999) Sutton, R. S. and Barto, A. G.  Reinforcement learning: An introduction.  _Robotica_ , 17(2):229–235, 1999.
  * Team et al. (2023) Team, O. M., Ghosh, D., Walke, H., Pertsch, K., Black, K., Mees, O., Dasari, S., Hejna, J., Xu, C., Luo, J., et al.  Octo: An open-source generalist robot policy, 2023.
  * Wu et al. (2023) Wu, X., Wen, X., Liu, X., and Zhao, H.  Masked scene contrast: A scalable framework for unsupervised 3d representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9415–9424, 2023.
  * Xie et al. (2020) Xie, S., Gu, J., Guo, D., Qi, C. R., Guibas, L., and Litany, O.  Pointcontrast: Unsupervised pre-training for 3d point cloud understanding.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pp. 574–591. Springer, 2020.
  * Yadav et al. (2022) Yadav, K., Ramrakhya, R., Majumdar, A., Berges, V.-P., Kuhar, S., Batra, D., Baevski, A., and Maksymets, O.  Offline visual representation learning for embodied navigation.  _arXiv preprint arXiv:2204.13226_ , 2022.
  * Yang et al. (2023a) Yang, H., He, T., Liu, J., Chen, H., Wu, B., Lin, B., He, X., and Ouyang, W.  Gd-mae: generative decoder for mae pre-training on lidar point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9403–9414, 2023a.
  * Yang et al. (2023b) Yang, H., Zhang, S., Huang, D., Wu, X., Zhu, H., He, T., Tang, S., Zhao, H., Qiu, Q., Lin, B., et al.  Unipad: A universal pre-training paradigm for autonomous driving.  _arXiv preprint arXiv:2310.08370_ , 2023b.
  * Yen-Chen et al. (2020) Yen-Chen, L., Zeng, A., Song, S., Isola, P., and Lin, T.-Y.  Learning to see before learning to act: Visual pre-training for manipulation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 7286–7293. IEEE, 2020.
  * Ze et al. (2023) Ze, Y., Yan, G., Wu, Y.-H., Macaluso, A., Ge, Y., Ye, J., Hansen, N., Li, L. E., and Wang, X.  Gnfactor: Multi-task real robot learning with generalizable neural feature fields.  In _Conference on Robot Learning_ , pp. 284–301. PMLR, 2023.
  * Zeng et al. (2021) Zeng, A., Florence, P., Tompson, J., Welker, S., Chien, J., Attarian, M., Armstrong, T., Krasin, I., Duong, D., Sindhwani, V., et al.  Transporter networks: Rearranging the visual world for robotic manipulation.  In _Conference on Robot Learning_ , pp. 726–747. PMLR, 2021.
  * Zhang et al. (2022) Zhang, R., Guo, Z., Gao, P., Fang, R., Zhao, B., Wang, D., Qiao, Y., and Li, H.  Point-m2ae: multi-scale masked autoencoders for hierarchical point cloud pre-training.  _Advances in neural information processing systems_ , 35:27061–27074, 2022.
  * Zhang et al. (2018) Zhang, T., McCarthy, Z., Jow, O., Lee, D., Chen, X., Goldberg, K., and Abbeel, P.  Deep imitation learning for complex manipulation tasks from virtual reality teleoperation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 5628–5635. IEEE, 2018.
  * Zhao et al. (2023) Zhao, T. Z., Kumar, V., Levine, S., and Finn, C.  Learning fine-grained bimanual manipulation with low-cost hardware.  _arXiv preprint arXiv:2304.13705_ , 2023.
  * Zhou et al. (2018) Zhou, T., Tucker, R., Flynn, J., Fyffe, G., and Snavely, N.  Stereo magnification: Learning view synthesis using multiplane images.  _arXiv preprint arXiv:1805.09817_ , 2018.
  * Zhou et al. (2019) Zhou, Y., Barnes, C., Lu, J., Yang, J., and Li, H.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5745–5753, 2019.
  * Zhu et al. (2023) Zhu, H., Yang, H., Wu, X., Huang, D., Zhang, S., He, X., He, T., Zhao, H., Shen, C., Qiao, Y., et al.  Ponderv2: Pave the way for 3d foundataion model with a universal pre-training paradigm.  _arXiv preprint arXiv:2310.08586_ , 2023.

##  Appendix A Task Details and Examples

###  A.1 Maniskill2

####  A.1.1 PickCube-v0

  * •

Objective: Pick up a cube and move it to a goal position.

  * •

Success Criteria: The cube is within 2.5 cm of the goal position, and the
robot is static.

  * •

Demonstration: 1000 successful trajectories.

  * •

Oracle Trajectory: Shown in Figure A.1.

Figure A.1: Illustrations on ManiSkill2 task: PickCube-v0.

####  A.1.2 StackCube-v0

  * •

Objective: Pick up a red cube and place it onto a green one.

  * •

Success Criteria: The red cube is placed on top of the green one stably and it
is not grasped..

  * •

Demonstration: 1000 successful trajectories.

  * •

Oracle Trajectory: Shown in Figure A.2.

Figure A.2: Illustrations on ManiSkill2 task: StackCube-v0.

####  A.1.3 TurnFaucet-v0

  * •

Objective: Turn on a faucet by rotating its handle.

  * •

Success Criteria: The faucet handle has been turned past a target angular
distance.

  * •

Demonstration: From the original set of 5510 trajectories (100 trajectories
per faucet for most of 60 models from PartNet-Mobility), our experiment
focused on 10 models, totaling 1000 trajectories.

  * •

Oracle Trajectory: Shown in Figure A.3.

Figure A.3: Illustrations on ManiSkill2 task: TurnFaucet-v0.

####  A.1.4 PegInsertionSide-v0

  * •

Objective: Pick up the peg, align it with the horizontal hole in the box, and
then insert the peg into the hole.

  * •

Success Criteria: The task is segmented into three stages of success.
Initially, the peg is picked up. Subsequently, the peg is aligned such that
both its head and entirety are within 1 cm of the target hole on the YZ plane.
The final stage is achieved when half of the peg is successfully inserted into
the hole.

  * •

Demonstration: 1000 successful trajectories.

  * •

Oracle Trajectory: Shown in Figure A.4.

Figure A.4: Illustrations on ManiSkill2 task: PegInsertionSide-v0.

####  A.1.5 Excavate-v0

  * •

Objective: Lift a specific amount of clay to a target height.

  * •

Success Criteria:The amount of lifted clay must be within a given range; the
lifted clay is higher than a specific height; fewer than 20 clay particles are
spilled on the ground; soft body velocity$<$0.05.

  * •

Demonstration: 200 successful trajectories.

  * •

Oracle Trajectory: Shown in Figure A.5.

Figure A.5: Illustrations on ManiSkill2 task: Excavate-v0.

####  A.1.6 Hang-v0

  * •

Objective: Hang a noodle on a target rod.

  * •

Success Criteria:Part of the noodle is higher than the rod; two ends of the
noodle are on different sides of the rod; the noodle is not touching the
ground; the gripper is open; soft body velocity$<$0.05.

  * •

Demonstration: 200 successful trajectories.

  * •

Oracle Trajectory: Shown in Figure A.6.

Figure A.6: Illustrations on ManiSkill2 task: Hang-v0.

####  A.1.7 Pour-v0

  * •

Objective: Pour liquid from a bottle into a beaker.

  * •

Success Criteria:The liquid level in the beaker is within 4mm of the red line;
the spilled water is fewer than 100 particles; the bottle returns to the
upright position in the end; robot arm velocity$<$0.05.

  * •

Demonstration: 200 successful trajectories.

  * •

Oracle Trajectory: Shown in Figure A.7.

Figure A.7: Illustrations on ManiSkill2 task: Pour-v0.

####  A.1.8 Fill-v0

  * •

Objective: Fill clay from a bucket into the target beaker.

  * •

Success Criteria:The amount of clay inside the target beaker$>$90%; soft body
velocity$<$0.05.

  * •

Demonstration: 200 successful trajectories.

  * •

Oracle Trajectory: Shown in Figure A.8.

Figure A.8: Illustrations on ManiSkill2 task: Fill-v0.

###  A.2 RLBench

All RLBench tasks are trained on 100 successful trajectories. Each task has
multiple variations with different language descriptions.

####  A.2.1 open drawer

  * •

Objective: Open one of the three drawers: top, middle, or bottom.

  * •

Success Criteria: The prismatic joint of the specified drawer is fully
extended.

  * •

Example description:Open the top drawer.

  * •

Oracle Trajectory: Shown in Figure A.9.

Figure A.9: Illustrations on RLBench task: open drawer.

####  A.2.2 sweep to dustpan of size

  * •

Objective: Use the broom to brush the dirt particles into either the short or
tall dustpan.

  * •

Success Criteria: All 5 dirt particles are inside the specified dustpan.

  * •

Example description: Sweep dirt to the short dustpan.

  * •

Oracle Trajectory: Shown in Figure A.10.

Figure A.10: Illustrations on RLBench task: sweep to dustpan of size.

####  A.2.3 meat off grill

  * •

Objective: Take either the chicken or steak off the grill and set it down on
the side.

  * •

Success Criteria: The specified meat is on the side, away from the grill.

  * •

Example description: Take the steak off the grill.

  * •

Oracle Trajectory: Shown in Figure A.11.

Figure A.11: Illustrations on RLBench task: meat off grill .

####  A.2.4 turn tap

  * •

Objective: Turn either the left or right handle of the tap. Left and right are
defined with respect to the faucet orientation.

  * •

Success Criteria: The revolute joint of the specified handle is at least
$90^{\circ}$ off from the starting position.

  * •

Example description: Turn right tap.

  * •

Oracle Trajectory: Shown in Figure A.12.

Figure A.12: Illustrations on RLBench task: turn tap .

####  A.2.5 reach and drag

  * •

Objective: Grab the stick and use it to drag the cube on to the specified
colored target square. The target colors are sampled from the full set of 20
color instances.

  * •

Success Criteria: Some part of the block is inside the specified target area.

  * •

Example description: Use the stick to drag the cube onto the navy target.

  * •

Oracle Trajectory: Shown in Figure A.13.

Figure A.13: Illustrations on RLBench task: reach and drag .

####  A.2.6 put money in safe

  * •

Objective: Pick up the stack of money and put it inside the safe on the
specified shelf. The shelf has three placement locations: top, middle, bottom.

  * •

Success Criteria: The stack of money is on the specified shelf inside the
safe.

  * •

Example description: Put the money away in the safe on the top shelf.

  * •

Oracle Trajectory: Shown in Figure A.14.

Figure A.14: Illustrations on RLBench task: put money in safe.

####  A.2.7 push buttons

  * •

Objective: Push the colored buttons in the specified sequence. The button
colors are sampled from the full set of 20 color instances. There are always
three buttons in scene.

  * •

Success Criteria: All the specified buttons were pressed.

  * •

Example description:Push the maroon button, then push the green button, then
push the navy button.

  * •

Oracle Trajectory: Shown in Figure A.15.

Figure A.15: Illustrations on RLBench task: push buttons.

####  A.2.8 close jar

  * •

Objective: Pick up the lid and close the jar of the specified color. The
target colors are sampled from the full set of 20 color instances.

  * •

Success Criteria:The lid is capped on a specific jar.

  * •

Example description: Close the red jar.

  * •

Oracle Trajectory: Shown in Figure A.16.

Figure A.16: Illustrations on RLBench task: close jar.

####  A.2.9 place wine at rack location

  * •

Objective: Grab the wine bottle and put it on the wooden rack at one of the
three specified locations: left, middle, right. The locations are defined with
respect to the orientation of the wooden rack.

  * •

Success Criteria: The wine bottle is at the specified placement location on
the wooden rack.

  * •

Example description: Stack the wine bottle to the left of the rack.

  * •

Oracle Trajectory: Shown in Figure A.17.

Figure A.17: Illustrations on RLBench task: place wine at rack location.

##  Appendix B Method Details

###  B.1 Encoders

$\mathbf{\circ}$ ResNet50. A key element in the $\mathbf{\circ}$ ResNet50 (He
et al., 2016) architecture is its utilization of deep residual learning with
skip connections, a design that effectively addresses the vanishing gradient
problem in deep networks. This structure allows $\mathbf{\circ}$ ResNet50 to
develop robust feature representations, crucial for complex image processing
tasks.

$\mathbf{\circ}$ ViT-B (Vision Transformer). Representing a significant shift
in image processing, $\mathbf{\circ}$ ViT-B (Dosovitskiy et al., 2020) applies
the transformer mechanism, traditionally used in natural language processing,
to visual data. It processes images by segmenting them into patches and
analyzing these via a transformer encoder, adept at capturing intricate
spatial hierarchies.

$\mathbf{\circ}$ MultiViT-B. An innovative adaptation of the Vision
Transformer, $\mathbf{\circ}$ MultiViT-B (Bachmann et al., 2022) is tailored
for multi-modal input, including RGB-D images. It features unique projection
layers for each modality, seamlessly integrating depth information with RGB
data. This fusion enriches the model’s perception of spatial relationships,
enhancing its environmental analysis capabilities.

$\mathbf{\circ}$ SpUNet34. In the realm of 3D vision, $\mathbf{\circ}$ SpUNet
(Contributors, 2022) is a prominent choice for processing sparse 3D data
structures like point clouds. It employs sparse convolution, efficiently
handling non-uniform data distribution in 3D space. The SpUNet architecture,
inspired by ResNet34, is modified to accommodate the nuances of 3D point cloud
data, ensuring effective processing and interpretation of these complex
structures.

###  B.2 Pretrained Visual Representations

$\bullet$ R3M. Employed for $\mathbf{\circ}$ ResNet50, $\bullet$ R3M (Nair et
al., 2023) uses time-contrastive learning, video-language alignment, and L1
regularization to produce sparse, compact representations. It’s trained on a
substantial 3,500 hours of human interaction videos from the Ego4D (Grauman et
al., 2022) dataset, demonstrating superior performance over other methods like
CLIP (Radford et al., 2021) and MoCo (He et al., 2020) in simulated robotic
manipulation tasks.

$\bullet$ VC-1. This approach pretrains a ViT using masked auto-encoding (MAE)
(He et al., 2022), similar to MVP (Radosavovic et al., 2023) but with a
broader dataset range, including Ego4D (Grauman et al., 2022), 100 Days of
Hands (100DOH) (Shan et al., 2020), Something-Something v2 (SS-V2) (Goyal et
al., 2017), Epic Kitchens (Damen et al., 2018), Real Estate 10K (Zhou et al.,
2018), and ImageNet (Deng et al., 2009). It excels in diverse embodied AI
tasks.

$\bullet$ MultiMAE (Multi-modal Multi-task Masked Autoencoders). $\bullet$
MultiMAE (Bachmann et al., 2022) is utilized for MultiViT, featuring masked
autoencoding across various modalities including RGB, depth, and semantics on
the ImageNet-1K dataset. Its cross-modality predictive coding significantly
enhances transfer to downstream tasks such as image classification and depth
estimation. In our study, we focus on its RGB and depth components.

$\bullet$ PonderV2. As the state-of-the-art self-supervised learning method
for point clouds, $\bullet$ PonderV2 (Zhu et al., 2023) employs differentiable
neural rendering. The method trains a 3D backbone ($\mathbf{\circ}$ SpUNet34)
within a volumetric neural renderer, focusing on learning detailed geometry
and appearance cues. $\bullet$ PonderV2 is effective across a wide range of
tasks, including high-level challenges like 3D detection and segmentation, as
well as low-level objectives like 3D reconstruction and image synthesis,
covering both indoor and outdoor scenarios. It demonstrates the framework’s
effectiveness across multiple benchmarks and showcases its potential in
advancing 3D foundation models.

###  B.3 Policy Network

In our study, we implemented the Action Chunking Transformer (ACT) (Zhao et
al., 2023) as our primary policy network. This choice was driven by ACT’s
proficiency in addressing the inherent challenges of imitation learning,
particularly in precision-demanding contexts where errors tend to accumulate
progressively.

ACT stands out for its innovative approach to generative modeling over action
sequences, significantly improving the robot’s ability to execute complex
tasks with heightened accuracy. One of the pivotal features of ACT is its
action chunking technique. This involves grouping sequences of actions into
cohesive units for execution, effectively shortening the task’s horizon and
minimizing the accumulation of errors. This aspect is particularly beneficial
for tasks that require detailed, fine-grained manipulations.

Furthermore, ACT integrates a conditional variational autoencoder (CVAE)
architecture. This design is adept at accommodating the variability and
stochastic nature found in human demonstrations, a common challenge in robot
learning. The CVAE framework facilitates the efficient learning of nuanced
manipulation skills, a capability that has led ACT to surpass previous
methodologies in both simulated and real-world applications.

##  Appendix C Implementation Details

General Implementations. Our implementation leverages PyTorch (Paszke et al.,
2019), a powerful framework for deep learning applications. For the
$\mathbf{\circ}$ ResNet50 encoder, we utilized the official model available in
TorchVision (Marcel & Rodriguez, 2010), aligning with the implementation in
$\bullet$ R3M. This approach ensures consistency with the original work in
terms of weights and feature extraction. For the $\mathbf{\circ}$ ViT and
$\bullet$ VC-1 models, we meticulously adhered to the original
implementations. We chose the ViT-base and corresponding VC-1-base models to
maintain parameter consistency with other methods used in our study, promoting
relative fairness. In the case of $\mathbf{\circ}$ MultiViT and $\bullet$
MultiMAE, our implementation is based on the official $\mathbf{\circ}$
MultiViT-B model. We employed the strongest pretrained weights available in
the official repository, pretrained simultaneously on RGB, depth, and semantic
modalities, enhancing the model’s robustness and versatility. For
$\mathbf{\circ}$ SpUNet and $\bullet$ PonderV2, we adopt the $\mathbf{\circ}$
SpUNet34 architecture as described in the official $\bullet$ PonderV2 paper,
ensuring alignment with their proposed methodology and optimizing the model’s
performance in handling point cloud data.

Input Preprocessing. For inputs to $\mathbf{\circ}$ ResNet50 and $\bullet$
R3M, we rigorously adhere to the $\bullet$ R3M preprocessing guidelines. This
includes resizing images to $224\times 224$ and normalizing them using the
ImageNet mean and standard deviation. For $\mathbf{\circ}$ ViT and $\bullet$
VC-1, we follow the$\bullet$ VC-1 preprocessing protocol. This process
involves resizing images to $256\times 256$ via bicubic interpolation, center
cropping them to $224\times 224$, and then normalizing with ImageNet mean and
std. For $\mathbf{\circ}$ MultiViT and $\bullet$ MultiMAE inputs, we
meticulously replicate the original method’s preprocessing. This includes
resizing, center cropping, and normalizing the RGB component, along with
truncating and normalizing the depth images. For point cloud data, we employ
grid sampling with a grid size of $0.005$, aligning with the PonderV2
methodology. Notably, the original PonderV2 model utilizes 6-channel inputs
(RGB and normals). In our case, given the absence of normal values in robotic
data, we substitute XYZ values alongside RGB. Despite this deviation from the
original training distribution, we surprisingly find that this approach yields
significant performance improvements after end-to-end finetuning.

Feature Extraction. In line with the original ACT implementation, we extract
features from the final convolutional layer of ResNet. To enhance these
features, we also follow ACT to incorporate 2D cosine position embeddings,
enriching the spatial context of the extracted feature map. For ViT and
MultiViT, we focus on the [CLS] token, recognized for encapsulating global
information of the input. We treat this token as a $1\times 768$ feature map
and apply a position embedding approach analogous to that used for ResNet’s
features. In processing point cloud data, our initial step involves utilizing
Farthest Point Sampling (FPS) to select $2048$ seed points. For each seed
point, we employ a K-Nearest Neighbors (KNN) algorithm to cluster $16$
neighboring points. Subsequently, each cluster undergoes max pooling following
a linear projection layer. The final step entails adding positional embeddings
based on the 3D coordinates of the seed points, which enhances the spatial
relevance of the extracted features.

ACT Policy Network. In implementing the Action Chunking Transformer (ACT)
policy network, we adhere to the original framework while tailoring the
encoder forward function to suit our specific needs. Key hyperparameters are
chosen as followings: we set the dropout rate to $0.1$, number of heads
(nhead) to $8$, dimension of feedforward network (dim_feedforward) to $32$,
number of encoder layers to $4$, and decoder layers to $7$. The hidden
dimension (hidden_dim) is fixed at $512$, with chunk sizes of $100$ for
ManiSkill2 and $20$ for RLBench tasks. The latent dimension (latent_dim) is
configured to $32$, and the weight for the KL divergence loss is set at
$10.0$. For tasks involving goal conditions, we project the goals into a
$512$-dimensional space to align with the hidden dimension of the transformer
policy. In the case of language-based goals in RLBench, we employ the language
encoder from CLIP to extract a feature vector of $512$ dimensions.

Training Details. For training each method, we employ the AdamW (Loshchilov &
Hutter, 2017) optimizer, known for its efficiency in weight optimization, with
a weight decay set at $0.05$. The OneCycle (Smith & Topin, 2019) learning rate
scheduler is configured to accelerate the learning process, featuring a
pct_start of $0.15$, anneal_strategy set to ’cos’, div_factor at $100$, and
final_div_factor at $1000$. The scheduler is updated at a step interval
frequency of $1$. Experiments are conducted across learning rates of
$1e-5,5e-5,1e-4$, each replicated twice to ensure consistency. Performance
evaluation relies on the checkpoint with the lowest validation loss.
ManiSkill2 experiments are carried out on an NVIDIA RTX 3090 GPU, with a batch
size of $16$ over $500$ epochs. RLBench experiments are conducted on an NVIDIA
A100 GPU. These experiments have a batch size of $32$ and also span $500$
epochs. We deviate from the quaternion, opting for a 6D rotation (Zhou et al.,
2019) representation for rotation actions, following Gervet et al. (2023). In
both sets of experiments, we apply action chunking and temporal ensembling
with an exponential decay factor of $k=0.01$, consistent with the original ACT
methodology.

##  Appendix D Full Zero-Shot Generalization Results

Here we give out full results on zero-shot generalization experiments.

###  D.1 Zero-Shot Generalization to Camera View

The results on camera view changes are listed in Tab. 6, Tab. 7, Tab. 8, Tab.
9.

Table 6: Full results on zero-shot generalization of scratch encoders to vertical camera view changes. |  |  Vertical 5∘ |  Vertical 10∘
---|---|---|---
Tasks |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34 |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34
ManiSkill2 |  |  |  |  |  |  |  |
PickCube | 0.66 | 0.13 | 0.02 | 0.64 | 0.58 | 0.03 | 0.02 | 0.44
StackCube | 0.32 | 0.00 | 0.00 | 0.21 | 0.26 | 0.00 | 0.00 | 0.22
TurnFaucet | 0.08 | 0.06 | 0.14 | 0.09 | 0.09 | 0.07 | 0.13 | 0.09
PegInsertionSide | Grasp | 0.62 | 0.33 | 0.15 | 0.81 | 0.45 | 0.27 | 0.14 | 0.81
Align | 0.09 | 0.02 | 0.00 | 0.30 | 0.04 | 0.02 | 0.00 | 0.28
Insert | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 | 0.00 | 0.00 | 0.01
Excavate | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.04
Hang | 0.75 | 0.51 | 0.81 | 0.62 | 0.61 | 0.33 | 0.67 | 0.71
Pour | 0.01 | 0.00 | 0.00 | 0.08 | 0.01 | 0.00 | 0.00 | 0.06
Fill | 0.29 | 0.05 | 0.75 | 0.53 | 0.34 | 0.10 | 0.76 | 0.15
RLBench |  |  |  |  |  |  |  |
open drawer | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.12
sweep to dustpan of size | 0.00 | 0.00 | 0.00 | 0.12 | 0.00 | 0.00 | 0.00 | 0.04
meat off grill | 0.04 | 0.08 | 0.00 | 0.56 | 0.00 | 0.00 | 0.00 | 0.12
turn tap | 0.08 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
reach and drag | 0.00 | 0.00 | 0.04 | 0.28 | 0.08 | 0.00 | 0.00 | 0.20
put money in safe | 0.08 | 0.00 | 0.16 | 0.32 | 0.00 | 0.00 | 0.08 | 0.08
push buttons | 0.08 | 0.00 | 0.00 | 0.08 | 0.04 | 0.00 | 0.00 | 0.04
close jar | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
place wine at rack location | 0.12 | 0.00 | 0.00 | 0.00 | 0.04 | 0.00 | 0.00 | 0.00
Mean Success rate |  | 0.17 | 0.06 | 0.11 | 0.24 | 0.13 | 0.04 | 0.09 | 0.18
Table 7: Full results on zero-shot generalization of PVRs to vertical camera view changes. |  |  Vertical 5∘ |  Vertical 10∘
---|---|---|---
Tasks |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2 |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2
ManiSkill2 |  |  |  |  |  |  |  |
PickCube | 0.84 | 0.74 | 0.52 | 0.82 | 0.84 | 0.71 | 0.42 | 0.66
StackCube | 0.29 | 0.07 | 0.25 | 0.37 | 0.23 | 0.07 | 0.18 | 0.36
TurnFaucet | 0.09 | 0.06 | 0.01 | 0.07 | 0.11 | 0.06 | 0.08 | 0.07
PegInsertionSide | Grasp | 0.69 | 0.49 | 0.61 | 0.66 | 0.51 | 0.37 | 0.48 | 0.65
Align | 0.16 | 0.04 | 0.09 | 0.23 | 0.09 | 0.03 | 0.06 | 0.23
Insert | 0.01 | 0.00 | 0.00 | 0.03 | 0.00 | 0.00 | 0.00 | 0.03
Excavate | 0.00 | 0.00 | 0.11 | 0.13 | 0.00 | 0.00 | 0.12 | 0.14
Hang | 0.76 | 0.75 | 0.81 | 0.76 | 0.54 | 0.68 | 0.67 | 0.56
Pour | 0.06 | 0.02 | 0.00 | 0.09 | 0.03 | 0.01 | 0.00 | 0.06
Fill | 0.38 | 0.14 | 0.69 | 0.80 | 0.47 | 0.22 | 0.70 | 0.42
RLBench |  |  |  |  |  |  |  |
open drawer | 0.04 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
sweep to dustpan of size | 0.00 | 0.04 | 0.28 | 0.36 | 0.00 | 0.00 | 0.00 | 0.16
meat off grill | 0.08 | 0.04 | 0.00 | 0.20 | 0.04 | 0.00 | 0.00 | 0.08
turn tap | 0.04 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.08
reach and drag | 0.12 | 0.00 | 0.12 | 0.24 | 0.16 | 0.00 | 0.04 | 0.00
put money in safe | 0.00 | 0.00 | 0.44 | 0.48 | 0.00 | 0.00 | 0.16 | 0.28
push buttons | 0.00 | 0.00 | 0.04 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00
close jar | 0.00 | 0.00 | 0.00 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00
place wine at rack location | 0.04 | 0.00 | 0.00 | 0.12 | 0.04 | 0.00 | 0.00 | 0.12
Mean Success rate |  | 0.19 | 0.13 | 0.21 | 0.29 | 0.16 | 0.11 | 0.15 | 0.21
Table 8: Full results on zero-shot generalization of scratch encoders to horizontal camera view changes. |  |  Horizontal 5∘ |  Horizontal 10∘
---|---|---|---
Tasks |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34 |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34
ManiSkill2 |  |  |  |  |  |  |  |
PickCube | 0.65 | 0.01 | 0.02 | 0.70 | 0.55 | 0.02 | 0.03 | 0.65
StackCube | 0.25 | 0.00 | 0.00 | 0.23 | 0.01 | 0.00 | 0.00 | 0.09
TurnFaucet | 0.14 | 0.09 | 0.13 | 0.06 | 0.13 | 0.09 | 0.14 | 0.08
PegInsertionSide | Grasp | 0.52 | 0.32 | 0.15 | 0.81 | 0.28 | 0.17 | 0.15 | 0.81
Align | 0.11 | 0.04 | 0.00 | 0.27 | 0.30 | 0.02 | 0.01 | 0.28
Insert | 0.00 | 0.00 | 0.00 | 0.01 | 0.00 | 0.00 | 0.00 | 0.01
Excavate | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
Hang | 0.81 | 0.76 | 0.47 | 0.79 | 0.74 | 0.33 | 0.45 | 0.71
Pour | 0.04 | 0.00 | 0.00 | 0.09 | 0.01 | 0.00 | 0.00 | 0.05
Fill | 0.30 | 0.08 | 0.74 | 0.03 | 0.31 | 0.88 | 0.74 | 0.02
RLBench |  |  |  |  |  |  |  |
open drawer | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
sweep to dustpan of size | 0.24 | 0.20 | 0.04 | 0.80 | 0.00 | 0.00 | 0.08 | 0.08
meat off grill | 0.16 | 0.12 | 0.00 | 0.48 | 0.04 | 0.20 | 0.00 | 0.32
turn tap | 0.00 | 0.00 | 0.00 | 0.00 | 0.04 | 0.04 | 0.00 | 0.00
reach and drag | 0.68 | 0.36 | 0.04 | 0.04 | 0.32 | 0.04 | 0.00 | 0.00
put money in safe | 0.60 | 0.32 | 0.16 | 0.36 | 0.24 | 0.20 | 0.00 | 0.24
push buttons | 0.28 | 0.16 | 0.08 | 0.04 | 0.12 | 0.08 | 0.04 | 0.00
close jar | 0.00 | 0.00 | 0.04 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
place wine at rack location | 0.04 | 0.00 | 0.00 | 0.00 | 0.08 | 0.00 | 0.00 | 0.00
Mean Success rate |  | 0.25 | 0.13 | 0.10 | 0.25 | 0.17 | 0.11 | 0.09 | 0.17
Table 9: Full results on zero-shot generalization of PVRs to horizontal camera view changes. |  |  Horizontal 5∘ |  Horizontal 10∘
---|---|---|---
Tasks |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2 |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2
ManiSkill2 |  |  |  |  |  |  |  |
PickCube | 0.82 | 0.72 | 0.49 | 0.85 | 0.80 | 0.62 | 0.44 | 0.84
StackCube | 0.28 | 0.05 | 0.20 | 0.23 | 0.00 | 0.04 | 0.17 | 0.09
TurnFaucet | 0.09 | 0.08 | 0.11 | 0.07 | 0.09 | 0.70 | 0.13 | 0.06
PegInsertionSide | Grasp | 0.15 | 0.18 | 0.28 | 0.65 | 0.56 | 0.43 | 0.55 | 0.65
Align | 0.00 | 0.00 | 0.00 | 0.02 | 0.11 | 0.06 | 0.10 | 0.22
Insert | 0.11 | 0.06 | 0.10 | 0.22 | 0.00 | 0.00 | 0.00 | 0.02
Excavate | 0.00 | 0.00 | 0.11 | 0.03 | 0.00 | 0.00 | 0.11 | 0.07
Hang | 0.84 | 0.78 | 0.45 | 0.77 | 0.80 | 0.67 | 0.46 | 0.72
Pour | 0.04 | 0.03 | 0.00 | 0.09 | 0.02 | 0.01 | 0.00 | 0.05
Fill | 0.46 | 0.11 | 0.70 | 0.03 | 0.02 | 0.09 | 0.70 | 0.02
RLBench |  |  |  |  |  |  |  |
open drawer | 0.00 | 0.00 | 0.00 | 0.08 | 0.00 | 0.04 | 0.04 | 0.04
sweep to dustpan of size | 0.28 | 0.24 | 0.56 | 0.72 | 0.00 | 0.04 | 0.24 | 0.04
meat off grill | 0.16 | 0.20 | 0.04 | 0.76 | 0.04 | 0.16 | 0.00 | 0.64
turn tap | 0.00 | 0.00 | 0.00 | 0.08 | 0.00 | 0.00 | 0.04 | 0.04
reach and drag | 0.00 | 0.16 | 0.60 | 0.36 | 0.00 | 0.00 | 0.04 | 0.32
put money in safe | 0.04 | 0.32 | 0.40 | 0.32 | 0.04 | 0.16 | 0.16 | 0.04
push buttons | 0.08 | 0.16 | 0.08 | 0.04 | 0.04 | 0.12 | 0.04 | 0.08
close jar | 0.00 | 0.04 | 0.20 | 0.12 | 0.00 | 0.00 | 0.08 | 0.16
place wine at rack location | 0.00 | 0.00 | 0.00 | 0.12 | 0.00 | 0.00 | 0.04 | 0.08
Mean Success rate |  | 0.18 | 0.16 | 0.23 | 0.29 | 0.13 | 0.17 | 0.18 | 0.22

###  D.2 Zero-Shot Generalization to Visual Changes

Table 10: Full results on visual change generalization capibilities. |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34 |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2
---|---|---|---|---|---|---|---|---
lighting intensity |  |  |  |  |  |  |  |
0.03 | 0.000 | 0.000 | 0.000 | 0.005 | 0.000 | 0.010 | 0.098 | 0.000
0.05 | 0.000 | 0.000 | 0.000 | 0.015 | 0.00 | 0.023 | 0.108 | 0.003
0.15 | 0.000 | 0.000 | 0.000 | 0.038 | 0.003 | 0.055 | 0.210 | 0.058
0.60 | 0.000 | 0.000 | 0.000 | 0.208 | 0.003 | 0.073 | 0.218 | 0.253
1.80 | 0.000 | 0.000 | 0.000 | 0.133 | 0.000 | 0.015 | 0.083 | 0.093
3.00 | 0.000 | 0.000 | 0.000 | 0.125 | 0.000 | 0.000 | 0.035 | 0.075
noise level |  |  |  |  |  |  |  |
2 | 0.008 | 0.000 | 0.000 | 0.098 | 0.003 | 0.000 | 0.203 | 0.163
16 | 0.008 | 0.000 | 0.000 | 0.120 | 0.008 | 0.000 | 0.215 | 0.148
32 | 0.008 | 0.000 | 0.000 | 0.098 | 0.008 | 0.050 | 0.225 | 0.138
64 | 0.008 | 0.000 | 0.000 | 0.085 | 0.002 | 0.005 | 0.210 | 0.150
background color |  |  |  |  |  |  |  |
R0.2 | 0.000 | 0.000 | 0.000 | 0.205 | 0.000 | 0.008 | 0.038 | 0.365
R0.6 | 0.000 | 0.000 | 0.000 | 0.213 | 0.000 | 0.008 | 0.000 | 0.330
R1.0 | 0.000 | 0.000 | 0.000 | 0.218 | 0.000 | 0.003 | 0.000 | 0.368
G0.2 | 0.000 | 0.000 | 0.000 | 0.223 | 0.000 | 0.025 | 0.105 | 0.340
G0.6 | 0.000 | 0.000 | 0.000 | 0.213 | 0.000 | 0.000 | 0.003 | 0.343
G1.0 | 0.000 | 0.000 | 0.000 | 0.218 | 0.000 | 0.003 | 0.000 | 0.343

The results on visual changes are listed in Tab. 10.

##  Appendix E Full Sample Efficiency Results

Full sample efficiency results on RLBench are listed in Tab. 11.

Table 11: Full results on sample efficiency. |  | sample 25% | sample 10%
---|---|---|---
Tasks |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34 |  $\mathbf{\circ}$ ResNet50 |  $\mathbf{\circ}$ ViT-B |  $\mathbf{\circ}$ MultiViT-B |  $\mathbf{\circ}$ SpUNet34
open drawer | 0.080 | 0.000 | 0.200 | 0.120 | 0.000 | 0.000 | 0.040 | 0.000
sweep to dustpan of size | 0.000 | 0.240 | 0.040 | 0.040 | 0.000 | 0.000 | 0.000 | 0.000
meat off grill | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
turn tap | 0.000 | 0.000 | 0.000 | 0.000 | 0.080 | 0.000 | 0.000 | 0.040
reach and drag | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
put money in safe | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
push buttons | 0.000 | 0.120 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
close jar | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
place wine at rack location | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
Mean Success rate | 0.009 | 0.040 | 0.027 | 0.018 | 0.009 | 0.000 | 0.004 | 0.004
Mean Rank | 1.670 | 1.330 | 1.110 | 1.220 | 0.650 | 0.650 | 0.590 | 0.590
|  |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2 |  $\bullet$ R3M |  $\bullet$ VC-1 |  $\bullet$ MultiMAE |  $\bullet$ PonderV2
open drawer | 0.040 | 0.000 | 0.000 | 0.200 | 0.040 | 0.000 | 0.000 | 0.000
sweep to dustpan_of_size | 0.440 | 0.240 | 0.200 | 0.080 | 0.000 | 0.000 | 0.000 | 0.000
meat off grill | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
turn tap | 0.040 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.120
reach and drag | 0.040 | 0.000 | 0.000 | 0.040 | 0.040 | 0.000 | 0.000 | 0.000
put money in safe | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
push buttons | 0.040 | 0.120 | 0.080 | 0.040 | 0.000 | 0.000 | 0.000 | 0.000
close jar | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
place wine at rack_location | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
Mean Success rate | 0.067 | 0.040 | 0.031 | 0.040 | 0.009 | 0.000 | 0.000 | 0.013
Mean Rank | 0.154 | 0.118 | 0.097 | 0.096 | 1.110 | 1.330 | 1.330 | 1.220