# NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using
Neural Radiance Fields

Arunkumar Byravan1, Jan Humplik1, Leonard Hasenclever1, Arthur Brussee1,
Francesco Nori,
Tuomas Haarnoja, Ben Moran, Steven Bohez, Fereshteh Sadeghi, Bojan Vujatovic
and Nicolas Heess
DeepMind, 1Equal Contributions

###### Abstract

We present a system for applying sim2real approaches to “in the wild” scenes
with realistic visuals, and to policies which rely on active perception using
RGB cameras. Given a short video of a static scene collected using a generic
phone, we learn the scene’s contact geometry and a function for novel view
synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering
of the static scene by overlaying the rendering of other dynamic objects (e.g.
the robot’s own body, a ball). A simulation is then created using the
rendering engine in a physics simulator which computes contact dynamics from
the static scene geometry (estimated from the NeRF volume density) and the
dynamic objects’ geometry and physical properties (assumed known). We
demonstrate that we can use this simulation to learn vision-based whole body
navigation and ball pushing policies for a 20 degrees of freedom humanoid
robot with an actuated head-mounted RGB camera, and we successfully transfer
these policies to a real robot. Project video is available at
https://sites.google.com/view/nerf2real/home.

##  I Introduction

Thanks to progress in large-scale deep reinforcement learning and scalable
simulation infrastructure, training control policies in simulation and
transferring them to real robots (sim2real) has become a popular paradigm in
robotics [1, 2, 3, 4]. This approach avoids many of the issues such as state
estimation, safety, and data efficiency which make it challenging to learn
directly on hardware. However, creating accurate and realistic simulations is
time consuming. Therefore, for sim2real to live up to its full potential, we
must make it easier to recreate real scenes in simulation while accurately
modelling how robots sense and interact with the world.

Reducing the gap between simulation and the real world often involves the
collection of small amounts of data followed by manual tuning, the use of
established system identification tools, or more recently by learning neural
network models of parts of the system, e.g. [1]. It is especially difficult to
accurately model the geometry and visual appearance of unstructured scenes
which affect how the robot makes contact with the world and how it senses its
surroundings e.g. when using a RGB camera. The need for modeling RGB cameras
can partially be alleviated by using depth sensors or LiDARs which are easier
to simulate and thus have a smaller sim2real gap, but such a compromise can
restrict the set of tasks a robot can learn. Existing approaches to
photorealistic scene reconstruction and rendering, e.g. those used for the
creation of the datasets in [5, 6, 7, 8], work poorly in outdoor scenes and
use specialized 3D scanning setups which are not widely available, hence
limiting their applicability.

In this paper we begin to address some of these challenges, and describe a
system for the semi-automated generation of simulation models for visually
complex scenes with highly realistic rendering of RGB camera views and
accurate geometry, primarily using videos from commodity mobile cameras. To
this end, we take advantage of recent advances in neural scene representations
using Neural Radiance Fields (NeRF) [9, 10]. NeRFs are a fast developing class
of scene representations that allow synthesizing novel photorealistic views
from a sparse set of input views. Unlike prior work, NeRFs can be learned
directly from videos or photographs from commodity mobile devices and admit
access not just to a rendering function but also to the underlying scene
geometry. They can be trained within minutes [11, 12], work in both indoor and
outdoor settings and scale well even to large scenes such as city blocks [13].
Together with extensions to handle dynamic scenes [14], deformable objects
[15], and scene decompositions with novel re-combinations of objects [16],
NeRFs can enable a general system for recreating the visuals of real-word
scenes in simulation.

Our primary contribution is an approach for combining NeRF scene
representations, specifically the rendering and static geometry, learned from
short (5-6) minute videos of a scene, with a physics simulation of dynamic
objects such as a robot and a ball whose physical and visual properties are
assumed known (see Fig. 1). We present a semi-automated pipeline for setting
up these simulations and demonstrate that they have high enough fidelity to
enable simulation-to-reality transfer of vision-guided control policies.
Specifically, we use a physically accurate simulation of a 20 degree-of-
freedom Robotis OP3 humanoid robot together with the NeRF and end-to-end deep
reinforcement learning to train vision-based whole-body navigation and ball
pushing policies and we show a strong alignment between the performance of
these policies in simulation and when transferred zero-shot to real robot (see
Fig. LABEL:fig:results_fig for a visualization of our results).

Figure 1: Overview of our system for recreating a scene in a simulator. A. We
collect a video of the scene using a generic phone. B. We use structure-from-
motion software to label a subset of the video with camera poses. C. We train
a NeRF on these labeled images. D. We render the scene from novel views using
the calibrated intrinsics of the robot’s head-mounted camera. E. We use the
same NeRF to extract the scene geometry as a mesh. We coarsen the mesh and
replace the floor with a flat primitive. F. We combine the simplified mesh
with a model of a robot, and any other dynamic objects, in a physics
simulator. See Fig. 2 for further details on this step.

##  II Related work

###  II-A Neural Radiance Fields

Neural Radiance Fields (NeRF) [9] have recently become popular as an implicit
scene representation capable of synthesizing novel photorealistic views. NeRF
and its variants can represent accurate scene geometry [17], capture large
scenes [13] and can be trained from photo collections without the need for
localization or specialized hardware [18, 19, 20]. Recent work has also shown
training and rendering can be extremely fast and efficient [12, 11], or even
be real-time on commodity handheld devices [21].

NeRF in Robotics: NeRFs have been used in robotics for pose estimation [22],
representation learning [23], grasping [24] and dynamics model learning [25].
There is also closely related work on obstacle avoidance within simulated NeRF
environments leveraging a traditional state estimation and planning pipeline
[26], but transfer to the real-world was not explored. Unlike this work we use
reinforcement learning to train a policy that tightly integrates perception
and control for a bipedal robot and demonstrate transfer to the real world on
both visual navigation and object interaction tasks.

###  II-B Visual Navigation with (Visually) Realistic Simulators

There is a long line of work on modeling real-world indoor scenes including
datasets such as Matterport3D [5], Gibson [6], Replica [7] and Habitat-
Matterport3D [27]. Unlike these datasets, which were predominantly created
using purpose-built scanning setups, often with access to depth or LIDAR, we
use a small amount of video data from off-the-shelf mobile cameras to train a
NeRF to represent our scene.

Visual Navigation in simulation: Several simulation suites have been proposed
for embodied visual navigation tasks combining 3D simulators with the
different 3D scene datasets mentioned previously, such as Habitat [28, 8],
iGibson [29] and AI2/ROBO-THOR [30, 31]. These simulators have been used for
learning visual navigation policies [32, 33, 34], solve object-based
navigation [35, 36], also incorporating language commands [37]. These
approaches primarily consider dynamically simple platforms (wheeled robots)
and operate purely in simulation.

Sim2Real for Visual Navigation: Several works have demonstrated that policies
trained in these photorealistic simulators can be transferred to real-world
robots. The majority focuses on wheeled-base robots [4, 38], but some recent
work has extended this to quadrupeds [39, 40]. These approaches use RGB-D
sensors and/or LiDAR, assume access to localization, and in case of the
quadrupeds, work on top of existing low-level controllers; all these
assumptions help reduce the sim2real gap and thereby result in good transfer
performance. In contrast, we successfully transfer whole-body vision-based
control policies for a bipedal robot using only an RGB camera (which has a
high sim2real gap) without access to localization or low-level controllers.

###  II-C Sim2Real in Robotics

Sim2real transfer has made it possible to use reinforcement learning to solve
several challenging real-world control problems. Careful system identification
and techniques such as Domain Randomization [41], Domain Adaptation [42] and
Real2sim [43] have helped to reduce the discrepancies between simulation and
reality for the system dynamics and sensor model, enabling successes on tasks
such as Rubik’s cube solving with a dexterous hand [3], grasping [44],
stacking [45], autonomous flight [46], quadruped [47, 48, 49, 50] and biped
locomotion [51, 52, 53]. We rely on many of the lessons learned in these
works, and propose a system for high-fidelity replication of real-world scenes
in simulation with which we demonstrate successful zero-shot transfer of
complex vision based policies on a 20 DoF humanoid robot.

##  III Integrating NeRF with a physics simulator

Fig. 1 presents an overview of our approach for recreating a static scene in
simulation, and its extension to scenes with simple dynamic objects. Our
approach consists of 6 steps: video recording, localization, NeRF training,
post-processing to extract a rendering function and a collision mesh, and
combining these with a physics simulator to create the simulation (see Fig 2).
We describe each step below.

###  III-A Capturing a video of the real world scene

We capture a short $\sim 5-6$ minute video of the scene using an off-the-shelf
mobile camera (Google Pixel 6’s rear camera in this work). A human operator
walks around the scene and captures it from different viewpoints while
ensuring that the camera moves slowly and evenly to reduce motion blur and
minimize drastic viewpoint changes. For consistent lighting we set the white
balance and brightness to a fixed (arbitrary) value. We found that high-
resolution ($\geq$1080p) videos led to better localization and improved NeRF
results.

###  III-B Localization

Next, we extract $N\sim 1000$ keyframes from the captured video. We use COLMAP
[54, 55, 56], an open-source Structure-from-Motion (SfM) package, to estimate
the intrinsics of the camera, and extrinsics for each keyframe (see VI-A for
details).

###  III-C NeRF training

Given a dataset of images and corresponding camera poses, we train a NeRF [9]
to render the scene from novel viewpoints. We use recent NeRF extensions for
better reconstructions, improved reconstructed geometry, and decreased
rendering times. To avoid artifacts while rendering at low resolutions, we
sample the average of the volume over a normal distribution [57]. We use a
space squashing formulation to support large capture areas, as well as a
separate ’proposal’ network, and a ’distortion’ loss that encourages compact
representations [10]. To improve the reconstructed geometry we optimise a
separate specular and diffuse color [17].

NeRF rendering can be compute intensive even at low resolutions. While this is
not critical in our context as we use the NeRF only for offline learning in
simulation, to allow for faster experiment turnaround we implement a multi
scale spatial hash grid approach [11]. This provides a significant speedup
(order of magnitude), enabling rendering one frame in 6ms on a V100 GPU. We
use a similar architecture as described in [11], adding a layer normalization
[58] before the final MLP layer, and use swish activations [59] rather than
ReLU activations. Additionally, we adapted this approach to allow sampling the
radiance volume over a distribution. We blur training samples with a Gaussian
blur with a random variance $\sigma_{blur}\in[\sigma_{min},\sigma_{max}]$, and
provide $\Sigma=\Sigma_{sample}*(1+(\sigma_{blur}-\sigma_{min}))$ as an extra
input to the final MLP [60]. This augmentation allows the network to
interpolate samples in scale-space and improves our reconstruction
significantly at lower resolutions ($\sim 31.5$ vs $\sim 35.4$ average PSNR in
a few held out images). NeRF hyperparameters are listed in Sec. III.

###  III-D Rendering in Simulation

The trained NeRF can be used for rendering the scene from novel viewpoints and
camera intrinsics. In particular, we will use it to model the robot’s camera
(Logitech C920) which is different from the one used to collect images for
training. We match camera intrinsics between sim and real by calibrating the
robot’s camera, and use the obtained focal length and distortion parameters to
render the NeRF.

###  III-E Collision mesh extraction

The NeRF learns a function to predict the radiance and occupancy in space,
i.e. the underlying scene geometry. We voxelize the predicted occupancy and
compute a mesh via the marching cubes algorithm [61]; this mesh is used for
collisions within our simulation.

The camera poses obtained from COLMAP, and hence also the collision mesh
vertices, are expressed in an arbitrary reference frame (including an
arbitrary scale). Therefore, we need to estimate a rigid transformation and
scale between this frame of reference and the simulator’s world frame. We do
this by solving a least-squares optimization that constrains the normal vector
to the dominant floor plane in the mesh to be aligned with the z-axis in the
simulator. We use Blender [62] to manually select points on the mesh’s floor
for this purpose. We then manually rotate the mesh around the z-axis to a
desired alignment with the simulator’s world frame and compute the relative
scale between the NeRF and the world by comparing the size of an object within
the mesh and the real world. We also replace the floor vertices in the mesh
(which can have artifacts due to a lack of texture) with a flat plane. Lastly,
for faster collision computation, we crop the mesh to the extents needed for
simulation. See Table IV for details.

###  III-F Physics simulation

We use MuJoCo [63] as our physics simulator. The simulated scene consists of
the mesh extracted from NeRF attached to the world frame as a fixed object
which can collide with the robot body or other simulated dynamic objects such
as a ball. Physical and visual properties of these additional virtual objects
are assumed known.

Figure 2: Our MuJoCo simulation is created by combining: (1) the learnt static
scene mesh (Section III-E), (2) the dynamic object meshes and (3) the learnt
static scene NeRF rendering (Section III-D) on which (4) the Mujoco rendering
of dynamic objects (a ball and robot’s left arm in the camera image above) are
overlaid. Other dynamic parameters (e.g. friction) are assumed known or
measured.

Realistic rendering of such composite scenes is an active area of research
[64]. We opt for a straightforward approach which is suitable for our tasks.
We assume that the dynamic objects (all rendered with the MuJoCo built-in
renderer) are always in the foreground of the static scene (rendered with
NeRF); note that this doesn’t handle occlusions. Under this assumption the
combined rendering is obtained by overlaying the dynamic objects rendering on
top of the NeRF rendering (see Fig. 2 for a visualization).

##  IV Sim2real training setup with NeRF + MuJoCo

Once the combined MuJoCo simulation is set up we can train a policy purely in
this simulation and deploy it directly in the real-world. We describe our
training setup below.

###  IV-A Humanoid Robot platform

We use a Robotis OP3 [65] robot for all our experiments. This low cost
platform is a small humanoid (about 35 $\mathrm{c}\mathrm{m}$ tall, 3.5
$\mathrm{k}\mathrm{g}$ in weight) with 20 actuated degrees of freedom (see
Fig. LABEL:fig:results_fig). Actuators, and hence our learned policies, are
operated in a position control mode with both D and I gains set to 0. Our
policies run at 40Hz and rely on the robot’s on-board computer (2-core Intel
NUC i3) and on-board sensors only. These include joint encoders, gyroscope,
accelerometer, and a Logitech C920 camera attached to the robot’s head which
is actuated via two joints attached to its torso. The gyroscope and
accelerometer data are filtered at 125Hz to obtain an estimate of the gravity
direction in the robot’s body frame using a Madgwick filter [66]. To encourage
smoother movements, we apply an exponential filter with strength 0.8 to the
control signals before passing them to the actuators.

###  IV-B Reducing the dynamics sim2real gap

We ensured that the robot’s sensors and actuators are modeled accurately in
simulation. Specifically, we ensured that the simulated gyroscope and
accelerometer data are low-pass filtered in the same way in simulation as on
the real IMU chip. With these accurate models we found that policies trained
in simulation transferred well to the real robot; hence we used only limited
domain randomization on top of these models. Particularly, we applied random
pushes to the robot during training. We also applied constant delays per
episode, sampled uniformly in the range of 10$\mathrm{m}\mathrm{s}$ \-
50$\mathrm{m}\mathrm{s}$ as well as a 5 $\mathrm{m}\mathrm{s}$ jitter to all
simulated sensor data to reflect various latencies on the robot. At the
beginning of each episode, we attach a random mass (up to 0.5
$\mathrm{k}\mathrm{g}$) to a random position on the robot’s torso and
randomize the IMU’s position on the torso (we shift it by up to 0.5
$\mathrm{c}\mathrm{m}$, and tilt it by up to 2 degrees). In tasks with a ball,
we additionally randomize the ball’s mass (0.5 - 0.9$\mathrm{k}\mathrm{g}$)
and radius (11.5 - 12.5$\mathrm{c}\mathrm{m}$) at the start of each episode
(the real ball weighs 0.651$\mathrm{k}\mathrm{g}$ with a radius of
12$\mathrm{c}\mathrm{m}$).

###  IV-C Regularizing policy learning for better sim2real transfer

Carefully choosing rewards for regularizing the robot’s behavior is important
for successful transfer. In all of our tasks, we use the following reward
components as a regularization: 1. a constant penalty whenever the robot’s yaw
angular speed is larger than $\pi$
$\mathrm{r}\mathrm{a}\mathrm{d}\,\mathrm{s}^{-1}$ to encourage the robot to
turn slowly; 2. L2 regularization on joint angles towards a default standing
pose; and, 3. a walking reward encouraging the average of feet velocities in
the robot’s forward direction to be 0.3 $\mathrm{m}\,\mathrm{s}^{-1}$. These
rewards encourage the agent to learn gaits that transfer better, and also
encourage better exploration for faster learning. See Section VI-D2 for an
exact specification of these rewards.

###  IV-D Tasks

To demonstrate that our approach can scale to realistic scenes with complex
geometries and supports simple object interactions we choose two tasks:

Navigation and obstacle avoidance:  We demonstrate our approach on a point to
point visual navigation task where the robot has to reach multiple goals
(specified as (x,y) co-ordinates) while avoiding different obstacles such as a
large plant, a chair, and walls; see Fig. LABEL:fig:results_fig (left) and
Fig. 4 (bottom) for a visualization of our scene which measures 5m x 4m.

We chose three targets in different parts of the space that the robot has to
reach. We automatically compute the free areas of the scene using the NeRF’s
mesh and, during simulation, we randomly initialize the robot to a position
and orientation within these areas.

The reward for training consists of the regularization terms described in
Section IV-E, and two task-specific terms: 1. a sparse bonus upon reaching the
goal location; 2. a walking reward like the one we use as a regularization but
instead encouraging moving in the direction of the goal at a speed of 0.3
$\mathrm{m}\,\mathrm{s}^{-1}$ (see Section VI-D2). Episodes terminate whenever
the robot’s body parts other than the feet touch the scene’s mesh. We consider
an episode to be successful if the robot gets to $\leq$25cm of the target
without falling and does not collide with any obstacles.

Ball pushing:  As a proof of concept that we can combine static NeRF scenes
with dynamic interacting objects, we consider a task in which the robot has to
move a basketball to a corner of a 3m x 3m workspace (see Fig.
LABEL:fig:results_fig, right). We model the basketball as a simple orange ball
ignoring the fine black print which is barely visible at the resolutions we
use (see Fig. 2 for an example simulated image). During training in
simulation, each episode starts with the ball and robot randomly positioned.
In half of all episodes, we initialize the ball just in front of the robot to
speed up learning.

We again use the regularization terms in Section IV-E and two task-specific
terms: 1. a reward for minimizing the distance between the ball and the goal
region; and, 2. a reward for minimizing the distance between the robot and the
ball if the ball is not moving towards the goal (see Section VI-D2). Episodes
are terminated whenever the robot falls. We consider an episode as successful
if the robot gets the ball to the correct 1m x 1m corner square within 60
seconds. This task is much more challenging than navigation due to significant
partial observability & interactions; the robot has to search for the ball,
localize itself and the ball, and move it to the goal.

Figure 3: The policy’s network architecture.

###  IV-E Policy training

All our policies are trained using DMPO [67], a state-of-the-art algorithm
which combines distributional deep reinforcement learning [68] and MPO [69,
70]. Our policies take vision and proprioception as input; the policy network
(see Fig. 3) consists of a recurrent image encoder (to handle partial
observability) which passes the RGB camera images (30x40 or 60x80 resolution)
through a small ResNet followed by a LSTM. The encoded images are then
combined with a history of past 5 proprioceptive observations (gyroscope,
accelerometer, gravity direction estimate, joint positions, and previous
control signal) and passed through an MLP which outputs a diagonal Gaussian
for sampling actions. Hyperparameters used for training are listed in Section
VI-D3.

We use an asymmetric actor-critic setup for training in simulation where the
critic, a separate neural network that is not evaluated on the robot, receives
privileged information. Specifically, the critic shares the same network
structure as the actor but we replace the image encoder with the simulation’s
ground truth state (robot/object poses and velocities). This step is crucial
for efficient learning in simulation.

Image augmentations:  While the NeRF significantly reduces the sim2real gap
with realistic scene renderings, we cannot easily modulate image intensity
properties such as brightness or gain (see Fig. LABEL:fig:results_fig (left)
for comparison of real vs rendered images). Thus, we apply image augmentations
during training: we randomize the brightness, saturation, hue, and contrast,
and apply random translations to the image, see Section VI-E for details.

Figure 4: Localization performance of the agent. Top row: Robot camera images
from a successful zero-shot transfer trial. The target is next to the potted
plant. Bottom row: Visualization of the robot’s belief over it’s 2D position
in the scene, shown as a heatmap on a top-down view of the scene. White X:
Ground truth position from motion capture (not input to policy); Green
Triangle: Target position.

##  V Experimental results

We now present our results. We highly encourage the reader to watch the
accompanying video at https://sites.google.com/view/nerf2real/home.

###  V-A Training time

We train policies in simulation and transfer them zero-shot to the robot for
evaluation. Given $\sim$1000 frames from a 5-6 minute video, COLMAP
localization takes about 3-4 hours. Training the NeRF on this dataset takes
about 20 minutes on a cluster of 8 V100s, though this could be significantly
sped up further [11]. Mesh extraction and post-processing for setting up the
simulation takes a few hours, and finally training the policies takes around
24 hours (about 8M gradient updates, and 128M environment steps) for
navigation and twice as long for ball pushing.

###  V-B Navigation and obstacle avoidance results

Evaluation setup & metrics: We compare the performance of learned goal-
conditioned policies in simulation to zero-shot transfer performance in real.
We use the following evaluation protocol: for each of the 3 goals we chose 3
unique initial positions and 3 orientations, forming a total of 27
combinations. We perform two trials for each combination, for a total of 54
real-world episodes. We consider an episode to be successful if the robot
reaches the goal ($\leq$25 cm distance) without falling. We report the overall
success percentage, i.e. the fraction of episodes the policy was successful,
and the median time taken to reach the target. As the evaluation space is
equipped with a motion capture system we can use this to compute the ground-
truth position of the robot–this ground-truth information is only used for
analysis and evaluation, not as an input to the policy.

As an analysis tool, our policies are trained together with an auxiliary
prediction MLP which predicts the robot’s belief about its 2D position and yaw
from the policy’s recurrent image embedding (we do not propagate any gradients
to the policy’s parameters). The belief is modelled as a mixture of five
Gaussians and we use it to help us disambiguate between policy failures due to
confusing visual inputs and those due to other factors (e.g. the robot
tripping). We use the difference between the mean of the belief distribution
and the ground-truth position when quantifying the policy’s localization error
which is averaged over the entire episode.

| Simulation results | Zero-shot transfer results
---|---|---
Policy resolution | Success | Time taken | Localization error | Success | Time taken | Localization error
30 x 40 | 73$\pm$3% | 10.3$\pm$0.3 sec | 0.17$\pm$0.01 m | 69$\pm$6% | 10.3$\pm$0.4 sec | 0.24$\pm$0.01 m
60 x 80 | 86$\pm$2% | 10.8$\pm$0.1 sec | 0.19$\pm$0.004 m | 87$\pm$5% | 11.2$\pm$0.4 sec | 0.27$\pm$0.04 m
TABLE I: Sim & Real performance (with standard errors) of the trained policies
on the navigation and obstacle avoidance task. Time taken & Localization error
values are median statistics across all evaluation episodes.

Results: Table I presents the results of our evaluation. We evaluated policies
with two different input resolutions, 30x40 and 60x80; due to hardware
limitations on the robot (2 CPU cores, no GPU) and the need to run a policy
step within 25ms, we were unable to evaluate higher resolutions.

We draw attention to a few interesting trends: 1) Our policies do not exhibit
a significant gap between performance in simulation and on the real robot. On
the real robot, the 60x80 policy successfully reached the goal in 47/54
episodes (87$\pm$5%). The lower resolution 30x40 policy performs slightly
worse and was successful in 37/54 episodes (69$\pm$6%). This is remarkably
similar to the performance of these policies in simulation. Based on
monitoring the policy’s belief about its pose, we estimate that about half of
the failures of the 40x30 policy were due to collisions with obstacles and/or
the robot falling down, and the rest due to localization failures, but it is
impossible to perfectly disambiguate failure modes. 2) The policies take
similar amounts of time to reach the goal in simulation and on the robot
demonstrating that dynamical properties of the behavior such as the gait
velocity also do not suffer from a sim2real gap. 3) As expected, our policies
learn representations which are informative about the robot’s current pose;
these transfer well to the real-world. The median localization error across
all control steps and trials is $\sim$0.25m. We saw that several failures of
the policy correlated with higher localization error, specifically on a single
target near the potted plant. This was particularly evident for the 40x30
policy (0.33m for failures vs 0.23m for successes). We show a visualization of
the belief for a successful trajectory in Fig. 4, which demonstrates that the
agent is able to quickly localize itself accurately with $\sim$2 seconds of
data and rarely loses track throughout the trial. Videos of both simulated and
real-world evaluations can be found on our website.

###  V-C Ball pushing results

Evaluation setup & metrics: Similar to the navigation task, we compare the
average success of policies between simulation and real-world. We consider two
different evaluation setups in real, a Center setting where the ball is
initialized in the center cell of the workspace (Fig. LABEL:fig:results_fig,
right) and the robot is initialized in the center of any of the eight cells
near the wall with 4 different orientations per cell (32 episodes total), and
a Wall setting where the robot is initialized in the center cell facing the
target corner, and the ball is initialized near the center of one of the
remaining 7 cells (except the target cell) with two trials each (14 episodes
total). The target is always fixed to be the corner with the two cones (see
Fig. LABEL:fig:results_fig, right), and an episode is considered successful if
the robot can get the ball to the 1m x 1m target cell within 60 seconds
without falling down.

| Sim Success | Real Success
---|---|---
Policy resolution | Center | Wall | Center | Wall
30 x 40 | 99% | 100% | 78$\pm$7% | 43$\pm$14%
TABLE II: Sim & real performance (with standard errors) of trained policies on
the ball pushing task with ball initialized in the center of the arena vs in
different cells near the wall.

Results: Table II presents the evaluation results. We highlight two key
points: 1) As can be seen in the accompanying video, our policies use the
robot’s hands to move the ball, and exhibits active perception when searching
and tracking the ball. These behaviors emerge from the task requirements and
are not explicitly encouraged by the reward function; they also transfer
successfully from simulation to the real world. 2) While our policies show
good performance, the sim2real gap is larger for this contact-rich task. This
is especially true for the Wall initializations; unless the robot executes a
perfect push, the ball often gets stuck near the walls and the robot has a
hard time moving it.

Videos of all our results, and comparisons of renderings from the NeRF, COLMAP
reconstructions (which we show as a baseline) and real images can also be
found in the supplementary material and on our website.

##  VI Discussion and Limitations

We have presented a pipeline for creating simulation environments of visually
complex scenes in a way that allows training vision guided policies for
sim2real transfer. To this end we combine the scene geometry and rendering
function derived from a NeRF with a known physics model of the robot and
(optionally) additional objects.

In principle, our approach is embodiment independent and it can be automated
further in future work. For instance, new NeRF-like models such as [17, 71,
72], may improve scene geometry reconstruction thus eliminating the need for
manual postprocessing of minimally textured areas like the floor. Evaluating
the approach on contact-rich tasks such as climbing on objects will allow us
to better assess the current limitations of the approach, and may guide future
NeRF and scene modeling developments.

We have currently opted for a very simple approach to composing the rendering
of an a priori known object with the rendering of a static NeRF scene.
However, the field has been actively working on NeRF-like approaches to
photorealistic rendering of composite scenes [73], including ways for
segmenting a static scene into dynamic objects and representing each using a
separate NeRF [16]. If necessary, our pipeline can leverage any of these
improvements (and will have to for more complex dynamic scenes).

Similarly, we believe that recent work on eliminating the computationally
expensive localization step from NeRF pipelines [18, 19, 20], and speeding up
both NeRF [11] and RL training [74] will soon enable going from a video of a
scene to a trained policy within minutes or hours instead of 1-2 days,
potentially enabling running our setup online on the robot during deployment.

Impact statement: This work presents an approach to train vision guided
policies for general robotics systems. While in its current form the approach
is unlikely to enable real world applications, future research may make
possible a range of applications that can benefit humanity. We strongly oppose
any applications designed to bring harm to humans.

## Acknowledgments

We would like to thank Neil Sreendra, Marlon Gwira, Kushal Patel, Nathan
Batchelor, and Federico Casarini for maintaining and repairing robots used in
this project, Jon Scholz and Francesco Romano for reviewing the paper, and
Claudio Fantacci for helping with paper figures.

## APPENDIX

###  VI-A COLMAP details

To train the NeRF model, we need a paired set of images and camera poses. We
divide the video into $N\sim 1000$ equal partitions, and for each partition we
use a heuristic to pick the least motion blurred frame. We use the average
variance of the Laplacian of each frame to approximate how sharp a frame is.

After extracting these keyframes, we feed them into COLMAP [54, 55, 56]. We
use the sequential matcher to generate camera poses with mostly default
settings, except for using affine SIFT features, guided matching, and forcing
a single OPENCV style camera.

For the comparisons between the NeRF reconstruction and COLMAP reconstruction
shown in the accompanying video and on our website, we reconstruct a dense
mesh from the sparse results using the Poisson mesher with the default
settings.

###  VI-B NeRF implementation details

As described in Section III-C, we use a NeRF with a similar architecture to
the one used in [11]. We use ’swish’ [59] rather than ’relu’ activations
however, and add a final layernorm [58] before the last MLP layer.

As described in Section III-C, we’ve also integrated mip-NeRF style sampling
with the hashtable style NeRF by appending the diagonalised variance to the
sample position, and feeding in blurred training samples. Figure 5 compares
results with and without this method when rendering at lower resolutions.

Figure 5: Two different views of one of the captured scenes. For each view we
show an example rendered from a model trained without (left) and with (right)
our gaussian blur augmentation. We concatenate the diagonalised variance and
train with augemented samples. This allows the network to effectively
interpolate in scale space. Zooming in shows the reduced ’stair-stepping’
artifacts in the renders.

Table III lists the hyperparameters of the NeRF architecture and
hyperparameters for training the model.

NeRF model parameters |
---|---
hashtable size | $2^{20}$
hashtable levels | $12$
samples | $64\times 64$
MLP depth | $2$
MLP width | $64$
MLP width viewdir | $32$
Proposal MLP depth | $2$
Proposal MLP width | $64$
sample dilation | $0.001$
activation type | swish
density activation |  squareplus [75]
density bias | $-5$
NeRF training parameters |
batch size | 1$6384$
weight decay | $5\text{\times}{10}^{-5}$
optimizer | adamw
lr init | $2\text{\times}{10}^{-3}$
lr final | $4\text{\times}{10}^{-5}$
warmup steps | $2048$
max steps | $40000$
charb padding | $0.001$
distortion loss mult | $0.01$
blur $\sigma_{max}$ | $12.0$
TABLE III: List of NeRF hyperparameters.

###  VI-C Mesh processing details

Table IV visualizes the different stages of the mesh processing needed to go
from the raw mesh extracted from NeRF’s occupancy network to a simplified mesh
suitable for MuJoCo simulation.

|  |
---|---|---
|  |
TABLE IV: Mesh preprocessing. Two examples of how the NeRF meshes are
processed to make them suitable for simulation. The leftmost column shows the
meshes of two different static environments, obtained by running marching
cubes on a discretized occupancy grid from the trained NeRF and rotating,
translating and cropping the resulting mesh to the extents of the scene. The
central column shows the same meshes with the floor removed (as described in
Section III-E). Note that for the raw mesh in the bottom left, a partial floor
removal was done by choosing a high threshold for the occupancy within the
marching cubes procedure. In the rightmost column the mesh is decomposed into
convex sub-components (each sub-components in a different colour), which are
passed to MuJoCo for collision detection. The convex decomposition step is
specific to MuJoCo as it does not handle collisions between non-convex
objects.

###  VI-D Policy training details

####  VI-D1 Control limits

We clipped the inputs to actuator position controllers to be within the limits
specified in Table V. For the ball pushing task, we further extended the
limits of the head_pan joint to be within (-2.5, 2.5) so that the policy can
learn to actively look at the ball. In both tasks we found it important to
limit the upper bound of the head_tilt joint to prevent the policy from
looking at the ceiling which is not well captured by the NeRF due to a lack of
data in the input video.

joint | reference pose [rad] | range [rad]
---|---|---
head pan | $0.043$ | (-0.79, 0.79)
head tilt | $-0.47$ | (-0.79, -0.157)
left ankle pitch | $0.638$ | (-0.25, 1.109)
left ankle roll | $-0.0414$ | (-0.4, 0.4)
left elbow | $-0.7915$ | (-0.8, 0.2)
left hip pitch | $-0.5553$ | (-0.45, 1.57)
left hip roll | $-0.0383$ | (-0.4, -0.1)
left hip yaw | $0$ | (-0.3, 0.3)
left knee | $1.0646$ | (-0.2, 2)
left shoulder pitch | $-0.0874$ | (-0.785, 0.785)
left shoulder roll | $0.7915$ | (0, 0.8)
right ankle pitch | $-0.638$ | (-1.109, 0.25)
right ankle roll | $0.0414$ | (-0.4, 0.4)
right elbow | $0.7915$ | (-0.2, 0.8)
right hip pitch | $0.5553$ | (-1.57, 0.45)
right hip roll | $0.0383$ | (0.1, 0.4)
right hip yaw | $0$ | (-0.3, 0.3)
right knee | $-1.0646$ | (-2, 0.2)
right shoulder pitch | $0.0874$ | (-0.785, 0.785)
right shoulder roll | $-0.7915$ | (-0.8, 0)
TABLE V: Joint reference poses and limits.

####  VI-D2 Rewards

We used the following regularization rewards (from Section IV-C) to shape the
robot’s behavior and improve sim2real transfer:

$r_{\text{turn}}=-1.\ \text{if}\ \omega_{yaw}>\pi\ \text{else}\ 0.,$
$r_{\text{pose}}=\sqrt{\frac{1}{20}\sum_{i=1}^{20}\frac{(\mathbf{q}_{i}-\mathbf{ref}_{i})^{2}}{\mathbf{range}_{i}^{2}}},$
$r_{\text{speed}}=\frac{0.3-|\mathbf{v}_{x}^{\text{feet}}-0.3|}{0.3},$

where $\omega_{yaw}$ is the angular velocity of the robot’s gravity-aligned
body frame. The gravity-aligned body frame is obtained from the robot’s torso
frame by rotating it using the smallest rotation which aligns robot’s frame
negative z-axis with gravity direction. $\mathbf{q}$ are the robot’s joint
positions, $\mathbf{ref}$ are the corresponding reference poses, and
$\mathbf{range}$ are the joint ranges (see Table V for specific values).
$\mathbf{v}^{\text{feet}}$ is the velocity of the midpoint of the robot’s feet
expressed in the gravity-aligned body frame.

The task-specific reward components we used for the navigation task (Section
IV-D) are

$r_{\text{navigate sparse}}=1.\ \text{if}\ ||\mathbf{x}_{\text{goal}}||<0.25\
\text{else}\ 0.,$
$r_{\text{navigate}}=\left(0.3-\left\lVert\mathbf{v}_{x,y}^{\text{feet}}-0.3\frac{\mathbf{x}_{\text{goal}}}{||\mathbf{x}_{\text{goal}}||}\right\rVert\right)/0.3,$

where $\mathbf{x}_{\text{goal}}$ is the 2d goal position in the robot’s
gravity-aligned frame. The full reward we used for training the navigation
policies is

$r_{\text{navigation
task}}=r_{\text{turn}}+0.5r_{\text{speed}}+0.5r_{\text{pose}}+\\\
r_{\text{navigate sparse}}+0.25r_{\text{navigate}}.$

The task-specific reward for the ball pushing task (Section IV-D) works in two
stages. If the ball is near the goal or is moving towards the goal, then we do
not encourage any robot behavior. If this is not the case, then we encourage
the robot to move towards the ball if it is away from it, otherwise we
encourage it to stay close to the ball. Specifically, let
$d_{ball}=||\mathbf{x}_{\text{robot}}-\mathbf{x}_{\text{ball}}||$,
$d_{goal}=||\mathbf{x}_{\text{ball}}-\mathbf{x}_{\text{goal}}||$, and
$v_{ball}$, $v_{goal}$ the time derivatives of these distances. Let

$\rho(d,v)=e^{-d^{2}-v^{2}}+\left(1-e^{-d^{2}}\right)\left(1-e^{-\min(0,v)^{2}}\right).$

The task specific reward is

$r_{\text{ball}}=\rho(d_{goal},v_{goal})+\left(1-\rho(d_{goal},v_{goal})\right)\rho(d_{ball},v_{ball})/2,$

and the full reward for training ball pushing policies is

$r_{\text{ball pushing
task}}=r_{\text{turn}}+0.5r_{\text{speed}}+0.5r_{\text{pose}}+r_{\text{ball}}.$

####  VI-D3 DMPO

Our DMPO implementation is based on the open-source reference implementation
https://github.com/deepmind/acme/tree/master/acme/agents/tf/dmpo [67]. Readers
can find more details about this algorithm in the MPO [69, 70] and
distributional RL [68] papers, as well as in [76] which used the exact same
implementation we are using. Here we only mention the differences between ours
and the linked reference implementation:

  * 1.

We use JAX [77] instead of TensorFlow.

  * 2.

We use a distributed asynchronous setting with separate actor and learner
processes.

  * 3.

In order to support recurrent architectures, we calculate losses and gradients
over batches of multi-step trajectories instead of batches of single-step
transitions.

  * 4.

N-step returns are estimated from multi-step trajectories inside the critic
loss rather than being accumulated in the actor process.

Hyperparameters are listed in Table VI.

policy learning rate | $0.0001$
---|---
critic learning rate | $0.0001$
dual variables learning rate | $0.01$
trajectory length | $48$
batch size | $32$
updates per environment step | $1/8$
discount | $0.99$
init log temperature | $10$
init log alpha mean | $10$
init log alpha stddev | $1000$
epsilon | $0.1$
epsilon mean | $0.0025$
epsilon stddev | $1\text{e}-6$
epsilon action penalty | $0.001$
per dim KL constraints | True
out-of-bounds action penalization | True
n step | $5$
target actor update period | $25$
target critic update period | $100$
vmin | $-150$
vmax | $150$
TABLE VI: List of DMPO hyperparameters.

###  VI-E Camera calibration and image augmentations

dm_pix function | function arguments
---|---
random_brightness | max_delta=32. / 255.
random_hue | max_delta=1. / 24.
random_contrast | lower=0.5, upper=1.5
random_saturation | lower=0.5, upper=1.5
TABLE VII: Image augmentation settings.

We calibrated the robot’s Logitech C920 camera’s focal length and distortion
parameters using the camera_calibration ROS package. In order to address the
mismatch between the image intensity settings of the robot’s camera and the
camera used for data collection, we use image augmentations during policy
training. Specifically we use the random_brightness, random_hue,
random_contrast, and random_saturation functions from the dm_pix library [78]
with the parameters listed in Table VII. We also apply random translations by
up to $5\%$ of the image height/width. However, these augmentations are not a
replacement for calibration. We found the gain parameter of the camera to be
particularly important and so we manually tuned it by comparing to the images
used for training NeRF (we used gain=50 for the navigation task, and gain=128
for the ball pushing task).

The sensitivity to gain is quantified in Table VIII and the effects of varying
gain on the camera images is visualized in Fig. 6. The policy is especially
sensitive to high gain values (leading to brighter images) which cause it to
walk in a small circle. It is less sensitive to small gain values for which it
still navigates to the target but sometimes gets lost or hits obstacles.

Gain | Success rate | Behavior
---|---|---
128 | 0/5 | Robot walks in a small circle.
90 | 0/5 | Robot walks in a small circle.
50 | 4/5 | Robot consistently reaches the target.
|  | This is the default setting for our evaluation
|  | results shown in Section V-B
10 | 1/5 | Robot often reaches the target (4/5) but
|  | hits obstacles on the way.
1 | 2/5 | Robot occasionally reaches the target but
|  | often gets lost.
TABLE VIII: Performance of the 80x60 resolution navigation policy when varying
the camera’s gain parameter. Both the goal position and the robot’s initial
position were fixed in all trials. Higher gain corresponds to brighter image,
and 50 is the default value used in the navigation experiments. Figure 6: The
effect of changing the camera gain. The policy breaks when the gain is 128,
and sometimes gets lost when it is 1.

###  VI-F Ablations and other results

We qualitatively evaluated our navigation policies in perturbed settings such
as moving some of the fixed objects in the scene or having humans to walk
around the scene. These evaluation runs are shown in the accompanying videos
on our website. We were surprised that the “low-level” locomotion of the robot
was disentangled from the “high-level” scene understanding even though the
policies were trained end-to-end. For example, when we changed the scene
setup, the robot might walk in the wrong direction but it would retain a
stable gait. Similarly, when we blocked the robot’s camera, the robot would
stop walking but it would not fall (and it would start walking again once we
unblocked its view). Additionally, we often saw that the robot would reach the
successful target even in these perturbed settings.

## References

  * [1] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Sci Robot, vol. 4, Jan. 2019.
  * [2] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, p. eabk2822, 2022.
  * [3] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al., “Solving rubik’s cube with a robot hand,” arXiv preprint arXiv:1910.07113, 2019.
  * [4] P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee, “Sim-to-real transfer for vision-and-language navigation,” in Conference on Robot Learning, pp. 671–681, PMLR, 2021.
  * [5] A. Chang, A. Dai, T. Funkhouser, M. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang, “Matterport3d: Learning from rgb-d data in indoor environments,” arXiv preprint arXiv:1709.06158, 2017.
  * [6] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: Real-world perception for embodied agents,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9068–9079, 2018.
  * [7] J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, et al., “The replica dataset: A digital replica of indoor spaces,” arXiv preprint arXiv:1906.05797, 2019\.
  * [8] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, et al., “Habitat 2.0: Training home assistants to rearrange their habitat,” Advances in Neural Information Processing Systems, vol. 34, pp. 251–266, 2021.
  * [9] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” CoRR, vol. abs/2003.08934, 2020.
  * [10] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” CoRR, vol. abs/2111.12077, 2021.
  * [11] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., vol. 41, pp. 102:1–102:15, July 2022.
  * [12] C. Reiser, S. Peng, Y. Liao, and A. Geiger, “Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps,” CoRR, vol. abs/2103.13744, 2021.
  * [13] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar, “Block-nerf: Scalable large scene neural view synthesis,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8248–8258, 2022.
  * [14] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, “D-nerf: Neural radiance fields for dynamic scenes,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318–10327, 2021\.
  * [15] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla, “Nerfies: Deformable neural radiance fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5865–5874, 2021.
  * [16] K. Stelzner, K. Kersting, and A. R. Kosiorek, “Decomposing 3d scenes into objects via unsupervised volume segmentation,” arXiv preprint arXiv:2104.01148, 2021.
  * [17] D. Verbin, P. Hedman, B. Mildenhall, T. E. Zickler, J. T. Barron, and P. P. Srinivasan, “Ref-nerf: Structured view-dependent appearance for neural radiance fields,” CoRR, vol. abs/2112.03907, 2021.
  * [18] Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu, “Nerf–: Neural radiance fields without known camera parameters,” arXiv preprint arXiv:2102.07064, 2021.
  * [19] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “imap: Implicit mapping and positioning in real-time,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6229–6238, 2021.
  * [20] R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth, “Nerf in the wild: Neural radiance fields for unconstrained photo collections,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210–7219, 2021.
  * [21] Z. Chen, T. Funkhouser, P. Hedman, and A. Tagliasacchi, “Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering on mobile architectures,” arXiv preprint arXiv:2208.00277, 2022\.
  * [22] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin, “inerf: Inverting neural radiance fields for pose estimation,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1323–1330, IEEE, 2021.
  * [23] L. Yen-Chen, P. Florence, J. T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola, “Nerf-supervision: Learning dense object descriptors from neural radiance fields,” arXiv preprint arXiv:2203.01913, 2022.
  * [24] J. Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg, “Dex-nerf: Using a neural radiance field to grasp transparent objects,” arXiv preprint arXiv:2110.14217, 2021.
  * [25] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint, “Learning multi-object dynamics with compositional neural radiance fields,” arXiv preprint arXiv:2202.11855, 2022.
  * [26] M. Adamkiewicz, T. Chen, A. Caccavale, R. Gardner, P. Culbertson, J. Bohg, and M. Schwager, “Vision-only robot navigation in a neural radiance world,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4606–4613, 2022\.
  * [27] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. Turner, E. Undersander, W. Galuba, A. Westbury, A. X. Chang, et al., “Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai,” arXiv preprint arXiv:2109.08238, 2021.
  * [28] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, et al., “Habitat: A platform for embodied ai research,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9339–9347, 2019.
  * [29] B. Shen, F. Xia, C. Li, R. Martín-Martín, L. Fan, G. Wang, C. Pérez-D’Arpino, S. Buch, S. Srivastava, L. Tchapmi, et al., “igibson 1.0: a simulation environment for interactive tasks in large realistic scenes,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7520–7527, IEEE, 2021.
  * [30] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “Ai2-thor: An interactive 3d environment for visual ai,” arXiv preprint arXiv:1712.05474, 2017.
  * [31] M. Deitke, W. Han, A. Herrasti, A. Kembhavi, E. Kolve, R. Mottaghi, J. Salvador, D. Schwenk, E. VanderBilt, M. Wallingford, et al., “Robothor: An open simulation-to-real embodied ai platform,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3164–3174, 2020.
  * [32] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, et al., “On evaluation of embodied navigation agents,” arXiv preprint arXiv:1807.06757, 2018.
  * [33] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra, “Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames,” arXiv preprint arXiv:1911.00357, 2019.
  * [34] T. Chen, S. Gupta, and A. Gupta, “Learning exploration policies for navigation,” arXiv preprint arXiv:1903.01959, 2019.
  * [35] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, “Simple but effective: Clip embeddings for embodied ai,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14829–14838, 2022\.
  * [36] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans, “Objectnav revisited: On evaluation of embodied agents navigating to objects,” arXiv preprint arXiv:2006.13171, 2020.
  * [37] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel, “Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3674–3683, 2018.
  * [38] J. Truong, S. Chernova, and D. Batra, “Bi-directional domain adaptation for sim2real transfer of embodied navigation agents,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2634–2641, 2021.
  * [39] J. Truong, D. Yarats, T. Li, F. Meier, S. Chernova, D. Batra, and A. Rai, “Learning navigation skills for legged robots with learned robot embeddings,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 484–491, IEEE, 2021.
  * [40] Z. Fu, A. Kumar, A. Agarwal, H. Qi, J. Malik, and D. Pathak, “Coupling vision and proprioception for navigation of legged robots,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17273–17283, 2022.
  * [41] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23–30, IEEE, 2017.
  * [42] K. Bousmalis, N. Silberman, D. Dohan, D. Erhan, and D. Krishnan, “Unsupervised pixel-level domain adaptation with generative adversarial networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3722–3731, 2017.
  * [43] Y. Chebotar, A. Handa, V. Makoviychuk, M. Macklin, J. Issac, N. Ratliff, and D. Fox, “Closing the sim-to-real loop: Adapting simulation randomization with real world experience,” in 2019 International Conference on Robotics and Automation (ICRA), pp. 8973–8979, IEEE, 2019.
  * [44] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, et al., “Using simulation and domain adaptation to improve efficiency of deep robotic grasping,” in 2018 IEEE international conference on robotics and automation (ICRA), pp. 4243–4250, IEEE, 2018.
  * [45] A. X. Lee, C. M. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, et al., “Beyond pick-and-place: Tackling robotic stacking of diverse shapes,” in 5th Annual Conference on Robot Learning, 2021.
  * [46] F. Sadeghi and S. Levine, “Cad2rl: Real single-image flight without a single real image,” arXiv preprint arXiv:1611.04201, 2016.
  * [47] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics, vol. 4, no. 26, 2019.
  * [48] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science Robotics, vol. 5, p. eabc5986, Oct 2020.
  * [49] X. B. Peng, E. Coumans, T. Zhang, T.-W. Lee, J. Tan, and S. Levine, “Learning agile robotic locomotion skills by imitating animals,” arXiv preprint arXiv:2004.00784, 2020.
  * [50] S. Bohez, S. Tunyasuvunakool, P. Brakel, F. Sadeghi, L. Hasenclever, Y. Tassa, E. Parisotto, J. Humplik, T. Haarnoja, R. Hafner, M. Wulfmeier, M. Neunert, B. Moran, N. Siegel, A. Huber, F. Romano, N. Batchelor, F. Casarini, J. Merel, R. Hadsell, and N. Heess, “Imitate and repurpose: Learning reusable robot movement skills from human and animal behaviors,” arXiv, Mar. 2022.
  * [51] W. Yu, V. C. Kumar, G. Turk, and C. K. Liu, “Sim-to-real transfer for biped locomotion,” arXiv preprint arXiv:1903.01390, 2019.
  * [52] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath, “Reinforcement learning for robust parameterized locomotion control of bipedal robots,” arXiv preprint arXiv:2103.14295, 2021.
  * [53] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst, “Blind bipedal stair traversal via sim-to-real reinforcement learning,” arXiv preprint arXiv:2105.08328, 2021.
  * [54] J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [55] J. L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm, “Pixelwise view selection for unstructured multi-view stereo,” in European Conference on Computer Vision (ECCV), 2016.
  * [56] J. L. Schönberger, T. Price, T. Sattler, J.-M. Frahm, and M. Pollefeys, “A vote-and-verify strategy for fast spatial verification in image retrieval,” in Asian Conference on Computer Vision (ACCV), 2016.
  * [57] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan, “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields,” CoRR, vol. abs/2103.13415, 2021.
  * [58] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” 2016.
  * [59] P. Ramachandran, B. Zoph, and Q. V. Le, “Searching for activation functions,” CoRR, vol. abs/1710.05941, 2017.
  * [60] H. Baatz, J. Granskog, M. Papas, F. Rousselle, and J. Novák, “Nerf-tex: Neural reflectance field textures,” in Eurographics Symposium on Rendering, The Eurographics Association, June 2021.
  * [61] W. E. Lorensen and H. E. Cline, “Marching cubes: A high resolution 3d surface construction algorithm,” ACM siggraph computer graphics, vol. 21, no. 4, pp. 163–169, 1987.
  * [62] B. O. Community, Blender - a 3D modelling and rendering package.  Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
  * [63] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033, IEEE, 2012.
  * [64] B. Yang, Y. Zhang, Y. Xu, Y. Li, H. Zhou, H. Bao, G. Zhang, and Z. Cui, “Learning object-compositional neural radiance field for editable scene rendering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13779–13788, 2021.
  * [65] “Robotis op3.” https://emanual.robotis.com/docs/en/platform/op3/introduction/.
  * [66] S. Madgwick et al., “An efficient orientation filter for inertial and inertial/magnetic sensor arrays,” Report x-io and University of Bristol (UK), vol. 25, pp. 113–118, 2010.
  * [67] M. Hoffman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas, “Acme: A research framework for distributed reinforcement learning,” arXiv preprint arXiv:2006.00979, 2020\.
  * [68] M. G. Bellemare, W. Dabney, and R. Munos, “A distributional perspective on reinforcement learning,” in International Conference on Machine Learning, pp. 449–458, PMLR, 2017.
  * [69] A. Abdolmaleki, J. T. Springenberg, Y. Tassa, R. Munos, N. Heess, and M. Riedmiller, “Maximum a posteriori policy optimisation,” 2018.
  * [70] A. Abdolmaleki, J. T. Springenberg, J. Degrave, S. Bohez, Y. Tassa, D. Belov, N. Heess, and M. Riedmiller, “Relative entropy regularized policy iteration,” 2018.
  * [71] L. Yariv, J. Gu, Y. Kasten, and Y. Lipman, “Volume rendering of neural implicit surfaces,” CoRR, vol. abs/2106.12052, 2021.
  * [72] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang, “Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction,” CoRR, vol. abs/2106.10689, 2021.
  * [73] M. Guo, A. Fathi, J. Wu, and T. Funkhouser, “Object-centric neural scene rendering,” arXiv preprint arXiv:2012.08503, 2020.
  * [74] N. Rudin, D. Hoeller, P. Reist, and M. Hutter, “Learning to walk in minutes using massively parallel deep reinforcement learning,” in 5th Annual Conference on Robot Learning, 2021.
  * [75] J. T. Barron, “Squareplus: A softplus-like algebraic rectifier,” CoRR, vol. abs/2112.11687, 2021.
  * [76] M. Bloesch, J. Humplik, V. Patraucean, R. Hafner, T. Haarnoja, A. Byravan, N. Y. Siegel, S. Tunyasuvunakool, F. Casarini, N. Batchelor, F. Romano, S. Saliceti, M. Riedmiller, S. M. A. Eslami, and N. Heess, “Towards real robot learning in the wild: A case study in bipedal locomotion,” in 5th Annual Conference on Robot Learning, 2021.
  * [77] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, “JAX: composable transformations of Python+NumPy programs,” 2018.
  * [78] I. Babuschkin, K. Baumli, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Budden, T. Cai, A. Clark, I. Danihelka, C. Fantacci, J. Godwin, C. Jones, R. Hemsley, T. Hennigan, M. Hessel, S. Hou, S. Kapturowski, T. Keck, I. Kemaev, M. King, M. Kunesch, L. Martens, H. Merzic, V. Mikulik, T. Norman, J. Quan, G. Papamakarios, R. Ring, F. Ruiz, A. Sanchez, R. Schneider, E. Sezener, S. Spencer, S. Srinivasan, L. Wang, W. Stokowiec, and F. Viola, “The DeepMind JAX Ecosystem,” 2020.