# iGibson 1.0: A Simulation Environment for Interactive Tasks
in Large Realistic Scenes

Bokui Shen∗, Fei Xia∗, Chengshu Li∗, Roberto Martín-Martín∗, Linxi Fan,
Guanzhi Wang, Claudia Pérez-D’Arpino,
Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, Micael Tchapmi, Kent Vainio,
Josiah Wong, Li Fei-Fei, Silvio Savarese ∗Equal contribution. All authors are
with the Stanford Vision & Learning Laboratory, Stanford University.

###### Abstract

We present iGibson 1.0, a novel simulation environment to develop robotic
solutions for interactive tasks in large-scale realistic scenes. Our
environment contains 15 fully interactive home-sized scenes with 108 rooms
populated with rigid and articulated objects. The scenes are replicas of real-
world homes, with distribution and the layout of objects aligned to those of
the real world. iGibson 1.0 integrates several key features to facilitate the
study of interactive tasks: i) generation of high-quality virtual sensor
signals (RGB, depth, segmentation, LiDAR, flow and so on), ii) domain
randomization to change the materials of the objects (both visual and
physical) and/or their shapes, iii) integrated sampling-based motion planners
to generate collision-free trajectories for robot bases and arms, and iv)
intuitive human-iGibson interface that enables efficient collection of human
demonstrations. Through experiments, we show that the full interactivity of
the scenes enables agents to learn useful visual representations that
accelerate the training of downstream manipulation tasks. We also show that
iGibson features enable the generalization of navigation agents, and that the
human-iGibson interface and integrated motion planners facilitate efficient
imitation learning of human demonstrated (mobile) manipulation behaviors.
iGibson 1.0 is open-source, equipped with comprehensive examples and
documentation. For more information, visit our project website:
http://svl.stanford.edu/igibson/.

##  I Introduction

Simulation environments have proliferated over the last few years as a way to
train robots and interactive agents in a rapid and safe manner. In these
environments, agents learn to engage in physical interactions [1, 2], navigate
based on sensor signals [3, 4, 5, 6, 7], or plan long-horizon tasks [8, 9, 10,
11]. In simulation, agents learn to perform interactions that actively change
the input sensor signals and the state of the environment towards a desired
configuration, capabilities at the core of what an embodied agent needs to
achieve.

However, existing simulation environments that combine physics simulation and
robotic tasks often cater to a narrow set of tasks and include only clean,
small-scale scenes [12, 13, 14, 15, 16, 17]. The few simulation environments
that include large scenes such as homes or offices either disable the
possibility of interacting with the scene, focusing only on navigation (e.g.
Habitat [18]), or use simplified modes of interaction (e.g. AI2Thor [19],
VirtualHome [20]). These simulators do not support the development of end-to-
end sensorimotor control loops for tasks that require rich, continuous
interaction with the scene. Such tasks are difficult to accomplish in the
aforementioned simulators, and the simplified modes of interaction lead to
difficulties in transferring the learned policy onto real robots.

We present iGibson 1.0 (alternative called just iGibson in this manuscript), a
novel simulation environment that enables the development of embodied agents
for interactive tasks in large-scale realistic scenes (Fig. 1). Interactivity
is achieved by leveraging a physics engine processing all elements in the
scene, enabling manipulation of rigid and articulated objects as well as
mobility. iGibson 1.0 aims at unifying several aspects of robot simulation
that are often available in different software tools, such as physics
simulation for interaction with objects and robot control, high-quality
simulated sensors, integration with reinforcement learning frameworks, and
realistic indoor scenes that reflect the objects distribution of real homes.
This integration allows fully physics based simulation of robot tasks (i.e.
simulating the full complexity of the task) and allows developing task and
motion planning, reinforcement learning or imitation learning solutions for
those tasks with virtual sensor signals.

Figure 1:  Robot performs an interactive task in iGibson 1.0. It operates in
the kitchen of one of iGibson’s fully interactive scenes, planning an
interaction with the arm using a integrated sampling-based motion planner and
receiving first-person view. Bottom: The same scene can be randomized with
different materials and/or object models

iGibson 1.0 contains 15 fully interactive and visually-realistic scenes with a
total of 108 rooms. These scenes were generated by annotating 3D
reconstructions of real-world scans (static scenes represented by a single
mesh) and converting them into fully interactive scene models (scenes filled
with articulated object models). In this process, we respect the original
object-instance layout and object-category distribution. The object models are
extended from open-source datasets [21, 22, 15] enriched with annotations of
material and dynamic properties. iGibson’s physics-based renderer leverages
the extra information provided in the material annotation (maps of metallic,
roughness and normals) to generate high-quality virtual images. To further
facilitate the training of more robust visuomotor agents, iGibson 1.0 offers
domain randomization procedures for materials (both visual appearances and
dynamics properties) and object shapes while respecting the distribution of
object placements and preserving interactability. iGibson 1.0 is also equipped
with a graphical user interface that allows human users to easily interact
with the scenes, enabling efficient collection of human demonstrations for
imitation learning.

In summary, iGibson 1.0 provides the following novel features that facilitate
developing and training robotic solutions:

  1. 1.

Fifteen fully interactive scenes containing 108 rooms modelled after real
world homes with articulated object models annotated with materials and
dynamics properties. Additionally, we support importing CubiCasa5K [23] and
3D-Front [24] layouts, giving access to more than 12000 additional fully
interactive scenes.

  2. 2.

Realistic virtual sensor signals, including a physics-based renderer (PBR) for
RGB images, rendering of normals, depth, point clouds, virtual LiDAR signals,
and optical/scene flow. We further integrate domain randomization
functionality (visual texture, dynamics properties and object instances) that
facilitates generalization to unseen scenes.

  3. 3.

Useful tooling for developing robotics solutions in simulation, such as a
human-computer interface for humans to provide interactive demonstrations, and
sampling-based motion planners for navigation and manipulation.

We demonstrate the benefits of these novel features in a comprehensive set of
experiments in which visual agents are trained for navigation and interactive
tasks. Our experiments show that iGibson 1.0 enables researchers to 1) train
more robust and generalizable sensorimotor policies thanks to its realistic
virtual sensor signals (including LiDAR) and domain randomization mechanisms,
2) collect human demonstrations and train imitation learning policies for
manipulation and mobile manipulation tasks, and 3) learn intermediate visual
representations linked to interactability of the scene that accelerate
training of downstream manipulation tasks. iGibson 1.0 is open-source and
academically developed, and available on our website
http://svl.stanford.edu/igibson/.

##  II Related Work

TABLE I: Comparison of Simulation Environments

| iGibson 1.0 (ours) | Gibson [25] | Habitat [18] | Sapien [15] | AI2Thor [19] | VirtualH [20] | TDW [26]
---|---|---|---|---|---|---|---
| Provided Large Scenes
---
Real-World / Designed
| 15 homes
---
(108 rooms) / –
1400 / – | – | – | – / 120 rooms | – / 7 | – / 25
| Provided Objects
---
Number / Materials
570 / Yes | – / – | – / – | 2346 / No | 609 / Yes | 308 / No | 200 / Yes
| Agent/World Interaction
---
Forces, Predefined Actions
F | – | – | F | F & PA | F & PA | F
| Physics Engine
---
Bullet | Bullet | Bullet | PhysX | Unity | Unity | Unity & Flex
| Type of Simulation
---
Rigid Bodies Physics,
Rigid Bodies with PA,
Particles and Fluids
| RBP
---
| RBP
---
| RBP
---
| RBP
---
|  | RBP&RBPA
---
| RBPA
---
| RBP&PF
---
| Supported Task
---
| Nav.&Manip.
---
| Nav.
---
| Nav.
---
| Nav.&Manip.
---
|  | Nav.&Manip.
---
| Nav.&Manip.
---
| Nav.&Manip.
---
| Type of Rendering
---
PBR | IBR | PBR | PBR,RTX | PBR | PBR | PBR
Virtual Sensor Signals |  | RGB,D,N
---
SS,FL,LiDAR
| RGB,D
---
N,SS
RGB,D,SS,S | RGB,D,SS | RGB,D,SS,S |  | RGB,D
---
SS,FL
RGB,D,SS,S
| Domain Randomization
---
Scene,Object,Materials
S,O,M | - | - | - | S | S | S,O
Speed | ++ | + | +++ | ++(PBR)/-(RTX) | + | + | +
Human Interface |  | Mouse
---
Keyboard
- |  | Mouse
---
Keyboard
- |  | Mouse
---
Keyboard
| Natural
---
Language
| Virtual
---
Reality
|  Integrated Motion Planner
---
Yes | No | No | No | No | No | No
gray!0 Specialty |  | Phys. Int. in
---
Large Scenes
| Nav.
---
| Fast,
---
Nav.
| Articulation,
---
Ray Tracing
| Object States,
---
Task Planning
| Object States,
---
Task Planning
| Audio,
---
Fluids
gray!0 Type of rendering: PBR:Physics-Based Rendering, IBR:Image-Based
Rendering, RTX:Ray Tracing
gray!0 Virtual sensor signals: RGB: Color Images, D:Depth, N:Normals,
SS:Semantic Segmentation, LiDAR:Lidar, FL:Flow (optical and/or scene), S:
Sounds

The use of simulation in robotics has significantly increased in recent years
and the research community has proposed a number of simulators for robotics
and embodied AI. Here, we use the terms physics simulator and simulation
environment as follows. A physics simulator is an engine capable of computing
the physical effect of actions on an environment (e.g. motion of bodies when a
force is applied, or flow of liquid particles when being poured) [27, 28, 29,
30, 31, 32]. On the other hand, a simulation environment is a framework that
includes a physics simulator, a renderer of virtual signals, and a set of
assets (i.e. models of scenes, objects, robots) ready to be used to study and
develop solutions for different tasks. Both components are crucial for
advancing embodied AI and robotics. Here, we focus on the discussion of
simulation environments.

Several simulation environments have been proposed recently to study
manipulation with stationary arms [13, 12, 17, 33, 16]. Most of them are based
on Bullet [27] or MuJoCo [28] for physics simulation, and render images with
the default renderer or a Unity [30] plugin. Different from these simulation
environments, iGibson focuses on large-scale (house-size) scenes and includes
fifteen fully interactive scenes with 108 rooms – such as kitchens and
bedrooms – where researchers can develop solutions for navigation,
manipulation and mobile manipulation.

Closer to iGibson are simulation environments that include large-scale
realistic scenes (e.g. homes or offices), which we summarize and compare in
Table II. Gibson [25] (now Gibson v1) was the precursor of iGibson. It
includes over 1400 3D-reconstructed floors of homes and offices with real-
world object distribution and layout. Although Gibson incorporates PyBullet as
its physics engine for simulating robot navigation, each scene is one single
fully rigid object (static mesh). Thus, it does not allow agents to interact
with the scenes other than collisions with the mesh, restricting its use to
only navigation. A similar environment is Habitat [18]. Despite its high
rendering speed, Habitat uses the non-interactive assets from Gibson v1 [25]
and Matterport [34] and therefore only supports navigation tasks and
simplified rearrangement of added objects. Recent work [35] introduced an
extension to the Gibson v1 environment to support Interactive Navigation [35]
where parts of the reconstructions corresponding to five object classes
(chairs, tables, desks, sofas, and doors) in several Gibson static models were
segmented and replaced with interactive versions. This enabled navigation
agents to interact with the scene, and thus allowed for the first benchmark
for Interactive Navigation. However, the simulators above are not interactive
[25, 18] or partially interactive [35]. iGibson encapsulates the
functionalities in all previous versions of Gibson with backwards
compatibility for the static environments.

A variety of simulation environments have been proposed recently for scene-
level interactive tasks, such as Sapien [15], AI2Thor [19], VirtualHome [20],
and ThreeDWorld (TDW [26]). These simulators adopt different ways of agent-
world interactions. Predefined Actions (PA) consist in the set of actions that
can be performed for each object type. When the agent is close enough to an
object and the object is in the right state (precondition), the agent can
select a predefined action, and the object is “transitioned” to the next state
(postcondition). In Table II we refer to this technique as Rigid Bodies with
Predefined Actions (RBPA). It is possible to combine Rigid Bodies with
Predefined Actions (RBPA) and Rigid Body Physics (RBP), such as first using
RBPA to grasp an object and then using RBP after releasing it. AI2Thor and
VirtualHome use predefined actions (PA) as an abstraction of physical
interactions and allow agents to modify the object states instantaneously
(e.g. open a closed cabinet), suitable to study high-level task planning.
While the availability of PA helps focusing on symbolic reasoning, the full
robotics problems require RBP to simulate the tasks in all its complexity. PA
limits access to all granularities of the task, which impedes robot learning
and sim2real transfer. With the purpose of simulating full robotics tasks,
iGibson uses RBP, simulating the physics behavior of all objects continuously,
with embodiment of real robots. This is crucial if the learned policy is to be
deployed in the real world. Similar to iGibson, Sapiens also uses RBP without
PA, but with smaller scenes, focusing on interaction with articulated objects.
In iGibson, we enrich the articulated objects models from the PartNet-Mobility
dataset introduced by Sapien with materials and dynamics properties. TDW is
also capable of continuous RBP, with simplifications that facilitate grasping
using robot avatars and not real robot platforms.

While other simulation environments focus on particular aspects of embodied
agent simulation (Table. II), iGibson uniquely unifies a set of important
tools for robotics that together enable robot learning in large scenes:
support of LiDAR and PBR rendering, speed that enable reinforcement learning,
robot integration (URDF, controllers, motion planners), and continuous physics
simulation of agents and objects. This integration of features enables tasks
such as mobile manipulation, illustrated in Fig. 2, including physics-based
robot simulation across the entire task.

(a) pick

(b) transport

(c) place

Figure 2: Simulated Fetch Robot performing a mobile manipulation task in a
cluttered environment.

##  III iGibson Simulation Environment

In this section, we discuss the main structure, properties and features of
iGibson that support training of robust sensor-guided policies for navigation
and manipulation.

###  III-A Simulation Characteristics and API

At the highest level, iGibson follows the OpenAI Gym [36] convention. The
environment receives an action and returns a new observation, reward and
additional meta-information (e.g. if the episode has ended). Environments are
specified with config files that determine scenes, tasks, robot embodiments,
sensors, etc. Given a config file, iGibson creates an Environment that
contains a Task and a Simulator. The Simulator contains a Scene, with a list
of interactive Objects and one or more Robot instances. It also contains a
Renderer that generates virtual visual signals from any point of view, such as
a camera mounted on a robot or an external third person view. The Task defines
the reward, initial and final conditions for the scene and the agents. While
modular and easy to extend, most users may only need to interface with
Environment after instantiating it with the appropriate config files.

iGibson comes with multiple easy-to-use configs, demos and Docker [37] files.
It has been extensively adopted to train visuo-motor policies that
successfully transfer to the real world [38, 39, 40, 41], and was the platform
for iGibson Sim2Real Challenge at CVPR20 [42] and iGibson Challenge at CVPR21
[43]. The provided virtual LiDAR sensor has been used for robotics research in
planning and reinforcement learning for social navigation [44] and mobile
manipulation [10]. iGibson is easily parallelizable and supports off-screen
rendering on clusters.

###  III-B Fully Interactive Assets

iGibson provides fifteen high quality fully interactive scenes with 108 rooms
(see Fig. 3), populated with interactable objects. The scenes are interactive
versions of fifteen 3D reconstructed scenes included in the Gibson v1 dataset.
To preserve the real-world layout and distribution of objects, we follow a
semi-automatic annotation procedure. This process is radically different from
the annotation we performed for the Interactive Gibson Benchmark [35]. Instead
of segmenting the original scene and replacing part of the meshes with
interactive object models, we create fully interactive counterparts of the 3D
reconstruction from scratch. This eliminates the need to fix artifacts in the
original mesh due to reconstruction noise or segmentation error, and allows us
to improve the overall quality of the scenes.

The scene generation process is composed of two annotation phases. First, the
layout of the scene is annotated with floors, walls, doors and window
openings. Then, all objects are annotated with 3D bounding boxes and class
labels. We annotate bounding boxes for 57 different object classes, including
all furniture types (doors, chairs, tables, cabinets, TVs, shelves, stoves,
sinks, etc) and some small objects (plants, laptops, speakers, etc); see
project website for the complete list. Annotating class-labeled bounding boxes
allows us to scale and use different models of the same object class, while
maintaining the real-world distribution of objects in the scene. In this way,
we are able generate realistic randomized versions of the scenes (see Sec.
III-D). To achieve the highest quality, for each class-labeled bounding box,
we select a best fitting object model. The scene is also annotated with
lights, with which we generate light probes for physics-based rendering (see
Sec. III-C). We also bake in a realistic ray-traced ambient light and other
light effects in the walls, floors and ceilings.

Figure 3: Fifteen interactive iGibson 1.0 scenes modelled after real-world
reconstructions, preserving layout, distribution and size of objects.

The object models are curated from open-source datasets: ShapeNet [21],
PartNet Mobility [15, 45], and SketchFab. To preserve visual realism of the
original reconstruction, we improve the object visual quality by annotating
different parts of the models with photorealistic materials, which are then
used by iGibson’s physics-based renderer. We utilize materials from
CC0Texture, including wood, marble, metal, etc. To achieve a high degree of
physics realism, we curate a mapping from visual materials to friction
coefficients. We additionally compute the collision mesh, center of mass and
inertia frame for each link of all objects. To assign realistic mass and
density for different objects, we take the the median values of the top 20
search results from Amazon.

Additionally, we provide compatibility with CubiCasa5K [23] and 3D-Front [24]
repositories of home scenes. We use their scene layouts and populate them with
our annotated object models, leading to additional more than 12000 interactive
home scenes. These scenes contain fewer objects than the fifteen iGibson
scenes, but provide a very large number of additional models to train tasks.

The fully interactive scenes we include in iGibson enable learning of
interactive tasks in large realistic home scenes; in Sec. IV-C we show that
the scenes can be used to learn a useful visual representation that
accelerates the learning of downstream manipulation tasks.

Figure 4: Robot interacting in iGibson 1.0 (large picture: 3rd person view)
and virtual sensor signals generated. Policies and solutions can make use of
the following channels: (from top to bottom, left to right) RGB, depth,
semantic/instance segmentation, normals, 16D LiDAR (point cloud), 1D LiDAR
(also as occupancy map). Not depicted: optical/scene flow, joint encoders for
robot’s and objects’ joints, poses, wrenches, contact points, and map
localization.

###  III-C Virtual Sensors

A crucial component of iGibson is the generation of high quality virtual
sensor signals, i.e. images and point clouds, for the simulated robots. In the
following, we summarize the most relevant of these signal generators (Fig. 4).

#### Physics Based Rendering

In iGibson, we include an open-source physics-based renderer, which implements
an approximation of BRDF models [46] with spatially varying material maps
including roughness, metallic and tangent-space surface normals, extending
[47].

#### LiDAR Sensing

Many real-world robots are equipped with LiDAR sensors for obstacle detection.
In iGibson, we support virtual LiDAR signals, with both 1 beam (e.g. Hokuyo)
and 16 beams (e.g. Velodyne VLP-16). We include a simple drop-out sensor noise
model to emulate the common failure case in real sensors in which some of the
laser pulses do not return. Additionally, we provide the functionality to turn
the 1D LiDAR scans into local occupancy maps, which are bird’s-eye view images
with three types of pixels indicating free, occupied, or unknown space.

#### Additional Visual Channels

In addition to RGB and LiDAR, we support a wide range of visual modalities,
such as depth maps, optical/scene flow and normals, segmentation of semantic
class, instance, material and movable parts. These modalities can support
research topics such as: depth/segmentation/normal/affordance prediction [48,
49, 50], action-conditioned flow prediction [51], multi-modal pose estimation
[52, 53, 54], and visuomotor policy training assuming perfect vision systems
[35, 55].

###  III-D Domain Randomization

It is standard practice for robot learning to partially randomize the
environment’s parameters in order to make the policy more robust [56, 57, 58,
59]. With the model being trained in a wide distribution of environments, it
will be more likely to generalize to unknown evaluation environments. The
evaluation environment may be the real world if we aim to train in simulation
and transfer the policy to a real robot. In iGibson, we include domain
randomization that leads to an endless variation of visual appearance,
dynamics properties and object instances with the same scene layout.

First, we provide object randomization. The original 3D reconstructions are
annotated with class-labeled object bounding boxes. These labels can be used
to instantiate any object model of the corresponding class into the given
bounding box (e.g. a bounding box labeled as “table” can be filled with any
table model). This randomization maintains the semantic layout of the scenes
(i.e. the object categories remain at the same 3D locations) while enabling
near-infinite combinations of object instances. It provides strong variation
in depth maps and LiDAR signals that helps robustify policies based on these
observations (see Sec. IV-A).

Second, we provide material randomization. In addition to high-quality
material annotation for object and scene models, we provide a mechanism to
randomize the specific material model associated with each object part (e.g.
associating a different type of wood or metal). The effect is a stark color
randomization that still represents plausible material combinations. This
randomization generates strong variations in the RGB images and helps
robustify policies based on this observation (see Sec. IV-A). Moreover, the
dynamics properties of all object links can be randomized based on a curated
mapping from visual materials to dynamics properties.

###  III-E Motion Planning

Motion planners provide collision-free trajectories to move a robot from an
initial to a final configuration [60]. They can be used to generate collision-
free navigation paths for robot bases and collision-free motion paths for
robot arms. In iGibson, we include implementations of the most popular
sampling-based motion planners: rapidly growing random trees (RRT [61]) and
its bidirectional variant (BiRRT [62]), and lazy probabilistic road-maps
(lazyPRM [63]), adapted from [64]. Sampling-based motion planners can have
rather suboptimal and intricate paths. To alleviate this, we include
acceleration-bounded shortcuts [65] for smoother paths.

###  III-F Human-iGibson Interface

We provide a human-iGibson interface that enables users to navigate and
interact in iGibson scenes using mouse and key commands on a viewer window.
The user can navigate and interact with (pull, push, pick and place) objects.
While a virtual reality or a 3D mouse interface may provide a more intuitive
experience, most users do not have the necessary hardware. This interface
offers a natural and simple way to demonstrations for imitation learning,
evaluate the difficulty or feasibility of a task, or change the scene into a
better initial state, for example. This interface is also integrated with the
motion planner to command the robot to desired base and/or arm configurations.
We verify this interface facilitates efficient development of interactive
robotic solutions in Sec. IV-B .

##  IV Experiments

The goal of our experiments is to study how iGibson’s features help to develop
AI agents. Specifically, we examine:

  * •

(Sec.IV-A) does iGibson’s domain randomization and realistic virtual sensor
signals (including LiDAR) allow navigation agents to generalize to unseen
scenes (including the real world)?

  * •

(Sec.IV-B) can the human-iGibson interface be used to efficiently train
imitation learning agents for manipulation and mobile manipulation tasks?

  * •

(Sec.IV-C) does the full interactivity in the scenes allow agents to learn
visual representations that accelerate learning of downstream manipulation
tasks?

###  IV-A Domain Randomization and Realistic Virtual Sensor Signals for Robot
Navigation

In the first three experiments, we evaluate the generalization benefits
brought by our realistic virtual sensor signals (including LiDAR) and domain
randomization. First, we compare the generalization capabilities of vision-
based reinforcement learning policies trained with and without domain
randomization. Concretely, we evaluate the performance of policies trained in
iGibson for PointGoal tasks [66] in held-out scenes with held-out visual
textures. The observations for the policy include the depth maps, the robot’s
linear and angular velocities, the goal location in the robot’s reference
frame, and the next 10 waypoints in the shortest path between the robot’s
current location and the goal location, separated by $0.2\text{\,}\mathrm{m}$.
The waypoints are computed only based on the room layout, not the objects, so
the robot mainly relies on depth maps for obstacle avoidance.

Second, we evaluate the performance of robot policies trained in iGibson to
navigate to a target object (a lamp) using virtual RGB images also in held-out
scenes with held-out visual textures. The task goal is to get at least 5% of
the image occupied by the pixels of the target object. The observation for the
policy only includes RGB images. In the above two experiments, we train in
eleven scenes and evaluate in four held-out scenes with held-out visual
textures.

Third, we evaluate the performance of the policies trained in iGibson for a
PointGoal tasks [66] using virtual LiDAR signals with no vision inputs, and
examine how well those policies transfer to the real world without adaptation,
a hard test for generalization. The observations for the policy include a 1D
LiDAR scan with 512 laser rays, the robot’s linear and angular velocities, and
the goal location in the robot’s reference frame. To focus on sim2real
transferability, we train in our scene Rs_int, for which we have access to the
real-world counterpart (see Fig. 5).

Figure 5: Robot navigating the real-world counterpart of the iGibson 1.0 scene
Rs_int. The robot executes a policy trained in simulation with virtual LiDAR
signals, without domain adaptation. The quality and realism of iGibson
facilitates zero-shot policy transfer.

##### Results

In the first two experiments, we observe better generalization capabilities in
policies using iGibson’s domain randomization. For PointGoal navigation based
on depth images, the performance goes from 0.27 to 0.40 SPL [66] and from
31.25% to 44.75% success rate when using randomization, indicating that the
larger variety of shapes observed in the training process generates more
robust depth-based policies. For object navigation based on RGB images, the
performance goes from 49.75% to 57.5% success rate, indicating that material
randomization helps in obtaining RGB-based policies that are more
generalizable to unseen scenes and textures. Finally, for PointGoal navigation
based on LiDAR signals, the policy achieves 33% success rate in Rs_int in
iGibson, and 24% success rate in the real-world apartment. With only a 9% drop
in performance and the failures mostly occurring in the same episodes (same
pairs of the initial and goal locations in iGibson and real world), this
experiment indicates that the LiDAR signals generated in iGibson are realistic
enough to facilitate zero-shot policy transfer. In summary, as shown in Table.
II, iGibson provides unique support to train with realistic virtual sensor
signals (e.g. LiDAR) and domain randomization, which leads to more robust
robot navigation policies that successfully transfer to novel scenes.

Figure 6: Imitation learning from human demonstration. Top: third-person view
of a robot performing a pick-and-place task. The policy trained using
demonstrations collected with our human-iGibson interface achieves 98% success
rate. Bottom: first-person view of a robot performing a mobile manipulation
task. The policy trained using teleoperated demonstrations achieved 70%
success rate.

###  IV-B Imitation Learning of Human Demonstrated (Mobile) Manipulation

In the second set of experiments, we evaluate iGibson as platform to train
robots to perform manipulation and mobile manipulation tasks with Imitation
Learning. First, we test the usability of the human-iGibson interface to
efficiently collect demonstrations of manipulation-only tasks. We collect 50
demonstrations of pick-and-place operations with 20 mug models: pick a mug and
place in the sink (Fig. II), and store pairs of state (object position) and
action (desired end-effector translation). We use the demonstrations to train
a behavioral cloning policy that maps states to actions at
$20\text{\,}\mathrm{Hz}$. The action space consists of two parts: a
3-dimensional continuous space for desired end-effector translation, and a
1-dimensional discrete space for gripper opening. The desired end-effector
translation is computed using inverse kinematics (IK) and executed with a
joint position controller. The evaluation is conducted using a simulated
mobile manipulator (Fetch robot), and generalization is tested with 5 unseen
mugs.

Second, we collect demonstrations for imitation learning through continuous
teleoperation using a system similar to Roboturk [67] for mobile-manipulation
tasks. We collect 200 demonstrations with a simulated Fetch robot for search-
and-pick operations: the robot must navigate and interact with a cabinet to
open drawers, find a bowl and pick it (Fig. II (bottom)). The bowl and robot
initial poses are randomized between episodes. Using these demonstrations, we
train a behavioral cloning policy mapping observations to actions at
$20\text{\,}\mathrm{Hz}$. The observation space includes the robot
proprioceptive information (joint positions and velocities) and RGB-D images
from virtual cameras on robot’s head and wrist. The action space consists of
four parts: a 1-dimensional discrete value indicating whether to extend the
arm, 2-dimensional continuous values representing the robot base’s linear and
angular velocities, 6-dimensional continuous values representing the desired
pose change of the end-effector, and 1-dimensional discrete value indicating
whether to open the gripper.

##### Results

In our first experiment, training manipulation-only policies with imitation
learning, we observe 98$\%$ success rate over 100 evaluation episodes. This
experiment showcases that the human-iGibson interface enables easy collection
of effective demonstrations for imitation learning, and the integrated motion
planner is helpful for policy training. These two integrated features, as
shown in Table. II, are novel combinations offered by iGibson. In the second
experiment, training mobile-manipulation policies with imitation learning, we
observe 70$\%$ success rate over 20 evaluation episodes. This experiment
indicates that the we can leverage iGibson to train imitation learning
algorithms for mobile manipulation tasks.

###  IV-C Pretraining in Fully Interactive Scenes

In the third and final set of experiments, we evaluate the potential of using
iGibson’s fully interactive scenes to learn an intermediate visual
representation that encodes the expected outcome of interactions with
different objects. Such an intermediate visual representation may be used to
accelerate robot learning of manipulation tasks, since they typically require
the agent to associate visual observations with promising areas of interaction
to change the state of the scene towards a manipulation goal.

Figure 7: Left: Example result of interaction pretraining. The agent receives
RGB input and predicts if the pixels are pushable (red: higher probability;
blue: lower) (Sec. IV-C). The model learns to associate edge of doors as the
most pushable points. Right: Training curves for two interactive tasks
(PushDrawer,PushCabinet) with and without interaction pretraining.

To learn such representations, we set up a virtual agent that interacts with
random points in the scenes and learns to predict the outcome of these
interactions. The interaction is parameterized as a coordinate in the virtual
agent’s image observation space. We emulate a pushing interaction by
displacing the corresponding 3D location of the selected pixel by
$30\text{\,}\mathrm{cm}$ in the opposite direction of the surface normal,
applying a maximum force of $60\text{\,}\mathrm{N}$ (a common payload of
commercial robots). A motion of the point for more than
$10\text{\,}\mathrm{cm}$ is considered a success. We sample 10 random pushes
at each location, 4,000 locations in each scene. We use the images annotated
with interaction successes/failures to train a U-Net [68]-based visual encoder
that predicts heatmaps of expected interaction success from RGB input.

For the second phase, we train two policy networks for two manipulation task
respectively (PushDrawer, PushCabinet). The goal is to close the drawers or
the cabinets. The policy outputs points to interact (push) that are given to
one of our integrated motion planners to generate an arm motion [10]. We use
DQN [69] as policy learning algorithm. The predicted interaction heatmaps are
used to gate the Q-value maps predicted by the network.

##### Results

Fig. 7 (left) depicts the result of the pretrained visual model. We observe
that the heatmap has stronger activation at the edge of the door than in the
area closer to the hinge, and weak activation on closed cabinets. The model
learns to identify the best areas to push to cause motion (further
visualizations on project website). For both downstream tasks, we observe that
using the pre-trained representation significantly accelerates training (Fig.
7 (right)). This suggests that the full interactability of iGibson can help
agents learn useful visual representation for downstream mobile manipulation
tasks. Having a subset of objects not physically interactable will lead to
false negatives during pretraining, and prevents successful representation
learning. As shown in Table. II and discussed in Sec. II, fully interactive
scenes with continuous robot actions is a specialty of iGibson.

##  V Conclusion

We presented iGibson, a novel simulation environment for developing
interactive robotic agents in large-scale realistic scenes. iGibson includes
15 fully interactive scenes with 108 rooms, and novel capabilities to generate
high-quality virtual sensor signals, domain randomization, integration with
motion planners, and human-iGibson interface. Through experiments, we
showcased that iGibson helps to develop robust policies for navigation and
manipulation. We hope that iGibson can aid researchers in solving complex
robotics problems in large-scale realistic scenes.

##  VI Acknowledgement

We thank NVIDIA, Google, ONR MURI (N00014-14-1-0671), ONR (1165419-10-TDAUZ),
Panasonic (1192707-1-GWMSX), Qualcomm and Samsung for their support.

## References

  * Lillicrap et al. [2016] T. P. Lillicrap _et al._ , “Continuous control with deep reinforcement learning,” _ICLR_ , 2016.
  * Levine et al. [2016] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _JMLR_ , 2016.
  * Mirowski et al. [2017] P. Mirowski _et al._ , “Learning to navigate in complex environments,” _ICLR_ , 2017.
  * Parisotto and Salakhutdinov [2018] E. Parisotto and R. Salakhutdinov, “Neural map: Structured memory for deep reinforcement learning,” _ICLR_ , 2018.
  * Zhu et al. [2017] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in _IEEE ICRA_ , 2017.
  * Shen et al. [2019] W. B. Shen _et al._ , “Situational fusion of visual representation for visual navigation,” in _IEEE ICCV_ , 2019.
  * Chen et al. [2019] K. Chen, J. P. de Vicente, G. Sepulveda, F. Xia, A. Soto, M. Vázquez, and S. Savarese, “A behavioral approach to visual navigation with graph localization networks,” _RSS_ , 2019.
  * Garrett et al. [2018] C. R. Garrett _et al._ , “Ffrob: Leveraging symbolic planning for efficient task and motion planning,” _IJRR_ , 2018.
  * Xu et al. [2018] D. Xu _et al._ , “Neural task programming: Learning to generalize across hierarchical tasks,” in _IEEE ICRA_ , 2018.
  * Xia et al. [2021] F. Xia, C. Li, R. Martín-Martín, O. Litany, A. Toshev, and S. Savarese, “Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation,” _IEEE ICRA_ , 2021.
  * Li et al. [2019] E. Li, R. Martín-Martín, F. Xia, and S. Savarese, “Hrl4in: Hierarchical reinforcement learning forinteractive navigation with mobile manipulators,” in _CoRL_ , 2019.
  * James et al. [2020] S. James, Z. Ma, D. R. Arrojo, and A. J. Davison, “Rlbench: The robot learning benchmark & learning environment,” _IEEE RA-L_ , 2020.
  * Yu et al. [2020] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, “Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning,” in _CoRL_ , 2020.
  * Urakami et al. [2019] Y. Urakami, A. Hodgkinson, C. Carlin, R. Leu, L. Rigazio, and P. Abbeel, “Doorgym: A scalable door opening environment and baseline agent,” _arxiv:1908.01887_ , 2019.
  * Xiang et al. [2020] F. Xiang _et al._ , “Sapien: A simulated part-based interactive environment,” in _IEEE CVPR_ , 2020.
  * Lee et al. [2021] Y. Lee, E. S. Hu, Z. Yang, A. Yin, and J. J. Lim, “IKEA furniture assembly environment for long-horizon complex manipulation tasks,” _IEEE ICRA_ , 2021\.
  * Zhu et al. [2020] Y. Zhu, J. Wong, A. Mandlekar, and R. Martín-Martín, “robosuite: A modular simulation framework and benchmark for robot learning,” _arxiv:2009.12293_ , 2020.
  * Manolis Savva* et al. [2019] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra, “Habitat: A Platform for Embodied AI Research,” in _IEEE ICCV_ , 2019.
  * Kolve et al. [2017] E. Kolve _et al._ , “Ai2-thor: An interactive 3d environment for visual ai,” _arxiv:1712.05474_ , 2017.
  * Puig et al. [2018] X. Puig _et al._ , “Virtualhome: Simulating household activities via programs,” in _IEEE CVPR_ , 2018.
  * Chang et al. [2015] A. X. Chang _et al._ , “Shapenet: An information-rich 3d model repository,” _arxiv:1512.03012_ , 2015.
  * Wang et al. [2019] X. Wang, B. Zhou, Y. Shi, X. Chen, Q. Zhao, and K. Xu, “Shape2motion: Joint analysis of motion parts and attributes from 3d shapes,” in _IEEE CVPR_ , 2019.
  * Kalervo et al. [2019] A. Kalervo, J. Ylioinas, M. Häikiö, A. Karhu, and J. Kannala, “Cubicasa5k: A dataset and an improved multi-task model for floorplan image analysis,” in _Scandinavian Conference on Image Analysis_ , 2019.
  * Fu et al. [2020] H. Fu _et al._ , “3d-front: 3d furnished rooms with layouts and semantics,” _arxiv:2011.09127_ , 2020.
  * Xia et al. [2018] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese, “Gibson env: Real-world perception for embodied agents,” in _CVPR_ , 2018.
  * Gan et al. [2020] C. Gan _et al._ , “Threedworld: A platform for interactive multi-modal physical simulation,” _arxiv:2007.04954_ , 2020.
  * Coumans et al. [2013] E. Coumans _et al._ , “Bullet physics library,” _Open source: bulletphysics. org_ , 2013.
  * Todorov et al. [2012] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _IROS_ , 2012.
  * Lee et al. [2018] J. Lee, M. X. Grey, S. Ha, T. Kunz, S. Jain, Y. Ye, S. S. Srinivasa, M. Stilman, and C. K. Liu, “Dart: Dynamic animation and robotics toolkit,” _Journal of Open Source Software_ , 2018.
  * [30] “Unity,” https://unity.com/, accessed: 2020-10-30.
  * Liang et al. [2018] J. Liang, V. Makoviychuk, A. Handa, N. Chentanez, M. Macklin, and D. Fox, “Gpu-accelerated robotic simulation for distributed reinforcement learning,” _CoRL_ , 2018.
  * [32] “Ode,” https://ode.org/, accessed: 2020-10-30.
  * Erickson et al. [2020] Z. Erickson _et al._ , “Assistive gym: A physics simulation framework for assistive robotics,” in _IEEE ICRA_ , 2020.
  * Chang et al. [2017] A. Chang _et al._ , “Matterport3d: Learning from rgb-d data in indoor environments,” in _3DV_ , 2017.
  * Xia et al. [2020] F. Xia _et al._ , “Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments,” _IEEE RA-L_ , 2020.
  * Brockman et al. [2016] G. Brockman _et al._ , “Openai gym,” _arxiv:1606.01540_ , 2016.
  * Merkel [2014] D. Merkel, “Docker: lightweight linux containers for consistent development and deployment,” _Linux journal_ , 2014.
  * Meng et al. [2020] X. Meng, N. Ratliff, Y. Xiang, and D. Fox, “Scaling local control to large-scale topological navigation,” in _IEEE ICRA_ , 2020.
  * Meng et al. [2019] ——, “Neural autonomous navigation with riemannian motion policy,” in _IEEE ICRA_ , 2019.
  * Kang et al. [2019] K. Kang _et al._ , “Generalization through simulation: Integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight,” _IEEE ICRA_ , 2019.
  * Hirose et al. [2019] N. Hirose, F. Xia, R. Martín-Martín, A. Sadeghian, and S. Savarese, “Deep visual mpc-policy learning for navigation,” _IEEE RA-L_ , 2019.
  * [42] “iGibson Sim2real Challenge at CVPR2020.” [Online]. Available: http://svl.stanford.edu/igibson/challenge2020.html
  * [43] “iGibson Challenge at CVPR2021.” [Online]. Available: http://svl.stanford.edu/igibson/challenge.html
  * Pérez-D’Arpino et al. [2021] C. Pérez-D’Arpino, C. Liu, P. Goebel, R. Martín-Martín, and S. Savarese, “Robot navigation in constrained pedestrian environments using reinforcement learning,” _IEEE ICRA_ , 2021.
  * Mo et al. [2019] K. Mo, S. Zhu, A. X. Chang, L. Yi, S. Tripathi, L. J. Guibas, and H. Su, “Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding,” in _IEEE CVPR_ , 2019.
  * Schlick [1994] C. Schlick, “An inexpensive brdf model for physically-based rendering,” in _Computer graphics forum_ , 1994.
  * [47] “Nadrin/pbr,” https://github.com/Nadrin/PBR, accessed: 2020-10-30.
  * Porzi et al. [2016] L. Porzi, S. R. Bulo, A. Penate-Sanchez, E. Ricci, and F. Moreno-Noguer, “Learning depth-aware deep representations for robotic perception,” _IEEE RA-L_ , 2016.
  * Casser et al. [2019] V. Casser, S. Pirk, R. Mahjourian, and A. Angelova, “Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular videos,” in _AAAI_ , 2019.
  * Xie et al. [2020] C. Xie, Y. Xiang, A. Mousavian, and D. Fox, “The best of both modes: Separately leveraging rgb and depth for unseen object instance segmentation,” in _CoRL_ , 2020.
  * Nematollahi et al. [2020] I. Nematollahi, O. Mees, L. Hermann, and W. Burgard, “Hindsight for foresight: Unsupervised structured dynamics models from physical interaction,” _IEEE IROS_ , 2020.
  * Choi and Christensen [2016] C. Choi and H. I. Christensen, “Rgb-d object pose estimation in unstructured environments,” _Robotics and Autonomous Systems_ , 2016.
  * Wang et al. [2020] C. Wang _et al._ , “6-pack: Category-level 6d pose tracker with anchor-based keypoints,” in _IEEE ICRA_ , 2020.
  * Hu et al. [2019] Y. Hu, J. Hugonot, P. Fua, and M. Salzmann, “Segmentation-driven 6d object pose estimation,” in _IEEE CVPR_ , 2019.
  * Yan et al. [2020] M. Yan, Q. Sun, I. Frosio, S. Tyree, and J. Kautz, “How to close sim-real gap? transfer with segmentation!” _arxiv:2005.07695_ , 2020.
  * Sadeghi and Levine [2016] F. Sadeghi and S. Levine, “Cad2rl: Real single-image flight without a single real image,” _RSS_ , 2016.
  * Tobin et al. [2017] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in _IROS_ , 2017.
  * James et al. [2019] S. James _et al._ , “Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks,” in _IEEE CVPR_ , 2019.
  * Andrychowicz et al. [2020] M. Andrychowicz _et al._ , “Learning dexterous in-hand manipulation,” _IJRR_ , 2020.
  * LaValle [2006] S. M. LaValle, _Planning algorithms_. Cambridge university press, 2006.
  * Lavalle [1998] S. M. Lavalle, “Rapidly-exploring random trees: A new tool for path planning,” Department of Computer Science; Iowa State University, Tech. Rep., 1998.
  * Qureshi and Ayaz [2015] A. H. Qureshi and Y. Ayaz, “Intelligent bidirectional rapidly-exploring random trees for optimal motion planning in complex cluttered environments,” _Robotics and Autonomous Systems_ , 2015.
  * Bohlin and Kavraki [2000] R. Bohlin and L. E. Kavraki, “Path planning using lazy prm,” in _IEEE ICRA_ , 2000.
  * Caelan Reed Garrett [2018] Caelan Reed Garrett, “PyBullet Planning.” https://pypi.org/project/pybullet-planning/, 2018.
  * Hauser and Ng-Thow-Hing [2010] K. Hauser and V. Ng-Thow-Hing, “Fast smoothing of manipulator trajectories using optimal bounded-acceleration shortcuts,” in _IEEE ICRA_ , 2010.
  * Anderson et al. [2018] P. Anderson _et al._ , “On evaluation of embodied navigation agents,” _arxiv:1807.06757_ , 2018.
  * Mandlekar et al. [2018] A. Mandlekar _et al._ , “Roboturk: A crowdsourcing platform for robotic skill learning through imitation,” in _CoRL_ , 2018.
  * Ronneberger et al. [2015] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on MICCAI_ , 2015.
  * Van Hasselt et al. [2016] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _AAAI_ , 2016.
  * Qiao et al. [2019] S. Qiao, H. Wang, C. Liu, W. Shen, and A. Yuille, “Weight standardization,” _arxiv:1903.10520_ , 2019.
  * Haarnoja et al. [2018] T. Haarnoja _et al._ , “Soft actor-critic algorithms and applications,” _arxiv:1812.05905_ , 2018.
  * Quigley et al. [2009] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “Ros: an open-source robot operating system,” in _ICRA workshop on open source software_ , 2009.
  * Kingma and Ba [2015] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _ICLR_ , 2015.

## Appendix

###  VI-A Details on the Experimental Evaluation

In this section, we provide additional information on the experiments
presented in Sec. IV such as training processes and architectures, or system
characteristics.

####  VI-A1 Domain Randomization for Visual Navigation

For PointGoal navigation, as mentioned in the main paper, the observations for
the policy include the depth maps, the robot’s linear and angular velocities,
the goal location in the robot’s reference frame, and the location of the next
10 waypoints in the shortest path between the robot’s current location and the
goal location. We concatenate the robot’s linear and angular velocities, the
goal location, and the location of the next 10 waypoints together and denote
them by sensor observations. For both depth maps and sensor observations, we
adopt a frame stack of 4. The encoder of the policy network consists of 3
parts: (a) a 9-layer ResNet with Weight Standardization [70] and GroupNorm,
followed by a 3-layer Conv1D block to encode depth maps; (b) a 2-layer MLP to
encode sensor observations; (c) a 2-layer fusion MLP to encode the
concatenation of depth maps embedding and sensor observations embedding. The
learning rate for SAC is 1e-4. For object navigation, the observation only
includes a frame stack of 4 RGB images. The encoder of the policy network is a
9-layer ResNet with Weight Standardization and GroupNorm, followed by a
3-layer Conv1D block to encode RGB images. The learning rate for SAC is 5e-4.
To accelerate the training of our visual policies, we implement a multi-GPU
distributed reinforcement learning pipeline based on SAC [71]. iGibson can be
easily parallelized and deployed on large computing clusters, which makes the
training highly efficient.

Tables II and III include a breakdown analysis of the results of the
experiment for different scenes and phases of the training process.

TABLE II: Quantitative results on PointGoal navigation. Domain | Training | Rs_int | Beechwood_0_int | Merom_0_int | Pomaria_0_int | Overall
---|---|---|---|---|---|---
Randomization | Steps | SPL | SR | SPL | SR | SPL | SR | SPL | SR | SPL | SR
w/o | 400k | 0.41 | 42% | 0.21 | 21% | 0.22 | 22% | 0.27 | 27% | 0.28 | 28%
w/ | 400k | 0.35 | 36% | 0.25 | 26% | 0.32 | 33% | 0.25 | 25% | 0.29 | 30%
w/o | 800k | 0.50 | 56% | 0.14 | 17% | 0.22 | 25% | 0.23 | 27% | 0.27 | 31.15%
w/ | 800k | 0.60 | 67% | 0.37 | 40% | 0.29 | 33% | 0.36 | 39% | 0.40 | 44.75%
TABLE III: Quantitative results on object navigation. Domain | Training | Rs_int | Beechwood_0_int | Merom_0_int | Pomaria_0_int | Overall
---|---|---|---|---|---|---
Randomization | Steps | SR | SR | SR | SR | SR
w/o | 100k | 43% | 34% | 47% | 45% | 42.25%
w/ | 100k | 40% | 45% | 52% | 51% | 47%
w/o | 200k | 50% | 46% | 48% | 55% | 49.75%
w/ | 200k | 55% | 54% | 64% | 57% | 57.5%

####  VI-A2 LiDAR-Based Point-to-Point Navigation

As mentioned in the main paper, the observations for the policy include a 1D
LiDAR scan with 512 laser rays, the robot’s linear and angular velocities, and
the goal location in the robot’s reference frame. Similar to PointGoal
navigation in Sec. VI-A1, we concatenate the robot’s linear velocity, angular
velocity and the goal location and denote them by sensor observations. The
controller runs at $10\text{\,}\mathrm{Hz}$. For both LiDAR scans and sensor
observations, we adopt a frame stack of 8. The encoder of the policy network
consists of 3 parts: (a) a 3-layer MLP to encode LiDAR scans; (b) a 3-layer
MLP to encode sensor observations; (c) a 3-layer fusion MLP to encode the
concatenation of the flattened (along temporal dimension) LiDAR scans
embedding and sensor observations embedding. The learning rate for SAC is
1e-4.

As real robot platform we use a Locobot, a non-holonomic mobile base, with an
additionally mounted Hokuyo 1 beam LiDAR sensor (see Fig. 5, left). The
simulated agents matches the characteristics of the real robot and sensor. The
robot is controlled via ROS [72]. Processing the images requires more
computation than the one provided by the onboard computer of the Locobot;
thus, we send the images to a desktop computer that hosts the policy and
generate commands that are sent back to the robot. We use the system we
developed for the CVPR Challenge “Sim2Real with iGibson” [42].

####  VI-A3 Imitation Learning: Human Demonstrated Manipulation

We collected 50 demonstrations of pick-and-place operations demonstrated with
our Human-iGibson interface to pick a mug and place in the sink (Fig. II,
right) with 20 different mug models. We split the demos into train/val/test
set with 35, 10, 5 demos and 1095, 304, 157 pairs of state and action,
respectively. The state includes the object position and the action includes
the desired delta translation to add to the object position in the next step
and whether to open gripper. The starting position of the mug is randomized
within a $50\text{\,}\mathrm{cm}$ by $10\text{\,}\mathrm{cm}$ area.

We trained a policy using behavioral cloning to map states to actions. The
policy network is composed by three MLP layers and ReLU activation. We used
MSE loss for the delta translation action and cross entropy loss for the
binary gripper action. We used ADAM optimizer [73] with learning rate 0.1 and
trained the policy for 1000 epochs after validation loss plateaus.

For evaluation, we deployed the policy on the simulated Fetch robot and
achieved 98% success rate for 100 evaluation episodes. The two failed episodes
are caused by the robot prematurely opening its gripper and the mug being
dropped outside the sink. The robot has a time budget of 500 action steps (25
seconds) to accomplish the task.

####  VI-A4 Pretraining in Fully Interactive Scenes

In this section we show the details of the network used in pretraining and the
details for policy training in downstream tasks. The network used in
pretraining is a UNet structure. The input to the UNet is an RGB image of size
$128\times 128$, and the output is a binary mask indicating which area is
interactable. The UNet consists of an encoder with ResNet9 architecture, 4
(Upsampling, Conv) blocks and finally a readout module consisting of 2 Conv
layers predicting the binary mask.

For the downstream task, we focused on pushing task. The observation space is
RGB images of size $128\times 128$, and the action space is a point on the
image. To give readers an intuitive sense, an example trajectory of the two
tasks are shown in Fig. 8. The baseline model for these tasks are DQN with
dense Q-value prediction. The policy network uses a 6-layer convolutional
neural network to predict an array of Q-values, with the same shape as the
input image. The agent picks the pixel with the highest associated Q-value,
and use a motion planner to plan a push motion. The method that integrates
pretraining modifies the baseline method, by adding a mask to the predicted
Q-values with predicted interaction mask, only pixels that are predicted to be
interactable keep the Q-values, and the rest are zeroed out. Both setups uses
Q-learning algorithm with $\epsilon$-greedy to update the network, we use a
learning rate of $1\times 10^{-3}$, discount factor of $0.99$, $\epsilon=0.2$
for exploration, and the policies are trained for $5\times 10^{4}$ steps.

As shown in the main paper, with pretraining, the agent learns faster. Fig. 8
depicts different stages of the interactions learned by an agent using our
learned intermediate representation.

(a) PushDrawer task

(b)  PushCabinet task

Figure 8: Execution of (a) PushDrawer task. The agent successfully push the
top drawers in. (b) PushCabinet task. The agent push and close the cabinet
doors.

###  VI-B Physics Based Rendering

In this section we include additional information about the shading model and
rendering process we use in iGibson.

Shading Models To improve the realism when rendering images of objects, we
represent the properties of the surface material of each object using four
layers, 1) metallic layer (single channel image), 2) roughness layer (single
channel image), 3) albedo (three channel image), and 4) tangent-space normal
mapping (three channel image). The information of the layers is combined by
our physics-based rendering model. To generate correct light effects on the
objects’ surfaces, we create environment maps that light the scene. We also
pre-integrate the Cook-Torrance specular bidirectional scattering distribution
function (BRDF) for varying roughness and viewing directions to accelerate and
enable real time rendering. The results are saved into 2-dimensional look up
table texture, which is used to scale and add bias to Fresnel reflectance at
normal incidence($F_{0}$) during rendering. The $F_{0}$ is then used to
calculate Shlick’s approximation of the Fresnel factor as follows:  F(θ) = F_0
+ (1-F_0)(1-cosθ) ^5

To calculate the specular highlights, we pre-filter environment cube map using
GGX normal distribution function importance sampling. The results are saved as
an image for later faster retrieval. For diffuse effect we use quasi Monte
Carlo sampling with Hammersley sequence to approximate the integration.

Light Probe Generation To generate light probes, we use Blender to bake high
resolution high dynamic range(HDR) environment textures within iGibson scenes.
The light sources are artistically designed and placed on the ceiling.

Shadow Mapping For generating shadows we use shadow mapping techniques and
simulate orthogonal uniform light in $+z$ direction. All the objects in the
scenes are shadow casters except for the ceiling. The shadows don’t
necessarily match the real world indoor lights, but create a realistic enough
effect that help perceiving depth.

###  VI-C iGibson Performance

In this section we evaluate the speed of the iGibson simulation environment,
analyzing the time for rendering and physics simulation, and its combination.

####  VI-C1 Rendering Performance

We evaluate the performance and speed of our novel iGibson renderer with
different working settings. There are many configurable options in iGibson to
control the rendering quality. We benchmark two typical use cases. The first
one is Reinforcement Learning (VisualRL), where the goal is to generate
simulated sensor data (low resolution) as fast as possible. The second one is
when higher quality images, as photorealistic as possible, are required, but
maintaining a minimum level of rendering speed (HighFidelity). This use case
is common when training perceptual solutions (e.g., segmentation, object
detection) and also to support collecting human demonstrations with
photorealistic enough images and effects such as shadows.

For the first use case, VisualRL, we render $128\times 128$ images, with
physically based rendering on, multi sample anti-aliasing off, and shadow
mapping off. For the second case, HighFidelity, we render $512\times 512$
images, with physically based rendering on, multi sample anti-aliasing on,
shadow mapping on. The results of our benchmark of the rendering time for
different modalities with these settings are shown in Table IV.

TABLE IV: Rendering Speed of iGibson Renderer [fps]  Preset | Modality | Mean | Max | Min
---|---|---|---|---
VisualRL | RGB | 409.8 | 1142.7 | 270.4
Normal | 530.9 | 1142.0 | 283.7
Point Cloud | 530.3 | 1129.0 | 282.1
Semantic Mask | 529.4 | 1140.4 | 281.7
Optical Flow | 528.1 | 1129.1 | 281.6
Scene Flow | 526.4 | 1129.9 | 280.1
HighFidelity | RGB | 188.4 | 289.3 | 114.3
Normal | 219.6 | 310.1 | 148.9
Point Cloud | 240.4 | 345.1 | 174.7
Semantic Mask | 221.5 | 313.8 | 163.5
Optical Flow | 240.1 | 348.9 | 170.8
Scene Flow | 240.5 | 345.9 | 173.9
TABLE V: Simulator Step Speed and Full Step Speed [$\text{\,}\mathrm{H}\mathrm{z}$]  | Robot | Mean | Max | Min
---|---|---|---|---
Simulator Physics Step | With robot | 175 | 311 | 130
Scene only | 310 | 797 | 92
Simulator Full Step | With robot | 100 | 150 | 68
Scene only | 136 | 230 | 84

####  VI-C2 Physics Simulator and Full Simulator Performance

The size and number of objects included in our scenes are beyond what is
typically included in other simulator or other projects based on PyBullet. We
benchmarked the performance of the physics simulator and the final performance
of iGibson, combining physics and rendering. In our experiments, the physics
simulation timestep is set to $1\text{/}120\text{\,}\mathrm{s}$. Each
simulator step includes four steps of the physics simulator and one rendering
pass, corresponding to rendering at 30 fps in simulation. The results of our
analysis are shown in Table. V. We achieve an average of
$100\text{\,}\mathrm{H}\mathrm{z}$ with robot for simulator full step, and
$136\text{\,}\mathrm{H}\mathrm{z}$ without robot, which correspond to
$3.33\times$ and $4.53\times$ real-time respectively.

###  VI-D Integration of Additional Datasets of Scenes

####  VI-D1 Integration of CubiCasa5K

CubiCasa5K [23] is a dataset of five thousand annotated floor plans of real
world homes in Finland. The annotated floor plans are semi-automatically
generated and include the structural elements (walls, doors, windows) and
position and size of fixed furniture items (closets, toilets, benches,
embedded cabinets, counters, sinks, …). We convert this annotations into
iGibson 3D fully interactive scenes with a two step procedure: 1) generate a
building based on the annotation of structural elements, and 2) populate the
building with object models from our dataset based on the description of poses
and sizes from CubiCasa5K. Some of the floor plans in CubiCasa5K included two
separate floors; since we do not include outdoor navigation in iGibson, we
split them into two separate indoor scenes. In total, we offer 6297 scene in
iGibson based on the real-world layouts of CubiCasa5K. Fig. 9 depicts some
examples of the scenes included created based on the CubiCasa5K dataset.

Figure 9: Examples of iGibson scenes created based on the CubiCasa5K and
3D-Front annotations. We provide over 12000 additional fully interactive
iGibson scenes based the two datasets.

####  VI-D2 Integration of 3D-Front

3D-Front [24] is a large dataset of layouts with room models populated with
furniture. The layouts have been created by artists and interior designers. It
includes 18,797 rooms, around 10,000 houses, and 7,302 furniture models. We
convert 3D-Front static scenes into iGibson 3D fully interactive scenes with a
two step procedure: 1) we keep the original building structural meshes as
visual meshes while procedurally generating collision meshes for structural
elements that approximate their geometry, and 2) we populate the building with
object models from our dataset based on the description of poses and sizes
from 3D-Front.

There are four challenges we faced when integrating 3D-Front to convert their
scenes into interactive scenes for iGibson. First, some 3D-Front scenes
contain objects of undefined categories, corresponding to one-of-a-kind
pieces. We skip these scenes, since we cannot generate appropriate low-poly
collision meshes for these objects. Second, some of the 3D data within
3D-Front is corrupted or contain shape errors (see reported issue in this
link). Third, the kitchen cabinets in 3D-Front are not annotated as objects,
but instead the entire kitchen furniture is a single object with each panel of
the furniture (front and lateral panels, internal shelves) annotated as a
separate part. This impedes us to generate interactive versions of the kitchen
cabinets. We include two alternative versions of the scenes: a) a version with
non-interactive kitchen cabinets, and b) a version without any kitchen
cabinets. We expect this problem to be solved in future annotations of
3D-Front. Fourth, while 3D-Front dataset includes a layout description of
rooms and elements, including their position and size, the furniture pieces
are sometimes defined as overlapping significantly with each other (see
reported issue in this link). This has a severe effect in our physics
simulation as it tries to solve the penetrating contact. To alleviate this
issue, we remove objects that overlap more than 80% of volume with others. For
objects that overlap less than 80%, we reduce their size (with a threshold of
reducing no more than 20%) until the overlapping is resolve. After all the
aforementioned filtering and fixing processes, we obtained 6049 scenes with
correct object placements without overlapping, ready to be used in interactive
tasks.

####  VI-D3 Comparing iGibson, CubiCasa5K and 3D-Front Scenes

There are two significant differences between the 15 fully interactive iGibson
scenes, and the ones obtained from the integration of Cubicasa5K and 3D-Front.
The first one is in density of objects. Our iGibson curated scenes contain a
density of 75 objects per room. This is more realistic and significantly
higher density that the one in 3D-Front rooms (37) and CubiCasa5K rooms (32).
A second difference between the scenes is in the albedo and material effects
of walls and floors. In the 15 iGibson scenes provided, we bake the effects of
the ambient lighting and the light sources on walls, floors and ceilings into
a RGB texture map. This provides a more realistic lighting effect.
Differently, the structural elements of CubiCasa5K and 3D-Front do not present
this enhanced diffuse color channel. To bake lighting sources in CubiCasa5K
and 3D-Front scenes, we need lighting information that are not contained as
part of the datasets. It would require a manual annotation of light location
and strength in thousands of scenes, an effort that is beyond our attempt to
make these datasets available to the community. Despite these differences, we
believe getting access to these two large datasets of scenes significantly
complement the lower number (15) but higher quality scenes provided with
iGibson.


New EG Logo

#  LaTeX Author Guidelines for EUROGRAPHICS Proceedings Manuscripts

D. W. Fellner1,20000-0001-7756-0901 and S. Behnke20000-0001-5923-423X
1TU Darmstadt & Fraunhofer IGD, Germany 2Graz University of Technology,
Institute of Computer Graphics and Knowledge Visualization, Austria  Chairman
Eurographics Publications Board

###### Abstract

The ABSTRACT is to be in fully-justified italicized text, between two
horizontal lines, in one-column format, below the author and affiliation
information. Use the word “Abstract” as the title, in 9-point Times, boldface
type, left-aligned to the text, initially capitalized. The abstract is to be
in 9-point, single-spaced type. The abstract may be up to 3 inches (7.62 cm)
long.
Leave one blank line after the abstract, then add the subject categories
according to the ACM Classification Index (see
https://www.acm.org/publications/class-2012) <ccs2012> <concept>
<concept_id>10010147.10010371.10010352.10010381</concept_id>
<concept_desc>Computing methodologies Collision detection</concept_desc>
<concept_significance>300</concept_significance> </concept> <concept>
<concept_id>10010583.10010588.10010559</concept_id> <concept_desc>Hardware
Sensors and actuators</concept_desc>
<concept_significance>300</concept_significance> </concept> <concept>
<concept_id>10010583.10010584.10010587</concept_id> <concept_desc>Hardware PCB
design and layout</concept_desc>
<concept_significance>100</concept_significance> </concept> </ccs2012>

[300]Computing methodologies Collision detection [300]Hardware Sensors and
actuators [100]Hardware PCB design and layout

††volume: 39††issue: 7

##  1 Introduction

Please follow the steps outlined in this document very carefully when
submitting your manuscript to Eurographics.

You may as well use the LaTeX source as a template to typeset your own paper.
In this case we encourage you to also read the LaTeX comments embedded in the
document.

##  2 Instructions

Please read the following carefully.

###  2.1 Language

All manuscripts must be in English.

###  2.2 Margins and page numbering

All printed material, including text, illustrations, and charts, must be kept
within a print area 7 inches (17.7 cm) wide by 9.44 inches (24 cm) high. Do
not write or print anything outside the print area. Number your pages on odd
sites right above, on even sites left above, no page number on the first site.
Do not use page numbering within the final version of your paper.

###  2.3 Formatting your paper

All text with the exception of the abstract must be in a two-column format.
The total allowable width of the text area – including header and footer lines
– is 177 mm (7 inch) wide by 245 mm (9.64 inch) high.

Columns are to be 84 mm (3.3 inch) wide, with a 8 mm (0.315 inch) space
between them.

The space between the header line and the first line of the text body and
between the last line of the text body and the footer line is 5 mm (0.196
inch) each.

###  2.4 Type-style and fonts

Wherever Times is specified, Times Roman may also be used. If neither is
available on your word processor, please use the font closest in appearance to
Times that you have access to. Only Type-1 fonts will be accepted.

MAIN TITLE. The title should be in Times 17-point, boldface type and centered.
Capitalize the first letter of nouns, pronouns, verbs, adjectives, and
adverbs; do not capitalize articles, coordinate conjunctions, or prepositions
(unless the title begins with such a word). Leave two blank lines after the
title.

AUTHOR NAME(s) and AFFILIATION(s) are to be centered beneath the title and
printed in Times 9-point, non-boldface type. This information is to be
followed by two blank lines.

The ABSTRACT ist to be in a one-column format. The MAIN TEXT is to be in a
two-column format.

MAIN TEXT. Type main text in 9-point Times, single-spaced. Do _not_ use
double-spacing. All paragraphs should be indented 1 em (the length of the dash
in the actual font). Make sure your text is fully justified – that is, flush
left and flush right. Please do not place any additional blank lines between
paragraphs. Figure and table captions should be 9-point Times boldface type as
in Figure 2.

Long captions should be set as in Figure 1 or Figure 3.

Figure 1:  ’Empty’ figure only to serve as an example of long caption
requiring more than one line. It is not typed centered but aligned on both
sides.

Figures which need the full textwidth can be typeset as Figure 3.

Callouts should be 9-point Times, non-boldface type. Initially capitalize only
the first word of section titles and first-, second-, and third-order
headings.

FIRST-ORDER HEADINGS. (For example, 1\. Introduction) should be Times 9-point
boldface, initially capitalized, flush left, with one blank line before, and
one blank line after.

SECOND-ORDER HEADINGS. (For example, 2.1. Language) should be Times 9-point
boldface, initially capitalized, flush left, with one blank line before, and
one after. If you require a third-order heading (we discourage it), use
9-point Times, boldface, initially capitalized, flush left, preceded by one
blank line, followed by a period and your text on the same line.

The headline _(authors / title)_ must be shortened if it uses the full two
column width of the main text. There must be enough space for the page
numbers. Please use “et al.” if there are more than three authors and specify
a shortened version for your title.

###  2.5 Footnotes

Please do _not_ use footnotes at all!

###  2.6 References

List all bibliographical references in 9-point Times, single-spaced, at the
end of your paper in alphabetical order. When referenced in the text, enclose
the citation index in square brackets, for example [Lous90]. Where
appropriate, include the name(s) of editors of referenced books.

For your references please use the following algorithm:

  * •

one author: first 3 chars plus year – e.g. [Lous90]

  * •

two, three or four authors: first char of each family name plus year – e.g.
[Fellner-Helmberg93] or [Kobbelt97-USHDR] or [Lafortune97-NARF]

  * •

more than 4 authors: first char of family name from first 3 authors followed
by a ’*’ followed by the year – e.g. [Buhmann:1998:DCQ] or
[FolDamFeiHug.etal93]

For BibTeX users a style file eg-alpha.bst and eg-alpha-doi.bst is available
which uses the above algorithm.

For Biber users a style file EG.bbx is available which uses the above
algorithm.

###  2.7 Illustrations, graphs, and photographs

All graphics should be centered.

For all figures please keep in mind that you must not use images with
transparent background!

Figure 2:  Here is a sample figure.

If your paper includes images, it is very important that they are of
sufficient resolution to be faithfully reproduced.

To determine the optimum size (width and height) of an image, measure the
image’s size as it appears in your document (in millimeters), and then
multiply those two values by 12. The resulting values are the optimum $x$ and
$y$ resolution, in pixels, of the image. Image quality will suffer if these
guidelines are not followed.

Example 1: An image measures 50 mm by 75 mm when placed in a document. This
image should have a resolution of no less than 600 pixels by 900 pixels in
order to be reproduced faithfully.

Example 2: Capturing a screenshot of your entire $1024\times 768$ pixel
display monitor may be useful in illustrating a concept from your research. In
order to be reproduced faithfully, that $1024\times 768$ image should be no
larger than 85 mm by 64 mm (approximately) when placed in your document.

###  2.8 Color

Please observe: as of 2003 publications in the proceedings of the Eurographics
Conference can use color images throughout the paper. No separate color tables
are necessary.

However, workshop proceedings might have different agreements! Figure 3 is an
example for creating color plates.

###  2.9 Embedding of Hyperlinks / Typesetting of URLs

Due to the use of the package hyperref the original behavior of the command
$\backslash$url from the package url is not available. To circumvent this
problem we either recommend to use the command $\backslash$httpAddr from the
included package egweblnk (see below) or to replace the command
$\backslash$url by the command $\backslash$webLink – e.g. in cases where
$\backslash$url has been used widely in BibTeX-References. In the latter case
we suggest to run BibTeX as usual and then replace all occurences of
$\backslash$url by $\backslash$webLink

The provided commands for hyperlinks, in a nutshell, are:

$\backslash$httpAddr {URL without leading ’http:’}

e.g. http://diglib.eg.org/handle/10.2312/306

$\backslash$httpsAddr {URL without leading ’https:’}

e.g. https://diglib.eg.org/handle/10.2312/306

$\backslash$ftpAddr {URL without leading ’ftp:’}

e.g. ftp://www.eg.org/EG/DL/ftpupload

$\backslash$URL {url}

e.g. http://diglib.eg.org/handle/10.2312/306

$\backslash$MailTo {Email addr}

e.g<EMAIL_ADDRESS>
$\backslash$MailToNA {emailName}{@emailSiteAddress}

e.g<EMAIL_ADDRESS>
$\backslash$webLink{URL without hyperlink creation}

e.g. http://www.eg.org/some_arbitrary_long/but_useless/URL

###  2.10 PDF Generation

Your final paper should be delivered as a PDF document with all typefaces
embedded. LaTeX users should use dvips and ps2pdf to create this PDF document.
Adobe Acrobat Distiller may be used in place of ps2pdf.

Adobe PDFWriter is _not_ acceptable for use. Documents created with PDFWriter
will be returned to the author for revision. pdftex and pdflatex (and its
variants) can be used only if the author can make certain that all typefaces
are embedded and images are not downsampled or subsampled during the PDF
creation process.

Users with no access to these PDF creation tools should make available a
PostScript file and we will make a PDF document from it.

The PDF file _must not_ be change protected.

#### Configuration Notes: dvips / ps2pdf / etc.

dvips should be invoked with the -Ppdf and -G0 flags in order to use Type 1
PostScript typefaces:

        dvips -t a4 -Ppdf -G0 -o my.ps my.dvi

If you are using version 7.x of GhostScript, please use the following method
of invoking ps2pdf, in order to embed all typefaces and ensure that images are
not downsampled or subsampled in the PDF creation process:

      ps2pdf -dMaxSubsetPct=100 \
             -dCompatibilityLevel=1.3 \
             -dSubsetFonts=true \
             -dEmbedAllFonts=true \
             -dAutoFilterColorImages=false \
             -dAutoFilterGrayImages=false \
             -dColorImageFilter=/FlateEncode \
             -dGrayImageFilter=/FlateEncode \
             -dMonoImageFilter=/FlateEncode \
             mypaper.ps mypaper.pdf

If you are using version 8.x of GhostScript, please use this method in place
of the example above:

      ps2pdf -dPDFSETTINGS=/prepress \
             -dCompatibilityLevel=1.3 \
             -dAutoFilterColorImages=false \
             -dAutoFilterGrayImages=false \
             -dColorImageFilter=/FlateEncode \
             -dGrayImageFilter=/FlateEncode \
             -dMonoImageFilter=/FlateEncode \
             -dDownsampleColorImages=false \
             -dDownsampleGrayImages=false \
             mypaper.ps mypaper.pdf

#### Configuration Notes: pdftex / pdflatex / etc.

Configuration of these tools to embed all typefaces can be accomplished by
editing the updmap.cfg file to enable inclusion of the standard (or base) 14
typefaces.

Linux users can run the updmap script to do this:

    updmap --setoption pdftexDownloadBase14 true

Windows users should edit the updmap.cfg files found in their TeX installation
directories (one or both of the following may be present):

      INSTALLDIR\texmf\web2c\updmap.cfg
      INSTALLDIR\localtexmf\miktex\config\updmap.cfg

Ensure the value for pdftexDownloadBase14 is "true," and then follow the
instructions found here: http://docs.miktex.org/manual/ to update your MikTeX
installation.

#### Configuration Notes: Acrobat Distiller

We recommend to use a Distiller job options file that embeds all typefaces and
does not downsample or subsample images when creating the PDF document.

###  2.11 Exclusive License Form

You must include your signed Eurographics Exclusive License Form when you
submit your finished paper. We MUST have this form before your paper can be
published in the proceedings.

###  2.12 Conclusions

Please direct any questions to the production editor in charge of these
proceedings.

Figure 3: For publications with color tables (i.e., publications not offering
color throughout the paper) please observe: for the printed version -- and
ONLY for the printed version -- color figures have to be placed in the last
page.
For the electronic version, which will be converted to PDF before making it
available electronically, the color images should be embedded within the
document. Optionally, other multimedia material may be attached to the
electronic version.