# iGibson 2.0: Object-Centric Simulation for
Robot Learning of Everyday Household Tasks

Chengshu Li*♠, Fei Xia*♡, Roberto Martín-Martín*♠
Michael Lingelbach♣, Sanjana Srivastava♠, Bokui Shen♠, Kent Vainio♠
Cem Gokmen♠, Gokul Dharan♠, Tanish Jain♠, Andrey Kurenkov♠
Karen Liu♠★, Hyowon Gweon♢★, Jiajun Wu♠★, Li Fei-Fei♠★, Silvio Savarese♠★

Department of Computer Science♠, Electrical Engineering♡, Neurosciences IDP♣,
Psychology♢
Institute for Human-Centered AI (HAI)★
Stanford University indicates equal contribution
correspondence to<EMAIL_ADDRESS>
###### Abstract

Recent research in embodied AI has been boosted by the use of simulation
environments to develop and train robot learning approaches. However, the use
of simulation has skewed the attention to tasks that only require what
robotics simulators can simulate: motion and physical contact. We present
iGibson 2.0, an open-source simulation environment that supports the
simulation of a more diverse set of household tasks through three key
innovations. First, iGibson 2.0 supports object states, including temperature,
wetness level, cleanliness level, and toggled and sliced states, necessary to
cover a wider range of tasks. Second, iGibson 2.0 implements a set of
predicate logic functions that map the simulator states to logic states like
Cooked or Soaked. Additionally, given a logic state, iGibson 2.0 can sample
valid physical states that satisfy it. This functionality can generate
potentially infinite instances of tasks with minimal effort from the users.
The sampling mechanism allows our scenes to be more densely populated with
small objects in semantically meaningful locations. Third, iGibson 2.0
includes a virtual reality (VR) interface to immerse humans in its scenes to
collect demonstrations. As a result, we can collect demonstrations from humans
on these new types of tasks, and use them for imitation learning. We evaluate
the new capabilities of iGibson 2.0 to enable robot learning of novel tasks,
in the hope of demonstrating the potential of this new simulator to support
new research in embodied AI. iGibson 2.0 and its new dataset are publicly
available at http://svl.stanford.edu/igibson/.

##  1 Introduction

Recent years, we have seen the emergence of many simulation environments for
robotics and embodied AI research [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. The main
function of these simulators is to compute the motion resulting from the
physical contact-interaction between (rigid) bodies, as this is the main
process that allows robots to navigate and manipulate the environment. This
kinodynamic simulation is sufficient for pick-and-place and rearrangement
tasks [11, 12, 13, 14]; however, as the field advances, researchers are taking
on more diverse and complex tasks that cannot be performed in these
simulators, e.g., household activities that involve changing the temperature
of objects, their dirtiness and wetness levels. There is a need for new
simulation environments that can maintain and update new types of object
states to broaden the diversity of activities that can be studied.

We present iGibson 2.0, an open-source extension of the kinodynamic simulator
iGibson with several novel functionalities. First and foremost, iGibson 2.0
maintains and updates new extended physical states resulting from the
approximation of additional physical processes. These states include not only
kinodynamics (pose, motion, forces), but also object’s temperature, wetness
level, cleanliness level, toggled and sliced state (functional states). These
states have a direct effect on the appearance of the objects, captured by the
high-quality virtual sensor signals rendered by the simulator.

Second, iGibson 2.0 provides a set of logical predicates that can be evaluated
with a single object (e.g. Cooked) or a pair of objects (e.g. InsideOf). These
logical predicates discriminate the continuous physical state maintained by
the simulator into semantically meaningful logical states (e.g. Cooked is True
if the temperature is above a certain threshold). Complementary to the
discriminative functions, iGibson 2.0 implements generative functions that
sample valid simulated physical states based on logical states. Scene
initialization can then be described as a set of logical states that the
simulator can translate into valid physical instances. This enables faster
prototyping and specification of scenes in iGibson 2.0, facilitating the
training of embodied AI agents in diverse instances of the same tasks. We
demonstrate the potential of this generative functionality with a new dataset
of home scenes densely populated with small objects. We generate this new
dataset by applying a set of hand-designed semantic-logic rules to the
original scenes of iGibson 1.0.

Figure 1: Simulating new activities with iGibson 2.0 (Left) iGibson 2.0’s
simulates a set of extended physical states (temperature, functional state,
cleanliness, wetness level) for objects, enabling studying and developing
solutions to new household tasks such as cooking. The full physical state can
be mapped to symbolic representation that facilitates sampling new instances
of tasks. (Right) Humans can provide demonstrations for the new tasks with a
novel virtual reality interface to enable policy learning.

Third, to facilitate the development of new embodied AI solutions to new tasks
in these new scenes, iGibson 2.0 includes a new virtual reality interface (VR)
compatible with the two main commercially available VR systems. All states are
logged during execution and can be replayed deterministically in the
simulator, enabling the generation a posteriori of additional virtual sensor
signals or visualizations of the interactions and the development of imitation
learning solutions.

We evaluate the new functionalities of iGibson 2.0 on six novel tasks for
embodied AI agents, and apply state-of-the-art robot learning algorithms to
solve them. These tasks were not possible before in iGibson or in alternative
simulation environments. Additionally, we evaluate the use of the new iGibson
2.0 VR interface to collect human demonstrations to train an imitation
learning policy for bimanual operations. While the previous version of iGibson
and other simulators provide interfaces to control an agent with a keyboard
and/or a mouse, these interfaces are insufficient for bimanual manipulation.

In summary, iGibson 2.0 presents the following contributions:

  * •

A set of new physical properties, e.g. temperature, wetness and cleanliness
level, maintained and updated by the simulator; and a set of unary and binary
logical predicates that map simulated states to a logical state that have a
direct connection to semantics and language,

  * •

A set of generative functions associated with the logical predicates to sample
valid simulated states from a given logical state, and a new rule-based
mechanism exploiting these functions to populate the iGibson scenes with small
objects placed at semantically meaningful locations to increase realism,

  * •

A novel virtual reality interface that allows humans to collect demonstrations
for robot learning,

We hope that iGibson 2.0 will open new avenues of research and development in
embodied AI, enabling solutions to household activities that have been under-
explored before.

##  2 Related Work

Simulation environments with (mostly) kinodynamic simulation: In the last
years, the robotics and AI communities have presented several impressive
simulation environments and benchmarks: iGibson [3, 1], Habitat AI [5],
AI2Thor (and variants) [15, 6], ThreeDWorld [7], Sapien [4], Robosuite [9],
VirtualHome [16], RLBench [8], MetaWorld [10], and more. They are based in
physics engines such as (py)bullet [17], MuJoCo [18], and Nvidia Physx [19,
20], combined with rendering capabilities and usually enriched with a dataset
of objects and/or scenes to use to develop embodied AI solutions. While these
simulators have fueled research with new possibilities for training, testing
and developing robotic solutions, they have skewed, with few exceptions, the
exploration towards activities related to what they can simulate accurately:
changes in kinematic states of (rigid or flexible) objects, i.e. Rearrangement
tasks [11]. However, many everyday activities require the simulation of other
physical states that can be modified by the agents, like the temperature of
objects and their level of wetness or cleanliness. Recent simulators have
attempted realistic simulation of fluids and flexible materials [19, 21, 22,
23] or extended rigid body simulators to approximate the dynamics of soft
materials [17, 24]. Compared with these simulators, iGibson 2.0 provides a
simple but effective mechanism to simulate the temperature of objects that
change based on proximity to heat sources. It also simulates fluids through a
system of droplets that can be absorbed by objects and change their wetness
level. Despite simpler than the accurate simulation of heat transfer, fluids
and soft materials used by a few other simulators, iGibson 2.0 object-centric
solution leads to realistic robotic behavior and motion in tasks involving
changing temperature, handling liquids, or soaking objects.

Simulation environments with object-centric representation: In robotics, some
simulators have adopted an object-centric representation with extended
physical states, e.g. AI2Thor [15] and VirtualHome [16]. Both are based on
Unity [25] and share a common set of functionalities. Actions in these
simulators are predefined, discrete and symbolic, and characterized by
preconditions (“what conditions need to be fulfilled for this action to be
executable?”) and postconditions (“what conditions will change after the
execution of this action?”), similar to actions in the planning domain
definition language (PDDL [26]) or STRIPS [27] but with the additional link to
visual rendering of the predefined action execution and outcome. In AI2Thor
and VirtualHome, some of the actions change the temperature of an object
between two or more discrete values pre- and post-execution of an action, e.g.
raw and cooked. This is fundamentally different to our approach in iGibson
2.0: instead of maintaining only a symbolic state (raw/cooked), we provide a
simple simulation of the underlying physical process (e.g. heat transfer)
leading to continuously varying values of temperature and other extended
states. These states are then mapped into a symbolic representation through
predicates (see Sec. 4). This provides a new level of detail in the execution
of actions, where the agent can and should control the specific value of the
object’s extended states (temperature, wetness, cleanliness) to achieve a
task, leading to more complex activities and more realistic execution. In the
Appendix, we include a detailed comparison between iGibson 2.0 and other
simulation environments in Table A.7.

Simulation environments with virtual reality interfaces: Researchers have used
virtual reality interfaces before to develop robotic solutions with real
robots [28, 29, 30]. VR has also been used in simulation environments to
collect demos. VRKitchen [31] collected demos for five cooking tasks in
simulation. While realistic looking, activities in VRKitchen are performed
with primitive actions similar to the pre-condition/post-condition system of
AI2Thor and VirtualHome, falling short of realistic motion. More physically
realistic are the VR interactions with UnrealROX/RobotriX [32, 33] and
ThreeDWorld [7], based on Unreal [34] and Nvidia Physx [20]. Our interface
also enables realistic manipulation of objects, with additional features such
as gaze tracking and assistive grasping to bridge the differences between
simulation and the real-world.

##  3 Extended Physical States for Simulation of Everyday Household Tasks

To perform household tasks, an agent needs to change objects’ states beyond
their poses. iGibson 2.0 extends objects with five additional states:
temperature, $T$, wetness level, $w$, cleanliness level (dustiness level, $d$,
or stain level, $s$), toggled state, TS, and sliced state, SS. While some of
these states could be different for different parts of an object, in iGibson
2.0 we simplify their simulation and adopt an object-centric representation:
the simulator maintains a single value of each extended state for every
simulated object (rigid, flexible, or articulated). This simplification is
sufficient to simulate realistically household tasks such as cooking or
cleaning. We assume that the extended properties are latent: agents are not
able to observe them directly. Therefore, iGibson 2.0 implements a mechanism
to change objects’ appearance based on their latent extended states (see Fig.
2), so that visually-guided agents can infer the latent states from sensor
signals.

We further impose in iGibson 2.0 that every simulated object should be an
instance of an existing object category in WordNet [35]. This semantic
structure allows us to associate characteristics to all instances of the same
category [36, 37]. For example, we further simplify the simulation of extended
states by annotating what extended states each category need. Not all object
categories need all five extended states (e.g., the temperature is not
necessary/relevant for non-food categories for most tasks of interest). The
extended states required by each object category are determined by a
crowdsourced annotation procedure in the WordNet hierarchy.

In the following, we explain the details of the five extended states and the
way they are updated in iGibson 2.0. For a full list of object states
(kinematics and extended), see Table A.4.

(a) Object temperature: (From left to right) A stove –toggleable heat source
by proximity– is toggled on (second from left), and starts to heat a nearby
fish. The temperature and appearance of the fish changes due to proximity to
the heat source reaching the temperature to be cooked (third from left).
Additional heat elevates further the fish’s temperature, burning it (fourth
from left). After the stove is toggled off, the fish changes back to room
temperature, keeping the appearance that corresponds to the maximum
temperature reached (most right). The temperature system with heat sources and
sinks, enable visually guided execution of household activities such as
cooking.

(b) Object wetness level: (From left to right) A sink –toggleable droplet
source– is toggled on, and starts to create droplets (second from left).
Droplets can be contained in a receptacle to be poured on other objects (third
from left). An object that can change its wetness level gets in contact with
the droplets and absorbs them, changing its appearance (most right). With the
droplets system, iGibson v2.0 provides a simplified liquid simulation
sufficient to perform common household activities.

Figure 2: Extended object states I): temperature and wetness level

Temperature: To update temperature, iGibson 2.0 needs to approximate the
dynamics of heat transfer from heat sources (hot) or sinks (cold). To that
end, we annotate object categories in the WordNet hierarchy as heat sources or
heat sinks. Heat sources elevate the objects’ temperature over room
temperature (23∘) towards the source’s heating temperature, also annotated per
category; similarly, heat sinks decrease the temperature under room
temperature towards the sink’s cooling temperature. The rate the objects
change their temperature towards the temperature of the source/sink is a
parameter annotated per category with units ∘ C$/$\text{\,}\mathrm{s}$$. We
consider two types of heat source/sink: source/sinks that change the
temperature of objects if they are proximal enough (e.g., a stove) and
source/sinks that change the temperature of objects if they are inside of them
(e.g., a fridge or an oven). Additionally, some of the sources/sinks need to
be toggled on (see Functional State below) before they can change the
temperature of other objects (e.g., a microwave). For each object that can
change temperature, the simulator first evaluates if it fulfills the
conditions to be heated or cooled by a heat source/sink. Assuming that an
object with temperature $T_{o}$ fulfills the conditions of a source/sink with
heating/cooling temperature $T$ and changing rate $r$, the temperature of the
object at each step is updated by
$T_{o}^{t+1}=T_{o}^{t}\left(1+\Delta_{sim}\times r\times(T-T_{o}^{t})\right)$,
where $\Delta_{sim}$ is the simulated time.

iGibson 2.0 maintains also a historical value of the maximum temperature that
each object has reached in the past, $T_{o}^{\text{max}}=\max T_{o}^{t}$ for
$t\in[0,\ldots,t_{\text{now}}]$. This value dictates the appearance of an
object: if the object reached cooking or burning temperature in the past, it
will look cooked or burned, even if its current temperature is low. Fig. 2(a)
depicts the temperature system in action.

Wetness Level: Similar to temperature, iGibson 2.0 maintains the level of
wetness for each object that can get soaked. This level corresponds to the
number of droplets that have been absorbed by the object. In iGibson 2.0, the
system of droplets approximates liquid/fluid simulation. Specifically,
droplets are small particles of liquid that are created in droplet sources
(e.g. faucets), destroyed by droplet sinks (e.g. sinks), and absorbed by
soakable objects (e.g. towels). They can also be contained in receptacles
(e.g. cups) and poured later, leading to realistic behavior for the simulation
of several household activities involving liquids, illustrated in Fig. 2(b).

(a) Object cleanliness level: (From left to right) An object is initialized
with dust particles that can be cleaned with a cloth; Another object is
initialized with stains that require a wet tool (a scrubber) to be removed;
Realistic behaviors and motion are required in iGibson v2.0 to change the
cleanliness level of the objects.

(b) Object toggling state: An object with the toggled state, an oven, is
initially off. When it is toggled on, its appearance changes indicating the
transition.

(c) Object slice state: An object with the sliced state, a peach, is sliced
after exerting enough force with the right part of a slicing tool (the sharp
edge of a knife)

Figure 3: Extended object states II): cleanliness level (dustiness, stains),
toggling, slice state

Cleanliness – Dustiness and Stain Level: A common task for robots in homes and
offices is to clean dirt. This dirt commonly appears in the form of dust or
stains. In iGibson 2.0, the main difference between dust and stains is the way
they get cleaned: while dust can be cleaned with a dry cleaning tool like a
cleaning cloth, stains can only be cleaned with a soaked cleaning tool like a
scrubber. To clean a particle of dirt (dust or stain), the right part of a
cleaning tool should get in physical contact with the particle. Once a dirt
particle is cleaned, it disappears from objects’ surface.

In iGibson 2.0, objects can be initialized with visible dust or stain
particles on its surface. The number of particles at initialization
corresponds to a 100% level of dustiness, $d$, or stains, $s$, as we assume
that dust/stain particles cannot be generated after initialization. As
particles are cleaned, the level decreases proportionally to the number of
particles removed, reaching a level of 0% dustiness, $d$, or stains when the
object is completely clean (no particles left). This extended state allows
simulating multiple cleaning tasks in our simulator: the agent needs to
exhibit a behavior (motion, use of tools) similar to the one necessary in the
real world. Fig. 3(a) depicts an example of the cleanliness level simulation
in iGibson 2.0, for both dust and stain particles.

Toggled State: Some object categories in iGibson 2.0 can be toggled on and
off. iGibson 2.0 maintains and updates an internal binary functional state for
those objects. The functional state can affect the appearance of an object,
but also activate/deactivate other processes, e.g. heating food inside a
microwave requires the microwave to be toggled on. To toggle the object, a
certain area needs to be touched. For object models of categories that can be
toggled on/off, we annotate a TogglingLink, an additional virtual fixed link
that needs to be touched by the agent to change the toggled state. Fig. 3(b)
depicts an example of an object that can change its toggled state, an oven.

Sliced State: Many cooking activities require the agent to slice objects, e.g.
food items. Slicing is challenging in simulation environments where objects
are assumed to have a fixed (rigid or flexible) 3D structure of vertices and
faces. To approximate the effect of slicing, iGibson 2.0 maintains and updates
a sliced state in instances of object categories that are annotated as
sliceable. When the sliced state transitions to True, the simulator replaces
the whole object with two halves. The two halves will be placed at the same
location and inherit the extended states from the whole object (e.g.
temperature). The transition is not reversible: the object will remain sliced
for all upcoming simulated time steps. Objects can only be sliced into two
halves, with no further division. The sliced state changes when the object is
contacted with enough force (over a slicing force threshold for the object) by
a slicing tool, e.g. a knife. Objects of these categories are annotated as
SlicingTool. If an object is a slicing tool, it will undergo a second
annotation process to obtain a new virtual fixed link that acts as
SlicingLink, the part of the slicing tool that can slice an object, e.g., the
sharp edge of a knife. Fig. 3(c) depicts an example of a peach being sliced by
a knife. For more information about update rules for all extended physical
states, please refer to see Sec. A.2.

##  4 Logical Representation of Physical States

The new extended object states from iGibson 2.0 are sufficient to simulate a
new set of household activities in indoor environments. However, there is a
semantic gap between the extended states (e.g. temperature or wetness level)
and the natural description of activities in a household setup (e.g. cooking
apples). To bridge this gap, we define a set of functions that map the
extended object states to logical states for single objects and pairs of
objects. The logical states are semantically grounded on common natural
language representing properties such as cooked or dusty.

The list of logical predicates covers kinematic states between pair of objects
(InsideOf, OnTopOf, NextTo, InContactWith, Under, OnFloor), states related to
the internal degrees of freedom of articulated objects (Open), states based on
the object temperature (Cooked, Burnt, Frozen), wetness and cleanliness level
(Soaked, Dusty, Stained), and functional state (ToggledOn, Sliced). A complete
list of the logic predicates with detailed explanation is included in Table
A.1. They allow iGibson 2.0 to map a physical simulated state into an
corresponding logical state.

###  4.1 Generative System based on Logical Predicates

Logical predicates map multiple physically simulated states to the same
logical state, e.g., all relative poses between to objects that correspond to
being onTop. In addition to this discriminative role, we include
functionalities in iGibson 2.0 to use logical predicates in a generative
manner, to describe initial states symbolically that can be used to initialize
the simulator. iGibson 2.0 includes a sampling mechanism to create valid
instances of tasks described with logical predicates. This mechanism
facilitates the creation of multiple semantically meaningful initial states,
without the laborious process of manually annotating the initial distributions
per scene.

The process of sampling valid object states is different depending on the
nature of the logical predicate. For predicates based on objects’ extended
states such as Cooked, Frozen or ToggledOn, we just sample values of the
extended states that satisfy the predicate’s requirements, e.g., a temperature
below the annotated freezing point of an object to fullfil the predicate
Frozen. Particles for Dusty and Stained are sampled on the surface of an
object following a pseudo-random procedure. Generating initial states to
fulfill kinematic predicates such as OnTopOf or Inside is a more complex
procedure as the underlying physical state (the object pose) must lead to a
stationary state (e.g., not falling) that does not cause penetration between
objects. Each kinematic predicate is implemented differently, combining
mechanisms that include ray-casting and analytical methods to verify the
validity of sampled poses. For example, to sample a state that satisfies
Inside(A, B), we implement a procedure that generates 6D poses inside of
object B and we evaluate that 1) a bounding box of the the size of object A
does not penetrate object B and rays cast from A intersect object B from
evaluated by casting rays from the the internal box. For more details on the
sampling mechanism, see Sec. A.2.

###  4.2 iGibson 2.0 Scenes with Realistic Object Distribution Created by
Generative System

Figure 4: Densely populated ecological scenes from the iGibson 2.0 dataset:
Objects are sampled in semantically meaningful locations using a set of
semantic rules and the generative system based on logical statements,
increasing realism for robot learning and VR experience.

One common issue that limits the realism of indoor scenes in simulation is
that they are less densely populated than those in the real world. Creating
highly populated simulated houses is usually a laborious process that requires
manually selecting and placing models of small objects in different locations.
Thanks to the generative system in iGibson 2.0, this process can be extremely
simplified. The users only need to specify a list of logical predicates that
represent a realistic distribution of objects in a house.

We provide as part of iGibson 2.0 a set of semantic rules to generate more
populated scenes, and a new version of the iGibson original 15 fully
interactive scenes, populated with additional small objects as a result of the
application of the rules. Given an indoor scene with multiple types of rooms
(kitchen, bathroom, …) that are populated with furniture containers and
appliances (fridge, cabinet, …), the semantic rules define the probabilities
for object instances of diverse categories to be sampled in certain container
type in a given room type, e.g. $p($InsideOf(Beer,
Fridge)$\land$InsideOf(Fridge, Kitchen)$)$. To generate the more densely
populated versions of the 15 scenes, we first collect a large number of small
3D object models created by artists and we annotate them with semantic
categories (e.g. cereal, apple, bowl) and realistic dimensions. Then, we apply
in a random sequence the logic predicates using the generative system
explained in Sec. 4.1, increasing the number of objects in the scene by over
100. The result is depicted in Fig. 4.

##  5 Virtual Reality Interface

iGibson 2.0’s new functionalities enable modeling new household activities and
generating multiple instances in more densely populated scenes. To facilitate
research in these new, complex tasks, iGibson 2.0 includes a novel virtual
reality (VR) interface compatible with major commercially available VR headset
through OpenVR [38]. One of the goals is to allow researchers to collect human
demonstrations, and use them to develop new solutions via imitation (see Sec.
6).

iGibson 2.0’s VR interface creates an immersive experience: humans embody an
avatar in the same scene and for the same task as the AI agents. The virtual
reality avatar (see Fig. 1) is composed of a main body, two hands and a head.
The human controls the motion of the head and the two hands via the VR headset
and hand controllers with an optional, additional tracker for control of the
main body. Humans receive stereo images as generated from the point of view of
the head of the virtual avatar, at at least 30 fps (up to 90 fps) using the
PBR rendering functionalities of iGibson [1].

Grasping in VR: Grasping in the real-world, while natural to adult humans, is
a complex experience that proves difficult to reproduce with virtual reality
controllers. Our empirical observations revealed that while using solely
physical grasping, user dexterity was significantly impaired relative to the
real-world resulting in unnatural behavior when manipulating in VR. To provide
a more natural grasping experience, we implement an assistive grasp (AG)
mechanism that enables an additional constraint between the palm and a target
object after the user passes a grasp threshold (50% actuation) and provided
the object is in contact with the hand, between the fingers and the palm. This
facilitates grasping of small objects, and prevents object slippage. To not
render grasping artificially trivial, the AG connection can break if the
constraint is violated beyond a set threshold, such as while lifting heavy
objects or during intense acceleration, encouraging natural task execution
that leverages careful motions and bimanual manipulation. Please refer to Sec.
A.1 for additional details about AG.

Navigating in VR: Navigation of the avatar is controlled by the locomotion of
the human. However, the VR space is much smaller than the typical size of
iGibson 2.0 scenes. To navigate between rooms, we configured a touchpad in the
hand controller that humans can use to translate the avatar.

##  6 Evaluation

In our evaluation, we test the new functionalities of iGibson 2.0 explained
above and that sets it apart from other existing environments. First, we
create a set of tasks that showcase the new extended states (see Fig. 5(a)) as
their modification is required to achieve the tasks. In our experiments, we
make use of our discriminative and generative logical engine to detect task
completion and create multiple instances of each task for training. Then, we
use our novel VR interface to collect human demonstrations to train an
imitation learning policy for a bimanual task.

(a) Six evaluated tasks: (top) 1) Grasping Book, 2) Soaking Towel, 3) Cleaning
Stained Shelf; (bottom) 4) Cooking Meat, 5) Slicing fruit, 6) Bimanual Pick
and Place.

(b) Success rate of IL in Bimanual Pick and Place

Figure 5: Evaluation: We create six novel household tasks in iGibson 2.0 where
the agents have to manipulate extended states. These tasks cannot be studied
in other simulation environments.

##### Experimental Setup:

To evaluate and demonstrate the new capabilities of iGibson 2.0, we create the
following six novel tasks for embodied AI agents: 1) Grasping Book: the agent
has to grasp a book OnTopOf a table and lift it. This task demonstrates the
kinematic logical states; 2) Soaking Towel: the agent has to soak a cleaning
tool (a towel) with water droplets from a sink. This task demonstrates the
liquid/droplets system. 3) Cleaning Stained Shelf: the agent has to clean a
stained shelf using a cleaning tool. This task showcases the capabilities to
update the cleanliness level of an object. 4) Cooking Meat: the agent has to
cook a piece of meat by placing it on a heat source (a burning stove) and
waiting for enough time for the temperature to rise. This task showcases the
capabilities to update the temperature of an object. 5) Slicing Fruit: the
agent has to slice a fruit by exerting enough force with the SlicingLink of a
slicing tool (a knife). This task showcases the capabilities to simulate
sliceable objects and to model the interaction between cutting tools and
sliceable objects. 6) Bimanual Pick and Place: The agent has to pick up a
heavy object (a cauldron) and place it OnTopOf a table. Manipulating this
object requires bimanual interaction as its mass is $50\text{\,}\mathrm{kg}$
and each hand can only exert at most $300\text{\,}\mathrm{N}$. Demonstrating
bimanual manipulation is natural and easy with our novel VR interface, which
is not the case with previous interfaces such as keyboard [1].

We conduct two sets of experiments to evaluate the current robot learning
algorithms on these tasks. First, we train agents with a state-of-the-art
reinforcement learning (RL) algorithm, Soft-Actor Critic (SAC [39]). For
embodiment, we use a bimanual humanoid robot (the one used in VR) and a Fetch
robot. The agent receives RGB-D images from its onboard sensors and
proprioception information as observation and outputs the desired linear and
angular velocities of the right hand (assuming the rest of the agent is
stationary). We adopt the “sticky mitten” simplification [11] for grasping
that creates a fixed constraint between the hand and the object when they get
into contact. We also conduct experiments that remove such simplification for
the Fetch robot, in which case the action space includes one additional DoF
for grasping. Second, we train agents with a standard imitation learning (IL)
algorithm, behavioral cloning [40] on Bimanual Pick and Place. We collected
$30$ demonstrations, more than $6500$ frames in total
($\sim$$215\text{\,}\mathrm{s}$). The agent receives ground-truth states of
the objects and proprioception information as observation, and outputs the
desired linear and angular velocities for both hands. Additional information
about the experimental setup can be found in Sec. A.3.

RL experiments: With the simplified grasping mechanism, the agents trained
with SAC for both the bimanual humanoid and the Fetch embodiment achieve
$100\%$ success rate for Grasping Book, Soaking Towel, Cleaning Stained Shelf,
Cooking Meat tasks. For Slicing Fruit, the agents achieve only $15\%$ and
$0\%$ for bimanual humanoid and Fetch robot respectively, due to the increased
accuracy necessary to align the knife blade with the fruit. The agent using
bimanual humanoid achieves $0\%$ success in the Bimanual Pick and Place task
because of the difficulties of controlling and coordinating both hands.
Additionally, we evaluate the performance with the Fetch robot in more
realistic conditions, without any simplification for grasping, and observe a
significant drop in performance, achieving $25\%$ success rate for 2 tasks and
$0\%$ for the other 3 (see Fig. A.2 in the Appendix). This indicates that
successful grasping for diverse objects is a significant challenge in these
manipulation tasks. To test generalization, we conducted an ablation study in
which we train policies with three different levels of variability in Soaking
Towel –no variations, different poses, different objects and poses– and
evaluate them on an unseen setup (an unseen object with randomized initial
pose). The policies achieve success rate of $19\%$, $79\%$, and $87\%$,
respectively. This study shows that it’s essential to train with diverse
object models and initial states to obtain robust policies, and the generative
system of iGibson 2.0 facilitates it by specifying a few logical states that
describe the initial scene. The attached video shows the policies performing
the tasks trained using iGibson 2.0’s new extended physical states and logical
predicates that help generating task instances and discriminating their
completion.

IL experiments in Bimanual Pick and Place with VR demonstrations: The trained
policy in the full task diverges and fails, even after training with 30
demonstrations. We hypothesize that the agent suffers from covariate shift
[41], visiting state space not covered by demonstrations, due to the different
strategies demonstrated in VR for this long task (300+ steps). We then
evaluate if the policy can successfully perform the last part of the task. We
take a successful human demonstration and initialize a few seconds before task
completion. We then query the IL policy from this point onward to control the
agent. The results of this experiment are shown in Fig 5(b). The policy
achieves $19\%$ and $46\%$ success rate when starting $6\text{\,}\mathrm{s}$
and $3\text{\,}\mathrm{s}$ away from the goal. This experiment demonstrates
the potential of the new VR interface of iGibson 2.0 for generating
demonstrations for IL. We believe that our new interface will open new avenues
for research in bimanual manipulation, hand-eye coordination, and coordination
of robot base and robot arm.

##  7 Conclusion

We presented iGibson 2.0, an open-source simulation environment for household
tasks with several key novel features: 1) an object-centric representation and
extended object states (e.g. temperature, wetness and cleanliness level), 2)
logical predicates mapping simulation states to logical states, and generative
mechanism to create simulated worlds based on a given logical description, and
3) a virtual reality interface to easily collect human demonstrations for
imitation. We demonstrate in multiple experiments the new avenues for research
enabled by iGibson 2.0. We hope iGibson 2.0 becomes a useful tool for the
community, and facilitates the development of novel embodied AI solutions.

#### Acknowledgments

This work is in part supported by ARMY MURI grant W911NF-15-1-0479 and
Stanford Institute for Human-Centered AI (SUHAI). S. S. is supported by the
National Science Foundation Graduate Research Fellowship Program (NSF GRFP)
and Department of Navy award (N00014-16-1-2127) issued by the Office of Naval
Research. S. S. and C. L. are supported by SUHAI Award # 202521. R. M-M is
supported by SAIL TRI Center – Award # S-2018-28-Savarese-Robot-Learn. F. X.
and B. S. are supported by Qualcomm Innovation Fellowship. M. L. is supported
by Regina Casper Stanford Graduate Fellowship.

## References

  * Shen et al. [2020] B. Shen, F. Xia, C. Li, R. Martín-Martín, L. Fan, G. Wang, S. Buch, C. D’Arpino, S. Srivastava, L. P. Tchapmi, M. E. Tchapmi, K. Vainio, L. Fei-Fei, and S. Savarese.  iGibson, a Simulation Environment for Interactive Tasks in Large Realistic Scenes, 2020.
  * Xia et al. [2018] F. Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and S. Savarese.  Gibson env: Real-world perception for embodied agents.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Xia et al. [2020] F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. Martín-Martín, and S. Savarese.  Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments.  _IEEE Robotics and Automation Letters_ , 5(2):713–720, 2020.
  * Xiang et al. [2020] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, et al.  SAPIEN: A simulated part-based interactive environment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11097–11107, 2020.
  * Manolis Savva* et al. [2019] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra.  Habitat: A Platform for Embodied AI Research.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Ehsani et al. [2021] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, and L. Weihs.  Manipulathor: A framework for visual object manipulation.  _arXiv preprint arXiv:2104.11213v1_ , 2021.
  * Gan et al. [2020] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, et al.  ThreeDWorld: A platform for interactive multi-modal physical simulation.  _arXiv preprint arXiv:2007.04954_ , 2020.
  * James et al. [2020] S. James, Z. Ma, D. Rovick Arrojo, and A. J. Davison.  Rlbench: The robot learning benchmark & learning environment.  _IEEE Robotics and Automation Letters_ , 2020.
  * Zhu et al. [2020] Y. Zhu, J. Wong, A. Mandlekar, and R. Martín-Martín.  robosuite: A modular simulation framework and benchmark for robot learning.  _arXiv preprint arXiv:2009.12293_ , 2020.
  * Yu et al. [2020] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In _Conference on Robot Learning_ , pages 1094–1100. PMLR, 2020.
  * Batra et al. [2020] D. Batra, A. X. Chang, S. Chernova, A. J. Davison, J. Deng, V. Koltun, S. Levine, J. Malik, I. Mordatch, R. Mottaghi, M. Savva, and H. Su.  Rearrangement: A challenge for embodied ai, 2020.
  * Weihs et al. [2021] L. Weihs, M. Deitke, A. Kembhavi, and R. Mottaghi.  Visual room rearrangement.  _arXiv preprint arXiv:2103.16544_ , 2021.
  * Gan et al. [2021] C. Gan, S. Zhou, J. Schwartz, S. Alter, A. Bhandwaldar, D. Gutfreund, D. L. Yamins, J. J. DiCarlo, J. McDermott, A. Torralba, et al.  The threedworld transport challenge: A visually guided task-and-motion planning benchmark for physically realistic embodied ai.  _arXiv preprint arXiv:2103.14025_ , 2021.
  * Liu et al. [2021] Z. Liu, W. Liu, Y. Qin, F. Xiang, S. Xin, M. A. Roa, B. Calli, H. Su, Y. Sun, and P. Tan.  Ocrtoc: A cloud-based competition and benchmark for robotic grasping and manipulation.  _arXiv preprint arXiv:2104.11446v1_ , 2021.
  * Kolve et al. [2017] E. Kolve et al.  AI2-THOR: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_ , 2017.
  * Puig et al. [2018] X. Puig et al.  Virtualhome: Simulating household activities via programs.  In _IEEE CVPR_ , 2018.
  * Coumans and Bai [2016] E. Coumans and Y. Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  2016\.
  * Todorov et al. [2012] E. Todorov, T. Erez, and Y. Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033. IEEE, 2012.
  * Rieffel et al. [2009] J. Rieffel, F. Saunders, S. Nadimpalli, H. Zhou, S. Hassoun, J. Rife, and B. Trimmer.  Evolving soft robotic locomotion in physx.  In _Proceedings of the 11th annual conference companion on genetic and evolutionary computation conference: Late breaking papers_ , pages 2499–2504, 2009.
  * Nvidia Corp. [2021a] Nvidia Corp.  Nvidia Physx.  https://developer.nvidia.com/physx-sdk, 2021a.  Accessed: 2021-06-17.
  * Nvidia Corp. [2021b] Nvidia Corp.  Nvidia FleX.  https://developer.nvidia.com/flex, 2021b.  Accessed: 2021-06-17.
  * Coevoet et al. [2017] E. Coevoet, T. Morales-Bieze, F. Largilliere, Z. Zhang, M. Thieffry, M. Sanz-Lopez, B. Carrez, D. Marchal, O. Goury, J. Dequidt, and C. Duriez.  Software toolkit for modeling, simulation and control of soft robots.  _Advanced Robotics_ , pages 1–26, Nov. 2017.  URL https://hal.inria.fr/hal-01649355.
  * Faure et al. [2012] F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, and S. Cotin.  SOFA: A Multi-Model Framework for Interactive Physical Simulation.  In Y. Payan, editor, _Soft Tissue Biomechanical Modeling for Computer Assisted Surgery_ , volume 11 of _Studies in Mechanobiology, Tissue Engineering and Biomaterials_ , pages 283–321. Springer, June 2012.  doi:10.1007/8415_2012_125.  URL https://hal.inria.fr/hal-00681539.
  * Lin et al. [2020] X. Lin, Y. Wang, J. Olkin, and D. Held.  Softgym: Benchmarking deep reinforcement learning for deformable object manipulation.  In _Conference on Robot Learning_ , 2020.
  * Goldstone [2009] W. Goldstone.  _Unity game development essentials_.  Packt Publishing Ltd, 2009.
  * McDermott et al. [1998] D. McDermott, M. Ghallab, A. Howe, C. Knoblock, A. Ram, M. Veloso, D. Weld, and D. Wilkins.  Pddl - the planning domain definition language.  Technical report, Technical Report 1165, Yale Computer Science, 1998.(CVC Report 98-003), 1998.
  * Fikes and Nilsson [1971] R. E. Fikes and N. J. Nilsson.  Strips: A new approach to the application of theorem proving to problem solving.  _Artificial intelligence_ , 2(3-4):189–208, 1971\.
  * Zhang et al. [2018] T. Zhang, Z. McCarthy, O. Jowl, D. Lee, X. Chen, K. Goldberg, and P. Abbeel.  Deep imitation learning for complex manipulation tasks from virtual reality teleoperation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 1–8. IEEE, 2018.
  * DelPreto et al. [2020] J. DelPreto, J. I. Lipton, L. Sanneman, A. J. Fay, C. Fourie, C. Choi, and D. Rus.  Helping robots learn: A human-robot master-apprentice model using demonstrations via virtual reality teleoperation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 10226–10233. IEEE, 2020.
  * Stramandinoli [2018] F. Stramandinoli.  Robot learning from human demonstration in virtual reality.  In _Human-Robot Interaction_ , 03 2018.
  * Gao et al. [2019] X. Gao, R. Gong, T. Shu, X. Xie, S. Wang, and S.-C. Zhu.  Vrkitchen: an interactive 3d virtual environment for task-oriented learning.  _arXiv preprint arXiv:1903.05757_ , 2019.
  * Martinez-Gonzalez et al. [2018] P. Martinez-Gonzalez, S. Oprea, A. Garcia-Garcia, A. Jover-Alvarez, S. Orts-Escolano, and J. Garcia-Rodriguez.  Unrealrox: An extremely photorealistic virtual reality environment for robotics simulations and synthetic data generation.  _arXiv preprint arXiv:1810.06936_ , 2018.
  * Garcia-Garcia et al. [2018] A. Garcia-Garcia, P. Martinez-Gonzalez, S. Oprea, J. A. Castro-Vargas, S. Orts-Escolano, J. Garcia-Rodriguez, and A. Jover-Alvarez.  The robotrix: An extremely photorealistic and very-large-scale indoor dataset of sequences with robot trajectories and interactions.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6790–6797. IEEE, 2018.
  * Epic Games [2021] Epic Games.  Unreal engine.  https://www.unrealengine.com/en-US/unreal, 2021.
  * Miller [1995] G. A. Miller.  WordNet: a lexical database.  _Communications of the ACM_ , 38(11):39–41, 1995\.
  * Tversky and Hemenway [1984] B. Tversky and K. Hemenway.  Objects, parts, and categories.  _Journal of experimental psychology: General_ , 113(2):169, 1984.
  * Barsalou et al. [1999] L. W. Barsalou et al.  Perceptual symbol systems.  _Behavioral and brain sciences_ , 22(4):577–660, 1999.
  * ValveSoftware [2021] ValveSoftware.  OpenVR.  https://github.com/ValveSoftware/openvr, 2021.  Accessed: 2021-06-17.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ , 2018.
  * Pomerleau [1989] D. A. Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In _Advances in neural information processing systems_ , pages 305–313, 1989.
  * Ross et al. [2011] S. Ross, G. Gordon, and D. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 627–635, 2011.
  * Ehsani et al. [2021] K. Ehsani, W. Han, A. Herrasti, E. VanderBilt, L. Weihs, E. Kolve, A. Kembhavi, and R. Mottaghi.  Manipulathor: A framework for visual object manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4497–4506, 2021.

## Appendix for iGibson 2.0: Object-Centric Simulation for Robot Learning of
Everyday Household Tasks

###  A.1 The iGibson 2.0 Virtual Reality Interface

In this section, we provide additional information about the implementation of
our virtual reality (VR) interface in iGibson 2.0.

Mapping human motion to the virtual embodiment: Humans in VR control a
bimanual embodiment in iGibson 2.0 composed of a head, two hands and a
torso/body. The hardware for VR control is composed of a headset and two hand
controllers, possibly an additional tracker for the human torso that maps
directly to the VR body. At each step, the pose of the agent’s head is
directly set to be the new headset’s pose as provided by the VR hardware. This
step bypasses physics simulation of the head motion to make sure that the
motion of the head in VR corresponds exactly and without delays to that in the
real world to avoid any discomfort. In contrast, both the hands and the main
body move as the result of physical simulation of a body constraint connecting
the simulated hands and body to the new poses provided by the VR hardware. In
other words, the VR hardware provides at each step new “desired” poses for the
hands and body that “pull” the body parts towards them with forces computed by
the simulator. This creates realistic interactions between the hands and body
and the objects in the scene, colliding with them and applying forces. The
constraint to move the hand has a maximal force of $300\text{\,}\mathrm{N}$,
simulating the lifting force of humans. The constraint to move the body is
$50\text{\,}\mathrm{N}$. While the desired new poses for the hands always come
from the hand-controllers, the desired pose for the body can be provided by a
body tracker on the human’s torso or be otherwise estimated based on the
headset position. This constraint-based system strikes a balance between
accurately following the human motion and realistic interactions with the
scene in VR.

Haptic feedback: To approximate the real-world experience, it is important to
create haptic feedback to human subjects when interacting with the scene. To
that end, in our VR interface collisions of the body and the hands trigger
haptic vibrations in the controllers. The body will trigger a strong vibration
in both controllers facilitating navigation in the scene. The hands also
generate low-strength vibration when they are in contact with an object, to
notify users that they are in contact with a virtual object. These mechanisms
create a multi-modal stream to the humans (vision and haptics) that help them
interact more dexterously and realistically.

Assistive Grasping: Creating realistic, robust and dexterous grasping in
virtual reality is challenging. Grasping objects in real-world involves
generating multiple frictional contact points and surfaces between the hand
and the object: simulating physically this complex process in a realistic
manner is non-trivial. Additionally, real-world grasping involves rich
multimodal signals that include tactile and haptic information, that is not
available in common virtual reality interfaces. To compensate for these
differences and generate a natural experience in simulation, we implement an
assistive grasping mechanism.

In our VR interface, grasping is performed by pressing the left or right
trigger, a one degree of freedom (DoF) actuation. This single DoF is mapped to
a closing motion of the avatar’s hand where all fingers move synchronously. If
the trigger is pressed more than 50% of its range, we activate the assistive
grasping (AG) mode. The AG mode facilitates grasping by creating an additional
fixed or point-to-point joint between the hand and the movable objects inside
the hand. We use three criteria to decide what object inside the hand should
be assisted for grasping: First, the object has to be inside the hand. We
evaluate this we a ray casting mechanism. Rays are shot between the following
two sets of points: (thumb tip, thumb middle, palm middle, palm base) and (4
non-thumb finger tips). These 16 rays return a list of all objects that are in
the hand. Second, the object has to be close to the palm. This is defined as
the distance between the center of mass of the palm link and the object is the
smallest among all objects in hand. And third, the object needs to be in
contact with the hand and the hand has to be applying force on it. We found
that, defined with these three criteria, the AG mechanism is realistic as
humans can grasp objects reliably with motions that are close to the ones used
in real-world for the same objects. While a user is grasping an object using
AG, collision of the hand with that object is also disabled, to avoid any
recurring collisions or simulation instability.

The AG breaks the connection between the object and the hand if the grasping
trigger goes under 50% pressed, or the center of mass of object being grasped
moves further away from the palm than a maximum threshold,
$D_{\mathit{AGMax}}$. We use
$D_{\mathit{AGMax}}=$10\text{\,}\mathrm{c}\mathrm{m}$$. This effectively
avoids that humans can use AG to grasp objects or pull from them in an
unrealistic manner, e.g., grasping a heavy object such a watermelon with a
single hand. Our AG system strikes a good balance between facilitating
grasping and interactions of humans in VR and simplifying the manipulation of
objects excessively. In this way, humans in VR still need to move and position
the hands in realistic ways to grasp and manipulate heavy objects. The AG mode
is also critical to grasp small objects with dexterity. In the VR
demonstrations created by the human, we can see that the AG mechanism works
robustly for all kind of rigid and articulated objects, ranging from fridge
handles to knives, from watermelons to strawberries. Please refer to the
supplementary video to see examples of AG mechanism.

Hardware compatibility: We have tested our iGibson 2.0 with the three main
commercially available headsets at the time, HTC Vive (Pro Eye), Oculus Rift S
and Oculus Quest. For the former, we include functionalities to observe and
record the eye gaze tracked by the device. To serve both devices, the iGibson
renderer needs to output 1296 x 1440 images at $30\text{\,}\mathrm{Hz}$, shown
to the human to create a immersive 3D experience. The system is capable of
rendering at up to $90\text{\,}\mathrm{Hz}$, but we had to reduce the frame
rate to accommodate additional expensive per-frame computations, including
physics and extended states update steps in our simulator.

Logging and replaying demonstrations: All information of the runs can be
logged and replay deterministically (same output for the same actions). The
logged information includes kinematics and extended iGibson 2.0 states, and VR
agent actions. We believe the logged information acquired with the iGibson 2.0
VR interface will facilitate research: the information can be analyzed to
understand human strategies, or used with modern robot learning techniques,
e.g. with imitation learning, to train embodied AI solutions.

###  A.2 Extended Object States, Logical Predicates and Generative System in
iGibson 2.0

In this section, we provide additional information on iGibson 2.0’s new
extended physical states, logical predicate system that map physical states to
logical states, and the generative system that sample valid simulated physical
states based on logical states.

Extended states associated to object categories: In iGibson 2.0, not all
extended states need to be maintained for instances of all object categories,
e.g., most objects cannot be Sliced and only food objects could be Cooked. We
assume that any object instance added to iGibson 2.0 belongs to a category
annotated with properties. The properties indicate what extended states should
be updated for object instances of that category. The exhaustive list of all
possible object category properties is shown in Table A.2. Some properties
will require additional annotation for each object model to estimate logical
states.

Object model annotations: Each model needs to be annotated with additional
physical and semantic information to simulate correctly interactions and their
associated logical states. The exhaustive list of all possible object model
properties are included in Table A.3. Some properties directly come from the
3D assets, such as Shape and KinematicStructure. We compute Weight based on
query results from Amazon Product API, and compute CenterOfMass and
MomentOfInertia accordingly for each link based on the assumption of uniform
density. Note that if an object category is annotated with a certain property,
e.g., stove is annotated as HeatSourceSink, we need to annotate
HeatSourceSinkLink, a virtual (non-colliding) fixed link that heats or cools
objects, for all object models of stove. For SlicingTool and CleaningTool, we
additionally annotate SlicingToolLink and CleaningToolLink, which are
colliding fixed links that can slice objects (the blade of a knife) and remove
dirt particles from objects (the bottom of a vacuum), respectively.

Updating object state: During simulation, our simulator maintains and updates
not only the kinematic states of the objects, such as Pose and InContactObjs,
using our underlying physics engine Bullet [17], but also the non-kinematic
states, such as Temperature and WetnessLevel with custom rules. These update
rules are explained in Sec. 3 and summarized in Table A.4.

Logical predicates as discriminative functions: As explained in Sec. 4, we
define a set of discriminative functions that map the extended physical states
to logical states that are semantically grounded on natural language, such as
Cooked and Sliced. These logical states can used for symbolic planning and
checking intermediate success for sub-tasks for reinforcement learning. The
details of the discriminative functions of all the logical predicates can be
found in Table A.1.

Logical predicates as generative functions: We also define a set of sampling
functions that can generate valid physical states that satisfy the given
logical states. For example, if the initial conditions of the task require a
book placed OnTopOf a table or a shelf being Stained, our system can
automatically sample concrete physical states that satisfy the requirements:
sampling a random position on the table and place the book there, and sample
stain particles on random locations on the shelf (see Fig. 5(a)). The details
of the generative functions of all the logical predicates can be found in
Table A.5. Please also refer to our supplementary video for more details.

Sampling extended states based on the given logical predicates is relatively
simple, e.g., sampling a temperature that corresponds to an object being
Frozen. However, sampling object poses to fulfill the given kinematic
predicates is more involved as it requires sampling values in the Special
Euclidean group SE(3) with additional constraints such as placing objects in
stable configurations and not causing penetrations between objects. In the
following, we describe our algorithm to sample valid poses based on kinematic
logical predicates.

##### Pose Sampling Algorithm:

Say we are sampling a valid pose for an object $o_{1}$ to be OnTopOf object
$o_{2}$. First, we query the set of stable orientations allowed for object
$o_{1}$. We assume these orientations are provided per object model, e.g., for
a book the orientations to place the book on its cover and last page or
upright. Each stable orientation is linked to an axis-aligned bounding box
with an associated bounding-box base area. The next step would be to find
areas on the surface of object $o_{2}$ that can hold the bounding-box area and
that are flat, unobstructed, and accessible. To find these areas of $o_{2}$
surface we use a ray-casting mechanism conditioned on the specific kinematic
logical predicate. For example, for our case of OnTopOf we will generate rays
starting immediately above $o_{2}$ by sampling points from the top face of its
axis-aligned bounding box, and marching downwards in the vertical direction.
The points where the rays intersect $o_{2}$ surface will be used to define
planes where we can attempt to sample object $o_{1}$ if they fulfill some
criteria such as providing stable support. We can repeat the procedure for
different stable orientations of $o_{1}$. Other logical predicates use a
similar generative procedure but with variations in the ray-tracing step. For
example, for InsideOf, we start our rays at different points inside the
$o_{2}$ bounding box rather than above it. In addition, for particle-based
states such as Dusty and Stained, we additionally allow casting rays in
horizontal directions.

Predicate |  Description
---|---
InsideOf($o_{1}$,$o_{2}$) |  Object $o_{1}$ is inside of object $o_{2}$ if we can find two orthogonal axes crossing at $o_{1}$ center of mass that intersect $o_{2}$ collision mesh in both directions.
OnTopOf($o_{1}$,$o_{2}$) |  Object $o_{1}$ is on top of object $o_{2}$ if $o_{2}\in\texttt{InSameNegativeVerticalAxisObjs}(o_{1})\land o_{2}\not\in\texttt{InSamePositiveVerticalAxisObjs}(o_{1})\land\texttt{InContactWith}(o_{1},o_{2})$, where $\texttt{InSamePositive/NegativeVerticalAxisObjs}(o_{1})$ is the list of objects in the same positive/negative vertical axis as $o_{1}$ and $\texttt{InContactWith}(o_{1},o_{2})$ is whether the two objects are in physical contact.
NextTo($o_{1}$,$o_{2}$) |  Object $o_{1}$ is next to object $o_{2}$ if $o_{2}\in\texttt{InSameHorizontalPlaneObjs}(o_{1})\land l2(o_{1},o_{2})<t_{\mathit{NextTo}}$, where $\texttt{InSameHorizontalPlaneObjs}(o_{1})$ is a list of objects in the same horizontal plane as $o_{1}$, $l2$ is the L2 distance between the closest points of the two objects, and $t_{\mathit{NextTo}}$ is a distance threshold that is proportional to the average size of the two objects.
InContactWith($o_{1}$,$o_{2}$) |  Object $o_{1}$ is in contact with $o_{2}$ if their surfaces are in contact in at least one point, i.e., $o_{2}\in\texttt{InContactObjs}(o_{1})$.
Under($o_{1}$,$o_{2}$) |  Object $o_{1}$ is under object $o_{2}$ if $o_{2}\in\texttt{InSamePositiveVerticalAxisObjs}(o_{1})$ $\land o_{2}\not\in\texttt{InSameNegativeVerticalAxisObjs}(o_{1})$.
OnFloor($o_{1}$,$o_{2}$) |  Object $o_{1}$ is on the room floor $o_{2}$ if $\texttt{InContactWith}(o_{1},o_{2})$ and $o_{2}$ is of Room type.
Open(o) |  Any joints (internal articulated degrees of freedom) of object $o$ are open. Only joints that are relevant to consider an object Open are used in the predicate computation, e.g. the door of a microwave but not the buttons and controls. To select the relevant joints, object models of categories that can be Open undergo an additional annotation that produces a RelevantJoints list. A joint is considered open if its joint state $q$ is 5% over the lower limit, i.e. $q>0.05(q_{\mathit{UpperLimit}}-q_{\mathit{LowerLimit}})+q_{\mathit{LowerLimit}}$.
Cooked(o) |  The temperature of object $o$ was over the cooked threshold, $T_{\mathit{cooked}}$, and under the burnt threshold, $T_{\mathit{burnt}}$, at least once in the history of the simulation episode, i.e., $T_{\mathit{cooked}}\leq T_{o}^{\mathit{max}}<T_{\mathit{burnt}}$. We annotate the cooked temperature $T_{\mathit{cooked}}$ for each object category that can be Cooked.
Burnt(o) |  The temperature of object $o$ was over the burnt threshold, $T_{\mathit{burnt}}$, at least once in the history of the simulation episode, i.e., $T_{o}^{\mathit{max}}\geq T_{\mathit{burnt}}$. We annotate the burnt temperature $T_{\mathit{burnt}}$ for each object category that can be Burnt.
Frozen(o) |  The temperature of object $o$ is under the freezing threshold, $T_{\mathit{frozen}}$, i.e., $T_{o}\leq T_{\mathit{frozen}}$. We assume as default freezing temperature $T_{\mathit{frozen}}=0^{\circ}C$, a value that can be adapted per object category and model.
Soaked(o) |  The wetness level $w$ of the object $o$ is over a threshold, $w_{\mathit{soaked}}$, i.e., $w\geq w_{\mathit{soaked}}$. The default value for the threshold is $w_{\mathit{soaked}}=1$, (the object is soaked if it absorbs one or more droplets), a value that can be adapted per object category and model.
Dusty(o) |  The dustiness level $d$ of the object $o$ is over a threshold, $d_{\mathit{dusty}}$, i.e., $d>d_{\mathit{dusty}}$. The default value for the threshold is $d_{\mathit{dusty}}=0.5$, (half of the dust particles have been cleaned), a value that can be adapted per object category and model.
Stained(o) |  The stain level $s$ of the object $o$ is over a threshold, $s_{\mathit{stained}}$, i.e., $s>s_{\mathit{stained}}$. The default value for the threshold is $s_{\mathit{stained}}=0.5$, (half of the stain particles have been cleaned), a value that can be adapted per object category and model.
ToggledOn(o) |  Object $o$ is toggled on or off. It is a direct query of the iGibson 2.0 objects’ extended state $\mathit{TS}$, the toggled state.
Sliced(o) |  Object $o$ is sliced or not. It is a direct access of the iGibson 2.0 objects’ extended state $\mathit{SS}$, the sliced state.
InFoVOfAgent(o) |  Object $o$ is in the field of view of the agent, i.e., at least one pixel of the image acquired by the agent’s onboard sensors corresponds to the surface of $o$.
InHandOfAgent(o) |  Object $o$ is grasped by the agent’s hands (i.e. assistive grasping is activated on that object).
InReachOfAgent(o) |  Object $o$ is within $d_{\mathit{reach}}=2$ meters away from the agent.
InSameRoomAsAgent(o) |  Object $o$ is located in the same room as the agent.
Table A.1: Logical Predicates: Description of the discriminative functions Property of an object category |  Required extended object states
---|---
Can be Cooked |  MaxTemperature, Temperature
Can be Burnt |  MaxTemperature, Temperature
Can be Frozen |  Temperature
Can be Soaked |  WetnessLevel
Can be Dusty |  DustinessLevel
Can be Stained |  StainLevel
Can be ToggledOn |  ToggledState
Can be Sliced |  SlicedState
Is a HeatSourceSink |  ToggledState
Is a DropletSource |  ToggledState
Table A.2: Extended states associated to properties of object categories Object Model Property |  Must be defined if the object category… |  Description
---|---|---
Shape |  |  Model of the 3D shape of each link of the object
Weight |  |  Weight of the object
CenterOfMass |  |  Mean position of the matter in the object
MomentOfInertia |  |  Resistance of the object to change its angular velocity
KinematicStructure |  |  Structure of links and joints connecting them in the form of URDF (non-articulated objects are composed of one link)
StableOrientations |  |  A list of stable orientations assuming the object is placed on a flat surface, computed using a 3D geometry library
HeatSourceSinkLink |  Is a HeatSourceSink |  Virtual (non-colliding) fixed link that generates/absorbs heat
CleaningToolLink |  Is a CleaningTool |  Fixed link that needs to contact dirt particles for the tool to clean them
DropletSourceLink |  Is a DropletSource |  Virtual (non-colliding) fixed link that generates droplets
DropletSinkLink |  Is a DropletSink |  Virtual (non-colliding) fixed link that absorbs droplets
TogglingLink |  Can be ToggledOn |  Virtual (non-colliding) fixed link that changes the toggled state of the object when contacted
SlicingLink |  Is a SlicingTool |  Fixed link that changes the sliced state of another object if it contacts it with enough force
RelevantJoints |  Can be Open |  List of joints that are relevant to indicate whether an object is open
Table A.3: Non-updatable object model properties (annotated) Object State |  Description and Update Rules
---|---
Pose |  6 DoF pose (position and orientation) of the object in world reference frame, updated by the underlying physics engine.
AABB |  Axis-aligned bounding box (coordinates of two opposite corners) of the object in the world reference frame, updated by the underlying physics engine.
JointStates |  State of all internal DoFs of the (articulated) object for the structure defined by KinematicStructure, updated by the underlying physics engine.
InContactObjs |  List of all objects in physical contact with the object, updated by the underlying physics engine.
InSamePositiveVerticalAxisObjs |  List of all objects in the positive vertical axis drawn from the object’s center of mass, updated by shooting a ray upwards in the positive z-axis and gather the objects hit by the ray.
InSameNegativeVerticalAxisObjs |  List of all objects in the negative vertical axis drawn from the object’s center of mass, updated by shooting a ray downwards in the negative z-axis and gather the objects hit by the ray.
InSameHorizontalPlaneObjs |  List of all objects in the horizontal plane drawn from the object’s center of mass, updated by shooting a number of ray in the x-y plane and gather the objects hit by the rays.
Temperature, $T$ |  Object’s current temperature in ∘C, updated by detecting if the object is affected by any heat source or heat sink.
MaxTemperature, $T_{\textit{max}}$ |  Maximum temperature of the object reached historically during this simulation run, updated by keeping track of all the Temperature in the history.
WetnessLevel, $w$ |  Amount of liquid absorbed by the object corresponding to the number of liquid droplets contacted, updated by detecting if the object is in contact with any liquid droplets.
DustinessLevel, $d$ |  Fraction of the initial amount of dust particles that remain on the object’s surface, updated by detecting if the particles are in contact with any CleaningTool.
StainLevel, $s$ |  Fraction of the initial amount of stain particles that remain on the object’s surface, updated by detecting if the particles are in contact with any Soaked CleaningTool.
ToggledState, TS |  Binary state indicating if the object is currently on or off, updated by detecting if the agent is in contact with the TogglingLink.
SlicedState, SS |  Binary state indicating whether the object has been sliced (irreversible), updated by detecting if the object is in contact with any CleaningTool that exerts a force above a certain threshold $F_{\mathit{sliced}}$. We assume as default force threshold of $F_{\mathit{sliced}}=$10\text{\,}\mathrm{N}$$, a value that can be configured per object category and model.
Table A.4: Object states maintained by iGibson 2.0  Predicate |  Sampling Mechanism
---|---
InsideOf($o_{1}$,$o_{2}$) |  Only InsideOf($o_{1}$,$o_{2}$) = True can be sampled. $o_{1}$ is randomly sampled within $o_{2}$ using the mechanism described in Pose Sampling Algorithm in Sec. A.2. $o_{1}$ is guaranteed to be supported fully by a surface and free of collisions with any other object except $o_{2}$.
OnTopOf($o_{1}$,$o_{2}$) |  Only OnTopOf($o_{1}$,$o_{2}$) = True can be sampled. $o_{1}$ is randomly sampled on top of $o_{2}$ using the mechanism described in Pose Sampling Algorithm in Sec. A.2. $o_{1}$ is guaranteed to be supported fully by a surface and free of collisions with any other object except $o_{2}$.
NextTo($o_{1}$,$o_{2}$) |  Not supported at the moment.
InContactWith($o_{1}$,$o_{2}$) |  Not supported at the moment.
Under($o_{1}$,$o_{2}$) |  Only Under($o_{1}$,$o_{2}$) = True can be sampled. $o_{1}$ is randomly sampled on top of the floor region beneath $o_{2}$ using the mechanism described in Pose Sampling Algorithm in Sec. A.2. $o_{1}$ is guaranteed to be supported fully by a surface and free of collisions with any other object except the floor.
OnFloor($o_{1}$,$o_{2}$) |  Only OnFloor($o_{1}$,$o_{2}$) = True can be sampled. $o_{1}$ is randomly sampled on top of $o_{2}$, which is the floor of a certain room, using the scene’s room segmentation mask. $o_{1}$ is guaranteed to be supported fully by a surface and free of collisions with any other object except $o_{2}$.
Open(o) |  To sample an object $o$ with the predicate Open(o) = True, a subset of the object’s relevant joints (using the RelevantJoints model property) are selected, and each selected joint is moved to a uniformly random position between the openness threshold and the joint’s upper limit. To sample an object $o$ with the predicate Open(o) = False, all of the object’s relevant joints (using the RelevantJoints model property) are moved to a uniformly random position between the joint’s lower limit and the openness threshold.
Cooked(o) |  To sample an object $o$ with the predicate Cooked(o) = True, the object’s MaxTemperature is updated to $\max(T_{o}^{\mathit{max}},T_{\mathit{cooked}})$. Similarly, to sample an object $o$ with the predicate Cooked(o) = False, the object’s MaxTemperature is updated to $\min(T_{o}^{\mathit{max}},T_{\mathit{cooked}}-1)$.
Burnt(o) |  To sample an object $o$ with the predicate Burnt(o) = True, the object’s MaxTemperature is updated to $\max(T_{o}^{\mathit{max}},T_{\mathit{burnt}})$. Similarly, to sample an object $o$ with the predicate Cooked(o) = False, the object’s MaxTemperature is updated to $\min(T_{o}^{\mathit{max}},T_{\mathit{burnt}}-1)$.
Frozen(o) |  To sample an object $o$ with the predicate Frozen(o) = True, the object’s Temperature is updated to a uniformly random temperature between $T_{\mathit{frozen}}-10$ and $T_{\mathit{frozen}}-50$. To sample an object $o$ with the predicate Frozen(o) = False, the object’s Temperature is updated to $T_{\mathit{frozen}}+1$.
Soaked(o) |  To sample an object $o$ with the predicate Soaked(o) = True, the object’s WetnessLevel $w$ is updated to match the Soaked threshold of $w_{\mathit{soaked}}$. To sample an object $o$ with the predicate Soaked(o) = False, the object’s WetnessLevel $w$ is updated to 0.
Dusty(o) |  To sample an object with Dusty(o) = True, a fixed number (currently 20) of dust particles are randomly placed on the surface of $o$ using the mechanism described in Pose Sampling Algorithm in Sec. A.2. To sample an object with Dusty(o) = False, all dust particles on the object are removed.
Stained(o) |  To sample an object with Stained(o) = True, a fixed number (currently 20) of stain particles are randomly placed on the surface of $o$ using the mechanism described in Pose Sampling ALgorithm in Sec. A.2. To sample an object with Stained(o) = False, all stain particles on the object are removed.
ToggledOn(o) |  The ToggledState of the object is updated to match the required predicate value.
Sliced(o) |  The SlicedState of the object is updated to match the required predicate value. Also, the whole object are replaced with the two halves, that will be placed at the same location and inherit the extended states from the whole object (e.g. Temperature).
Table A.5: Logical Predicates: Description of the generative functions

###  A.3 Experimental Setup and Additional Results

In this section we provide the experimental setup for the reinforcement
learning and imitation learning experiments described in Sec. 6.

##### Reinforcement Learning Experiments with Bimanual Humanoid Robot:

The observation space include $128\times 128$ RGB-D images from the onboard
sensor on the agent’s head, and proprioceptive information (hand poses in
agent’s local frame, and a fraction indicating how much each hand is closed).
The action space is 6-dimensional representing the desired linear and angular
velocities of the right hand, where the rest of the agent is stationary. For
grasping, we adopt the “sticky mitten” simplification from other works [11]:
we create a fixed constraint between the hand and the object as soon as they
get in contact.

The agent receives a one-time success reward if it satisfies the single
predicate (e.g. Cooked(meat)). Additionally, we provide distance-based reward
shaping for each experiment to encourage the hand to approach activity-
relevant objects, e.g. encourage the hand to approach the meat and the meat to
approach to stove. Finally, for the Cleaning Stained Shelf task, we provide
partial progress reward for each stain particle that has been cleaned. The
episode terminates if the agent achieves success or times out (200 timesteps,
or equivalently 20 seconds). We train for 10K episodes, evaluate on the same
setups, and report the results. The training reward curves can be found in
Fig. A.1.

We use Soft Actor-Critic [39] for training. The policy network has two
encoders for RGB-D images and proprioceptive information. With RGB-D images as
input, we use a 3-layer convolutional neural network to encode the image into
a 256 dimensional vector. The proprioceptive information is encoded into a 256
dimensional vector with an MLP. The features are concatenated and pass through
additional MLP layers to generate the action.

##### Reinforcement Learning Experiments with Fetch Robot:

We also conducted the same RL experiments with a Fetch robot. The observation
space include $128\times 128$ RGB-D images from the onboard sensor on the
agent’s head, and proprioceptive information (the end effector pose in agent’s
local frame, joint configurations, and whether the end effector is currently
grasping something). The action space is 6-dimensional representing the
desired linear and angular velocities of the end effector, where the rest of
the agent is stationary. We experimented with both the “sticky mitten”
grasping simplification (the same as Bimanual Humanoid) and without such
simplification. For the later setup, the Fetch robot has to rely on the
friction between the gripper fingers and the objects to grasp them with
realistic physics simulation. Its action space also includes one additional
DoF for closing the gripper. With this later setup, we hope to minimize the
sim2real gap as much as possible. Due to the additional complexity of
grasping, we add one more reward shaping terms to encourage the gripper to
grasp the task-relevant object and penalize the agent for dropping it. We use
different reward scaling. The termination conditions remain the same. We also
use the same policy network architecture and training schema as before. The
training reward curves can be found in Fig. A.2.

##### Imitation Learning Experiments with Bimanual Humaniod Robot

: The observation space includes the ground truth poses of the task-relevant
objects, and proprioceptive information, the same as the RL setup. In the
Bimanual Pick and Place experiment, task relevant objects include the
cauldron, the table and the agent itself. The agent can control both of its
hands with the desired linear and angular velocities, which result in 12
degree of freedom. The hand closing action is not learned, but replayed from
the human demonstrations.

We collected $6500+$ state-action pairs from $30$ human demos and used
behavior cloning to predict the action based on the state. The policy network
has two MLP encoders for proprioceptive information and ground truth object
poses. The features are concatenated and pass through additional MLP layers to
generate the action.

The network is trained until validation loss plateaus, and evaluated on a test
set of demonstrations. As discussed in the main paper, the entire task is long
horizon (>300 steps) and the policy diverges due to covariate shift [41]. We
then evaluate if the policy can successfully perform the task if we initialize
the simulator a few seconds before task completion of a successful human demo.
The success rate with respect to different policy starting time can be found
in Fig. 5(b). We show a successful sequence after rewinding 2 seconds in Fig.
A.3.

Figure A.1: Reward curves for RL experiments (Bimanual Humanoid): We observe
steady training progress across all the tasks and the RL agent achieve perfect
reward for task 1)-4). For Slicing Fruit, the agent achieves only $15\%$
success rate because the task requires a precise alignment between the knife
blade and the fruit. For Bimanual Pick and Place, the agent fails to succeed;
although it receives partial reward for approaching the cauldron, bimanual
manipulation of large heavy objects requires careful coordination between
hands and remains challenging learn with RL.

Figure A.2: Reward curves for the five task trained with RL (Fetch): When we
adopt the same “sticky mitten” grasping simplifcation as Bimanual Humanoid for
Fetch, the RL agent achieves very similar result (blue) as the one shown in
Fig. A.1. Once we remove such simplification, the RL agent, however, struggles
to solve the tasks, achieving around $25\%$ success rate for Grasping Book and
Soaking Towel, and $0\%$ for the other three. Although the robot learns to
approach the task-relevant objects with its end effector, grasping a diverse
set of objects (e.g. towel, knife, meat) with a parallel gripper remains a
challenging robotics problem.

Figure A.3: A sequence of IL execution of bimnual grasping: the blue bar
indicates that we are replaying the demonstration, and the green bar is
indicating the policy is taking over. The sequence shows that we are able to
complete the task of placing a heavy object on table by starting at
$T-2\text{s}$.

###  A.4 Performance Benchmark of iGibson 2.0

To evaluate whether iGibson 2.0 can be used in computationally expensive
embodied AI research, we benchmarked the performance (simulation time) and
compared with the previous version. The benchmark setup is the same as in Shen
et al. [1], which considered an “idle” setup, in which we place a robot (a
TurtleBot model) in the scene and run the physics simulation and extended
physical state simulation loop. The benchmark runs on 15 scenes, and
statistics are collected. The agent applies zero actions and stays still. We
use action time step of $t_{a}=\frac{1}{30}\text{s}$ and physics time step of
$t_{s}=\frac{1}{120}\text{s}$ to be consistent with Shen et al. [1]. Both
settings are benchmarked on a computer with Intel 5930k CPU and Nvidia GTX
1080 Ti GPU, in a single process setting, rendering $512\times 512$ RGB-D
images.

The simulator speed is shown in Table A.6. Although we added many extended
physical states, we still achieved a $25\%$ increase in average performance
compared with iGibson 1.0 [1]. In iGibson 2.0, the main source of speed up
with respect to the previous version of iGibson is obtained from better usage
of the object sleeping mechanism, and lazy update of object poses in the
renderer. This allows us to simulate much larger scenes with many more objects
with extended physical states tracked, and as a result, more diverse everyday
household activities.

Simulator | Mean | Max | Min
---|---|---|---
iGibson 1.0 | 100 | 150 | 68
iGibson 2.0 | 125 | 217 | 73
Table A.6: Simulation Speed Comparison for iGibson 1.0 and iGibson 2.0. The
unit is steps per second; each step simulates $\frac{1}{30}\text{s}$.

###  A.5 Feature Comparisons of Simulators

In Table A.7, we provide a detailed comparison across multiple simulation
environments. The table is adapted from Table I of [1]. We include more recent
simulation environments as columns and more feature comparisons as rows.

Table A.7: Comparison of Simulation Environments

| iGibson 2.0 (ours) | iGibson 1.0 [1] | Gibson [2] | Habitat [5] | Sapien [4] | AI2Thor [15] | ManipulaTHOR [42] | VirtualH [16] | TDW [7, 13]
---|---|---|---|---|---|---|---|---|---
| Provided Large Scenes
---
Real-World / Designed
| 15 homes
---
(108 rooms) / –
| 15 homes
---
(108 rooms) / –
1400 / – | – | – | – / 120 rooms | – / 30 rooms | – / 7 | – / 25
Provided Objects | 1217 (*) | 570 | – | – | 2346 | 609 | 150 | 308 | 200
Continuous Extended States |  |  |  |  |  |  |  |  |
Non-kinematic States |  |  |  |  |  |  |  |  |
| Geometric Sampling
---
Based on Logical States
|  |  |  |  |  |  |  |
Human Interface |  | Mouse, VR
---
| Mouse
---
- |  | Mouse
---
- |  | Mouse
---
| Mouse
---
| Natural
---
Language
| VR
---
| Agent/World Interaction
---
Forces, Predefined Actions
F | F | – | – | F | F & PA | F & PA | F & PA | F
| Physics Engine
---
Pybullet | Pybullet | Pybullet | Bullet | PhysX | Unity | Unity | Unity | Unity & Flex
| Supported Task
---
| Nav.&Manip.
---
| Nav.&Manip.
---
| Nav.
---
| Nav.
---
| Nav.&Manip.
---
|  | Nav.&Manip.
---
|  | Nav.&Manip.
---
| Nav.&Manip.
---
| Nav.&Manip.
---
Speed | ++ | ++ | + | +++ | ++(PBR)/-(RTX) | + | + | + | +
|  Integrated Motion Planner
---
|  |  |  |  |  |  |  |
Specialty |  | Continuous Extended
---
States, VR
| Phys. Int. in
---
Large Scenes
| Nav.
---
| Fast,
---
Nav.
| Articulation,
---
Ray Tracing
| Object States,
---
Task Planning
| Mobile
---
Manipulation
| Object States,
---
Task Planning
| Audio,
---
Fluids
Type of rendering: PBR:Physics-Based Rendering, IBR:Image-Based Rendering, RTX:Ray Tracing |  |
Virtual sensor signals: RGB: Color Images, D:Depth, N:Normals, SS:Semantic Segmentation, LiDAR:Lidar, FL:Flow (optical and/or scene), S: Sounds |  |
* included in a parallel submission BEHAVIOR and fully compatible with iGibson 2.0 |  |

###  A.6 Limitations and Future Work

Although iGibson 2.0 has made several significant contributions towards
simulating complex, everyday household tasks for robot learning, it is not
without limitation. First of all, iGibson 2.0 doesn’t support soft bodies /
flexible material in a scalable way at the moment, due to the limitation of
our underlying physics engine. This prevents us from simulating tasks like
folding laundry and making bed in large, interactive scenes. Also, iGibson 2.0
doesn’t support accurate human behavior modeling (other than goal-oriented
navigation), and thus prevent us from simulating tasks that are inherently
rich in human-robot interaction (e.g. elderly care). With the recent
advancement of physics engines, and human behavior modeling and motion
synthesis, we plan to overcome these limitations in the future. In addition,
we also plan to support a more diverse set of extended object states (e.g.
Filled, Hung, Assembled, etc) as well as bi-directional transitions for some
of our existing states (e.g. Soaked and Stained/Dusty), which can unlock even
more household tasks. Finally, we plan to transfer mobile manipulation
policies trained in iGibson 2.0 to the real world.