#  ToolTango: Common sense Generalization in Predicting Sequential Tool
Interactions for Robot Plan Synthesis

Shreshth Tuli<EMAIL_ADDRESS>
Department of Computing
Imperial College London, UK Rajas Bansal∗<EMAIL_ADDRESS>
Department of Computer Science
Stanford University, USA Rohan Paul<EMAIL_ADDRESS>
Department of Computer Science and Engineering / Yardi School of Artificial
Intelligence
Indian Institute of Technology Delhi, India Mausam<EMAIL_ADDRESS>
Department of Computer Science and Engineering / Yardi School of Artificial
Intelligence
Indian Institute of Technology Delhi, India Most work done when the authors
were undergraduate students at Indian Institute of Technology Delhi, India.

###### Abstract

Robots assisting us in environments such as factories or homes must learn to
make use of objects as tools to perform tasks, for instance using a tray to
carry objects. We consider the problem of learning commonsense knowledge of
when a tool may be useful and how its use may be composed with other tools to
accomplish a high-level task instructed by a human. Specifically, we introduce
a novel neural model, termed ToolTango, that first predicts the next tool to
be used, and then uses this information to predict the next action. We show
that this joint model can inform learning of a fine-grained policy enabling
the robot to use a particular tool in sequence and adds a significant value in
making the model more accurate. ToolTango encodes the world state, comprising
objects and symbolic relationships between them, using a graph neural network
and is trained using demonstrations from human teachers instructing a virtual
robot in a physics simulator. The model learns to attend over the scene using
knowledge of the goal and the action history, finally decoding the symbolic
action to execute. Crucially, we address generalization to unseen environments
where some known tools are missing, but alternative unseen tools are present.
We show that by augmenting the representation of the environment with pre-
trained embeddings derived from a knowledge-base, the model can generalize
effectively to novel environments. Experimental results show at least
48.8-58.1% absolute improvement over the baselines in predicting successful
symbolic plans for a simulated mobile manipulator in novel environments with
unseen objects. This work takes a step in the direction of enabling robots to
rapidly synthesize robust plans for complex tasks, particularly in novel
settings.

##  1 Introduction

Advances in autonomy have enabled robots to enter human-centric domains such
as homes and factories where we envision them performing general purpose tasks
such as transport, assembly, and clearing. Such tasks require a robot to
interact with objects, often using them as _tools_. For example, a robot asked
to “take fruits to the kitchen”, can use a _tray_ for carrying items, a
_stick_ to fetch objects beyond physical reach and may use a _ramp_ to reach
elevated platforms. Previous work has shown that the ability to predict the
possible use of tools for a given task is often useful in guiding a robot’s
task planner towards plans likely to be feasible (?, ?, ?, ?, ?). In this work
we consider the problem of predicting _which_ objects could be used as tools
and _how_ their use can be composed for a task. In essence, we focus on the
ability to predict appropriate tools that can guide the robot towards feasible
and efficient plans and delegate the issue of dexterous tool manipulation to
prior work (?, ?).

Learning to predict task-directed tool interactions poses several challenges.
First, real environments (a household or factory-like domain) are typically
large where an expansive number of tool interactions may be possible (e.g.,
objects supporting others while transporting). Acquiring data for all feasible
tool objects or exploring the space of tool interactions is challenging for
any learner. Second, the usefulness of a tool varies with context. For
example, placing milk in the cupboard may require the robot to elevate itself
vertically using a ramp if the milk is placed at a height unreachable by the
robot, but if the milk is kept on a table, a simple tray might suffice. Third,
the robot may encounter new environments populated with novel objects not
encountered during training. Hence, the agent’s model must be able to
_generalize_ by reasoning about interactions with novel objects unseen during
training. Humans possess innate commonsense knowledge about contextual use of
tools for an intended goal (?). For example, a human actor when asked to move
objects is likely to use trays, boxes, or even improvise with a new object
with a flat surface. We aim at providing this commonsense to a robotic agent,
so that it can generalize its knowledge to unseen tools, based on shared
context and attributes of seen tools (see Figure 1).

Figure 1:  ToolTango acquires commonsense knowledge from human demonstrations
leveraging graph-structured world representation, knowledge-based corpora and
goal-conditioned attention to perform semantic tasks. Our aim is to acquire
commonsense knowledge to develop a generalized goal-conditioned policy for a
robot.

This paper takes a step in the direction of enabling robots to learn how to
perform high-level tasks such as compositional tool use in semantic tasks,
particularly in novel environments. This paper makes four main contributions.
As the first contribution, we present a crowd-sourced dataset of human-
instructed plans where a human teacher guides a simulated mobile manipulator
to leverage objects as tools to perform multi-step actions such as assembly,
transport and fetch tasks. The process results in a corpus of
$\sim\\!\\!1,500$ human demonstrated robot plans involving diverse goal
settings and environment scenes.

Second, the dataset mentioned above is first used to supervise a (1-step)
neural imitation learner that predicts tool applicability given the knowledge
of the world state and the intended goal. We show how learning a dense
embedding for the environment and that of a background knowledge base can
enable the model to generalize to novel scenes with new object instances that
may share semantic attributes with objects seen during training; a common
problem in state of the art task and policy learning approaches. We introduce
a graph neural architecture, ToolNet, that encodes both the metric and
relational attributes of the world state as well as available taxonomic
resources such as $\mathrm{ConceptNet}$ (?). The ToolNet model predicts tool
use by learning an attention over entities that can potentially serve as
tools. Implicitly, the model acquires knowledge about primitive spatial
characteristics (typically an output of a mapping system) and semantic
attributes (typically contained in taxonomic resources) enabling
generalization to novel contexts with previously unseen objects.

Third, we present an imitation learner that uses the same dataset to make
action predictions towards the intended goal. We term the model, Tool
Interaction Prediction Network for Generalized Object environments (Tango).
Similar to ToolNet, Tango also encodes the world state using a graph neural
network and learns to attend over the scene using knowledge of the goal and
the action history, finally decoding the symbolic action to execute. The
action predictions are interleaved with physics simulation (or execution)
steps, which ameliorates the need for modeling the complex effects of actions
inherent in tool interactions.

As a final contribution, we combine the two models ToolNet and Tango in a
single architecture. Specifically, the joint model first makes the predictions
for the next tool, and then uses this information to make a better action
prediction. We show that this can inform learning of a fine-grained policy
enabling the robot to use a particular tool in sequence and adds a significant
value in making the model more accurate. We term this joint model ToolTango.

Experimental evaluation with a simulated mobile manipulator demonstrates (a)
accurate prediction of tool interaction sequences with high
executability/goal-attainment likelihood, (b) common sense generalization to
novel scenes with unseen object instances, and (c) robustness to unexpected
errors during execution. Additionally, compared to Tango, we demonstrate the
benefit of using ToolNet predictions for specific complex settings requiring
multiple tools to reach the goal state. Experiments show that in previously
seen settings, ToolTango gives an absolute improvement of 3.38-5.59% in
reaching a goal state for a simulated mobile manipulator compared to the
state-of-the-art Tango model. In unseen settings, the unified model gives
2.48-3.58% improvement compared to Tango. In comparison to a simple affordance
prediction approach, the proposed model performs better in learning the pre-
conditions for tool use. Further, the use of a neural model enables higher
goal-reaching performance and faster training compared to a vanilla
reinforcement learning approach.

A preliminary version of ToolNet that performs single tool prediction using
only the initial environment state was presented as part of the Workshop on
Advances & Challenges in Imitation Learning in Robotics at Robotics Science
and Systems (RSS) 2020 conference (?). Our Tango model was presented as part
of the International Joint Conference in Artificial Intelligence (IJCAI) 2021
(?). This journal submission presents a substantially detailed exposition of
the ToolNet and Tango models and includes additional supporting background
material. Moreover, it also extends ToolNet to predict _next_ tool at each
step of a multi-step action execution, instead of a single tool for the whole
sequence, as in the original paper. This paper also combines the two models
into ToolTango, and establishes its superior performance.

The remainder of the paper is organized as follows. Section 2 overviews
related work. Section 3 formulates the problem of predicting tool interaction
sequences. Section 4 details the technical approach and presents the ToolTango
model. The data collection platform and the dataset in detailed in Section 5.
Section 6 provides the experimental details and results. Finally, Sections 7
and 8 summarizes the work and lays out avenues for future work. The associated
code, data set and videos are available at https://github.com/reail-
iitd/tango. Links to all supplementary material are given in Appendix A.

##  2 Related Work

###  2.1 Classical Planning

Reaching goal states from a given world state through symbolic actions is
closely tied to the domain of task and motion planning (TAMP) (?). Literature
presents a large volume of work in this domain, ranging from constraint
satisfaction to search based methods. For instance, ? (?) a planner to compose
physics tool interactions using a logic-based symbolic planner. Similarly, ?
(?) provide an implementation agnostic interface between a task and motion
planner to combine both for goal-directed planning. ? (?) extend PDDL
descriptions to include generic and declarative specifications enabling the
planning to be domain independent. These methods only utilize the task
properties through action constraints. For instance, the knowledge that a tray
can carry multiple objects is only encoded through the constraint that an
object can be placed on it. This allows such methods to create any feasible
plan and not leverage commonsense knowledge to ensure that a goal state is
reached in a few steps. Some recent works such as by ? (?) aim to learn object
importance through human demonstrations. However, such methods can only work
on objects seen previously in training and cannot generalize to unseen
objects. We consider a part of the broad TAMP framework that focuses on
determining the set of actions that are likely to take an agent to the goal
state while taking the motion planning as a behavioural routine. We build our
work on the observation that in domain with a large number of states and
possible interactions, the task planning itself becomes challenging. We
consider home and factory-like domains inspired from VirtualHome (?) and
similar related works. We ensure that in our domains the number of objects is
large and they can be contained within/supported or transported by other
objects as tools (details in Section LABEL:sec:data_collection). Similar to
the work by ? (?), our model learns to prune-away irrelevant objects,
additionally considering a domain with richer inter-object interactions.
Consequently, our learner makes additional use of semantic properties and
exploits correlations between actions and outputs interactions that are likely
to lead to successful plans.

###  2.2 Learning tool manipulation skills

Learning control policies for manipulating tools has received recent attention
in robotics. ? (?) and ? (?) learn tool manipulation policies from human
demonstrations. ? (?) and ? (?) learn physics models and _effects_ enabling
goal-directed compositional use. ? (?) address the problem of learning
primitive physical decomposition of tool like object through its physical and
geometric attributes enabling their human-like use. ? (?) learn physical
properties of objects from unlabeled videos. ? (?) and ? (?) learn to interact
with objects in a self-supervised setup. Efforts such as ? (?), and ? (?) plan
tool interactions modeling contact and force interactions. Another set of
efforts focuses on incorporating the ability to discover the use of objects as
tools and using them for plan completion. Rich symbolic architectures such as
ICARUS (?), KDAP (?) and ROAR (?, ?) attempt to model tool use via PDDL-like
descriptions expressing the applicability and post effects of tool use. In
particular, ICARUS (?) models bridge or staircase construction via lifted
symbolic concepts which can be grounded to real or imagined objects in the
environment. The framework presented in this paper takes inspiration from such
classic works and builds on the use of learned representations for
generalization to new scenes and objects. Further, instead of encoding such
knowledge via a symbolic representation for each tool, the framework acquires
such knowledge in a data-driven way from human demonstration. Thus, the
aforementioned works focus on learning _how_ to manipulate a tool. Our paper
considers the complementary problem of predicting _which_ objects may serve as
tools for a given task while delegating the issue of tool manipulation to the
works as mentioned earlier.

The works of ? (?) and ? (?) considers the specific problem of learning the
physical motion of tools (such as spatial, hammer, mug, etc.) from kinesthetic
demonstration trajectories. Further, the authors present a framework to learn
corrections or replacements enabling improvisation when the actual task
execution scenario differs from the taught demonstration. The focus of the
work is in learning physical motion of tools for short range tasks. This work
lifts the problem of dexterous tool manipulation and focuses on utilizing
tools to reach goal states (e.g., using a box to transfer multiple objects).
This work also extends planning to leverage multiple tools to attain goals
(using boxes and later using ramps). Rather than use fine-grained kinesthetic
teaching, we instead use demonstrations of.long range plan executions. The two
efforts are highly complementary. The tool-use trajectories learned by prior
work could potentially be used in our framework to accomplish long-range
tasks. Similarly, such works can use the framework presented here to compose
skills and generalize to unseen tools.

###  2.3 Learning symbolic action sequences

Others address the problem of acquiring knowledge for completing high-level
task specifications. ? (?, ?) create a knowledge base of task decompositions
as _action sketches_ and learn to translate sketches to executable plans.
These efforts rely on the causal knowledge of sequences on sub-steps required
to achieve an activity which are then contextually grounded. Instead, this
work learns compositional tool use required to achieve the task without any
causal sequence as input. ? (?) learn task decompositions from human
demonstration videos. However, their work does not explicitly model the
physical constraints of the robot and does not generalize to new environments.
? (?) and ? (?) learn trajectory-aware manipulation of deformable objects
using a non-rigid registration method and human demonstrations. These methods
can effectively handle visual variation in manipulating objects; however, they
cannot be extended to generate action sequences to reach goal constraints
given an unseen environment. ? (?) take a similar approach by collecting
natural language corpora describing high-level tasks and learn to associate
instructions to spatial attention over the scene.

Other works study _relational_ planning domains, where model is trained using
RL on small problems, but tested zero-shot on large problems (?, ?). ? (?)
extend this to imitation learning framework, but these works cannot perform
tool generalizations. ? (?) present a symbolic system where a robot imitates
demonstrations from a single teacher. In new environments, it adapts the plan
by performing object replacements using ConceptNet relation edges. A rich set
of approaches learn affordances by mapping objects to preferred locations or
learning the co-use of objects to accomplish a task. For instance, ? (?)
provide a method of acquiring and transferring such knowledge to new tasks by
learning to map between objects in the old and new environments in a task-
dependent manner. This work does not explicitly attempt to learn such
associations and presents a restricted generalization capacity. Instead, such
learning is implicit in the learned policy that predicts a sequence of robot
actions to achieve the goal while using tools (e.g., fetching the tool, using
the tools and attaining its post effects).

Our approach draws inspiration from the above-mentioned works in that, we
learn to predict tools that can be considered as sub-goals to guide planning
for a high-level task. In comparison to these approaches, our method provides
two key contributions. First, we explicitly model the physical constraints
arising from a mobile manipulator interacting in the work space. Second,
instead of learning actions predicated on specific object instances, we
address generalization to new object instances using primitive spatial and
semantic characteristics. We propose a neural model trained using a corpus of
multiple and varied demonstrations provided by several teachers. Our model
uses a dense embedding of semantic concepts, enabling generalization beyond
relationships explicitly stored in ConceptNet.

###  2.4 Commonsense knowledge in instruction following

Acquisition of common sense knowledge has been previously explored for the
task of robot instruction following (?). ? (?) present a symbolic knowledge
base for procedural knowledge of tasks that is utilized for interpreting
underspecified task instructions. Efforts such as ? (?) propose a similar
database encoding common sense knowledge about object affordances (objects and
their common locations). Others such as ? (?) learn motion preferences
implicit in commands. ? (?) ground instructions for recipe preparation tasks.
Their model can generalize to new recipes, but only in environments with
previously _seen_ objects. In contrast, our model generalizes to worlds with
previously _unseen_ tools. Others, such as ? (?) and (?) explore the problem
of robot tool construction, i.e., creating tools from parts available in the
environment. However, the limited set of commonsense concepts, such as
attachment in (?) restricts the space for generalization in such works. A
rule-based approach typically does not scale well to develop a robust
generalization model. We thus utilize a commonsense embedding based approach
that utilizes ConceptNet vectors to encapsulate the concepts and generalize to
unseen tools and object settings.

? (?) present an instruction grounding model that leverages common sense
taxonomic and affordance knowledge learned from linguistic co-associations. ?
(?) consider the problem of learning physical common sense associated with
objects and interactions required to achieve tasks from language only data
sets. They study this problem in the context of question-answering to enable
synthesis of textual responses that capture such physical knowledge. The
aforementioned approaches predict latent constraints or affordances for a
specified task. This work, additionally predicts the _sequence_ of tool
interactions implicitly learning the causal relationships between tools use
and effects. Specifically, this paper focuses on a learning common sense tool
use in the context of following instructions that require multiple object
interactions to attain the intended goal.

###  2.5 Synthetic Interaction Datasets

Virtual environments have been used to collect human demonstrations for high-
level tasks. ? (?) introduce a knowledge base of actions required to perform
activities in a virtual home environment. ? (?) provide a vision-language
dataset translating symbolic actions for a high-level activity to attention
masks in ego-centric images. ? (?) curated data sets that provide a sequence
of _How-To_ instructions for tasks such as preparing recipes. ? (?) present an
affordance detection dataset for tool parts with geometric features. Others
such as ? (?), ? (?) and ? (?) present simulation environments and data sets
for tasks such as learning spatial affordances, situated interaction or
learning low-level motor skills. The present data sets possess two limitations
that make them less usable for the learning task addressed in this work.
First, the data sets are collected using human actors or avatars but do not
explicitly model a robot in their environment. Though virtual agents serve as
a proxy for the robot, they preclude modeling of the physical constraints and
the range of tasks an robot can perform. Second, a majority of the data sets
aim at visual navigation and limited physical interaction with objects. They
are less amenable to interactions (e.g., containment, pushing and attachment
etc.) inherent in tool use. Data sets utilized in robotic tool use literature,
including the UMD Part Affordance Dataset (?), are confined mainly to local
use, such as finding the appropriate tool part for tool use and manipulation.
Such data sets are less amenable to multi-stage plans in large workspaces.

##  3 Problem Formulation

###  3.1 Robot and Environment Model

We consider a mobile manipulator operating in a known environment populated
with objects. An object is associated with a pose, a geometric model and
symbolic states such as $\mathrm{Open/Closed}$, $\mathrm{On/Off}$ etc. We
consider object relations such as (i) _support_ e.g., a block supported on a
tray, (ii) _containment_ : items placed inside a box/carton and (iii)
_attachment_ : a nail attached to a wall, and (iv) _contact_ : a robot
grasping an object. Let $s$ denote the world state that maintains (i) metric
information: object poses, and (ii) symbolic information: object states, class
type and object relations as $\mathrm{OnTop}$, $\mathrm{Near}$,
$\mathrm{Inside}$ and $\mathrm{ConnectedTo}$. Let $s_{0}$ denote the initial
world state and $\mathcal{O}(\cdot)$ denote a map from world state $s$ to the
set of object instances $O=\mathcal{O}(s)$ populating state $s$. Let $\tau$
denote the set of tool objects that the robot can use in its plan. Note that
only movable objects in the scene are considered as potential tools. Hence,
$\tau\subseteq\mathcal{O}(s)$. Online, the robot may encounter _unseen_
objects in its environment.

Let $A$ denote the robot’s symbolic action space. An action $a\in A$ is
abstracted as $I(o^{1},o^{2})$, with an action type predicate
$I\in\mathcal{I}$ that affects the states of objects $o^{1}\in O$ and
$o^{2}\in O$, for instance, $\mathrm{Move(fruit_{0},tray_{0})}$. Here the
arity of the interaction can be 2 or 1 depending on the interaction type. In
case of arity of 1, we can drop the second object. Each action in our
formulation is realized as a set of object relations in the environment that
belongs to
$\\{\mathrm{OnTop},\mathrm{Near},\mathrm{Inside},\mathrm{ConnectedTo}\\}$. The
precondition and postconditions of actions are taken directly from prior work
(?). Realization of these conditions using geometric properties is described
in Appendix D. We shall also use the notion of a timestamp as a subscript to
indicate prediction for each state in the execution sequence. The space of
robot interactions include grasping, releasing, pushing, moving an object to
another location or inducing discrete state changes (e.g.. opening/closing an
object, operating a switch or using a mop). We assume the presence of an
underlying low-level metric planner, encapsulated as a robot _skill_ , which
realizes each symbolic action or returns if the action is infeasible. Robot
actions are stochastic, modeling execution errors (unexpected collisions) and
unanticipated outcomes (objects falling, changing the symbolic state). Let
$\mathcal{T}(\cdot)$ denote the transition function. The successor state
$s_{t+1}$ upon taking the action $a_{t}$ in state $s_{t}$ is generated from a
physics simulator. Let $\eta_{t}=\\{a_{0},a_{1},\dots,a_{t-1}\\}$ denote the
_action history_ till time $t$.

###  3.2 Semantic Goals and Interactions

The robot’s goal is to perform tasks such as transporting or delivering
objects to appropriate destinations, making an assembly, clearing or packing
items or performing abstract tasks such as illuminating or cleaning the room.
The robot is instructed by providing a _declarative_ goal $g$ expressing the
symbolic constraint between world objects (?). For example, the _declarative_
goal, “place milk in fridge” can be expressed as a constraint
$\mathrm{Inside(milk_{0},fridge_{0})}$ between specific object instances.
There may be multiple instance of the same object, for eg. milk carton, in the
environment. The sub-index is used to specify which instance is being referred
to. Another example is of the task of moving all fruits onto the kitchen table
that can be expressed as a list of constraints
$\mathrm{OnTop(apple_{0},table_{0})}$, $\mathrm{OnTop(orange_{0},table_{0})}$
and $\mathrm{OnTop(banana_{0},table_{0})}$. Finally, the robot must synthesize
a plan to satisfy the goal constraints. Goal-reaching plans may require using
some objects as tools, for instance, using a container for moving items, or a
ramp negotiate an elevation.

###  3.3 Predicting Tool Interactions

We assume that the robot is primed with a set of primitive symbolic actions
but lacks knowledge about how object characteristics can facilitate their use
as in attaining high-level goals. Hence, the robot cannot predict the use of
tray-like objects in transportation tasks, or the use of a stick to fetch an
object at a distance. An exception is by discovering such characteristics via
explicit simulation, which may be infeasible or intractable in large planning
domains. Our goal is to learn common sense knowledge about _when_ an object
can be used as a tool and _how_ their use can be sequenced for goal-reaching
plans. We aim at learning a policy $\pi$ that estimates the next action
$a_{t}$ conditioned on the the goal $g$ and the initial state $s$ (including
the action history $\eta_{t}$, such that the robot’s _goal-reaching_
likelihood is maximized. We adopt the MAXPROB-MDP (?) formalism and estimate a
policy that maximizes the goal-reaching likelihood from the given state.
MAXPROB-MDP can be equivalently viewed as an infinite horizon, un-discounted
MDP with a zero reward for non-goal states and a positive reward for goal
states (?). Formally, let $P^{\pi}\left(s,g\right)$ denote the _goal-
probability_ function that represents the likelihood of reaching the goal $g$
from a state $s$ on following $\pi$. Let $S^{\pi_{s}}_{t}$ be a random
variable denoting the state resulting from executing the policy $\pi$ from
state $s$ for $t$ time steps. Let $\mathcal{G}(s,g)$ denote the Boolean _goal
check_ function that determines if the intended goal $g$ is entailed by a
world state $s$ as
$\mathcal{G}(s,g)\in\\{\mathrm{True(T)},\mathrm{False(F)}\\}$. The policy
learning objective is formulated as maximizing the likelihood of reaching a
goal-satisfying state $g$ from an initial state $s_{0}$, denoted as

$\displaystyle\max_{\pi}P^{\pi}(s_{0},g)=\max_{\pi}\sum_{t=1}^{\infty}P\bigg{(}\mathcal{G}(S^{\pi_{s_{0}}}_{t},g)=\mathrm{T}:\mathcal{G}(S^{\pi_{s_{0}}}_{t^{\prime}},g)=\mathrm{F},~{}\forall
t^{\prime}\in[1,t)\bigg{)}.$ (1)

The policy is modeled as a function $f_{\theta}(.)$ parameterized by
parameters $\theta$ that determines the next action for a given world state,
the robot’s action history and the goal as
$a_{t}=f_{\theta}\left(s_{t},g,\eta_{t}\right)$. We adopt imitation learning
approach and learn the function $f_{\theta}(.)$ from demonstrations by human
teachers. Let $\mathcal{D}_{\mathrm{Train}}$ denote the corpus of $N$ goal-
reaching plans,

$\mathcal{D_{\mathrm{Train}}}=\\{(s_{0}^{i},g^{i},\\{s_{j}^{i},a_{j}^{i}\\})\mid
i\in\\{1,\ldots,N\\},j\in\\{0,t_{i}-1\\}\\},$ (2)

where the $i^{th}$ datum consists of the initial state $s^{i}_{0}$, the goal
$g^{i}$ and a state-action sequence

$\\{(s^{i}_{0},a^{i}_{0}),\dots,(s^{i}_{t-1},a^{i}_{t-1})\\}$

of length $t_{i}$. The set of human demonstrations elucidate common sense
knowledge about _when_ and _how_ tools can be used for attaining provided
goals. The data set $\mathcal{D}_{\mathrm{Train}}$ supervises an imitation
loss between the human demonstrations and the model predictions, resulting in
learned parameters $\theta^{*}$. Online, the robot uses the learned model to
sequentially predict actions and execute in the simulation environment till
the goal state is attained. We also consider the _open-world_ case where the
robot may encounter instances of _novel_ object categories _unseen_ in
training, necessitating a _zero-shot_ generalization.

##  4 Technical Approach

We aim at predicting the next robot action $a_{t}$, given the world state
$s_{t}$, the intended goal $g$ and the history $\eta_{t}$ of past actions
taken the robot. We consider learning to predict multi-step plans requiring
complex interaction of objects as tools in possibly novel environments unseen
during training. Our technical approach is built on three insights. First, we
factor the overall learning task of predicting the action for each environment
state by first predicting the tool required to perform the task and then
predicting required interaction. This tool conditioned action prediction
enables us to decouple the learning problem and make the model training
tractable. Second, we incorporate learning dense embeddings of the robot’s
environment as well as semantic representations for words trained on existing
symbolic knowledge corpora. The learned representation allows the robot to
generalize its predictions to novel environments populated with novel objects
that may share semantic attributes with those encountered during training.
Finally, we turn to a corpus of human demonstrated plans to train our learner.
The corpus elucidates the commmon sense knowledge of sequencing tools
interactions for a task. Formally, we introduce an imitation learner, realized
as a hyper-parametric function (a neural network model) denoted as
$f_{\theta}$ as follows:

$a_{t}=f_{\theta}\left(s_{t},g,\eta_{t}\right)=f^{act}_{\theta}\left(f^{goal}_{\theta}\left(f^{state}_{\theta}\left(s_{t}\right),g,f^{hist}_{\theta}\left(\eta_{t}\right)\right),f^{tool}_{\theta}(s_{t},g,\eta_{t})\right).$
(3)

It adopts an object-centric graph representation, learning a state encoding
that fuses metric-semantic information about objects in the environment via
function $f^{state}_{\theta}\left(\cdot\right)$. The function
$f^{hist}_{\theta}\left(\cdot\right)$ encodes the action history. The model
learns to attend over the world state conditioned on the declarative goal and
the history of past actions through $f^{goal}_{\theta}\left(\cdot\right)$.  We
leverage an adapted version of our tool likelihood prediction model ToolNet,
denoted as $f^{tool}_{\theta}\left(\cdot\right)$. Finally, the learned
encodings are decoded as the next action for the robot to execute via
$f^{act}_{\theta}\left(\cdot\right)$. Crucially, the predicted action is
grounded over an _a-priori_ unknown state and type of objects in the
environment. The predicted action is executed in the environment and the
updated state action history is used for estimation at the next time step.
Using a neural network as a hyper-parametric function for action prediction
enables us to leverage ConceptNet embeddings to generalize and scale with the
size of object sets, goal and interaction types. We utilize an imitation
learning model with the dataset described in Equation 2 to learn the $\theta$
parameters of the neural network. The constituent model components are
detailed next.

###  4.1 Graph Structured World Representation

Figure 2:  Updated ToolNet model encodes the metric-semantic world state using
graph convolution (GGCN) and fully connected (FCN) layers. The model uses goal
information and the robot’s action history to attend over a task-specific
context, finally predicting the likelihood scores for each tool in the
environment. A graph structured representation and inclusion of pre-trained
word embeddings (from a knowledge base) facilitate generalization in
predicting interactions in novel contexts with new objects unseen in training.

####  4.1.1 Semantic Reasoning

We first describe the $f^{state}_{\theta}\left(\cdot\right)$ function in
equation (3). We denote the robot’s current world state $s_{t}$ as an object-
centric graph $G_{t}=(O,R)$. Each node in the graph represents an object
instance $o\in O=\mathcal{O}(s_{t})$. The edge set consists of binary
relationships $\mathrm{OnTop}$, $\mathrm{ConnectedTo}$, $\mathrm{Near}$ and
$\mathrm{Inside}$ between objects $R\subseteq O\times O$. Let
$l_{o}\in\\{0,1\\}^{p}$ represents discrete object states for the object $o$
(e.g. $\mathrm{Open/Closed}$, $\mathrm{On/Off}$). Here, $p$ represents the
number of possible states of all objects. Next, it incorporates a pre-trained
function $\mathcal{C}(\cdot)$ that embeds a word (like token of an object
class or a relation) to a dense distributed representation, such that
semantically close tokens appear close in the learned space (?). The use of
such embeddings enables generalization, which we discuss subsequently.
Consider the example goal of “Place the box in the cupboard”. In this case we
want to satisfy the relationship $\mathrm{Inside}$ for cupboard and box. The
entire environment can be considered as the graph where relations like
$\mathrm{OnTop}(box,table)$ may be true. The pretrained function
$\mathcal{C}(\cdot)$ can be any pretrained language model.

Let $e_{o}=\mathcal{C}(o)\in\mathcal{R}^{q}$ denote the $q$-dimensional
embedding for an object instance $o$. The embeddings $l_{o}$ and $e_{o}$ model
object attributes that initialize the state of each object node in the graph.
The local context for each $o$ is incorporated via a Gated Graph Convolution
Network (GGCN) (?), which performs message passing between 1-hop vertices on
the graph. Following (?), the gating stage is realized as a Gated Recurrent
Unit (GRU) resulting in _graph-to-graph_ updates as:

$\displaystyle\begin{split}r_{o}^{0}&=\mathrm{tanh}\left(W_{r}\left[l_{o};e_{o}\right]+b^{r}\right),\\\\[-3.0pt]
x^{k}_{o}&=\sum_{j\in R}\sum_{o^{\prime}\in
N^{j}(o)}W_{j}^{k}r_{o^{\prime}}^{k-1},\\\\[-3.0pt]
r^{k}_{o}&=\mathrm{GRU}\left(r^{k-1}_{o},x^{k}_{o}\right).\\\\[-3.0pt]
\end{split}$ (4)

Here, the messages for object $o$ are aggregated over neighbors $N^{j}(o)$
connected by relation $j$ ($\forall j\in R$) during $n$ convolutions,
resulting in an embedding $r^{n}_{o}$ for each object instance in the
environment. Unlike a fully-connected neural network, a GGCN model facilitates
inference over the relations across objects for an input object centric graph.
These relations are crucial to predict the relevant actions and achieve the
intended goal. GRUs in the convolution iterations facilitate stable training
of the model (?).

####  4.1.2 Fusing Metric Information.

Next, we incorporate the metric information associated with objects. Let
$pose_{o}$ and $size_{o}$ represent the pose and size/extent (along xyz axes)
for each object instance. The pose includes the spatial position and the
orientation of the object. We do not explicitly provide the point-cloud
geometric model and abstract out the surface information that may be required
to infer the utility of objects as tools and instead rely on ConceptNet
embeddings for the same. However, in our evaluation (Section 6), we utilize
CAD models that closely represent the surface properties to model the physics
of tool use. Another point to note here is that for the specific task of
utilizing an object as a tool the pose information has little contribution.
However, the pose information is required to accurately identify the system
state. For instance, the pose information of the door of a fridge indicates
whether the fridge is open or closed. This information is given as a
categorical value i.e. the state vector $l_{o}$ to the GGCN model. To
explicitly pass this information to the FCN model, we use the pose vector of
objects as inputs. The properties are encoded using a $d$-layer Fully
Connected Network (FCN) with a Parameterized ReLU (PReLU) (?) activation as:

$\displaystyle\begin{split}m^{0}_{o}&=\mathrm{PReLU}\left(W_{mtr}^{0}[pose_{o};size_{o}]+b_{mtr}^{0}\right)\\\\[-3.0pt]
m^{k}_{o}&=\mathrm{PReLU}\left(W_{mtr}^{k}m^{k-1}_{o}+b_{mtr}^{k}\right),\end{split}$
(5)

resulting in the metric encoding $m^{d}_{o}$ for each object in the scene. A
world state encoding (for $s_{t}$) is obtained by fusing the semantic and
metric embeddings as

$f^{state}_{\theta}(s_{t})=\\{\tilde{s}_{t}^{o}\\!=\\![r^{n}_{o};m^{d}_{o}]|\
\forall o\in{\cal O}(s_{t})\\}.$ (6)

_Late fusion_ of the two encodings allows downstream predictors to exploit
them independently.

###  4.2 Encoding the Action History

Next, we define the $f^{hist}_{\theta}\left(\cdot\right)$ function in equation
(3). The task of deciding the next action is informed by the agent’s action
history in two ways. First, sequential actions are often temporally
correlated. For example, a placing task often involves moving close to the
box, opening it and then placing an item inside it.  If the previous action
involved moving close to a box, this information facilitates the model to
leverage localized contextual information and not jump to manipulate another
object in the goal specification. This, in tandem with the goal-conditioned
attention facilitates seamless action sequence generation. Hence, maintaining
the action history can help in prediction of the next action. Second, the set
of actions the robot executed in the past provides a local context indicating
the objects the the robot may utilize in the future. Formally, we encode the
temporal action history $\eta_{t}$ using an $\mathrm{LSTM}$. We define action
encoding $\mathcal{A}(a_{t-1})$ of $a_{t-1}=I_{t}(o^{1}_{t-1},o^{2}_{t-1})$
independent of the object set, as
$\mathcal{A}(a_{t-1})=[\vec{I}_{t-1};\mathcal{C}(o^{1}_{t-1});\mathcal{C}(o^{2}_{t-1})]$,
where $\vec{I}_{t-1}$ is a one-hot vector over possible interaction types
$\mathcal{I}$, and $\mathcal{C}(o^{1}_{t-1})$ and $\mathcal{C}(o^{2}_{t-1})$
represent the word embeddings of the object instances $o^{1}_{t-1}$ and
$o^{2}_{t-1}$. At each time step $t$, the $\mathrm{LSTM}$ encoder takes in the
encoding of previous action, $\mathcal{A}(a_{t-1})$ and outputs the updated
encoding $\tilde{\eta}_{t}$, given as
$\tilde{\eta}_{t}=\mathrm{LSTM}(\mathcal{A}(a_{t-1}),\tilde{\eta}_{t-1})$.
This results in the embedding vector

$f^{hist}_{\theta}(\eta_{t})=\tilde{\eta}_{t}.$ (7)

###  4.3 Goal-conditioned Attention

We now define the $f^{goal}_{\theta}\left(\cdot\right)$ function in equation
(3). The goal $g$ consists of symbolic relations (e.g. $\mathrm{Inside}$,
$\mathrm{OnTop}$ etc.) between object instances (e.g., carton and cupboard)
that must be true at the end of the robot’s plan execution. The declarative
goal input to the model is partitioned as relations $g_{rel}$ and the object
instances specified in the goal $g_{obj}$. The resulting encondings are
denoted as $\tilde{g}_{rel}$ and $\tilde{g}_{obj}$:

$\tilde{g}_{rel}=\frac{1}{|g_{rel}|}\sum_{j\in g_{rel}}\mathcal{C}(j)\hskip
5.69054pt\mathrm{and}\hskip
5.69054pt\tilde{g}_{obj}=\frac{1}{|g_{obj}|}\sum_{o\in
g_{obj}}\mathcal{C}(o).$

Next, the goal encoding and the action history encoding $\tilde{\eta}_{t}$ is
used to learn attention weights over objects in the environment (?) such that

$\displaystyle\begin{split}\epsilon_{o}=\mathrm{softmax}\left(W_{g}[\tilde{s}_{t}^{o};\tilde{g}_{obj};\tilde{\eta}_{t}]+b_{g}\right),\end{split}$
(8)

where $\tilde{s}_{t}^{o}$ is obtained from $f^{state}_{\theta}(s_{t})$ in (6)
and $\tilde{\eta}_{t}$ is obtained from $f^{hist}_{\theta}(\eta_{t})$ in (7).
This results in the attended scene encoding $\Omega_{t}$ as:

$\Omega_{t}=f^{goal}_{\theta}(\tilde{s}_{t}^{o},\tilde{g}_{obj},\tilde{\eta}_{t})=\sum_{o\in
O}\mathrm{\epsilon_{o}}\tilde{s}_{t}^{o}.$ (9)

The attention mechanism aligns the goal information with the scene learning a
task-relevant context, relieving the model from reasoning about objects in the
environment unrelated to the task, which may be numerous in realistic
environments.

###  4.4 Tool Likelihood Prediction

Now we define the function that predicts the likelihood scores for each tool
in the environment state, i.e., the $f^{tool}_{\theta}\left(\cdot\right)$
function in equation (3). This likelihood score corresponds to the probability
that a particular tool could be used to reach a goal state for a given
environment state and declarative goal specification. In order to allow the
model to generalize to unseen tools, instead of prediction over the pre-
defined tool set $\hat{\tau}$, we allow the model to predict a likelihood
score of a tool $t$ (which may not be present in any of the scenes in the
training set) to be used as a tool using the object embedding
($e_{t}=\mathcal{C}(t)$ as described in Section 4.1.1). This recurrence is
shown in the factored tool likelihood module in Figure 2. The prediction is
made using the encoding of the state, i.e the attended scene embedding
$\Omega_{t}$, the relational description of the goal $\tilde{g}_{rel}$ and the
action history encoding $\tilde{\eta}_{t}$. Likelihood of each tool is
computed for each $t$ using,

$p_{t}=\mathrm{sigmoid}(W[\Omega_{t};\tilde{g}_{rel};\tilde{\eta}_{t};e_{t}])\
\forall\ t\in\hat{\tau},$ (10)

where $\Omega_{t}$ is obtained using (9) and $\tilde{\eta}_{t}$ is obtained
using (7). We now predict over a tool set $\tau$, which may be different from
the fixed toolset seen during training set. We include the possibility of
having tools present at inference time that are unseen during training. The
factored style likelihood prediction enables our model to be flexible to make
likelihood predictions for unseen tools. We now use this to define the
likelihood score for each object as $p_{t}$ for tools and $0$ otherwise. Here
$p_{t}$ is the probability that tool t can be used to complete the goal.
Specifically,

$f^{tool}_{\theta}(s_{t},g_{t},\eta_{t})=\\{p_{o}\text{ if }o\in\tau\text{
else }0\ |\ \forall o\in{\cal O}(s_{t})\\}.$ (11)

Unlike the original ToolNet model (?), $f^{tool}_{\theta}\left(\cdot\right)$
function in this work can predict the tool likelihood scores for each time-
step $t$ and utilizes action history encoding $\tilde{\eta}_{t}$ to capture
temporal context.

Figure 3:  ToolTango neural model is architecturally similar to ToolNet but
also incorporates tool likelihood scores predicted from the ToolNet model,
finally decoding the next symbolic action for the robot to execute. The
interaction type and objects are predicted auto-regressively and in a factored
style.

###  4.5 Robot Action Prediction

We now take the encoded information about the world state, goal and action
history to decode the next symbolic action $a_{t}=I_{t}(o^{1}_{t},o^{2}_{t})$.
The three components $I_{t}$, $o^{1}_{t}$ and $o^{2}_{t}$ are predicted auto-
regressively, where the prediction of the interaction, $I_{t}$ is used for the
prediction of the first object, $o^{1}_{t}$ and both their predictions are
used for the second object prediction, $o^{2}_{t}$. For the object predictors
$o^{1}_{t}$ and $o^{2}_{t}$, instead of predicting over a predefined set of
objects, our model predicts a likelihood score of each object $o\in O$ based
on its object embedding $\tilde{s}_{t}^{o}$ and tool likelihood score $p_{o}$
from (11). It then selects the object with highest likelihood score. The
resulting factored likelihood allows the model to generalize to an _a-priori_
unknown number and types of object instances:

$\displaystyle
I_{t}=\mathrm{argmax}_{I\in\mathcal{I}}\left(\mathrm{softmax}(W_{I}[\Omega_{t};\tilde{g}_{rel};\tilde{\eta}_{t}]+b_{I})\right),$
(12)
$\displaystyle\begin{split}o^{1}_{t}&=\mathrm{argmax}_{o\in{O}}\alpha_{t}^{o}\\\
&=\mathrm{argmax}_{o\in{O}}\
(\sigma(W_{\alpha}[\Omega_{t};\tilde{g}_{rel};\tilde{\eta}_{t};e_{o};p_{o};\vec{I}_{t}]+b_{\alpha}))\mathrm{,}\end{split}$
(13) $\displaystyle o^{2}_{t}=\mathrm{argmax}_{o\in{O}}\
(\sigma(W_{\beta}[\Omega_{t};\tilde{g}_{rel};\tilde{\eta}_{t};e_{o};p_{o};\vec{I}_{t};\alpha_{t}^{o}]+b_{\beta})).$
(14)

Here $\alpha^{o}_{t}$ denotes the likelihood prediction of the first object.
Finally, we impose grammar constraints (denoted as $\Lambda$) at inference
time based on the number of arguments that the predicted interaction $I_{t}$
accepts. If $I_{t}$ accepts only one argument, only $o^{1}_{t}$ is selected,
otherwise both are used. Thus, predicted action for time-step $t$ is denoted
as

$a_{t}=f^{act}_{\theta}(\Omega_{t},\tilde{g}_{rel},\tilde{\eta}_{t})=\Lambda[I_{t}(o^{1}_{t},o^{2}_{t})].$
(15)

This action is then executed by the robot in simulation. Our simulation model
follows closely to the one developed by ? (?) wherein symbolic actions such as
$\mathrm{moveTo}$, $\mathrm{pick}$ and $\mathrm{place}$ are realized as
relations in the environment state using a PyBullet simulator. For instance,
the $\mathrm{moveTo}$ action changes the xyz coordinates of the robot such
that it is $\mathrm{Near}$ the target object. Similarly, $\mathrm{pick}$
establishes a $\mathrm{connectedTo}$ relation between the target object and
robot arm. For implementation specific details, visit
https://github.com/reail-iitd/tango/wiki. The executed action and resulting
world state is provided as input to the model for predicting the action at the
next time step. The $f^{act}_{\theta}\left(\cdot\right)$ function denotes the
ToolTango model. In our original Tango model presented in (?), the $p_{o}$
tool likelihood score was not part of equations (13) and (14).

###  4.6 Word Embeddings Informed by a Knowledge Base

ToolTango uses word embedding function $\mathcal{C}(\cdot)$ that provides a
dense vector representation for word tokens associated with object class and
relation types. Contemporary models use word embeddings acquired from language
modeling tasks (?). We adopt embeddings that are additionally informed by an
existing knowledge graph $\mathrm{ConceptNet}$ (?) that provides a
sufficiently large knowledge graph connecting words with edges expressing
relationships such as $\mathrm{SimilarTo}$, $\mathrm{IsA}$,
$\mathrm{UsedFor}$, $\mathrm{PartOf}$ and $\mathrm{CapableOf}$. Word
embeddings (?) can be _retro-fitted_ such that words related using knowledge
graph embeddings are also close in the embedding space (?). Using such (pre-
trained) embeddings incorporates _general purpose_ relational knowledge to
facilitate richer generalization for downstream policy learning. Thus, given
an object, say $apple$, the word embedding function $\mathcal{C}(.)$ returns a
dense $\mathrm{ConceptNet}$ vector that corresponds to this token. The
complete sequence of steps is summarized in Figure 3.

###  4.7 Model Training

We decompose the action prediction task into first predicting tool, i.e., the
$f^{tool}_{\theta}\left(\cdot\right)$ function and then evaluating the
$f^{act}_{\theta}\left(\cdot\right)$ function. We also decompose model
training. First, we train the tool prediction model. The loss used to train
this model is Binary Cross-Entropy with each $t\in\hat{\tau}$ acting as a
class. The class label, $y^{i}_{t}$ is assigned 1 if it is used in a given
demonstration, $i$ and 0 otherwise. We also use categorical weights based on
plan execution time to encourage shorter plans (?). However, the knowledge of
the time taken for different plans has not been injected into the model. In
order to make this notion explicit to the model we use loss weighting such
that for the dataset $\mathcal{D_{\mathrm{Train}}}$ defined in (2),

$\mathcal{L}=-\sum_{i}\alpha_{i}\sum_{j}\sum_{t\in\tau}y^{i}_{j,t}\log(p^{i}_{j,t})+(1-y^{i}_{j,t})\log(1-p^{i}_{j,t}),$
(16)

where, $p^{i}_{j,t}$ is obtained from
$f^{tool}_{\theta}(s^{i}_{j},g^{i},\eta^{i}_{j})$ for each datapoint in
$\mathcal{D_{\mathrm{Train}}}$ and $\alpha_{i}$ is a multiplier that is high
for optimal plans (shortest among human demonstrations) and low otherwise.

For a trained tool prediction model, we then train the
$f^{act}_{\theta}\left(\cdot\right)$ ToolTango model, fine-tuning the weights
of our updated ToolNet function $f^{tool}_{\theta}\left(\cdot\right)$. Here
too we use the Binary Cross-Entropy loss, with the loss for the three
predictors (action and the two objects) being added independently. Additional
model training and hyperparameter details are given in Appendix B.

This decomposed training style has direct implications on the training
stability. First training ToolNet to predict the likelihood scores for tools
enables action prediction informed by the likelihood scores of each tool in
the environment.

##  5 Data Collection Platform and Annotation

Domain | Plan lengths | Objects interact- | Tools used | Sample objects | Sample goal specifications
---|---|---|---|---|---
ed with in a plan | in a plan
Home | 23.25$\pm$12.65 | 4.12$\pm$1.97 | 0.93$\pm$0.70 |  floor1, wall, fridge123, cupboard123, tables1, couch1, big-tray1, tray1, book1, paper, cubes, light switch4, bottle, box2, fruits, chair15, stick, dumpster2, milk carton, shelf1, glue6, tape6, stool15, mop8, sponge8, vacuum8, dirt7, door2 |  1\. Place milk in fridge, 2. Place fruits in cupboard, 3. Remove dirt from floor, 4. Stick paper to wall, 5. Put cubes in box, 6. Place bottles in dumpster, 7. Place a weight on paper, 8. Illuminate the room.
Factory | 38.77$\pm$23.17 | 4.38$\pm$1.85 | 1.44$\pm$0.97 |  floor1, wall, ramp, worktable1, tray1, box2, crates1, stick, long-shelf1, lift1, cupboard123, drill4, hammer49, ladder5, trolley2, brick, blow dryer48, spraypaint4, welder4, generator4, gasoline, coal, toolbox2, wood cutter4, 3D printer4, screw9, nail9, screwdriver49, wood, platform1, oil7, water7, board, mop8, glue6, tape6, stool15 |  1\. Stack crated on platform, 2. Stick paper to wall, 3. Fix board on wall, 4. Turn on the generator, 5. Assemble and paint parts, 6. Move tools to workbench, 7. Clean spilled water, 8. Clean spilled oil.

Table 1: Dataset characteristics. The average plan length measured as the
length of action sequence), number of objects interacted in plan and number of
tools used in plans with object and goal sets for Home and Factory domains.
Object positions were sampled using Gaussian distribution. Objects in bold can
be used as tools. Legend:- 1: surface, 2: can open/close, 3: container, 4: can
operate, 5: can climb, 6: can apply, 7: can be cleaned, 8: cleaning agent, 9:
can 3D print. Objects in bold can be used as tools. Object affordances derived
from properties in the ConceptNet graph (?). Stool/ladder are objects used to
represent a tool for raising the height of the robot.

###  5.1 Data Collection Environment

We develop a low-fidelity simulation environment where the robot can take
actions and interact with objects. The low-level physics of motion is less
important as we are primarily concerned with the symbolic effects on the world
state. To do this, we use PyBullet, a physics simulator (?), to generate home
and factory-like environments populated with a virtual mobile manipulator (a
Universal Robotics (UR5) arm mounted on a Clearpath Husky mobile base). The
world state is object-centric including the metric locations of objects and
the discrete states of symbolic attributes and relations. The objects in the
domains were derived from real-world home and factory scenes that span
Facebook Replica Dataset (?). These scenes were made to look as photo
realistic as possible. The objects on other hand were chosen to span the YCB
dataset (?). The CAD model for each object obtained from open-source
repositories such as the Google 3D Warehouse111
https://3dwarehouse.sketchup.com/. This was done in order to keep the
simulated physics as real as possible. The set of objects in the two domains
are listed in Table 1.

Each object in the environment is associated with a metric location and
physical extent and may optionally possess discrete states such as
$\mathrm{Open/Closed}$, $\mathrm{On/Off}$, etc. The world model is assumed to
possess spatial notions such as $\mathrm{Near}$ or $\mathrm{Far}$. Objects in
the world model can be supported by, contained within or connected with other
objects (or the agent). Hence, we include semantic relations such as
$\mathrm{OnTop}$, $\mathrm{Inside}$, $\mathrm{ConnectedTo}$ etc. More details
in Appendix D.

Robot Actions
---
Push, Climb up/down, Open/Close, Switch on/off, Drop, Pick, Move to, Operate
device, Clean, Release material on surface, Push until force
Object Attributes
Grabbed/Free, Outside/Inside, On/Off, Open/Close, Sticky/Not Sticky,
Dirty/Clean, Welded/Not Welded, Drilled/Not Drilled, Driven/Not Driven,
Cut/Not Cut, Painted/Not Painted
Semantic Relations
On top, Inside, Connected to, Near
Metric Properties
Position, Orientation, Size

Table 2: Domain Representation. Robot symbolic actions, semantic attributes,
relations to describe the world state and objects populating the scene in Home
and Factory Domains.

The robot possesses a set of behaviours or symbolic actions such as
$\mathrm{Moving}$ towards an object, $\mathrm{Grasping}$,
$\mathrm{Releasing/Dropping}$ or $\mathrm{Pushing}$ an object or
$\mathrm{Operating}$ an entity to imply actions that induce discrete state
changes such as opening the door before exiting, turning on a switch etc. We
assume that the robot’s actions can be realized by the presence of an
underlying controller. We encode the geometric requirements for actions as
symbolic pre-conditions. Examples include releasing an object from the gripper
before grasping another, opening the door before trying to exit the room. The
set of possible actions and the range of interactions are listed in Table 2.

The robot is tasked with goals that involve multiple interactions with objects
derived from standardized data sets (?). These goals include: (a)
_transporting_ objects from one region to another (including space on top of
or inside other objects), (b) _fetching_ objects, which the robot must reach,
grasp and return with, and (c) _inducing state changes_ such as illuminating
the room or removing dirt from floor. We assume that the robot is instructed
by providing _declarative_ goals. For example, the task of moving all fruits
on top of the kitchen table can be modeled as a set intended constraints among
the interacting objects. The effects of actions such as pushing or moving are
simulated via a motion planner and propagated to the next time step. Abstract
actions such as attachment, operating a tool or grasping/releasing objects are
encoded symbolically as the establishment or release constraints. The
simulation for these actions is coarse and considers their symbolic effects
forgoing the exact motion/skills needed to implement them. We assume that the
robot can realize abstract actions through low-level routines.

We present the human agents with a scene and goal conditions we want to attain
and receive actions in a custom grammar. The instructions are grounded in the
world and the agent and the virtual environment show its effects to input the
next action. This is repeated till a goal state is reached. Figure 4
illustrates the interactive platform used for data collection. Using a curated
dataset, the robot must synthesize a plan of executable actions to satisfy the
goal constraints. The presence of a rich space of interactions gives rise to
plans with multiple interactions between objects. For example, “packing items
into a basket and carrying the basket to the goal region”, “using a stick to
fetch and drop an object beyond reach into a box”, “using a ramp/stool to
elevate itself to fetch an object”.

###  5.2 Annotated Corpus

To curate an imitation learning dataset, we recruit human instructors and
provide them with goals. They instruct the robot by specifying a sequence of
symbolic actions (one at a time) to achieve each goal. Each action is
simulated so that they can observe its effects and the new world state. We
encourage the instructors to to complete the task as quickly as possible,
making use of available tools in the environment. To familiarize them with the
simulation platform, we conduct tutorial sessions before data collection. Our
resulting dataset consists of diverse plans with different action sets and
object interactions. We collected plan traces from $\mathrm{12}$ human
subjects using domain randomization with $\mathrm{10}$ scenes and $8$ semantic
goals resulting in a corpus of $\mathrm{708}$ and $\mathrm{784}$ plans for the
home and factory domains. Figure 5(a) and 5(b) show number of interactions
with 10 most interacted objects and frequency of 10 most frequent actions
respectively. The complete set of objects and goals is given in Table 1. We
also perform data augmentation by perturbing the metric states in the human
plan trace, performing random replacements of scene objects and validating
plan feasibility in simulation. The process results in $\mathrm{3540}$ and
$\mathrm{3920}$ plans, respectively. Variation was observed in tool use for an
instructor for different goals, and within similar goals based on context. The
annotated corpus was split as $(75\%:25\%)$ forming the Training data set and
the Test data set to evaluate model accuracy. A $10\%$ fraction of the
training data was used as the Validation set for hyper-parameter search. No
data augmentation was performed on the Test set.

Figure 4: Data Collection Interface. The human teacher instructs (left) a
virtual mobile manipulator robot by specifying symbolic actions. The human-
instructed plan is simulated and visualized (right). The user is shown the
goal needed to be completed in text form. They select the user interaction,
first object and second object to instruct the robot. The interface can be
seen in action in the https://www.youtube.com/watch?v=lUWU3rK1Gno

.

(a) Object interactions

(b) Symbolic actions

Figure 5: Data set Characteristics. Distribution of plans with plan length for
home and factory domains. Frequency of interaction of top 10 objects and
frequency of actions for top 10 actions. The collected data set contains
diverse interactions in complex spaces.

###  5.3 Generalization Test Set

In order to asses the model’s capacity to generalize to unseen worlds, we
curate a second test environment populated by instances of novel object types
placed at randomized locations. The following sampling strategies were used:
(i) _Position_ : perturbing and exchanging object positions in a scene. (ii)
_Alternate_ : removal of the most frequently used tool in demonstrated plans
evaluating the ability to predict the alternative, next best tool to use.
(iii) _Unseen_ : replacing an object with a similar object, which is not
present in training. (iv) _Random_ : replacing a tool with a randomly picked
object which is _unrelated_ to the task. (v) _Goal_ : replacing the goal
objects (objects included as part of the goal specifications). For example:
milk and fridge in the goal “put milk inside fridge”. Here milk may be
replaced with another similar object, such as apple and fridge by a container,
such as cupboard. This process resulted in a Generalization Test set with
$7460$ (goal, plan) pairs.

##  6 Experiments

We first present the results corresponding to high-level tool prediction that
leverages an updated ToolNet model. We then present the goal-reaching
performance of Tango and the unified ToolTango model. Finally, we present a
detailed analysis of the resulting plans in terms of generalization,
robustness to execution errors and plan efficiency.

###  6.1 Sequential Tool Prediction

####  6.1.1 Baseline

We compare against the basic GGCN model of Section 4.1.1 as our baseline
model. This model is similar to the ResActGraph model proposed by ? (?) and
its encoder incorporates technical ideas from recent imitation learning works
(?) on action prediction. In the subsequent discussion, we consider similar
size of the hyperparameter set across all models performing the same task.

####  6.1.2 Evaluation Metrics

We first compare the accuracy of the updated ToolNet model on the test and
generalization test sets. We test ToolNet’s tool prediction capabilities in
two settings. In the first setting, we use the dataset as described in Section
5 and split it according to the scene instance, i.e., home and factory. We use
accuracy as our performance measure. Here, a tool prediction is deemed correct
if the predicted tool is used in at least one of the various annotated plans
for the $(\rm{goal,scene})$ pair and incorrect otherwise.

Model |  Test Set |  Generalization | Generalization Test cases
---|---|---|---
| Home | Factory  | Home | Factory  | Position | Alternate | Unseen | Random | Goal
Baseline (GGCN)  | 32.01 | 20.89  | 21.22 | 19.50  | 8.73 | 6.29 | 12.22 | 9.81 | 46.31
Updated ToolNet | 79.87 |  75.12 | 80.16 |  78.01 | 75.55 | 58.83 | 60.74 | 49.97 | 94.10

Table 3: A comparison of the updated ToolNet model with its previous version
and GGCN baseline for the accuracy of prediction of tool sequences instead of
single tool. Highest scores are shown in bold.

####  6.1.3 Results

Table 3 shows the final accuracies on the test-set and generalization test-set
with individual accuracies for each test-type. On the regular test set,
ToolNet outperforms the baseline by 47.87 (149% higher) and 54.23 (259%
higher) accuracy points on Home and Factory domains, respectively.

A similar pattern is found in the generalization test set, where each
improvement is 58.94 (278% higher) and 58.51 (300% higher) accuracy points for
Home and Factory domains. The reasoning behind the improvements is as follows.
The goal-conditioned attention gives major improvement in the Goal
generalization test type, since goal objects get replaced in those examples.
Explicitly biasing the model to use the features of those objects (through
conditioned attention) increases their importance, and likely reduces
overfitting. An example for Alternate case is when _generator_ is specified in
the goal, and wood (the fuel for generator, and the most likely tool) is made
absent from the scene. The model could err in giving attention to the _wood-
cutter_ tool, which is often correlated with wood. However, conditioned
attention gives low attention to _wood-cutter_ and predicts _gasoline_ ,
instead.

Moreover, factored likelihood predictions helps the most in Unseen cases,
since without this component, the model cannot predict any unseen tool. A
decent performance of earlier models in this case is attributed to alternative
possible correct answers (any alternative seen tool or no-tool) due to
multiple annotations per scenario. The $\mathrm{ConceptNet}$ embeddings likely
contain commonsense knowledge about unseen tools and objects, for example,
whether a new tool is flat or not (which should help in ascertaining whether
it can be used for transport or not). Using these embeddings makes huge
improvement in Unseen cases where entirely new objects are to be predicted as
tools. Finally, giving higher weight to optimal plans allows the model to
differentiate tools by plan execution time and not human usage frequency. This
helps in improved metric generalization, predicting nearby tools in the
Position test case. Overall, the complete architecture provides the maximum
generalization accuracy among all models.

###  6.2 Action Prediction

####  6.2.1 Baselines

We compare to the following three baseline models. (1) _ResActGraph_ model
(?), augmented with $\mathrm{FastText}$ embeddings (?). (2) _Affordance-only_
baseline inspired from (?) that learns a co-association between tasks and
tools, implemented by excluding the graph convolutions and attention from
Tango. (3) _Vanilla Deep Q-Learning (DQN)_ approach (?) that learns purely by
interactions with a simulator, receiving positive reward for reaching a goal
state.

####  6.2.2 Tool Prediction Accuracy

Our experiments use the following accuracy metrics for model evaluation: (i)
_Action prediction accuracy_ : the fraction of tool interactions predicted by
the model that matched the human demonstrated action $a_{t}$ for a given state
$s_{t}$, and (ii) _Plan execution accuracy_ : the fraction of estimated plans
that are successful, i.e., can be executed by the robot in simulation and
attain the intended goal (in max. $50$ steps).

We first present the results for the vanilla Tango model and later present
improvements with the ToolTango.

####  6.2.3 Comparing Tango with Baselines

Table 4 (top half) compares the Tango model performance with the baseline
models. The Tango model shows a $14-23$ point increase in _Action prediction
accuracy_ and a $66-71$ points increase in the _Plan execution accuracy_ when
compared to the _ResActGraph_ baseline. Note that the ResActGraph model learns
a scene representation assuming a fixed and known set of object types and
hence can only generalize to new randomized scenes of known objects. In
contrast, the Tango model can not only generalize to randomized scenes with
known object types (sharing the GGCN backbone with ResActGraph) but can to
novel scenes new object types (relying on dense semantic embeddings) and an
a-priori unknown number of instances (enabled by a factored likelihood).

The _Affordance-only_ baseline model is confined to learning the possible
association between a tool object type and the task specified by the human
(largely ignoring the environment context). This approach addresses only a
part of our problem as it ignores the sequential decision making aspect, where
tools may need to be used in sequence to attain a goal. Finally, the vanilla
DQN baseline achieves less than $20\%$ policy accuracy (even after a week of
training). In contrast, the Tango model shows accurate results after training
on imitation data for $12-18$ hours. The challenges in scaling can be
attributed to the problem size ($\approx$1000 actions), long plans, sparse and
delayed rewards (no reward until goal attainment).

Model |  Action Prediction |  Plan Execution | Generalization Plan Execution Accuracy
---|---|---|---
| Home | Factory  | Home | Factory  | Home | Factory | Position | Alternate | Unseen | Random | Goal
Baseline (ResActGraph)  | 27.67 | 45.81  | 26.15 | 0.00  | 12.38 | 0.00 | 0.00 | 0.00 | 0.00 | 25.10 | 9.12
Affordance Only  | 46.22 | 52.71  | 52.12 | 20.39  | 44.10 | 4.82 | 17.84 | 47.33 | 29.31 | 29.57 | 34.85
DQN  | - | \-  | 24.82 | 17.77  | 15.26 | 2.23 | 0.00 | 0.00 | 12.75 | 9.67 | 4.21
Tango | 59.43 | 60.22  | 92.31 | 71.42  | 91.30 | 60.49 | 93.44 | 77.47 | 81.60 | 59.68 | 59.41
Model Ablations
\- GGCN (World Representation)  | 59.43 | 60.59  | 84.61 | 27.27  | 78.02 | 38.70 | 70.42 | 58.79 | 60.00 | 56.35 | 38.64
\- Metric (World Representation)  | 58.8 | 60.84  | 84.61 | 62.34  | 72.42 | 51.83 | 59.68 | 67.19 | 60.79 | 84.47 | 21.70
\- Goal-Conditioned Attn  | 53.14 | 60.35  | 53.85 | 11.69  | 37.02 | 8.80 | 35.33 | 15.05 | 32.14 | 41.67 | 6.51
\- Temporal Action History  | 45.91 | 49.94  | 24.61 | 0.00  | 8.55 | 0.00 | 0.00 | 0.00 | 0.00 | 30.56 | 1.15
\- Factored Likelihood  | 61.32 |  61.34 | 95.38 |  85.71 | 34.22 | 43.44 | 90.50 | 14.82 | 30.65 | 64.67 | 53.26
\- ConceptNet  | 63.52 | 60.35  | 89.23 | 57.14  | 81.86 | 56.97 | 82.33 | 68.61 | 74.57 | 65.73 | 47.92
\- Constraints  | 57.23 | 57.74  | 64.62 | 37.66  | 62.98 | 41.95 | 84.95 | 45.39 | 39.99 | 36.11 | 84.85
\- Auto-regression  | 56.60 | 60.22  | 69.23 | 50.65  | 73.75 | 53.32 | 71.24 | 66.43 | 61.27 | 70.51 | 34.66

Table 4: A comparison of _Action prediction_ and _Plan execution_ accuracies
for the baseline, the proposed Tango model, and ablations. Results are
presented for test and generalization data sets (under five sampling
strategies) derived from the home and factory domains. Accuracy Prediction is
the percentage of predicted actions matching the human input on the Test set.
Plan Execution is the percentage of plans successfully executed in the Test
set. Generalization Plan Accuracy is the percentage of plans successfully
executed in the set. Highest scores shown in bold.

Next, we assess the zero-shot transfer setting, i.e., whether the model can
perform common sense generalization in worlds with new objects unseen in
training. The same table shows that the plans predicted by Tango lead to an
increase of up to $56$ points in plan execution accuracy on Generalization
Test set over the best-performing baseline model. This demonstrates accurate
prediction and use of unseen tool objects for a given goal. Specifically, in
the home domain, if the _stool_ is not present in the scene, the model is able
to use a _stick_ instead to fetch far-away objects. Similarly, if the robot
can predict the use of a box for transporting objects even if it has only seen
the use of a tray for moving objects during training. The ResActGraph model is
unable to adapt to novel worlds and obtains zero points in several
generalization tests.

The poorer performance of the _Affordance-only_ model can again be attributed
to the fact that planning tool interactions involves sequential decision-
making. Even if the robot can use affordance similarity to replace a _tray_
object with a _box_ , it still needs to predict the opening of the _box_
before placing an item in its plan for a successful execution. This is
corroborated by the drop in performance for the Unseen generalization tests
for this model by $52.3$ points. Finally, the vanilla DQN model lacks a clear
mechanism for transferring to novel settings, hence shows poor generalization
in our experiments.

####  6.2.4 Ablation Analysis of Tango Components

We analyze the importance of each component of the Tango model by performing
an ablation study. Table 4 (lower half) presents the results. For a fair
comparison, the model capacities remain the same during the ablation
experiments.

The model builds on the _GGCN_ environment representation encoding the inter-
object and agent-object relational properties. The ablation of the GGCN
component results in a reduction of 22% in the generalization accuracy in the
factory domain (where tools may be placed at multiple levels in the factory).
The inclusion of this component allows the robot to leverage relational
properties such as OnTop to predict the use of tools such as a ramp to
negotiate an elevation or a stick to fetch an object immediately beyond the
manipulator’s reach.

The _Metric_ component encodes the metric properties of objects in the scene
such as positions, size etc. Experiments demonstrate its effectiveness in
prediction tool interactions based on relative sizes of interacting objects.
E.g., the model successfully predicts that _fruits_ can be transported using a
_tray_ but larger _cartons_ require a _box_ for the same task. The ablation of
this component leads to a reduction of $10.2$ points in the Alternate
generalization tests as the ablated model unable to adapt the tool when there
are unseen objects with different sizes than those seen during training.

Next, we assess the impact of removing the _Goal-Conditioned Attention
component_. This experiment shows a a significant reduction ($\approx 50$
points) in the _Plan execution accuracy_ on the Generalization Test set,
particularly in scenes with a large number of objects. The attention mechanism
allows learning of a restricted context of tool objects that may be useful for
attaining the provided goal, in essence, filtering away goal-irrelevant
objects populating the scene. Additionally, note that the inclusion of this
component allows tool predictions to be _goal-aware_. Consequently, we observe
ablating this component leads to a reduction of $53$ points in the Goal
generalization test set where the goal objects are perturbed.

The _Action History_ component utilizes the agent’s past interactions for the
purpose of predicting the next tool interaction. The inclusion of this
component allows learning of correlated and commonly repeated action
sequences. For instance, the task of exiting from a room typically involves a
plan fragment that includes moving to a door, opening it and exiting from the
door and are commonly observed in a number of longer plans. The ablation of
this component leads to erroneous predictions where a particular action in a
common plan fragment is missing or incorrectly predicted. E.g., a robot
attempting to pick an object inside an enclosure without opening the lid. In
our experiments, we observe that ablating the model leads to a significant
decrease in goal reach-ability, causing a $70$ point decrease in the _Plan
Execution accuracy_ and $72$ point drop in the _Generalization accuracy_.

The need for generalization to novel scenes implies that our model cannot
assume a fixed and a-priori known set of objects that the robot can interact
with. Generalization to an arbitrary number of objects in the scene is
accomplished by factoring model predictions over individual objects in a
recurrent manner. Ablating the factored likelihood components results in a
simpler model that performs predictions over a known fixed-size object set.
The simplified model displays a higher action-prediction and plan-execution
accuracies in known world. Crucially, we observe that ablating this component
results in a significant decrease of $51$ and $63$ points in the Unseen and
the Alternate generalization test sets.

Finally, the _ConceptNet embeddings_ are important for semantic generalization
to unseen tool types. We replace ConceptNet embeddings with
$\mathrm{FastText}$ (?) embeddings in the _-ConceptNet_ model to show their
importance. The _-ConceptNet_ model shows poorer generalization (6.5%
decrease) as it models word affinity as expressed in language only.
$\mathrm{ConceptNet}$ embedding space models relational affinity between
objects as present in the knowledge-base.

Model |  Action Prediction |  Plan Execution | Generalization Plan Execution Accuracy
---|---|---|---
| Home | Factory  | Home | Factory  | Home | Factory | Position | Alternate | Unseen | Random | Goal
TANGO | 59.43 | 60.22  | 92.31 | 71.42  | 91.30 | 60.49 | 93.44 | 77.47 | 81.60 | 59.68 | 59.41
ToolTANGO | 64.12 |  63.72 | 95.69 |  77.01 | 92.88 | 62.97 | 94.38 | 88.12 | 92.71 | 78.01 | 64.43

Table 5: A comparison of TANGO with ToolTANGO model.

####  6.2.5 Comparing Tango with ToolTango

Table 5 compares the performance of ToolTango with the Tango model. ToolTango
gives improved scores in both domains and for every generalization test case.
The Action prediction accuracy of ToolTango is higher than Tango by 4.69 and
3.50 points for Home and Factory domains, respectively. Similarly, we see an
increase in the Plan execution accuracy both for Test and Generalization Test
sets. We get 3.38- 5.59 higher points on Test and 1.58-2.48 points in
Generalization Test set for Home and Factory domains. The improvement is
higher in Factory domain where the probability of using multiple tools in the
same plan to reach a goal state is higher. This is particularly due to the
independent tool prediction in ToolTango that aids action prediction in
complex settings requiring longer action sequences (more details in Section
6.3). The performance improvement is highest for the Alternate and Random
cases where the tools are replaced by an alternate tool or another non-tool
object. This shows the importance of independent tool-likelihood score
prediction in the ToolTango model.

Figure 6:  A simulated robot manipulator uses ToolTango to synthesize tool
interactions in novel contexts with unseen objects. (top) ToolTango predicts
the use of a box when tray is unavailable. (bottom) ToolTango predicts the use
of hammer and nails when screws are unavailable.  Figure 7: The model predicts
the instance of tray (on the left) which is closer to the fruits (goal
objects) other instance (one on the right).

Figure 8: Interleaved action prediction and execution enables adaptation in
case of unexpected errors during action execution. (top) robot recovers after
the milk carton falls. (bottom) recovers after a fruit falls from the tray by
picking it up again.

###  6.3 Analysis of Resulting Plans

####  6.3.1 Evidence of Generalization

Figure 6 shows the robot using the learned model to synthesize a plan for a
declarative goal. Here, if the goal is to transport fruits and human
demonstrates usage of _tray_ and the model never sees _box_ while training,
ToolTango uses _box_ in a scene where tray is absent, showing that it is able
to predict semantically similar tools for task completion. Similarly, for the
goal of fixing a board on the wall, if humans use _screws_ the agent uses
_nails_ and _hammer_ when screws are absent from the scene. Figure 7 shows how
the model uses the position information of tool objects to predict the tool
closer to the goal object or the agent. The world representation encodes the
metric properties of objects (position and orientation) that allows the robot
to interact with nearer tool objects.

Figure 9: Scatter plot comparing the lengths of plans obtained from model
predictions and those from human demonstrations. The plans generated from
ToolTango are of similar length as that of human demonstrators.

####  6.3.2 Robustness to Errors

Figure 8 shows the robustness to unexpected errors and stochasticity in action
execution. Consider the task of “fetching a carton”, where the milk carton is
on an elevated platform, the model predicts the uses a stool to elevate
itself. The carton falls due to errors during action execution. Following
which, the robot infers that the stool is no longer required and directly
fetches the carton. Similarly, for the task of “storing away the fruits in the
cupboard”, the robot predicts the use of tray for the transport task. During
execution the apple falls off the tray. The robot correctly re-collects the
apple.

####  6.3.3 Plan Efficiency Comparison

Figure 9 compares the length of robot plans predicted by the learned model
against human demonstrated plans. We observe that, on average, the predicted
plan lengths are close to the human demonstrated ones. In 12% cases, the plans
predicted by ToolTango utilize tools satisfying the goal condition in fewer
steps compared to the human demonstrated plan.

Figure 10: Test cases where TANGO fails but ToolTANGO reaches the goal state.
Typically in cases with complex and long plans with multiple tools being used.
Tango predicts stool in the first case and coal in the second case, both of
which are unreachable. ToolTango predicts reachable tools. (Top) Here the
robot needs to place the milk carton which is at a higher location which is
unreachable without a stool. In this test case, the stool in unreachable so
the model uses a stick to reach the milk carton. (Bottom) Switching on the
generator requires a fuel. In this test case coal is unreachable/unavailable
and so the model instead uses gasoline to turn on the generator.

Figure 11: Execution accuracy of inferred plans with plan length for Tango and
ToolTango. The latter gives a higher goal-reaching performance than the former
for plans with longer action sequences.

Figure 12: An analysis of fractional errors during plan execution using the
learned ToolTango model. The horizontal axis denotes the total errors for the
ablated model. The absolute value of the total errors are shown in Table 4

####  6.3.4 Analysis of Independent Tool Prediction

Figure 10 demonstrates the advantage of independent tool prediction and how it
augments the ToolTango to improve goal-reaching performance. The figure shows
two cases where the Tango model is unable to reach the goal state, but
ToolTango reaches a goal. The first example shows a setting where a milk
carton is placed on top of the fridge and stool is kept directly in front of
the fridge. In this case, Tango predicts the use of stool to reach the carton;
however, it first opens the fridge door, making stool inaccessible. On the
other hand, ToolTango predicts using a stick to drop the carton on top of the
table and is able to place it inside the fridge. The second example shows a
case where the generator needs to be powered on after adding a fuel source.
Here, ToolTango predicts the picking coal that is on top of the shelf.
However, it does not use any tool to elevate itself and the execution returns
an error of unreachable object. Instead, ToolTango predicts the use of
gasoline that is placed on the ground and is successfully able to execute the
plan.

Independent tool prediction also allows the model to perform better in complex
settings requiring longer action sequences and using multiple tools. Figure 11
assesses the model accuracy with the lengths of the inferred plans. We observe
that ToolTango has a higher plan execution accuracy in case of (goal, scene)
pairs where the average plan length to attain a goal state is high. For
instance, if the goal is to place fruits in side the cupboard and all the
fruits are on top of the fridge, the robot needs to use a tool to first reach
the fruits and then use another tool to carry them to the cupboard. In this
case the Tango model directly tries to pick unreachable fruits or predicts an
incorrect tool to reach the fruits. ToolTango is able to execute complex task
by subsequently using a stool and tray.

##  7 Limitations and Future Work

###  7.1 Scaling to Longer Plans

As we observed in Figure 11, the plan execution accuracy decreases by 20% on
the Test sets and 30% on Generalization Test sets. This merits investigation
into planning abstractions (?) for scaling to longer plan lengths in realistic
domains. Figure 12 analyzes the errors encountered during plan execution using
actions predicted by the proposed model. In 27% of the cases, the model misses
a pre-requisite actions required for a pre-condition for initiating the
subsequent action. For example, missing the need to open the door before
exiting the room (object unreachable $19\%$) or missing opening the cupboard
before picking an object inside it (object inside enclosure $8\%$). There is
scope for improvement here by incorporating explicit causal structure (?).

###  7.2 Partially observable environments

Figure 13: Likelihood estimates of exploration for different objects where
“screw” might be found.

In realistic scenarios, the robot’s environment may only be partially-known
due to the limited view and scope of the robot’s sensors. For example, tools
such as screws may be stored away in a tool box and may not be easily observed
by the robot. In such a scenario, we expect the robot to learn to predict
possible locations for exploration based on common sense knowledge. For
example, the robot should explore the tool box. Further, the subgeometry
information may be fed into the model to determine the set of objects in the
environment. In order to address partially-observed worlds, we can extend the
prediction model as follows. Instead of learning attention only over candidate
objects, we can learn a _generalized_ attention over spatial relations modeled
in the graph network. Such an extension allows the model to predict a
preference order for locations that the robot should explore to find the
required tool. Figure 13 illustrates an example, where the robot shows an
example, where the robot predicts that the screws may be found in toolbox or
workbench. Please note that this result is indicative of the possibility of
using the model in partially-known world and will be investigated in detail as
part of future work.

###  7.3 Extension to realistic robot control settings

The action representations adopted in this work are inspired from task and
motion planning communities (?, ?). We simulate a mobile manipulator
(Clearpath Robotics Husky with a mounted Universal Robotics UR5 arm). The
robot’s action space is modeled as high-level skills (or behaviors), each in-
turn realized using a low-level motion plan or controller that is
parameterized at run time. For navigation actions, a standard state-space
planner (A* with Manhattan distance as a heuristic) for a coarse collision
free path and realize point to point motion through a velocity-based
controller. For the manipulation, we consider a 6 degree of freedom model of
the manipulator. For object manipulation a crane-grasp was implemented where
the arm would move to a pre-grasp position to the intended object of interest.
The arm motion was realized using a joint state controller moving the arm to
the end-effector pose. The precise manipulation of tools is abstracted as
follows: once the robot end-effector makes contact a rigid joint is
established for subsequent motions till it is released. As mentioned in
Section 1, learning the precise manipulation of tools is not the focus of this
work and can be delegated to a method such as the one proposed by ? (?). All
robot actions are executed in the PyBullet physics engine with a mesh-based
model of objects and a URDF-model for the robot manipulation system. The
technical details appear in Appendix D. The entire implementation of the robot
behaviors and the simulation environment is available at
https://github.com/reail-iitd/tango for the use of the research community.

##  8 Conclusions

This paper proposes ToolTango, a novel neural architecture that learns a
policy to attain intended goals as tool interaction sequences leveraging
fusion of semantic and metric representations, goal-conditioned attention,
knowledge-base corpora. ToolTango is trained using a data set of human
instructed robot plans with simulated world states in home and factory like
environments. It uses an independent model that predicts likelihood scores for
each tool at each time-step. The imitation learner demonstrates accurate
commonsense generalization to environments with novel object instances using
the learned knowledge of shared spatial and semantic characteristics. It also
shows the ability to adapt to erroneous situations and stochasticity in action
execution. Finally, ToolTango synthesizes a sequence of tool interactions with
a high accuracy of goal-attainment.

Acknowledgments

Mausam is supported by an IBM SUR award, grants by Google, Bloomberg and 1MG,
Jai Gupta chair fellowship, and a Visvesvaraya faculty award by Govt. of
India. Rohan Paul acknowledges support from Pankaj Gupta Faculty Fellowship
and DST’s Technology Innovation Hub (TIH) for Cobotics. Shreshth Tuli is
supported by the President’s PhD Scholarship at Imperial College London. We
thank the IIT Delhi HPC facility and Prof. Prem Kalra and Mr. Anil Sharma at
the CSE VR Lab for compute resources. We thank Mr. Pulkit Sapra and Prof. P.
V. M. Rao for assistance with CAD model creation. We thank ? (?) for sharing
the implementation for baseline comparison. We are grateful to anonymous
turkers and student volunteers for assisting in the data collection.

## References

  * Allen et al. Allen, K. R., Smith, K. A., and Tenenbaum, J. (2019).  Rapid trial-and-error learning in physical problem solving..  In CogSci, p. 90.
  * Antunes et al. Antunes, A., Saponaro, G., Dehban, A., Jamone, L., Ventura, R., Bernardino, A., and Santos-Victor, J. (2015).  Robotic tool use and problem solving based on probabilistic planning and learned affordances.  In Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst. Workshop Learn. Object Affordances Fundamental Step to Allow Prediction Plan. Tool use.
  * Bae et al. Bae, H., Kim, G., Kim, J., Qian, D., and Lee, S. (2019).  Multi-robot path planning method using reinforcement learning.  Applied Sciences, 9(15), 3057.
  * Bahdanau et al. Bahdanau, D., Cho, K. H., and Bengio, Y. (2015).  Neural machine translation by jointly learning to align and translate.  In 3rd International Conference on Learning Representations, ICLR 2015.
  * Bansal et al. Bansal, R., Tuli, S., Paul, R., and Mausam (2020).  Toolnet: Using commonsense generalization for predicting tool use for robot plan synthesis.  In Workshop on Advances & Challenges in Imitation Learning for Robotics at Robotics Science and Systems (RSS).
  * Bisk et al. Bisk, Y., Zellers, R., Bras, R. L., Gao, J., and Choi, Y. (2020).  Piqa: Reasoning about physical commonsense in natural language.  In AAAI.
  * Boteanu et al. Boteanu, A., Kent, D., Mohseni-Kabir, A., Rich, C., and Chernova, S. (2015).  Towards robot adaptability in new situations.  In 2015 AAAI Fall Symposium Series. Citeseer.
  * Calli et al. Calli, B., Singh, A., Bruce, J., Walsman, A., Konolige, K., Srinavasa, S. S., Abbeel, P., and Dollar, A. M. (2017).  YCB Benchmarking Project: Object Set, Data Set and Their Applications.  Journal of The Society of Instrument and Control Engineers, 56(10), 792–797.
  * Chen et al. Chen, H., Tan, H., Kuntz, A., Bansal, M., and Alterovitz, R. (2020).  Enabling robots to understand incomplete natural language instructions using commonsense reasoning.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 1963–1969. IEEE.
  * Choi et al. Choi, D., Langley, P., and To, S. T. (2018).  Creating and using tools in a hybrid cognitive architecture..  In AAAI Spring Symposia.
  * Coumans and Bai Coumans, E., and Bai, Y. (2016).  Pybullet, a python module for physics simulation for games, robotics and machine learning.  In GitHub repository.
  * Driess et al. Driess, D., Oguz, O., Ha, J.-S., and Toussaint, M. (2020).  Deep visual heuristics: Learning feasibility of mixed-integer programs for manipulation planning.  In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 9563–9569. IEEE.
  * Finn et al. Finn, C., Yu, T., Zhang, T., Abbeel, P., and Levine, S. (2017).  One-shot visual imitation learning via meta-learning.  In Conference on Robot Learning, pp. 357–368. PMLR.
  * Fitzgerald et al. Fitzgerald, T., Goel, A., and Thomaz, A. (2018).  Human-guided object mapping for task transfer.  ACM Transactions on Human-Robot Interaction (THRI), 7(2), 1–24.
  * Fitzgerald et al. Fitzgerald, T., Goel, A., and Thomaz, A. (2021).  Modeling and learning constraints for creative tool use.  Frontiers in Robotics and AI, 8.
  * Gajewski et al. Gajewski, P., et al. (2019).  Adapting everyday manipulation skills to varied scenarios.  In 2019 International Conference on Robotics and Automation (ICRA), pp. 1345–1351. IEEE.
  * Garg et al. Garg, S., Bajpai, A., and Mausam (2019).  Size independent neural transfer for RDDL planning.  In Benton, J., Lipovetzky, N., Onaindia, E., Smith, D. E., and Srivastava, S. (Eds.), Proceedings of the Twenty-Ninth International Conference on Automated Planning and Scheduling, ICAPS 2018, Berkeley, CA, USA, July 11-15, 2019, pp. 631–636. AAAI Press.
  * Garg et al. Garg, S., Bajpai, A., and Mausam (2020).  Symbolic network: Generalized neural policies for relational mdps.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Vol. 119 of Proceedings of Machine Learning Research, pp. 3397–3407. PMLR.
  * Garrett et al. Garrett, C. R., et al. (2021).  Integrated task and motion planning.  Annu. Rev. Control Robot. Auton. Syst, 2021(4), 1–30.
  * Garrett et al. Garrett, C. R., Lozano-Pérez, T., and Kaelbling, L. P. (2020).  Pddlstream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning.  In Proceedings of the International Conference on Automated Planning and Scheduling, Vol. 30, pp. 440–448.
  * He et al. He, K., Zhang, X., Ren, S., and Sun, J. (2015).  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE international conference on computer vision, pp. 1026–1034.
  * Hermans et al. Hermans, T., Rehg, J. M., and Bobick, A. (2011).  Affordance prediction via learned object attributes.  In ICRA: Workshop on Semantic Perception, Mapping, and Exploration, pp. 181–184.
  * Holladay et al. Holladay, R., Lozano-Pérez, T., and Rodriguez, A. (2019).  Force-and-motion constrained planning for tool use.  In IROS.
  * Huang et al. Huang, D.-A., Nair, S., Xu, D., Zhu, Y., Garg, A., Fei-Fei, L., Savarese, S., and Niebles, J. C. (2019).  Neural task graphs: Generalizing to unseen tasks from a single video demonstration.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8565–8574.
  * Huang et al. Huang, S. H., Pan, J., Mulcaire, G., and Abbeel, P. (2015).  Leveraging appearance priors in non-rigid registration, with application to manipulation of deformable objects.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 878–885. IEEE.
  * Hübner et al. Hübner, J. F., Bordini, R. H., and Wooldridge, M. (2006).  Programming declarative goals using plan patterns.  In International Workshop on Declarative Agent Languages and Technologies, pp. 123–140. Springer.
  * Jain et al. Jain, A., Das, D., Gupta, J. K., and Saxena, A. (2015).  Planit: A crowdsourcing approach for learning to plan paths from large scale preference feedback.  In ICRA, pp. 877–884.
  * Kho et al. Kho, G., Hung, C., and Cunningham, H. (2014).  Robo brain: Massive knowledge base for robots.  In Cornell Univ., USA, Tech. Rep.
  * Kingma and Ba Kingma, D. P., and Ba, J. (2014).  Adam: A method for stochastic optimization.  In CoRR arXiv:1412.6980.
  * Kolobov et al. Kolobov, A., Mausam, Weld, D. S., and Geffner, H. (2011).  Heuristic search for generalized stochastic shortest path mdps.  In ICAPS.
  * Kroemer et al. Kroemer, O., Ugur, E., Oztop, E., and Peters, J. (2012).  A kernel-based approach to direct action perception.  In 2012 IEEE international Conference on Robotics and Automation, pp. 2605–2610. IEEE.
  * Lee et al. Lee, A. X., Gupta, A., Lu, H., Levine, S., and Abbeel, P. (2015).  Learning from multiple demonstrations using trajectory-aware non-rigid registration with applications to deformable object manipulation.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5265–5272. IEEE.
  * Levihn and Stilman Levihn, M., and Stilman, M. (2014).  Using environment objects as tools: Unconventional door opening.  In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2502–2508. IEEE.
  * Li et al. Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R. (2015).  Gated graph sequence neural networks.  In arXiv preprint arXiv:1511.05493.
  * Liao et al. Liao, Y.-H., Puig, X., Boben, M., Torralba, A., and Fidler, S. (2019).  Synthesizing environment-aware activities via activity sketches.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6291–6299.
  * Liu et al. Liu, Z., Freeman, W. T., Tenenbaum, J. B., and Wu, J. (2018).  Physical primitive decomposition.  In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19.
  * Lynch et al. Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., and Sermanet, P. (2020).  Learning latent plans from play.  In Conference on Robot Learning, pp. 1113–1132. PMLR.
  * Mandlekar et al. Mandlekar, A., Zhu, Y., Garg, A., Booher, J., Spero, M., Tung, A., Gao, J., Emmons, J., Gupta, A., Orbay, E., et al. (2018).  Roboturk: A crowdsourcing platform for robotic skill learning through imitation.  In Conference on Robot Learning, pp. 879–893. PMLR.
  * Mausam and Kolobov Mausam, and Kolobov, A. (2012).  Planning with Markov decision processes: An AI perspective.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 6(1), 1–210.
  * Migimatsu and Bohg Migimatsu, T., and Bohg, J. (2020).  Object-centric task and motion planning in dynamic environments.  IEEE Robotics and Automation Letters, 5(2), 844–851.
  * Mikolov et al. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., and Joulin, A. (2018).  Advances in pre-training distributed word representations.  In LREC.
  * Misra et al. Misra, D. K., Sung, J., Lee, K., and Saxena, A. (2016).  Tell me dave: Context-sensitive grounding of natural language to manipulation instructions.  IJRR, 35(1-3), 281–300.
  * Myers et al. Myers, A., Teo, C. L., Fermüller, C., and Aloimonos, Y. (2015).  Affordance detection of tool parts from geometric features.  In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 1374–1381. IEEE.
  * Nair et al. Nair, A., Chen, D., Agrawal, P., Isola, P., Abbeel, P., Malik, J., and Levine, S. (2017).  Combining self-supervised learning and imitation for vision-based rope manipulation.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 2146–2153. IEEE.
  * Nair and Chernova Nair, L., and Chernova, S. (2020).  Feature guided search for creative problem solving through tool construction.  In Frontiers in Robotics and AI, p. 205. Frontiers.
  * Nair et al. Nair, L., Srikanth, N. S., Erickson, Z. M., and Chernova, S. (2019a).  Autonomous tool construction using part shape and attachment prediction..  In Robotics: Science and Systems.
  * Nair et al. Nair, S., Zhu, Y., Savarese, S., and Fei-Fei, L. (2019b).  Causal induction from visual observations for goal directed tasks.  In CoRR arXiv:1910.01751.
  * Nyga and Beetz Nyga, D., and Beetz, M. (2018).  Cloud-based probabilistic knowledge services for instruction interpretation.  In Robotics Research, pp. 649–664. Springer.
  * Nyga et al. Nyga, D., Roy, S., Paul, R., Park, D., Pomarlan, M., Beetz, M., and Roy, N. (2018).  Grounding robot plans from natural language instructions with incomplete world knowledge.  In Conference on Robot Learning, pp. 714–723.
  * Park et al. Park, D., Noseworthy, M., Paul, R., Roy, S., and Roy, N. (2019).  Inferring task goals and constraints using bayesian nonparametric inverse reinforcement learning.  In Proceedings of the 3rd Conference on Robot Learning (CoRL).
  * Puig et al. Puig, X., Ra, K., Boben, M., Li, J., Wang, T., Fidler, S., and Torralba, A. (2018).  Virtualhome: Simulating household activities via programs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8494–8502.
  * Sarathy and Scheutz Sarathy, V., and Scheutz, M. (2018).  Macgyver problems: Ai challenges for testing resourcefulness and creativity.  Advances in Cognitive Systems, 6, 31–44.
  * Scalise et al. Scalise, R., Li, S., Admoni, H., Rosenthal, S., and Srinivasa, S. S. (2018).  Natural language instructions for human–robot collaborative manipulation.  The International Journal of Robotics Research, 37(6), 558–565.
  * Sharma et al. Sharma, S., Gupta, J., Tuli, S., Paul, R., and Mausam (2022a).  Goalnet: Inferring conjunctive goal predicates from human plan demonstrations for robot instruction following.  In International Conference on Automated Planning and Scheduling (ICAPS) - Planning and Reinforcement Learning Workshop.
  * Sharma et al. Sharma, V., Arora, D., Geißer, F., Mausam, and Singla, P. (2022b).  Symnet 2.0: Effectively handling non-fluents and actions in generalized neural policies for RDDL relational MDPs.  In Proceedings of the 38th Conference on Uncertainty in Artificial Intelligence (UAI).
  * Shridhar et al. Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. (2020).  Alfred: A benchmark for interpreting grounded instructions for everyday tasks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740–10749.
  * Silver et al. Silver, T., et al. (2021).  Planning with learned object importance in large problem instances using graph neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, pp. 11962–11971.
  * Speer et al. Speer, R., Chin, J., and Havasi, C. (2017).  ConceptNet 5.5: An open multilingual graph of general knowledge.  In AAAI.
  * Speer et al. Speer, R., Chin, J., and Havasi, C. (2019).  ConceptNet Numberbatch, the best pre-computed word embeddings you can use.  In GitHub repository.
  * Srivastava et al. Srivastava, S., Fang, E., Riano, L., Chitnis, R., Russell, S., and Abbeel, P. (2014).  Combined task and motion planning through an extensible planner-independent interface layer.  In 2014 IEEE international conference on robotics and automation (ICRA), pp. 639–646. IEEE.
  * Straub et al. Straub, J., Whelan, T., Ma, L., Chen, Y., Wijmans, E., Green, S., Engel, J. J., Mur-Artal, R., Ren, C., Verma, S., et al. (2019).  The replica dataset: A digital replica of indoor spaces.  In CoRR arXiv:1906.05797.
  * Toussaint et al. Toussaint, M. A., Allen, K. R., Smith, K. A., and Tenenbaum, J. B. (2018).  Differentiable physics and stable modes for tool-use and manipulation planning.  In Robotics: Science and Systems Foundation.
  * Tuli et al. Tuli, S., Bansal, R., Paul, R., and Mausam (2021).  Tango: Commonsense generalization in predicting tool interactions for mobile manipulators.  In International Joint Conference on Artificial Intelligence (IJCAI).
  * Vega-Brown and Roy Vega-Brown, W., and Roy, N. (2020).  Asymptotically optimal planning under piecewise-analytic constraints.  In Algorithmic Foundations of Robotics XII, pp. 528–543. Springer.
  * Wu et al. Wu, J., Lim, J. J., Zhang, H., Tenenbaum, J. B., and Freeman, W. T. (2016).  Physics 101: Learning physical object properties from unlabeled videos.  In British Machine Vision Conference.
  * Wu et al. Wu, J., Yildirim, I., Lim, J. J., Freeman, B., and Tenenbaum, J. (2015).  Galileo: Perceiving physical object properties by integrating a physics engine with deep learning.  In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M., and Garnett, R. (Eds.), Advances in Neural Information Processing Systems 28, pp. 127–135. Curran Associates, Inc.
  * Xie et al. Xie, A., Ebert, F., Levine, S., and Finn, C. (2019).  Improvisation through physical understanding: Using novel objects as tools with visual foresight.  In Robotics at Robotics Science and Systems (RSS).
  * Zhu et al. Zhu, Y., Tremblay, J., Birchfield, S., and Zhu, Y. (2021).  Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6541–6548. IEEE.

##  A Supplementary Material

All our code is available as a GitHub repository under BSD-2 License
https://github.com/reail-iitd/tango. Instructions for reproducing the results
are given at https://github.com/reail-iitd/tango/wiki. A supplementary video
demonstrating our data collection platform and model’s generalization
capability is available at https://www.youtube.com/watch?v=lUWU3rK1Gno.

##  B Hyperparameter Details

We detail the hyper-parameters for the ToolTango architecture introduced in
this paper.

  * •

_Graph Structured World Representation._ The Gated Graph Convolution Network
(GGCN) was implemented with $4$-hidden layers, each of size $128$, with
convolutions across $2$ time steps for every relation passing through a layer
normalized GRU cell. The Parameterized ReLU activation function with a $0.25$
negative input slope was used in all hidden layers.

  * •

_Word Embeddings._ The word embeddings (derived from $\mathrm{ConceptNet}$)
were of size $300$. Additionally, the semantic state of each object was
encoded as a one-hot vector of size $29$. Typically, there were $35$ and $45$
objects in the home and factory domains respectively.

  * •

_Fusing Metric Information._ The metric encodings were generated from the
metric information associated with objects using a $2$-layer Fully Connected
Network (FCN) with $128$-sized layers.

  * •

_Encoding Action History._ A Long Short Term Memory (LSTM) layer of size $128$
was used to encode the action history using the generalized action encoding
$\mathcal{A}(I_{t}(o^{1}_{t},o^{2}_{t}))$.

  * •

_Goal-conditioned Attention._ The attention network was realized as a
$1$-layer FCN of layer size $128$ with a $\mathrm{softmax}$ layer at the end.

  * •

_Tool Prediction._ To predict the tool-likelihood score $p_{t}$, a $3$-layer
FCN was used, each hidden layer with size $128$ and output layer with size $1$
with sigmoid activation function.

  * •

_Action Prediction._ To predict the action $I_{t}$, a $3$-layer FCN was used,
each hidden layer with size $128$ and output layer with size $|\mathcal{I}|$.
$I_{t}$ was converted to a one-hot encoding $\vec{I}_{t}$. This, with the
object embedding $e_{o}$ was passed to the $o^{1}_{t}$ predictor via an FCN.
This FCN consists of 3-hidden layers of size $128$ and a final layer of size
$1$ with a sigmoid activation (for likelihood). The $\vec{I}_{t}$ and
$o^{1}_{t}$ likelihoods were sent to the $o^{2}_{t}$ predictor to predict
likelihoods for all object embeddings $e_{o}$. This part was realized as a
$3$-layer FCN with hidden layer size $128$ and final layer of size $1$ with a
sigmoid activation function.

  * •

_Training parameters._ Model training used a learning rate of $5\times
10^{-4}$. The Adam optimizer (?) with a weight decay parameter of $10^{-5}$
and a batch size of $1$ was used. An early stopping criterion was applied for
convergence. The _action prediction accuracy_ was used as the comparison
metric on the validation set or up to a maximum of $200$ epochs.

##  C World Scenes

The figure 14 illustrates the object-centric graph representation of the 10
world scenes we used for Home and Factory domains. The agent node is shown in
green, tools in black and objects with states in red. The relations of Close
are shown in green (only populated for agent), On in black, Inside in red and
Stuck to in blue.

The legend for node IDs to objects are given below:

Home: 0: floor, 1: walls, 2: door, 3: fridge, 4: cupboard, 5: husky, 6: table,
7: table2, 8: couch, 9: big-tray, 10: book, 11: paper, 12: paper2, 13: cube
gray, 14: cube green, 15: cube red, 16: tray, 17: tray2, 18: light, 19: bottle
blue, 20: bottle gray, 21: bottle red, 22: box, 23: apple, 24: orange, 25:
banana, 26: chair, 27: ball, 28: stick, 29: dumpster, 30: milk, 31: shelf, 32:
glue, 33: tape, 34: stool, 35: mop, 36: sponge, 37: vacuum, 38: dirt.

Factory: 0: floor warehouse, 1: 3D printer, 2: assembly station, 3: blow
dryer, 4: board, 5: box, 6: brick, 7: coal, 8: crate green, 9: crate peach,
10: crate red, 11: cupboard, 12: drill, 13: gasoline, 14: generator, 15: glue,
16: hammer, 17: ladder, 18: lift, 19: long shelf, 20: mop, 21: nail, 22: oil,
23: paper, 24: part1, 25: part2, 26: part3, 27: platform, 28: screw, 29:
screwdriver, 30: spraypaint, 31: stick, 32: stool, 33: table, 34: tape, 35:
toolbox, 36: trolley, 37: wall warehouse, 38: water, 39: welder, 40: wood, 41:
wood cutter, 42: worktable, 43: ramp, 44: husky, 45: tray.

(a) A sample home scene.

(b) A sample factory scene.

Figure 14: Sample World Scenes in Home and Factory domains

##  D Realization of Object Relations

Figure 15: Manipulation region of an object

To implement various relations among objects PyBullet allows to define
_manipulation regions_ around objects. These regions are virtual regions
inside which a robot can manipulate that object. A visual description of this
region is shown in Figure 15.

We now define two object types.

  1. 1.

Standalone object: Such an object is defined as a collection of stereo-
lithographic tetrahedrons with visual and collision properties. Each such
object has a single manipulation region, denoted as $MR(object)$, which is the
region around the object where it is considered to be in “vicinity” to the
object and hence can be manipulated by the robot. Examples: cubes, fruits,
milk, etc.

  2. 2.

Containing object: Such an object is a standalone object with an additional
containment region, denoted as $CR(object)$, which is a proper subset of the
manipulation region where if another object’s center lies is called to be
“contained inside” this object. Examples: box, cupboard and fridge.

Every object has a center defined arithmetic mean of the centers of the
tetrahedrons, denoted by $C(object)$. We now define the two relation
constraints of which others are different versions of combinations of:

  1. 1.

Inside: An object $A$ is said to be contained inside a containing object $B$
if the $C(A)\in CR(B)$.

  2. 2.

On Top: An object $A$ is said to be on top on another object $B$ if
$C(A)_{z}>C(B)_{z}$ and $MR(A)\cap MR(B)\neq\phi$, where $C(O)_{z}$ denotes
the z component of the center of object $O$.