# Graph Inverse Reinforcement Learning
from Diverse Videos

Sateesh Kumar Jonathan Zamora* Nicklas Hansen*
Rishabh Jangir Xiaolong Wang
UC San Diego

###### Abstract

Research on Inverse Reinforcement Learning (IRL) from third-person videos has
shown encouraging results on removing the need for manual reward design for
robotic tasks. However, most prior works are still limited by training from a
relatively restricted domain of videos. In this paper, we argue that the true
potential of third-person IRL lies in increasing the diversity of videos for
better scaling. To learn a reward function from diverse videos, we propose to
perform graph abstraction on the videos followed by temporal matching in the
graph space to measure the task progress. Our insight is that a task can be
described by entity interactions that form a graph, and this graph abstraction
can help remove irrelevant information such as textures, resulting in more
robust reward functions. We evaluate our approach, _GraphIRL_ , on cross-
embodiment learning in X-MAGICAL and learning from human demonstrations for
real-robot manipulation. We show significant improvements in robustness to
diverse video demonstrations over previous approaches, and even achieve better
results than manual reward design on a real robot pushing task. Videos are
available at https://sateeshkumar21.github.io/GraphIRL.

Figure 1: GraphIRL. We propose an approach for performing inverse
reinforcement learning from diverse third-person videos via graph abstraction.
Based on our learned reward functions, we successfully train image-based
policies in simulation and deploy them on a real robot.

> Keywords: Inverse Reinforcement Learning, Third-Person Video, Graph Network

##  1 Introduction

00footnotetext:  * indicates equal contribution.

Deep Reinforcement Learning (RL) is a powerful general-purpose framework for
learning behavior policies from high-dimensional interaction data, and has led
to a multitude of impressive feats in application areas such as game-playing
[1] and robotics [2, 3]. Through interaction with an unknown environment, RL
agents iteratively improve their policy by learning to maximize a reward
signal, which has the potential to be used in lieu of hand-crafted control
policies. However, the performance of policies learned by RL is found to be
highly dependent on the careful specification of task-specific reward
functions and, as a result, crafting a good reward function may require
significant domain knowledge and technical expertise.

As an alternative to manual design of reward functions, inverse RL (IRL) has
emerged as a promising paradigm for policy learning. By framing the reward
specification as a learning problem, operators can specify a reward function
based on video examples. While imitation learning typically requires
demonstrations from a first-person perspective, IRL can in principle learn a
reward function, i.e., a measure of task progression, from any perspective,
including third-person videos of humans performing a task. This has positive
implications for data collection, since it is often far easier for humans to
capture demonstrations in third-person.

Although IRL from third-person videos is appealing because of its perceived
flexibility, learning a good reward function from raw video data comes with a
variety of challenges. This is perhaps unsurprising, considering the visual
and functional diversity that such data contains. For example, the task of
pushing an object across a table may require different motions depending on
the embodiment of the agent. A recent method for cross-embodiment IRL, dubbed
XIRL [4], learns to capture task progression from videos in a self-supervised
manner by enforcing temporal cycle-consistency constraints. While XIRL can in
principle consume any video demonstration, we observe that its ability to
learn task progression degrades substantially when the visual appearance of
the video demonstrations do not match that of the target environment for RL.
Therefore, it is natural to ask the question: can we learn to imitate others
from (a limited number of) diverse third-person videos?

In this work, we demonstrate that it is indeed possible. Our key insight is
that, while videos may be of great visual diversity, their underlying scene
structure and agent-object interactions can be abstracted via a graph
representation. Specifically, instead of directly using images, we extract
object bounding boxes from each frame using an off-the-shelf detector, and
construct a graph abstraction where each object is represented as a node in
the graph. Often – in robotics tasks – the spatial location of an object by
itself may not convey the full picture of the task at hand. For instance, to
understand a task like Peg in Box (shown in Figure 1), we need to also take
into account how the agent _interacts_ with the object. Therefore, we propose
to employ _Interaction Networks_ [5] on our graph representation to explicitly
model interactions between entities. To train our model, we follow [4, 6] and
apply a temporal cycle consistency loss, which (in our framework) yields task-
specific yet embodiment- and domain-agnostic feature representations.

We validate our method empirically on a set of simulated cross-domain cross-
embodiment tasks from X-MAGICAL [4], as well as three vision-based robotic
manipulation tasks. To do so, we collect a diverse set of demonstrations that
vary in visual appearance, embodiment, object categories, and scene
configuration; X-MAGICAL demonstrations are collected in simulation, whereas
our manipulation demonstrations consist of real-world videos of humans
performing tasks. We find our method to outperform a set of strong baselines
when learning from visually diverse demonstrations, while simultaneously
matching their performance in absence of diversity. Further, we demonstrate
that vision-based policies trained with our learned reward perform tasks with
greater precision than human-designed reward functions, and successfully
transfer to a real robot setup with only approximate correspondence to the
simulation environment. Thus, our proposed framework completes the cycle of
learning rewards from real-world human demonstrations, learning a policy in
simulation using learned rewards, and finally deployment of the learned policy
on physical hardware.

##  2 Related Work

Learning from demonstration. Conventional imitation learning methods require
access to expert demonstrations comprised of observations and corresponding
ground-truth actions for every time step [7, 8, 9, 10], for which kinesthetic
teaching or teleoperation are the primary modes of data collection in
robotics. To scale up learning, video demonstrations are recorded with human
operating the same gripper that the robot used, which also allows direct
behaviro cloning [11, 12]. More recently, researchers have developed methods
that instead infer actions from data via a learned forward [13] or inverse
[14, 15] dynamics model. However, this approach still makes the implicit
assumption that imitator and demonstrator share a common observation and
action space, and are therefore not directly applicable to the cross-domain
cross-embodiment problem setting that we consider.

Inverse RL. To address the aforementioned limitations, inverse RL has been
proposed [16, 17, 18, 19, 20, 21] and it has recently emerged as a promising
paradigm for cross-embodiment imitation in particular [22, 23, 24, 25, 26, 27,
28, 4, 29]. For example, Schmeckpeper et al. [22] proposes a method for
integrating video demonstrations without corresponding actions into off-policy
RL algorithms via a latent inverse dynamics model and heuristic reward
assignment, and Zakka et al. [4] (XIRL) learns a reward function from video
demonstrations using temporal cycle-consistency and trains an RL agent to
maximize the learned rewards. In practice, however, inverse RL methods such as
XIRL are found to require limited visual diversity in demonstrations. Our work
extends XIRL to the setting of diverse videos by introducing a graph
abstraction that models agent-object and object-object interactions while
still enforcing temporal cycle-consistency.

Object-centric representations. have been proposed in many forms at the
intersection of computer vision and robotics. For example, object-centric
scene graphs can be constructed for integrated task and motion planning [30,
31, 32], navigation [33, 34], relational inference [35, 36], dynamics modeling
[5, 37, 38, 39, 40], model predictive control [41, 42, 43] or visual imitation
learning [44]. Similar to our work, Sieb et al. [44] propose to abstract video
demonstrations as object-centric graphs for the problem of single-video cross-
embodiment imitation, and act by minimizing the difference between the
demonstration graph and a graph constructed from observations captured at each
step. As such, their method is limited to same-domain visual trajectory
following, whereas we learn a general alignment function for cross-domain
cross-embodiment imitation and leverage Interaction Networks [5] for modeling
graph-abstracted spatial interactions rather than relying on heuristics.

Figure 2: Overview. We extract object bounding boxes from video sequences
using an off-the-shelf detector, and construct a graph abstraction of the
scene. We model graph-abstracted object interactions using _Interaction
Networks_ [5], and learn a reward function by aligning video embeddings
temporally. We then train image-based RL policies using our learned reward
function, and deploy on a real robot.

##  3 Our Approach

In this section, we describe our main contribution, which is a self-supervised
method for learning a reward function directly from a set of diverse third-
person video demonstrations by applying temporal matching on graph
abstractions. Our Graph Inverse Reinforcement Learning (GraphIRL) framework,
shown in Figure 2, consists of building an object-centric graph abstraction of
the video demonstrations and then learn an embedding space that captures task
progression by exploiting the temporal cue in the videos. This embedding space
is then used to construct a _domain invariant_ and _embodiment invariant_
reward function which can be used to train any standard reinforcement learning
algorithm.

Problem Formulation. Given a task $T$, our approach takes a dataset of video
demonstrations $D=\\{V_{1},V_{2},\dots,V_{n}\\}$. Each video consists of image
frames $\\{I_{1}^{i},I_{2}^{i},\dots,I_{k}^{i}\\}$ where $i$ denotes the video
frame index and $k$ denotes the total number of frames in $V_{i}$. Given $D$,
our goal is to learn a reward function that can be used to solve the task $T$
for any robotic environment. Notably, we do _not_ assume access to any action
information of the expert demonstrations, and our approach does _not_ require
objects or embodiments in the target environment to share appearance with
demonstrations.

###  3.1 Representation Learning

To learn task-specific representations in a self-supervised manner, we take
inspiration from Dwibedi et al. [6] and employ a temporal cycle consistency
loss. However, instead of directly using images, we propose a novel object-
centric graph representation, which allows us to learn an embedding space that
not only captures task-specific features, but depends _solely_ on the spatial
configuration of objects and their interactions. We here detail each component
of our approach to representation learning.

Object-Centric Representation. Given video frames
$\\{I_{1}^{i},I_{2}^{i},\dots,I_{k}^{i}\\}$, we first extract object bounding
boxes from each frame using an off-the-shelf detector. Given $N$ bounding
boxes for an image, we represent each bounding box as a $4+m$ dimensional
vector $o_{j}=\\{x_{1},y_{1},x_{2},y_{2},d_{1},d_{2},\dots,d_{m}\\}$, where
the first $4$ dimensions represent the leftmost and rightmost corners of the
bounding box, and the remaining $m$ dimensions encode distances between the
centroids of the objects. For each frame $I^{i}_{j}$ we extract an object-
centric representation $I^{\prime i}_{j}=\\{o_{1},o_{2},\dots,o_{m}\\}$ such
that we can represent our dataset of demonstrations as
$D^{\prime}=\\{V^{\prime}_{1},V^{\prime}_{2},\dots,V^{\prime}_{n}\\}$ where
$V^{\prime}_{i}$ is the sequence of bounding boxes corresponding to video
$V_{i}$. Subsequent sections describe how we learn representations given
$D^{\prime}$.

Spatial Interaction Encoder. Taking inspiration from recent approaches on
modeling physical object dynamics [5, 37], we propose a _Spatial Interaction
Encoder Network_ to explicitly model object-object interactions. Specifically,
given a sequence $V^{\prime}_{i}$ from $D^{\prime}$, we model each element
$I^{\prime}$ as a graph, $G=(O,R)$, where $O$ is the set of objects
$\\{o_{1},o_{2},\dots,o_{m}\\}$, $m$ is the total number of objects in
$I^{\prime}$, and $R$ denotes the relationship between objects (_i.e._ ,
whether two objects interact with each other). For simplicity, all objects are
connected with all other objects in the graph such that $R=\\{(i,j)\mid i\neq
j\land i\leq m\land j\leq m\\}$. We compose an object embedding for each of
$o_{i}\in O$ by combining _self_ and _interactional_ representations as
follows:

$\displaystyle
f_{o}(o_{i})=\phi_{\text{agg}}(f_{\text{s}}+f_{\text{in}})\quad\text{with}\quad
f_{s}(o_{i})=\phi_{s}(o)\,,\quad
f_{\text{in}}(o_{i})=\sum_{j=1}^{m}\phi_{\text{in}}((o_{i},o_{j}))\mid(i,j)\in\mathbb{R}\,,$
(1)

where $f_{s}(o_{i})$ represents the _self_ or independent representation of an
object, $f_{\text{in}}$ represents the _interactional_ representation, _i.e._
, how it interacts with other objects in the scene, $f_{o}$ is the final
object embedding, and $(,)$ represents concatenation. Here, the encoders
$\phi_{\text{s}}$, $\phi_{\text{in}}$ and $\phi_{\text{agg}}$ denote Multi
layer Perceptron (MLP) networks respectively. We emphasize that the expression
for $f_{\text{in}}(\cdot)$ implies that the object embedding $f_{o}(.)$
depends on _all_ other objects in the scene; this term allows us to model
relationships of an object with the others. The final output from the spatial
interaction encoder $\psi(\cdot)$ for object representation $I^{\prime}$ is
the mean of all object encodings:

$\displaystyle\psi(I^{\prime})=\frac{1}{m}\sum_{i}^{m}f(o_{i})\,.$ (2)

The spatial interaction encoder is then optimized using the temporal alignment
loss introduced next.

Temporal Alignment Loss. Taking inspiration from prior works on video
representation learning [6, 45, 46, 47, 48], we employ the task of temporal
alignment for learning task-specific representations. Given a pair of videos,
the task of self-supervised alignment implicitly assumes that there exists
true semantic correspondence between the two sequences, i.e., both videos
share a common semantic space. These works have shown that optimizing for
alignment leads to representations that could be used for tasks that require
understanding task progression such as action-classification. This is because
in order to solve for alignment, a learning model has to learn features that
are (1) common across most videos and (2) exhibit temporal ordering. For a
sufficiently large dataset with single task, the most common visual features
would be distinct phases of a task that appear in all videos and if the task
has small permutations, these distinct features would also exhibit temporal
order. In such scenarios, the representations learned by optimizing for
alignment are _task-specific_ and invariant to changes in viewpoints,
appearances and actor embodiments.

In this work, we employ Temporal Cycle Consistency (TCC) [6] loss to learn
temporal alignment. TCC optimizes for alignment by learning an embedding space
that maximizes one-to-one nearest neighbour mappings between sequences. This
is achieved through a loss that maximizes for cycle-consistent nearest
neighbours given a pair of video sequences. In our case, the cycle consistency
is applied on the _graph abstraction_ instead of image features as done in the
aforementioned video alignment methods. Specifically, given $D^{\prime}$, we
sample a pair of bounding box sequences $V^{\prime}_{i}=\\{I^{\prime
i}_{1},\dots,I^{\prime i}_{m_{i}}\\}$ and
$V^{\prime}_{j}=\\{I^{\prime}{j}_{1},\dots,I^{\prime j}_{m_{j}}\\}$ and
extract embeddings by applying the spatial interaction encoder defined in
Equation 2. Thus, we obtain the encoded features $S_{i}=\\{\psi(I^{\prime
i}_{1}),\dots,\psi(I^{\prime i}_{m_{i}})\\}$ and $S_{j}=\\{\psi(I^{\prime
j}_{1}),\dots,\psi(I^{\prime j}_{m_{j}})\\}$. For the $n$th element in
$S_{i}$, we first compute its nearest neighbour, $\upsilon^{n}_{ij}$, in
$S_{j}$ and then compute the probability that it cycles-back to the $k$th
frame in $S_{i}$ as:

$\displaystyle\beta^{k}_{ijn}=\frac{e^{-||\upsilon^{n}_{ij}-S_{i}^{k}||^{2}}}{{\sum_{k}^{m_{j}}e^{-||\upsilon^{n}_{ij}-S^{k}_{i}||^{2}}}}\,,\upsilon^{n}_{ij}=\sum_{k}^{m_{j}}\alpha_{k}s^{k}_{j}\,,\alpha_{k}=\frac{e^{-||S^{n}_{i}-S^{k}_{j}||^{2}}}{{\sum_{k}^{m_{j}}e^{-||S^{n}_{i}-S^{k}_{j}||^{2}}}}\,.$
(3)

The cycle consistency loss for $n$th element can be computed as
$L_{n}^{ij}=(\mu^{n}_{ij}-n)^{2}$, where
$\mu^{n}_{ij}=\sum^{mi}_{k}\beta^{k}_{ijn}k$ is the expected value of frame
index $n$ as we cycle back. The overall TCC loss is then defined by summing
over all pairs of sequence embeddings $(S_{i},S_{j})$ in the data, i.e.,
$L^{n}_{ij}=\sum_{ijn}L^{n}_{ij}$.

###  3.2 Reinforcement Learning

We learn a task-specific embedding space by optimizing for temporal alignment.
In this section, we define how to go from this embedding space to a reward
function that measures task progression. For constructing the reward function,
we leverage the insight from Zakka et al. [4] that in a task-specific
embedding space, we can use euclidean distance as a notion of task
progression, i.e., frames far apart in the embedding space will be far apart
in terms of task progression and vice versa. We therefore choose to define our
reward function as

$\displaystyle r(o)=-\frac{1}{c}||\psi(o)-g||^{2}\,,\quad\text{with}\quad
g=\sum_{i=1}^{n}\psi(I^{\prime i}_{m_{i}})\,,$ (4)

where $o$ is the current observation, $\psi$ is the Spatial Interaction
Encoder Network from Section 3, $g$ is the representative goal frame, $m_{i}$
is the length of sequence $V^{\prime i}$ and $c$ is a scaling factor. The
scaling factor _c_ is computed as the average distance between the first and
final observation of all the training videos in the learned embedding space.
Note, that the range of the learned reward is $(-\infty,0]$. Defining the
reward function in this way gives us a dense reward because as the observed
state gets closer and closer to the goal, the reward starts going down and
approaches zero when the goal and current observation are close in embedding
space. After constructing the learned reward, we can use it to train any
standard RL algorithm. We note that, unlike previous approaches [22, 4], our
method does not use _any_ environment reward to improve performance, and
instead relies _solely_ on the learned reward, which our experiments
demonstrate is sufficient for solving diverse robotic manipulation tasks.

##  4 Experiments

In this section, we demonstrate how our approach uses diverse video
demonstrations to learn a reward function that generalizes to unseen domains.
In particular, we are interested in answering the questions: (1) How do
vision-based methods for IRL perform when learning from demonstrations that
exhibit _domain shift_? and (2) is our approach capable of learning a stronger
reward signal under this challenging setting? To that end, we first conduct
experiments X-MAGICAL benchmark [4]. We then evaluate our approach on multiple
robot manipulation tasks using a diverse set of demonstrations.

Implementation Details. All MLPs defined in Equation 2 have 2 layers followed
by a ReLU activation, and the embedding layer outputs features of size $128$
in all experiments. For training, we use ADAM [49] optimizer with a learning
rate of $10^{-5}$. We use Soft Actor-Critic (SAC) [50] as backbone RL
algorithm for all methods. For experiments on X-MAGICAL, we follow Zakka et
al. [4] and learn a state-based policy; RL training is performed for $500$k
steps for all embodiments. For robotic manipulation experiments, we learn a
multi-view image-based SAC policy [51]. We train RL agent for $300$k, $800$k
and $700$k steps for _Reach_ , _Push_ and _Peg in Box_ respectively. For fair
comparison, we only change the learned reward function across methods and keep
the RL setup identical. The success rates presented for all our experiments
are averaged over $50$ episodes. Refer to Appendix B for further
implementation details.

Baselines. We compare against multiple vision-based approaches that learn
rewards in a self-supervised manner: (1) XIRL [4] that learns a reward
function by applying the TCC [6] loss on demonstration video sequences, (2)
TCN [52] which is a self-supervised contrastive method for video
representation learning that optimizes for temporally disentangled
representations, and (3) LIFS [53] that learns an invariant feature space
using a dynamic time warping-based contrastive loss. Lastly, we also compare
against the manually designed (4) Environment Rewards from Jangir et al. [51].
For vision-based baselines, we use a ResNet-18 encoder pretrained on ImageNet
[54] classification. We use the hyperparameters, data augmentation schemes and
network architectures provided in Zakka et al. [4] for all vision-based
baselines. Please refer to Appendix E.1 for description of environment rewards
and Zakka et al. [4] for details on the vision-based baselines.

Standard Environment Diverse Environment
Gripper

S-stick

M-stick

L-stick

Gripper

S-stick

M-stick

L-stick

Figure 3: Overview of X-MAGICAL task variants. We consider two environment
variants and four embodiments for our simulated sweeping task experiments. Our
work assesses the performance of IRL algorithms in both the Diverse and
Standard environments across all four embodiments in the Same-Embodiment and
Cross-Embodiment settings.

Figure 4: Cross-Embodiment Cross-Environment. Success rates of our method
_GraphIRL_ and baselines on (top) Standard Environment Pretraining
$\rightarrow$ Diverse Environment RL and (bottom) Diverse Environment
Pretraining $\rightarrow$ Standard Environment RL. All reported numbers are
averaged over 5 seeds. Our approach performs favorably when compared to other
baselines on both settings.

###  4.1 Experimental Setup

We conduct experiments under two settings: the _Sweep-to-Goal_ task from
X-MAGICAL [4], and robotic manipulation tasks with an xArm robot both in
simulation and on a real robot setup. We describe our experimental setup under
these two settings in the following.

X-MAGICAL. We choose to extend X-MAGICAL [4], a 2D simulation environment for
cross-embodiment imitation learning. On this benchmark, we consider a multi-
object sweeping task, where the agent must push three objects towards a static
goal region. We utilize two variants of the X-MAGICAL benchmark, which we
denote as Standard (original) and Diverse (ours) environments, shown in Figure
3. Standard only randomizes the position of objects, whereas Diverse also
randomizes visual appearance. We consider a set of four unique embodiments
$\\{$gripper, short-stick, medium-stick, long-stick$\\}$. In particular, we
conduct experiments in the _cross-environment_ and _cross-embodiment setting_
where we learn a reward function in the Standard environment on 3 held-out
embodiments and do RL in the Diverse environment on 1 target embodiment, or
vice-versa. This provides an additional layer of difficulty for the RL agent
as visual randomizations show the brittleness of vision-based IRL methods.
Refer to Appendix C for more details on performed randomizations.

Robotic Manipulation. Figure 1 shows initial and success configurations for
each of the three task that we consider: (1) Reach in which the agent needs to
reach a goal (red disc) with its end-effector, (2) Push in which the goal is
to push a cube to a goal position, and (3) Peg in Box where the goal is to put
a peg tied to the robot’s end-effector inside a box. The last task is
particularly difficult because it requires geometric 3D understanding of the
objects. Further, a very specific trajectory is required to avoid collision
with the box and complete the task. We collect a total of $256$ and $162$
video demonstrations for _Reach_ and _Peg in Box_ , respectively, and use
$198$ videos provided from Schmeckpeper et al. [22] for _Push_. The videos
consist of human actors performing the same tasks but with a number of diverse
objects and goal markers, as well as varied positions of objects. Unlike the
data collected by Schmeckpeper et al. [22], we do not fix the goal position in
our demonstrations. In order to detect objects in our training demonstrations,
we use a trained model from Shan et al. [55]. The model is trained on a large-
scale dataset collected from YouTube and can detect hands and objects in an
image.; refer to Appendix E.2 for more details on data collection.
Additionally, we do not require the demonstrations to resemble the robotic
environment in terms of appearance or distribution of goal location. We use an
xArm robot as our robot platform and capture image observations using a static
third-person RGB camera in our real setup; details in Appendix G.

###  4.2 Results

X-MAGICAL. Results for the _cross-embodiment and cross-environment_ setting
are shown in Figure 4. When trained on Standard, our method performs
significantly better than vision-based baselines (e.g., $0.58$ GraphIRL for
gripper vs $0.35$ for XIRL and $0.99$ GraphIRL for longstick vs $0.56$ XIRL).
We conjecture that vision-based baselines struggle with visual variations in
the environment, which our method is unaffected by due to its graph
abstraction. Additionally, when trained on _diverse_ environment, GraphIRL
outperforms $3$ out of $4$ embodiments.

Figure 5: Robotic Manipulation. Success rates of our method _GraphIRL_ and
baselines on the tasks of _Reach_ , _Push_ and _Peg in Box_. All results are
averaged over 5 seeds. We observe significant gains in performance specially
over vision-based baselines due to large _domain-gap_

Real | XIRL | Env. Reward | GraphIRL (Ours)
---|---|---|---
Push | $0.27$ | ${0.47}$ | $\mathbf{0.60}$
Reach | $0.26$ | $\mathbf{0.93}$ | ${0.86}$
Peg in Box | $0.06$ | $\mathbf{0.60}$ | ${0.53}$

Table 1: Real robot experiments. Success rate on robot manipulation tasks on
physical hardware. We evaluate each method for 15 trials using a fixed set of
goal and start state configurations. Best results are in bold.

Robotic manipulation in simulation. In this section, we answer the core
question of our work: _can we learn to imitate others from diverse third-
person videos?_ In particular, we collect human demonstrations for
manipulation tasks as explained in Section 4.1 and learn a reward function as
explained in Section 3. This is a challenging setting because as shown in
Figure 1, the collected data and robotic environments belong to different
domains and do not share any appearance characteristics. Further, unlike
previous works [22, 4], we do not use any environment reward as an additional
supervision to the reinforcement learning agent. Figure 5 presents our
results. For the Reach task, GraphIRL and environment reward are able to
achieve a success rate of $1.0$, while other baseline methods are
substantially behind GraphIRL (e.g. $0.477$ XIRL and $0.155$ TCN). The poor
performance of vision-based baselines could be attributed to substantial
visual domain shift. Due to domain shift, the learned rewards for these
baselines produce low rewards for successful episodes, please refer to
Appendix A for a more detailed qualitative analysis. In the Push setting, we
find that vision-based baseline methods still perform poorly. Similar to
Reach, XIRL performs the best out of the vision-based baselines with a success
rate of $0.187$, and GraphIRL performs better than environment reward (e.g.
$0.832$ GraphIRL vs $0.512$ Environment Reward). This result shows clear
advantage of our method as we are able to outperform a hand-designed reward
function without using any task specific information. The Peg in Box task is
rigorous to solve since it requires 3-d reasoning and a precise reward
function. Here, while all vision-based methods fail, our GraphIRL method is
able to solve the task with a success rate comparable to that achieved with
the hand-designed environment reward. Overall, our GraphIRL method is able to
solve 2D and 3D reasoning tasks with a real-robot without a hand-designed
reward function or access to 3D scene information.

Real robot experiments. Finally, we deploy the learned policies on a real
robot. For each experiment, we conduct $15$ trials per method and report the
average success rate. Results are shown in Table 1. Interestingly, we find
that GraphIRL outperforms XIRL in all three tasks on the real robot setup
(e.g. $0.26$ XIRL vs $0.86$ GraphIRL on _Reach_ and $0.27$ XIRL vs $0.60$
GraphIRL on _Push_), and on _Push_ , GraphIRL performs better than the
environment reward specifically designed for the task (e.g. $0.47$ Environment
Reward vs $0.6$ GraphIRL) which is in line with our findings in simulation.

Variant | Success Rate
---|---
MLP | $0.61\scriptstyle{\pm 0.116}$
IN | $\mathbf{0.804\scriptstyle{\pm 0.054}}$
Table 2: Impact of modelling object-object interaction on _Push_ task. MLP;
Multi-layer perceptron and IN: Spatial Interaction Network Encoder. Results
averaged over 5 seeds. Best results are in bold.

% Videos Used | Success Rate
---|---
25% | $0.670\scriptstyle{\pm 0.176}$
50% | $0.755\scriptstyle{\pm 0.019}$
75% | $0.776\scriptstyle{\pm 0.04}$
100% | $\mathbf{0.804\scriptstyle{\pm 0.054}}$
Table 3: Impact of reducing number of pretraining demonstrations on Push task.
Results averaged over 5 seeds. Best results are in bold.

###  4.3 Ablations

In this section, we perform ablation study using the _Push_ task to validate
our design choices in Section 3. In the experiments below, we perform RL
training for $500$k steps and report the final success rate.

Impact of Modelling Spatial Interactions. We study the impact of modeling
object-object spatial interactions using Spatial Interaction Encoder Network
described (IN) in Section 3.1. Specifically, we replace our proposed encoder
component with an Multi-Layer Perceptron (MLP) by concatenating
representations of all objects into a single vector and then feeding it to a
3-layer MLP network. As shown in Table 3, IN leads to a $20\%$ improvement in
the reinforcement learning success rate.

Impact of Decreasing Number of Demonstration Videos. As shown in Table 3, the
performance of our approach gradually decreases as we decrease demonstration
data. However, we note that GraphIRL achieves $67\%$ success rate with $25$%
of total training videos (49 videos). This demonstrates that our approach is
capable of learning meaningful rewards even with a small number of videos.

##  5 Conclusions and Limitations

We demonstrate the effectiveness of our proposed method, GraphIRL, in a number
of IRL settings with diverse third-person demonstrations. In particular, we
show that our method successfully learns reward functions from human
demonstrations with diverse objects and scene configurations, that we are able
to train image-based policies in simulation using our learned rewards, and
that policies trained with our learned rewards are more successful than both
prior work and manually designed reward functions on a real robot. With
respect to limitations, while our method relaxes the requirements for human
demonstrations, collecting the demonstrations still requires human labor; and
although our results indicate that we can learn from relatively few videos,
eliminating human labor entirely remains an open problem.

## References

  * Mnih et al. [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , 518:529–533, 2015.
  * Levine et al. [2018] S. Levine, P. Pastor, A. Krizhevsky, and D. Quillen.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.  _The International Journal of Robotics Research_ , 37:421 – 436, 2018.
  * Andrychowicz et al. [2020] O. M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ , 39:20 – 3, 2020.
  * Zakka et al. [2022] K. Zakka, A. Zeng, P. Florence, J. Tompson, J. Bohg, and D. Dwibedi.  Xirl: Cross-embodiment inverse reinforcement learning.  In _Conference on Robot Learning_ , pages 537–546. PMLR, 2022.
  * Battaglia et al. [2016] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende, et al.  Interaction networks for learning about objects, relations and physics.  _Advances in neural information processing systems_ , 29, 2016.
  * Dwibedi et al. [2019] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman.  Temporal cycle-consistency learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1801–1810, 2019.
  * Pomerleau [1988] D. A. Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In D. Touretzky, editor, _Advances in Neural Information Processing Systems_ , volume 1. Morgan-Kaufmann, 1988.
  * Atkeson and Schaal [1997] C. G. Atkeson and S. Schaal.  Robot learning from demonstration.  In _ICML_ , 1997.
  * Argall et al. [2009] B. Argall, S. Chernova, M. Veloso, and B. Browning.  A survey of robot learning from demonstration.  _Robotics and Autonomous Systems_ , 57:469–483, 05 2009\.
  * Ravichandar et al. [2020] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard.  Recent advances in robot learning from demonstration.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 3(1):297–330, 2020.
  * Song et al. [2020] S. Song, A. Zeng, J. Lee, and T. Funkhouser.  Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations.  _Robotics and Automation Letters_ , 2020.
  * Young et al. [2020] S. Young, D. Gandhi, S. Tulsiani, A. Gupta, P. Abbeel, and L. Pinto.  Visual imitation made easy.  _arXiv_ , 2020.
  * Pathak et al. [2018] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell.  Zero-shot visual imitation.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , pages 2131–21313, 2018.
  * Torabi et al. [2018] F. Torabi, G. Warnell, and P. Stone.  Behavioral cloning from observation.  _ArXiv_ , abs/1805.01954, 2018.
  * Radosavovic et al. [2021] I. Radosavovic, X. Wang, L. Pinto, and J. Malik.  State-only imitation learning for dexterous manipulation.  _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7865–7871, 2021.
  * Ng et al. [2000] A. Y. Ng, S. J. Russell, et al.  Algorithms for inverse reinforcement learning.  In _ICML_ , 2000.
  * Abbeel and Ng [2004] P. Abbeel and A. Y. Ng.  Apprenticeship learning via inverse reinforcement learning.  In _ICML_ , 2004.
  * Ho and Ermon [2016] J. Ho and S. Ermon.  Generative adversarial imitation learning.  In _NeurIPS_ , 2016.
  * Fu et al. [2017] J. Fu, K. Luo, and S. Levine.  Learning robust rewards with adversarial inverse reinforcement learning.  _arXiv_ , 2017.
  * Aytar et al. [2018] Y. Aytar, T. Pfaff, D. Budden, T. Paine, Z. Wang, and N. de Freitas.  Playing hard exploration games by watching youtube.  In _NIPS_ , 2018.
  * Torabi et al. [2018] F. Torabi, G. Warnell, and P. Stone.  Generative adversarial imitation from observation.  _arXiv_ , 2018.
  * Schmeckpeper et al. [2020] K. Schmeckpeper, O. Rybkin, K. Daniilidis, S. Levine, and C. Finn.  Reinforcement learning with videos: Combining offline observations with interaction.  _arXiv preprint arXiv:2011.06507_ , 2020.
  * Jin et al. [2020] J. Jin, L. Petrich, Z. Zhang, M. Dehghan, and M. Jagersand.  Visual geometric skill inference by watching human demonstration.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 8985–8991. IEEE, 2020.
  * Xiong et al. [2021] H. Xiong, Q. Li, Y.-C. Chen, H. Bharadhwaj, S. Sinha, and A. Garg.  Learning by watching: Physical imitation of manipulation skills from human videos.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7827–7834. IEEE, 2021.
  * Lee et al. [2021] Y. Lee, A. Szot, S.-H. Sun, and J. J. Lim.  Generalizable imitation learning from observation via inferring goal proximity.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Chen et al. [2021] A. S. Chen, S. Nair, and C. Finn.  Learning generalizable robotic reward functions from” in-the-wild” human videos.  _arXiv preprint arXiv:2103.16817_ , 2021.
  * Fickinger et al. [2021] A. Fickinger, S. Cohen, S. Russell, and B. Amos.  Cross-domain imitation learning via optimal transport.  _arXiv preprint arXiv:2110.03684_ , 2021.
  * Qin et al. [2021] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang, R. Yang, Y. Fu, and X. Wang.  Dexmv: Imitation learning for dexterous manipulation from human videos.  _arXiv preprint arXiv:2108.05877_ , 2021.
  * Arunachalam et al. [2022] S. P. Arunachalam, S. Silwal, B. Evans, and L. Pinto.  Dexterous imitation made easy: A learning-based framework for efficient dexterous manipulation.  _arXiv preprint arXiv:2203.13251_ , 2022.
  * Fainekos et al. [2009] G. Fainekos, A. Girard, H. Kress-Gazit, and G. J. Pappas.  Temporal logic motion planning for dynamic robots.  _Autom._ , 45:343–352, 2009.
  * Srivastava et al. [2014] S. Srivastava, E. Fang, L. Riano, R. Chitnis, S. Russell, and P. Abbeel.  Combined task and motion planning through an extensible planner-independent interface layer.  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 639–646, 2014.
  * Zhu et al. [2021] Y. Zhu, J. Tremblay, S. Birchfield, and Y. Zhu.  Hierarchical planning for long-horizon manipulation with geometric and symbolic scene graphs.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6541–6548. IEEE, 2021.
  * Gupta et al. [2019] S. Gupta, V. Tolani, J. Davidson, S. Levine, R. Sukthankar, and J. Malik.  Cognitive mapping and planning for visual navigation.  _International Journal of Computer Vision_ , 128:1311–1330, 2019.
  * Yang et al. [2019] W. Yang, X. Wang, A. Farhadi, A. K. Gupta, and R. Mottaghi.  Visual semantic navigation using scene priors.  _ArXiv_ , abs/1810.06543, 2019.
  * Xu et al. [2017] D. Xu, Y. Zhu, C. B. Choy, and L. Fei-Fei.  Scene graph generation by iterative message passing.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3097–3106, 2017.
  * Li et al. [2017] Y. Li, W. Ouyang, B. Zhou, K. Wang, and X. Wang.  Scene graph generation from objects, phrases and region captions.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 1270–1279, 2017.
  * Watters et al. [2017] N. Watters, D. Zoran, T. Weber, P. Battaglia, R. Pascanu, and A. Tacchetti.  Visual interaction networks: Learning a physics simulator from video.  _Advances in neural information processing systems_ , 30, 2017.
  * Materzynska et al. [2020] J. Materzynska, T. Xiao, R. Herzig, H. Xu, X. Wang, and T. Darrell.  Something-else: Compositional action recognition with spatial-temporal interaction networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1049–1059, 2020.
  * Ye et al. [2019] Y. Ye, M. Singh, A. Gupta, and S. Tulsiani.  Compositional video prediction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10353–10362, 2019.
  * Qi et al. [2021] H. Qi, X. Wang, D. Pathak, Y. Ma, and J. Malik.  Learning long-term visual dynamics with region proposal interaction networks.  In _ICLR_ , 2021.
  * Sanchez-Gonzalez et al. [2018] A. Sanchez-Gonzalez, N. Heess, J. T. Springenberg, J. Merel, M. Riedmiller, R. Hadsell, and P. Battaglia.  Graph networks as learnable physics engines for inference and control.  In _International Conference on Machine Learning_ , pages 4470–4479. PMLR, 2018.
  * Li et al. [2019] Y. Li, H. He, J. Wu, D. Katabi, and A. Torralba.  Learning compositional koopman operators for model-based control.  _arXiv preprint arXiv:1910.08264_ , 2019.
  * Ye et al. [2020] Y. Ye, D. Gandhi, A. Gupta, and S. Tulsiani.  Object-centric forward modeling for model predictive control.  In _Conference on Robot Learning_ , pages 100–109. PMLR, 2020.
  * Sieb et al. [2020] M. Sieb, Z. Xian, A. Huang, O. Kroemer, and K. Fragkiadaki.  Graph-structured visual imitation.  In _Conference on Robot Learning_ , pages 979–989. PMLR, 2020.
  * Haresh et al. [2021] S. Haresh, S. Kumar, H. Coskun, S. N. Syed, A. Konin, Z. Zia, and Q.-H. Tran.  Learning by aligning videos in time.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5548–5558, 2021.
  * Liu et al. [2021] W. Liu, B. Tekin, H. Coskun, V. Vineet, P. Fua, and M. Pollefeys.  Learning to align sequential actions in the wild.  _arXiv preprint arXiv:2111.09301_ , 2021.
  * Wang et al. [2020] J. Wang, Y. Long, M. Pagnucco, and Y. Song.  Dynamic graph warping transformer for video alignment.  2020\.
  * Hadji et al. [2021] I. Hadji, K. G. Derpanis, and A. D. Jepson.  Representation learning via global temporal alignment and cycle-consistency.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11068–11077, 2021.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement, 2018\.
  * Jangir et al. [2022] R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang.  Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation.  _IEEE Robotics and Automation Letters_ , 2022.
  * Sermanet et al. [2018] P. Sermanet, C. Lynch, Y. Chebotar, J. Hsu, E. Jang, S. Schaal, S. Levine, and G. Brain.  Time-contrastive networks: Self-supervised learning from video.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 1134–1141. IEEE, 2018.
  * Gupta et al. [2017] A. Gupta, C. Devin, Y. Liu, P. Abbeel, and S. Levine.  Learning invariant feature spaces to transfer skills with reinforcement learning.  _arXiv preprint arXiv:1703.02949_ , 2017.
  * Russakovsky et al. [2015] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi:10.1007/s11263-015-0816-y.
  * Shan et al. [2020] D. Shan, J. Geng, M. Shu, and D. F. Fouhey.  Understanding human hands in contact at internet scale.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9869–9878, 2020.
  * Yarats et al. [2020] D. Yarats, I. Kostrikov, and R. Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In _International Conference on Learning Representations_ , 2020.
  * Hansen et al. [2021] N. Hansen, H. Su, and X. Wang.  Stabilizing deep q-learning with convnets and vision transformers under data augmentation.  _Advances in Neural Information Processing Systems_ , 34:3680–3693, 2021.

##  Appendix A Qualitative Analysis of Learned Reward

In this section, we present qualitative analysis of the reward learned using
_GraphIRL_. We plot the reward as defined in Equation 4 for GraphIRL and two
baseline IRL methods for three test examples across three tasks. The tasks we
evaluate with are Peg in Box, Push, and Reach. For each task, we use show two
successful episodes and one unsuccessful episode. The length of each episode
is 50, and for each figure we have included, we provide images that align with
critical points in the completion of the task.

(a)

(b)

(c)

(a)

(b)

(c)

Figure 6: Peg in Box Task Progress: Success. For the Peg in Box task setting,
we find that GraphIRL provides an accurate measurement of task progress.
Pictured are video frames (a), (b), (c) which denote critical points of task
progress. Task progress is measured using video frames from a 50-step
evaluation episode.

(a)

(b)

(c)

Figure 7: Peg in Box Task Progress: Failure. GraphIRL measures positive task
progress until the peg goes into the table, a critical failure point for the
task. The physical interaction between the peg and table is unnatural, and our
method succeeds in recognizing this.

(a)

(b)

(c)

(a)

(b)

(c)

Figure 8: Push Task Progress: Success. The Push task setting is often
completed within the first 10 steps of the evaluation episode, and as shown
between Steps 1 through 10 in both success examples, GraphIRL measures high
task progress. XIRL and TCN on the other hand, incorrectly show much lower
task progress.

(a)

(b)

(c)

Figure 9: Push Task Progress: Failure. GraphIRL’s understanding of object
relationships is made clear in this Push task failure, since without any
forward movement of the box object towards the goal, no positive task progress
is made. Other baselines rely on direct visual input of the task, and because
of this, they inaccurately align visual states (a), (b), (c) of the task with
positive task progress.

(a)

(b)

(c)

(a)

(b)

(c)

Figure 10: Reach Task Progress: Success. In the Reach task setting, positive
task progress is measured by GraphIRL with forward movement of the end-
effector gripper towards the goal location. The image frames (a), (b), (c)
reflect the alignment between measured task progress and visual state of the
task.

(a)

(b)

(c)

Figure 11: Reach Task Progress: Failure. Our GraphIRL method measures an
approximately linear task progress in this failure example for Reach. The
gripper’s distance to the goal region is indeed minimized over time, though
since it does not get within close-enough distance to the goal, the measured
task progress is lower compared to success examples shown in Figure 10.

We find that our method provides a superior and accurate reward signal to the
agent compared to the baseline visual IRL methods. We observe that if a task
is being completed successfully or unsuccessfully in a video, our method can
obtain a reward that accurately reflects how close the agent is to completing
the task. Additionally, both _XIRL_ and _TCN_ yield low reward even for
successful episodes due to large distance between the current observation and
the representative goal observation in the embedding space which could be
attributed to visual domain shift.

##  Appendix B Additional Implementation Details

Hyperparameter | Value
---|---
# of sampled frames | $90$
Batch Size | $2$
Learning Rate | $10^{-5}$
Weight Decay | $10^{-5}$
# of training iterations | $12000$
Embedding Size | $128$
Softmax Temperature | $0.1$
Table 4: Hyperparameters for Representation Learning with GraphIRL.

Representation Learning. Each MLP in the Spatial Interaction Encoder Network
defined in Equation 3.1 is implemented as a 2-layer network with a ReLU
activation. The size of the final embedding $\psi(\cdot)$ is $128$ in our
experiments. Please see Table 4 for a detailed list of hyperparameters for
representation learning. All the hyperparameters in Table 4 are kept fixed for
all tasks considered in this work.

Reinforcement Learning. For X-MAGICAL, we follow Zakka et al. [4] and learn a
state based policy. The state vector has dimensions of 16 and 17 for the
_Standard_ and _Diverse_ environments respectively. The _Diverse_ environment
state has an additional dimension to represent the size of blocks. For xArm,
we learn an image based policy. Specifically, we use first-person and third-
person cameras to learn a policy from multi-view image data. We extract
$84\times 84$ image from both cameras and concatenate them channel-wise. We
use the network architecture and attention mechanism proposed in Jangir et al.
[51]. Additionally, we apply data augmentation techniques: random $\pm 4$
pixel shift [56] and color jitter [57].

Extracting Reward. In order to compute the reward during Reinforcement
Learning (RL) training, we use the locations of objects available in
simulation to extract the bounding boxes corresponding to the current
observation. The bounding boxes are used to construct the object
representation which is then passed to the trained Spatial Interaction Encoder
Network to get the reward.

Criterion for Success. We use distance threshold to determine the success of
an episode. The thresholds are $5$cms, $10$cms and $8$cms for _Reach_ , _Push_
and _Peg in Box_ respectively. The distance refers to distance between goal
position and end-effector for _Reach_ , and goal position and object position
for _Push_ and _Peg in Box_.

Baseline Implementation Details. For all the vision-based baselines, we use
the hyperparameters, data augmentation schemes and network architectures
provided in Zakka et al. [4]. Readers are encouraged to read Zakka et al. [4]
for more details on the vision-based baselines.

##  Appendix C X-MAGICAL Experiment Details

###  C.1 Demonstration Data

For collecting demonstration data in the X-MAGICAL Diverse environment, we
trained 5 uniquely-seeded Soft Actor-Critic (SAC) RL policies for $2$ million
steps for each embodiment using the environment reward. We collect $1000$
successful episode rollouts for each embodiment using the 5 trained policies.
In particular, each policy is used to produce $200$ episode rollouts for a
given embodiment.

###  C.2 Diverse Environment

Below, we explain the randomization performed on the blocks in the diverse
environment that we use in our experiments:

  * •

Color: We randomly assign 1 out of 4 colors to each block.

  * •

Shape: Each block is randomly assigned 1 out of 6 shapes.

  * •

Size: The block sizes are also varied. In particular, we generate a number
between $0.75$ and $1.25$ and multiply the default block size by that factor.

  * •

Initial Orientation: The initial orientation of the blocks is also randomized.
We randomly pick a value between $0$ to $360$ degrees.

  * •

Initial Location: The initial location of the boxes is randomized by first
randomly picking a position for the y-coordinate for all blocks and then
randomly selecting x-coordinate separately for each block. This randomization
is also performed in the standard environment.

##  Appendix D Additional Results on X-MAGICAL Benchmark

Figure 12: Cross-Embodiment Same-Environment: We further evaluate GraphIRL in
the cross-embodiment same-environment setting (top) Standard Environment
(bottom) Diverse Environment, and it continues to provide competitive success
rates akin to those achieved by XIRL. These results confirm that GraphIRL is a
consistent and reliable method for learning from video demonstrations in
visually similar environments.

To complement our cross-embodiment cross-environment results from the main
paper, we also report results for X-MAGICAL in the _cross-embodiment_ _same-
environment_ setting. As shown in Figure 12, we outperform _TCN_ and _LIFS_ by
significant margins and achieve comparable results to _XIRL_. These results
reflect the effectiveness of GraphIRL when learning in a visually similar
environment with visually different agents.

##  Appendix E Appendix E: xArm Experiment Details

###  E.1 Description of Environment Rewards

In this section, we define the environment rewards for xArm environments that
were compared against GraphIRL in robot manipulation experiments under Section
4. We define $p_{g}$, $p_{o}$, and $p_{e}$ as the positions of the goal,
object and robot end-effector respectively. The reward for _Push_ is defined
as $||p_{o}-p_{g}||^{2}$, for reach it becomes $||p_{e}-p_{g}||^{2}$ and
finally for _Peg in Box_ , the reward is $||p_{o}-p_{g}||^{2}$. Note that the
distances are computed using 2-d positions in the case of _Reach_ and _Push_
and 3-d positions in the case of _Peg in Box_.

###  E.2 Demonstration Data

We use data from [22] for _Push_. We collect $256$ and $162$ demonstrations
respectively for _Reach_ and _Peg in Box_. For _Reach_ , we use 18 visually
distinct goal position markers i.e. $3$ different shapes and each shape with
$6$ different colors in order to ensure visual diversity. _Reach_
demonstrations have minimum, average and maximum demonstration lengths of
$1.76$ seconds, $4.51$ seconds and $9.23$ seconds respectively. For _Peg in
Box_ , we use 4 visually distinct objects. In this case, the minimum, average
and maximum demonstration lengths are $1.73$ seconds, $4.74$ seconds and
$11.7$ seconds respectively. For both _Reach_ and _Peg in Box_ , the goal and
object positions are also varied across demonstrations to diversify
trajectories. Please see https://sateeshkumar21.github.io/GraphIRL/ for
examples of collected demonstrations.

##  Appendix F Additional Results on Robot Manipulation in Simulation

Figure 13: MuJoCo State Pusher Task Progress: Success GraphIRL provides a
reward signal that is both better than all other vision-based baselines and
nearly as good as the task-specific environment reward. This indicates that
the reward learned from GraphIRL could be used across multiple environments of
the same task, showing strong generalization capabilities.

We also experiment with the _MuJoCo State Pusher_ environment used by
Schmeckpeper et al. [22] and Zakka et al. [4]. However, we make two changes,
(1) Instead of using a fixed goal position, we use a randomized goal position
and learn a goal-conditioned policy and (2) we do not use the sparse
environment reward and instead only use the learned rewards for GraphIRL and
learning-based baselines. Figure 13 presents our results, we note that
GraphIRL achieves slightly lower success rate than the task-specific
environment reward (e.g. GraphIRL $0.455$ vs Environment Reward $0.6133$).
Further, all vision-based baselines perform significantly lower than GraphIRL
(e.g. GraphIRL $0.455$ vs XIRL $0.125$ and TCN $0.005$). For all learning-
based methods, we use the data from Schmeckpeper et al. [22] as training
demonstrations similar to _Push_ experiments conducted in Section 4.

(a)

(b)

(c)

(d)

Figure 14: Real Robot Setup. In (a) and (b), we provide images of our real-
world environment for the Peg in Box task. We use a static third-person camera
and an egocentric camera which moves with the arm while completing the task.
Pictured in (c) and (d) are single image frames captured by our third-person
and egocentric cameras.

##  Appendix G Robot Setup

We use a Ufactory xArm 7 robot for our real robot experiments. As shown in
Figure 14, we use a fixed third-person camera and an egocentric camera that is
attached above the robot’s gripper. Example images of the egocentric and
third-person camera feeds passed to the RL agent are shown in Figure 14 (c)
and Figure 14 (d).


# Measurements of the observed cross sections for $e^{+}e^{-}\to$ exclusive
light hadrons containing $K_{S}^{0}$ meson at $\sqrt{s}=3.773$ and 3.650 GeV

M. Ablikim1, J. Z. Bai1, Y. Bai1, Y. Ban11, X. Cai1, H. F. Chen15, H. S.
Chen1, H. X. Chen1, J. C. Chen1, Jin Chen1, X. D. Chen5, Y. B. Chen1, Y. P.
Chu1, Y. S. Dai17, Z. Y. Deng1, S. X. Du1, J. Fang1, C. D. Fu14, C. S. Gao1,
Y. N. Gao14, S. D. Gu1, Y. T. Gu4, Y. N. Guo1, K. L. He1, M. He12, Y. K.
Heng1, J. Hou10, H. M. Hu1, T. Hu1, G. S. Huang1a, X. T. Huang12, Y. P.
Huang1, X. B. Ji1, X. S. Jiang1, J. B. Jiao12, D. P. Jin1, S. Jin1, Y. F.
Lai1, H. B. Li1, J. Li1, L. Li1, R. Y. Li1, W. D. Li1, W. G. Li1, X. L. Li1,
X. N. Li1, X. Q. Li10, Y. F. Liang13, H. B. Liao1b, B. J. Liu1, C. X. Liu1,
Fang Liu1, Feng Liu6, H. H. Liu1c, H. M. Liu1, J. B. Liu1d, J. P. Liu16, H. B.
Liu4, J. Liu1, R. G. Liu1, S. Liu8, Z. A. Liu1, F. Lu1, G. R. Lu5, J. G. Lu1,
C. L. Luo9, F. C. Ma8, H. L. Ma1, L. L. Ma1e, Q. M. Ma1, M. Q. A. Malik1, Z.
P. Mao1, X. H. Mo1, J. Nie1, R. G. Ping1, N. D. Qi1, H. Qin1, J. F. Qiu1, G.
Rong1, X. D. Ruan4, L. Y. Shan1, L. Shang1, D. L. Shen1, X. Y. Shen1, H. Y.
Sheng1, H. S. Sun1, S. S. Sun1, Y. Z. Sun1, Z. J. Sun1, X. Tang1, J. P.
Tian14, G. L. Tong1, X. Wan1, L. Wang1, L. L. Wang1, L. S. Wang1, P. Wang1, P.
L. Wang1, W. F. Wang1f, Y. F. Wang1, Z. Wang1, Z. Y. Wang1, C. L. Wei1, D. H.
Wei3, Y. Weng1, N. Wu1, X. M. Xia1, X. X. Xie1, G. F. Xu1, X. P. Xu6, Y. Xu10,
M. L. Yan15, H. X. Yang1, M. Yang1, Y. X. Yang3, M. H. Ye2, Y. X. Ye15, C. X.
Yu10, G. W. Yu1, C. Z. Yuan1, Y. Yuan1, S. L. Zang1g, Y. Zeng7, B. X. Zhang1,
B. Y. Zhang1, C. C. Zhang1, D. H. Zhang1, H. Q. Zhang1, H. Y. Zhang1, J. W.
Zhang1, J. Y. Zhang1, X. Y. Zhang12, Y. Y. Zhang13, Z. X. Zhang11, Z. P.
Zhang15, D. X. Zhao1, J. W. Zhao1, M. G. Zhao1, P. P. Zhao1, B. Zheng1, H. Q.
Zheng11, J. P. Zheng1, Z. P. Zheng1, B. Zhong9 L. Zhou1, K. J. Zhu1, Q. M.
Zhu1, X. W. Zhu1, Y. C. Zhu1, Y. S. Zhu1, Z. A. Zhu1, Z. L. Zhu3, B. A.
Zhuang1, B. S. Zou1
(BES Collaboration)
1 Institute of High Energy Physics, Beijing 100049, People’s Republic of China
2 China Center for Advanced Science and Technology(CCAST), Beijing 100080,
People’s Republic of China
3 Guangxi Normal University, Guilin 541004, People’s Republic of China
4 Guangxi University, Nanning 530004, People’s Republic of China
5 Henan Normal University, Xinxiang 453002, People’s Republic of China
6 Huazhong Normal University, Wuhan 430079, People’s Republic of China
7 Hunan University, Changsha 410082, People’s Republic of China
8 Liaoning University, Shenyang 110036, People’s Republic of China
9 Nanjing Normal University, Nanjing 210097, People’s Republic of China
10 Nankai University, Tianjin 300071, People’s Republic of China
11 Peking University, Beijing 100871, People’s Republic of China
12 Shandong University, Jinan 250100, People’s Republic of China
13 Sichuan University, Chengdu 610064, People’s Republic of China
14 Tsinghua University, Beijing 100084, People’s Republic of China
15 University of Science and Technology of China, Hefei 230026, People’s
Republic of China
16 Wuhan University, Wuhan 430072, People’s Republic of China
17 Zhejiang University, Hangzhou 310028, People’s Republic of China
a Current address: University of Oklahoma, Norman, Oklahoma 73019, USA
b Current address: DAPNIA/SPP Batiment 141, CEA Saclay, 91191, Gif sur Yvette
Cedex, France
c Current address: Henan University of Science and Technology, Luoyang 471003,
People’s Republic of China
d Current address: CERN, CH-1211 Geneva 23, Switzerland
e Current address: University of Toronto, Toronto M5S 1A7, Canada
f Current address: Laboratoire de l’Accélérateur Linéaire, Orsay, F-91898,
France
g Current address: University of Colorado, Boulder, CO 80309, USA

###### Abstract

By analyzing the data sets of 17.3 pb-1 taken at $\sqrt{s}=3.773$ GeV and of
6.5 pb-1 taken at $\sqrt{s}=3.650$ GeV with the BES-II detector at the BEPC
collider, we measure the observed cross sections for the exclusive light
hadron final states of $K_{S}^{0}K^{-}\pi^{+}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}$, $K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ produced in $e^{+}e^{-}$ annihilation at
the two energy points. We set the upper limits on the observed cross sections
and the branching fractions for $\psi(3770)$ decay to these final states at
$90\%$ C.L..

##  I INTRODUCTION

The $\psi(3770)$ resonance is expected to decay almost entirely into
$D\bar{D}$ meson pairs since its width is almost two orders of magnitude
larger than that of $\psi(3686)$ prl39_526 . In recent years, the study of the
$\psi(3770)$ non-$D\bar{D}$ decays becomes an attractive study field in the
charmonium energy region due to the existing puzzle that about $38\%$ of
$\psi(3770)$ does not decay into $D\bar{D}$ meson pairs hepex_0506051 . To
understand the possible excess of the $\psi(3770)$ cross section relative to
the $D\bar{D}$ cross section, BES and CLEO Collaborations made many efforts to
study the $\psi(3770)$ non-$D\bar{D}$ decays. The CLEO Collaboration measured
the $e^{+}e^{-}\to\psi(3770)\to$ non-$D\bar{D}$ cross section to be $(-0.01\pm
0.08^{+0.41}_{-0.30})$ nb prl96_092002 . While the BES Collaboration measured
the branching fraction for $\psi(3770)\to$ non$-D\bar{D}$ decay to be $(15\pm
5)\%$ plb641_145 ; prl97_121801 ; plb659_74 ; prd76_000000 ; pdg07 , which
indicates that, contrary to what is generally expected, the $\psi(3770)$ might
substantially decay into non$-D\bar{D}$ final states or there are some new
structure or physics effects which may partially be responsible for the
largely measured non-$D\bar{D}$ branching fraction of the $\psi(3770)$ decays
prl101_102004 ; plb668_263 . BES Collaboration observed the first
non$-D\bar{D}$ decay mode for $\psi(3770)\to J/\psi\pi^{+}\pi^{-}$, and
measured its decay branching fraction to be ${\mathcal{B}}[\psi(3770)\to
J/\psi\pi^{+}\pi^{-}]=(0.34\pm 0.14\pm 0.09)\%$ hepnp28_325 ; plb605_63 . This
was confirmed by CLEO Collaboration prl96_082004 . Latter, CLEO Collaboration
observed more $\psi(3770)$ exclusive non-$D\bar{D}$ decays, $\psi(3770)\to
J/\psi\pi^{0}\pi^{0}$, $J/\psi\pi^{0}$, $J/\psi\eta$ prl96_082004 ,
$\gamma\chi_{cJ}(J=0,1,2)$ prl96_182002 ; prd74_031106  and $\phi\eta$
prd74_012005 , etc. Summing over these measured branching fractions yields the
sum of the branching fractions for the $\psi(3770)$ exclusive non-$D\bar{D}$
decays not more than 2%. In addition, BES and CLEO Collaborations also
attempted to search for other $\psi(3770)$ exclusive charmless decays
prd70_077101 ; prd72_072007 ; plb650_111 ; plb656_30 ; epjc52_805
prd74_012005 ; prl96_032003 ; prd73_012002 . However, the existing results can
not clarify the possible excess. For better understanding the origin of the
possible excess, search for more $\psi(3770)$ exclusive charmless decays will
be helpful.

In this Letter, we report measurements of the observed cross sections for the
exclusive light hadron final states of $K_{S}^{0}K^{-}\pi^{+}$ (Throughout the
Letter, charge conjugation is implied), $K_{S}^{0}K^{-}\pi^{+}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ at the center-of-mass energies of 3.773
and 3.650 GeV with the same method as the one used in our previous works
plb650_111 ; plb656_30 ; epjc52_805 . With the measured cross sections at the
two energy points, we set the upper limits on the observed cross sections and
the branching fractions for $\psi(3770)$ decay to these final states. The
measurements are made by analyzing the data set of 17.3 pb-1 collected at
$\sqrt{s}=3.773$ GeV [called as the $\psi(3770)$ resonance data] and the data
set of 6.5 pb-1 collected at $\sqrt{s}=3.650$ GeV (called as the continuum
data) with the BESII detector at the BEPC collider.

##  II BESII detector

The BES-II is a conventional cylindrical magnetic detector that is described
in detail in Refs. nima344_319 ; nima458_627 . A 12-layer vertex chamber (VC)
surrounding the beryllium beam pipe provides input to the event trigger, as
well as coordinate information. A forty-layer main drift chamber (MDC) located
just outside the VC yields precise measurements of charged particle
trajectories with a solid angle coverage of $85\%$ of 4$\pi$; it also provides
ionization energy loss ($dE/dx$) measurements which are used for particle
identification. Momentum resolution of $1.7\%\sqrt{1+p^{2}}$ ($p$ in GeV/$c$)
and $dE/dx$ resolution of $8.5\%$ for Bhabha scattering electrons are obtained
for the data taken at $\sqrt{s}=3.773$ GeV. An array of 48 scintillation
counters surrounding the MDC measures the time of flight (TOF) of charged
particles with a resolution of about 180 ps for electrons. Outside the TOF, a
12 radiation length, lead-gas barrel shower counter (BSC), operating in
limited streamer mode, measures the energies of electrons and photons over
$80\%$ of the total solid angle with an energy resolution of
$\sigma_{E}/E=0.22/\sqrt{E}$ ($E$ in GeV) and spatial resolutions of
$\sigma_{\phi}=7.9$ mrad and $\sigma_{z}=2.3$ cm for electrons. A solenoidal
magnet outside the BSC provides a 0.4 T magnetic field in the central tracking
region of the detector. Three double-layer muon counters instrument the magnet
flux return and serve to identify muons with momentum greater than 500
MeV/$c$. They cover $68\%$ of the total solid angle.

##  III  EVENT SELECTION

In the reconstruction of the $K_{S}^{0}K^{-}\pi^{+}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}$, $K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ final states, the $K^{0}_{S}$ and
$\pi^{0}$ mesons are reconstructed through the decays of
$K^{0}_{S}\to\pi^{+}\pi^{-}$ and $\pi^{0}\to\gamma\gamma$.

For each candidate event, we require that at least four charged tracks are
well reconstructed in the MDC with good helix fits, and the polar angle of
each charged track satisfies $|\rm cos\theta|<0.85$. The charged tracks
(except for the $K_{S}^{0}$ meson reconstruction) are required to originate
from the interaction region $V_{xy}<2.0$ cm ($V_{xy}<8.0$ cm) and
$|V_{z}|<20.0$ cm, where $V_{xy}$ and $|V_{z}|$ are the closest approaches in
the $xy$-plane and the $z$ direction, respectively.

The charged particles are identified by using the $dE/dx$ and TOF
measurements, with which the combined confidence levels $CL_{\pi}$ and
$CL_{K}$ for pion and kaon hypotheses are calculated. The pion and kaon
candidates are required to satisfy $CL_{\pi}>0.001$ and $CL_{K}>CL_{\pi}$,
respectively. To reconstruct $K^{0}_{S}$ mesons, we require that the
$\pi^{+}\pi^{-}$ meson pairs must originate from a secondary vertex which is
displaced from the event vertex at least by 4 mm in the $xy$-plane.

The photons are selected with the BSC measurements. The good photon candidates
are required to satisfy the following criteria: the energy deposited in the
BSC is greater than 50 MeV, the electromagnetic shower starts in the first 5
readout layers, the angle between the photon and the nearest charged track is
greater than $22^{\circ}$ plb_597_39 ; plb_608_24 , and the opening angle
between the cluster development direction and the photon emission direction is
less than $37^{\circ}$ plb_597_39 ; plb_608_24 .

For each candidate event, there may be several different charged and/or
neutral track combinations satisfying the above selection criteria for
exclusive light hadron final states. Each combination is subjected to an
energy-momentum conservation kinematic fit. For the processes containing
$\pi^{0}$ meson in the final states, an additional constraint kinematic fit is
imposed on $\pi^{0}\to\gamma\gamma$. Candidates with a fit probability larger
than 1$\%$ are accepted. If more than one combination satisfies the selection
criteria in an event, only the combination with the longest decay distance of
the reconstructed $K_{S}^{0}$ mesons is retained.

To suppress the background from the $D\bar{D}$ decays, we use the double tag
method npb727_395  to remove the $D\bar{D}$ events. For example, for the
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}$ final state, we exclude the all possible events
from $D\bar{D}$ decays by rejecting those in which the $D$ and $\bar{D}$
mesons can be reconstructed in the decay modes of $D^{-}\to K_{S}^{0}K^{-}$
and $D^{+}\to\pi^{+}\pi^{0}$, $D^{-}\to K^{-}\pi^{0}$ and $D^{+}\to
K_{S}^{0}\pi^{+}$, $\bar{D}^{0}\to K_{S}^{0}\pi^{0}$ and $D^{0}\to
K^{-}\pi^{+}$ npb727_395 . For the other final states, the events from
$D\bar{D}$ decays are suppressed similarly. The remaining contaminations from
$D\bar{D}$ decays due to particle misidentification or missing photon(s) are
accounted by using Monte Carlo simulation, as discussed in Section V.

##  IV DATA ANALYSIS

In the data analysis, these processes containing $K^{0}_{S}$ meson in the
final state are studied by examining the invariant mass spectra of the
$\pi^{+}\pi^{-}$ combinations satisfying the above selection criteria for the
$K^{0}_{S}$ meson reconstruction. The invariant masses of the $\pi^{+}\pi^{-}$
combinations are calculated with the momentum vectors from the $K^{0}_{S}$
reconstruction. Figure 1 shows the resulting distribution of the invariant
masses of the $\pi^{+}\pi^{-}$ combinations from the selected candidates for
the $K_{S}^{0}K^{-}\pi^{+}$, $K_{S}^{0}K^{-}\pi^{+}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ final states. In each figure, the peak
around the $K^{0}_{S}$ nominal mass indicates the production of
$e^{+}e^{-}\to$ exclusive light hadrons containing $K_{S}^{0}$ meson. Fitting
the $\pi^{+}\pi^{-}$ invariant mass spectra with a Gaussian function for the
$K^{0}_{S}$ signal and a flat background yields the number of the events for
each process observed from the $\psi(3770)$ resonance data and the continuum
data. In the fit, the $K^{0}_{S}$ mass and its mass resolution are fixed at
the values obtained by analyzing Monte Carlo samples.

(a)(a’)(b)(b’)(c)(c’)(d)(d’)(e)(e’)(f)(f’)Invariant mass (GeV/c${}^{2})$

Events/0.005GeV/c2

Fig. 1:  The $\pi^{+}\pi^{-}$ invariant mass spectra of the candidates for the
(a) $K_{S}^{0}K^{-}\pi^{+}$, (b) $K_{S}^{0}K^{-}\pi^{+}\pi^{0}$, (c)
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$, (d)
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$, (e)
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ and (f)
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ final states selected from the
$\psi(3770)$ resonance data (left) and the continuum data (right).

##  V BACKGROUND SUBTRACTION

Some other events may contribute to the selected candidate events for
$e^{+}e^{-}\to f$ ($f$ represents exclusive light hadron final state). These
include the events from $J/\psi$ and $\psi(3686)$ decays due to ISR returns,
the events from the other final states due to misidentifying a pion as a kaon
or reverse, and the events from $D\bar{D}$ decays. The number $N^{\rm b}$ of
these contaminations should be subtracted from the number $N^{\rm obs}$ of the
candidates for $e^{+}e^{-}\to f$. The estimation of them can be done based on
Monte Carlo simulation. The details about the background subtraction have been
described in Ref. plb650_111 . For each background channel except $D\bar{D}$
decays, 50,000 or 100,000 Monte Carlo events are used in the background
estimation. The Monte Carlo sample of each different background channel is
from ten to several thousands times larger than the data in size.

Monte Carlo study shows that the contaminations from $\psi(3770)\to
J/\psi\pi^{+}\pi^{-}$, $\psi(3770)\to J/\psi\pi^{0}\pi^{0}$, $\psi(3770)\to
J/\psi\pi^{0}$ and $\psi(3770)\to\gamma\chi_{cJ}\hskip 2.84544pt(J=0,1,2)$ can
be neglected.

Even though we have removed the main contaminations from $D\bar{D}$ decays in
the previous event selection (see section III), there are still some events
from $D\bar{D}$ decays satisfying the selection criteria for the light hadron
final states due to particle misidentification or missing photon(s). The
number of these contaminations from $D\bar{D}$ decays are further removed by
analyzing a Monte Carlo sample which is about forty times larger than the
$\psi(3770)$ resonance data. The Monte Carlo events are generated as
$e^{+}e^{-}\to D\bar{D}$ at $\sqrt{s}=$ 3.773 GeV, where the $D$ and $\bar{D}$
mesons are set to decay into all possible final states with the branching
fractions quoted from PDG pdg07 .

Subtracting the number $N^{\rm b}$ of these contaminations from the number
$N^{\rm obs}$ of the candidate events, we obtain the net number $N^{\rm net}$
of the signal events for each process. For the $K_{S}^{0}K^{-}\pi^{+}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ final states, for which
only a few signal events are observed from the continuum data, we set the
upper limits $N^{\rm up}$ on the number of the signal events at 90% C.L..
Here, we use the Feldman-Cousins method prd57_3873  and assume that the
background is absent. The numbers of $N^{\rm obs}$, $N^{\rm b}$ and $N^{\rm
net}$ (or $N^{\rm up}$) are summarized in the second, third and fourth columns
of Tabs. 1 and 2. For each process, the background events in the $\psi(3770)$
resonance data are dominant by $D\bar{D}$ decays and $\psi(3686)$ decays.
While, there is no $D\bar{D}$ decay in the continuum data, and the
$\psi(3686)$ production cross section at $\sqrt{s}=$ 3.650 GeV is much less
than that at $\sqrt{s}=$ 3.773 GeV. So, the number of the background events in
the continnum data can almost be negligible.

##  VI  RESULTS

###  VI.1  Monte Carlo efficiency

To estimate the detection efficiency $\epsilon$ for $e^{+}e^{-}\to f$, we use
a phase space generator including initial state radiation and vacuum
polarization corrections yf41_377  with $1/s$ energy dependence in cross
section. Final state radiation cpc79_291  decreases the detection efficiency
not more than 0.5%. Detailed analysis based on Monte Carlo simulation for the
BES-II detector nima552_344  gives the detection efficiencies for each process
at $\sqrt{s}=3.773$ and 3.650 GeV, which are summarized in the fifth columns
of Tabs. 1 and 2, where the detection efficiencies do not include the
branching fractions for $K_{S}^{0}\to\pi^{+}\pi^{-}$ and
$\pi^{0}\to\gamma\gamma$, ${\mathcal{B}}(K_{S}^{0}\to\pi^{+}\pi^{-})$ and
${\mathcal{B}}(\pi^{0}\to\gamma\gamma)$.

###  VI.2 Observed cross sections

Let ${\mathcal{B}}_{\pi^{0}}$ = ${\mathcal{B}}(\pi^{0}\to\gamma\gamma)$ for
the modes of $K_{S}^{0}K^{-}\pi^{+}\pi^{0}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$, ${\mathcal{B}}_{\pi^{0}}$ =
${\mathcal{B}}^{2}(\pi^{0}\to\gamma\gamma)$ for the mode of
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ and ${\mathcal{B}}_{\pi^{0}}$ = 1 for
$K_{S}^{0}K^{-}\pi^{+}$, $K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$, where
${\mathcal{B}}(\pi^{0}\to\gamma\gamma)$ is the branching fraction for the
decay of $\pi^{0}\to\gamma\gamma$, then the observed cross section for
$e^{+}e^{-}\to f$ can be determined by

$\sigma_{e^{+}e^{-}\to f}=\frac{N^{\rm
net}}{{\mathcal{L}}\times\epsilon\times{\mathcal{B}}(K_{S}^{0}\to\pi^{+}\pi^{-})\times{\mathcal{B}}_{\pi^{0}}},$
(1)

where ${\mathcal{L}}$ is the integrated luminosity of the data set, $N^{\rm
net}$ is the number of the signal events, $\epsilon$ is the detection
efficiency and ${\mathcal{B}}(K_{S}^{0}\to\pi^{+}\pi^{-})$ is the branching
fraction for the decay of $K_{S}^{0}\to\pi^{+}\pi^{-}$. Inserting these
numbers in Eq. (1), we obtain the observed cross sections for each process at
$\sqrt{s}=3.773$ and 3.650 GeV. They are summarized in Tabs. 1 and 2, where
the first error is statistical and the second systematic. In the measurements
of the observed cross sections, the systematic errors arise from the
uncertainties in integrated luminosity of the data set ($2.1\%$ plb641_145 ;
prl97_121801 ), photon selection ($2.0\%$ per photon), tracking efficiency
($2.0\%$ per track), particle identification ($0.5\%$ per pion or kaon),
kinematic fit ($1.5\%$), $K_{S}^{0}$ reconstruction ($1.1\%$ plb_608_24 ),
branching fractions quoted from PDG jpg33_1  ($0.03\%$ for
${\mathcal{B}}(\pi^{0}\to\gamma\gamma)$ and $0.07\%$ for
${\mathcal{B}}(K_{S}^{0}\to\pi^{+}\pi^{-})$), Monte Carlo modeling ($6.0\%$
plb650_111 ; plb656_30 ; epjc52_805 ), Monte Carlo statistics ($1.4\%\sim
4.4\%$), background subtraction ($0.0\%\sim 3.0\%$) and fit to mass spectrum
($0.4\%\sim 8.5\%$). Adding these uncertainties in quadrature yields the total
systematic error $\Delta_{\rm sys}$ for each mode at $\sqrt{s}=3.773$ and
3.650 GeV.

The upper limit $\sigma_{e^{+}e^{-}\to f}^{\rm up}$ on the observed cross
sections for the $K_{S}^{0}K^{-}\pi^{+}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ final states at
$\sqrt{s}=3.650$ GeV are set with Eq. (1) by substituting $N^{\rm net}$ with
$N^{\rm up}/(1-\Delta_{\rm sys})$, where $N^{\rm up}$ is the upper limit on
the number of the signal event, and $\Delta_{\rm sys}$ is the systematic error
in the cross section measurement. Inserting the corresponding numbers in the
equation, we obtain the upper limits on the observed cross sections for
$e^{+}e^{-}\to K_{S}^{0}K^{-}\pi^{+}$ and $e^{+}e^{-}\to
K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ at $\sqrt{s}=3.650$ GeV,
which are also listed in Tab. 2.

###  VI.3 Upper limits on the observed cross sections and the branching
fractions for $\psi(3770)\to f$

If we ignore the possible interference effects between the continuum and
resonance amplitudes, and the difference of the vacuum polarization
corrections at $\sqrt{s}=3.773$ and 3.650 GeV, we can determine the observed
cross section $\sigma_{\psi(3770)\to f}$ for $\psi(3770)\to f$ at
$\sqrt{s}=3.773$ GeV by comparing the observed cross sections
$\sigma_{e^{+}e^{-}\to f}^{3.773\hskip 1.42271pt\rm GeV}$ and
$\sigma_{e^{+}e^{-}\to f}^{3.650\hskip 1.42271pt\rm GeV}$ for $e^{+}e^{-}\to
f$ measured at $\sqrt{s}=3.773$ and 3.650 GeV, respectively. It can be given
by

$\sigma_{\psi(3770)\to f}=\sigma_{e^{+}e^{-}\to f}^{3.773\hskip 1.42271pt\rm
GeV}-f_{\rm co}\times\sigma_{e^{+}e^{-}\to f}^{3.650\hskip 1.42271pt\rm GeV},$
(2)

where $f_{\rm co}=3.650^{2}/3.773^{2}$ is the normalization factor to consider
the $1/s$ cross section dependence. The results are summarized in the second
column of Tab. 3, where the first error is the statistical, the second is the
independent systematic arising from the uncertainties in the Monte Carlo
statistics, in the fit to the mass spectrum and in the background subtraction,
and the third is the common systematic error arising from the other
uncertainties as discussed in the subsection B.

The upper limit on the observed cross section $\sigma^{\rm up}_{\psi(3770)\to
f}$ for $\psi(3770)\to f$ at $\sqrt{s}=3.773$ GeV is set by shifting the cross
section by 1.64$\sigma$, where $\sigma$ is the total error of the measured
cross section. The results on $\sigma^{\rm up}_{\psi(3770)\to f}$ are
summarized in the third column of Tab. 3.

The upper limit on the branching fraction ${\mathcal{B}}^{\rm
up}_{\psi(3770)\to f}$ for $\psi(3770)\to f$ is set by dividing its upper
limit on the observed cross section $\sigma^{\rm up}_{\psi(3770)\to f}$ by the
observed cross section $\sigma^{\rm obs}_{\psi(3770)}=(7.15\pm 0.27\pm 0.27)$
nb plb650_111  for the $\psi(3770)$ production at $\sqrt{s}=3.773$ GeV and a
factor $(1-\Delta\sigma^{\rm obs}_{\psi(3770)})$, where $\Delta\sigma^{\rm
obs}_{\psi(3770)}$ is the relative error of the $\sigma^{\rm
obs}_{\psi(3770)}$. The results on ${\mathcal{B}}^{\rm up}_{\psi(3770)\to f}$
are summarized in the last column of Tab. 3.

##  VII SUMMARY

In this Letter, we present the measurements of the observed cross sections for
$K_{S}^{0}K^{-}\pi^{+}$, $K_{S}^{0}K^{-}\pi^{+}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$,
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ and
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ produced in $e^{+}e^{-}$ annihilation at
$\sqrt{s}=3.773$ and 3.650 GeV. These cross sections are obtained by analyzing
the data sets of 17.3 pb-1 taken at $\sqrt{s}=3.773$ GeV and of 6.5 pb-1 at
$\sqrt{s}=3.650$ GeV with the BES-II detector at the BEPC collider. By
comparing the observed cross sections for each process measured at
$\sqrt{s}=3.773$ and 3.650 GeV, we set the upper limits on the observed cross
sections and the branching fractions for $\psi(3770)$ decay to these final
states at $90\%$ C.L.. These measurements provide helpful information to
understand the mechanism of the continuum light hadron production and the
discrepancy between the observed cross sections for $D\bar{D}$ and
$\psi(3770)$ production.

##  VIII Acknowledgments

The BES collaboration thanks the staff of BEPC for their hard efforts. This
work is supported in part by the National Natural Science Foundation of China
under contracts Nos. 10491300, 10225524, 10225525, 10425523, the Chinese
Academy of Sciences under contract No. KJ 95T-03, the 100 Talents Program of
CAS under Contract Nos. U-11, U-24, U-25, the Knowledge Innovation Project of
CAS under Contract Nos. U-602, U-34 (IHEP), the National Natural Science
Foundation of China under Contract No. 10225522 (Tsinghua University).

## References

  * (1) MARK-I Collaboration, P. A. Rapidis, et al., Phys. Rev. Lett. 39 (1978) 526.
  * (2) G. Rong, D. H. Zhang, J. C. Chen, hep-ex/0506051.
  * (3) CLEO Collaboration, D. Besson, et al., Phys. Rev. Lett. 96 (2006) 092002.
  * (4) BES Collaboration, M. Ablikim et al., Phys. Lett. B 641 (2006) 145.
  * (5) BES Collaboration, M. Ablikim et al., Phys. Rev. Lett. 97 (2006) 121801.
  * (6) BES Collaboration, M. Ablikim et al., Phys. Lett. B 659 (2007) 74.
  * (7) BES Collaboration, M. Ablikim et al., Phys. Rev. D 76 (2007) 122002.
  * (8) Particle Data Group, 2007 partial updata for edition 2008 (URL:http://pdg.lbl.gov).
  * (9) BES Collaboration, M. Ablikim et al., Phys. Rev. Lett. 101 (2008) 102004, arXiv: 0807.0494/hep-ex.
  * (10) BES Collaboration, M.Ablikim et al., Phys. Lett. B 668 (2008) 263.
  * (11) BES Collaboration, J. Z. Bai et al., HEP $\&$ NP 28(4) (2004) 325.
  * (12) BES Collaboration, M.Ablikim et al., Phys. Lett. B 605 (2005) 63.
  * (13) CLEO Collaboration, N. E. Adam et al., Phys. Rev. Lett. 96 (2006) 082004.
  * (14) CLEO Collaboration, T. E. Coans et al., Phys. Rev. Lett. 96 (2006) 182002.
  * (15) CLEO Collaboration, B. A. Briere et al., Phys. Rev. D 74 (2006) 031106.
  * (16) CLEO Collaboration, D. Cronin-Hennessy et al., Phys. Rev. D 74 (2006) 012005.
  * (17) BES Collaboration, M.Ablikim et al., Phys. Rev. D 70 (2004) 077101.
  * (18) BES Collaboration, M.Ablikim et al., Phys. Rev. D 72 (2005) 072007.
  * (19) BES Collaboration, M.Ablikim et al., Phys. Lett. B 650 (2007) 111.
  * (20) BES Collaboration, M.Ablikim et al., Phys. Lett. B 656 (2007) 30.
  * (21) BES Collaboration, M.Ablikim et al., Eur. Phys. J. C 52 (2007) 805.
  * (22) CLEO Collaboration, G. S. Huang et al., Phys. Rev. Lett. 96 (2006) 032003.
  * (23) CLEO Collaboration, G. S. Adams et al., Phys. Rev. D 73 (2006) 012002.
  * (24) BES Collaboration, J. Z. Bai et al., Nucl. Instrum. Methods A 344 (1994) 319.
  * (25) BES Collaboration, J. Z. Bai et al., Nucl. Instrum. Methods A 458 (2001) 627.
  * (26) BES Collaboration, M. Ablikim et al., Phys. Lett. B 597 (2004) 39.
  * (27) BES Collaboration, M. Ablikim et al., Phys. Lett. B 608 (2005) 24.
  * (28) BES Collaboration, M. Ablikim et al., Nucl. Phys. B 727 (2005) 395.
  * (29) G. J. Feldman and R. D. Cousins, Phys. Rev. D 57 (1998) 3873.
  * (30) E. A. Kuraev and V. S. Fadin, Yad. Fiz. 41 (1985) 377.
  * (31) E. Barberio, Z. Was, Comput. Phys. Commun. 79 (1994) 291.
  * (32) BES Collaboration, M. Ablikim et al., Nucl. Instrum. Methods A 552 (2005) 344.
  * (33) Particle Data Group, W.-M. Yao, et al., J. Phys. G 33 (2006) 1.

Tab. 1: The observed cross sections for $e^{+}e^{-}\to$ exclusive light hadrons at $\sqrt{s}=3.773$ GeV, where $N^{\rm obs}$ is the number of events observed from the $\psi(3770)$ resonance data, $N^{\rm b}$ is the total number of background events, $N^{\rm net}$ is the number of the signal events, $\epsilon$ is the detection efficiency, $\Delta_{\rm sys}$ is the relative systematic error of the observed cross section, and $\sigma$ is the observed cross section. $e^{+}e^{-}\to$ | $N^{\rm obs}$ | $N^{\rm b}$ | $N^{\rm net}$ |  $\epsilon$($\%$) |  $\Delta_{\rm sys}$($\%$) | $\sigma^{\rm obs}[{\rm pb}]$
---|---|---|---|---|---|---
$K^{0}_{S}K^{-}\pi^{+}$ | $18.4\pm 4.6$ | $0.1\pm 0.0$ | $18.3\pm 4.6$ | $10.02\pm 0.14$ | 10.7 | $15.2\pm 3.8\pm 1.6$
$K^{0}_{S}K^{-}\pi^{+}\pi^{0}$ | $41.2\pm 6.6$ | $1.1\pm 0.2$ | $40.1\pm 6.6$ | $3.52\pm 0.08$ | 11.6 | $96.2\pm 15.9\pm 11.1$
$K^{0}_{S}K^{-}\pi^{+}\pi^{+}\pi^{-}$ | $40.0\pm 6.5$ | $1.0\pm 0.2$ | $38.9\pm 6.5$ | $3.56\pm 0.06$ | 14.2 | $91.5\pm 15.3\pm 13.0$
$K^{0}_{S}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$ | $24.5\pm 5.2$ | $1.5\pm 0.3$ | $23.0\pm 5.2$ | $0.77\pm 0.03$ | 15.2 | $253.0\pm 57.1\pm 38.4$
$K^{0}_{S}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ | $4.8\pm 2.2$ | $0.3\pm 0.1$ | $4.5\pm 2.2$ | $0.84\pm 0.03$ | 18.4 | $44.4\pm 21.9\pm 8.2$
$K^{0}_{S}K^{-}\pi^{+}\pi^{0}\pi^{0}$ | $19.8\pm 4.9$ | $2.8\pm 0.5$ | $17.0\pm 4.9$ | $0.99\pm 0.04$ | 14.3 | $147.0\pm 42.4\pm 21.0$
Tab. 2:  The observed cross sections for $e^{+}e^{-}\to$ exclusive light hadrons at $\sqrt{s}=3.650$ GeV, where $N^{\rm obs}$ is the number of events observed from the continuum data, $N^{\rm up}$ is the upper limit on the number of the signal events, $\sigma^{\rm up}$ is the upper limit on the observed cross section set at 90% C.L., and the definitions of the other symbols are the same as those in Tab. 1.  $e^{+}e^{-}\to$ | $N^{\rm obs}$ | $N^{\rm b}$ | $N^{\rm net}({\rm or}\hskip 4.5ptN^{\rm up})$ |  $\epsilon$($\%$) |  $\Delta_{\rm sys}$($\%$) | $\sigma^{\rm obs}(\sigma^{\rm up})[\rm pb]$
---|---|---|---|---|---|---
$K^{0}_{S}K^{-}\pi^{+}$ | $2$ | 0.0 | $<5.91$ | $10.55\pm 0.15$ | 12.9 | $<14.3$
$K^{0}_{S}K^{-}\pi^{+}\pi^{0}$ | $7.7\pm 2.9$ | 0.0 | $7.7\pm 2.9$ | $3.62\pm 0.09$ | 11.6 | $47.9\pm 18.0\pm 5.6$
$K^{0}_{S}K^{-}\pi^{+}\pi^{+}\pi^{-}$ | $13.4\pm 3.8$ | 0.0 | $13.4\pm 3.8$ | $3.66\pm 0.06$ | 14.1 | $81.4\pm 23.1\pm 11.5$
$K^{0}_{S}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$ | $4.6\pm 2.5$ | 0.0 | $4.6\pm 2.5$ | $0.87\pm 0.03$ | 17.2 | $119.0\pm 64.7\pm 20.5$
$K^{0}_{S}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ | $0$ | 0.0 | $<2.44$ | $0.95\pm 0.03$ | 18.1 | $<69.7$
$K^{0}_{S}K^{-}\pi^{+}\pi^{0}\pi^{0}$ | $3.3\pm 2.0$ | 0.0 | $3.3\pm 2.0$ | $1.12\pm 0.05$ | 14.1 | $67.1\pm 40.7\pm 9.5$
Tab. 3: The upper limits on the observed cross section $\sigma^{\rm up}_{\psi(3770)\to f}$ at $\sqrt{s}=3.773$ GeV and the branching fraction ${\mathcal{B}}^{\rm up}_{\psi(3770)\to f}$ for $\psi(3770)\to f$ are set at 90% C.L.. The $\sigma_{\psi(3770)\to f}$ is calculated with Eq. (2), where the first error is the statistical, the second is the independent systematic, and the third is the common systematic error. The upper ∗ denotes that we neglect the contributions from the continuum production. Decay Mode |  $\sigma_{\psi(3770)\to f}$ (pb) |  $\sigma^{\rm up}_{\psi(3770)\to f}$(pb) | ${\mathcal{B}}^{\rm up}_{\psi(3770)\to f}$
---|---|---|---
$K_{S}^{0}K^{-}\pi^{+}$ | $15.2\pm 3.8\pm 0.2\pm 1.6^{*}$ | $<22.0$ | $3.2\times 10^{-3}$
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}$ | $51.4\pm 23.2\pm 2.6\pm 5.8$ | $<90.7$ | $13.3\times 10^{-3}$
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}$ | $15.3\pm 26.5\pm 2.2\pm 2.1$ | $<59.0$ | $8.7\times 10^{-3}$
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{-}\pi^{0}$ | $141.6\pm 83.2\pm 14.7\pm 20.7$ | $<284.3$ | $41.8\times 10^{-3}$
$K_{S}^{0}K^{-}\pi^{+}\pi^{+}\pi^{+}\pi^{-}\pi^{-}$ | $44.4\pm 21.9\pm 2.1\pm 7.9^{*}$ | $<82.7$ | $12.2\times 10^{-3}$
$K_{S}^{0}K^{-}\pi^{+}\pi^{0}\pi^{0}$ | $84.2\pm 57.2\pm 8.3\pm 11.2$ | $<180.4$ | $26.5\times 10^{-3}$