2023

# On Robot Grasp Learning Using Equivariant Models

Xupeng Zhu<EMAIL_ADDRESS>Dian Wang<EMAIL_ADDRESS>Guanang Su<EMAIL_ADDRESS>Ondrej Biza<EMAIL_ADDRESS>Robin
Walters<EMAIL_ADDRESS>Robert Platt<EMAIL_ADDRESS>Khoury College of Computer Science, Northeastern University, Huntington Ave,
Boston, 02115, Massachusetts, USA

###### Abstract

Real-world grasp detection is challenging due to the stochasticity in grasp
dynamics and the noise in hardware. Ideally, the system would adapt to the
real world by training directly on physical systems. However, this is
generally difficult due to the large amount of training data required by most
grasp learning models. In this paper, we note that the planar grasp function
is $\mathrm{SE}(2)$-equivariant and demonstrate that this structure can be
used to constrain the neural network used during learning. This creates an
inductive bias that can significantly improve the sample efficiency of grasp
learning and enable end-to-end training from scratch on a physical robot with
as few as $600$ grasp attempts. We call this method Symmetric Grasp learning
(SymGrasp) and show that it can learn to grasp “from scratch” in less that 1.5
hours of physical robot time.

###### keywords:

Grasping, equivariant models, on robot learning, sample efficiency,
reinforcement learning, transparent object grasping

##  1 Introduction

_Grasp detection_ detects good grasp poses in a scene directly from raw visual
input (e.g., RGB or depth images) using machine learning. The learning-based
method generalizes to novel objects. This is in contrast to classical model-
based methods that attempt to reconstruct the geometry and the pose of objects
in a scene and then reason geometrically about how to grasp those objects.

Most current grasp detection models are data-driven, i.e., they must be
trained using large offline datasets. For example, Mousavian et al. (2019)
trains on a dataset consisting of over 7M simulated grasps, Breyer et al.
(2021) trains on over 2M simulated grasps, Mahler et al. (2017) trains on
grasp data drawn from over 6.7M simulated point clouds, and ten Pas et al.
(2017a) trains on over 700k simulated grasps. Some models are trained using
datasets obtained via physical robotic grasp interactions. For example, Pinto
and Gupta (2015) trains on a dataset created by performing 50k grasp attempts
over 700 hours, Kalashnikov et al. (2018) trains on over 580k grasp attempts
collected over the course of 800 robot hours, and Berscheid et al. (2021)
train on a dataset obtained by performing 27k grasps over 120 hours.

Such reliance on collecting large datasets necessitates either learning in
simulation or using significant amounts of robot time to generate data,
motivating the desire for a more sample efficient grasp detection model, i.e.,
a model that can achieve good performance with a smaller dataset. In this
paper, we propose a novel grasp detection strategy that improves sample
efficiency significantly by incorporating the equivariant structure into the
model. We term our strategy Symmetric Grasp learning as SymGrasp. Our key
observation is that the target grasp function (from images onto grasp poses)
is $\mathrm{SE}(2)$-equivariant. That is, rotations and translations of the
input image should correspond to the same rotations and translations of the
detected grasp poses at the output of the function. In order to encode the
$\mathrm{SE}(2)$-equivariance in the target function, we constrain the layers
of our model to respect this symmetry. Compared with conventional grasp
detection models that must be trained using tens of thousands of grasp
experiences, the equivariant structure we encode into the model enables us to
achieve good grasp performance after only a few hundred grasp attempts.

This paper makes several key contributions. First, we recognize that the grasp
detection function from images to grasp poses is a
$\mathrm{SE}(2)$-equivariant function. Then, we propose a neural network model
using equivariant layers to encode this property. Finally, we introduce
several algorithmic optimizations that enable us to learn to grasp online
using a contextual bandit framework. Ultimately, our model is able to learn to
grasp opaque (using depth images) and transparent objects (using RGB images)
with a good success rate after only approximately 600 grasp trials – 1.5 hours
of robot time. Although the model we propose here is only for 2D grasping
(i.e., we only detect top down grasps rather than all six dimensions as in
6-DOF grasp detection), the sample efficiency is still impressive and we
believe the concepts could be extended to higher-DOF grasp detection models in
the future.

These improvements in sample efficiency are important for several reasons.
First, since our model can learn to grasp in only a few hundred grasp trials,
it can be trained easily on a physical robotic system. This greatly reduces
the need to train on large datasets created in simulation, and it therefore
reduces our exposure to the risks associated with bridging the sim2real domain
gap – we can simply do all our training on physical robotic systems. Second,
since we are training on a small dataset, it is much easier to learn on-policy
rather than off-policy, i.e., we can train using data generated by the policy
being learned rather than with a fixed dataset. This focuses learning on areas
of state space explored by the policy and makes the resulting policies more
robust in those areas. Finally, since we can learn efficiently from a small
number of experiences, our policy has the potential to adapt relatively
quickly at run time to physical changes in the robot sensors and actuators.

##  2 Related Work

###  2.1 Equivariant convolutional layers

Equivariant convolutional layers incorporate symmetries into the structure of
convolutional layers, allowing them to generalize across a symmetry group
automatically. This idea was first introduced as G-Convolution (Cohen and
Welling, 2016a) and Steerable CNN (Cohen and Welling, 2016b). E2CNN is a
generic framework for implementing $\mathrm{E}(2)$ Steerable CNN layers
(Weiler and Cesa, 2019). In applications such as dynamics (Walters et al.,
2020; Wang et al., 2020b) and reinforcement learning (van der Pol et al.,
2020; Mondal et al., 2020; Wang et al., 2021, 2022) equivariant models
demonstrate improvements over traditional approaches.

###  2.2 Sample efficient reinforcement learning

Recent work has shown that data augmentation using random crops and/or shifts
can improve the sample efficiency of standard reinforcement learning
algorithms (Laskin et al., 2020a; Kostrikov et al., 2020). It is possible to
improve sample efficiency even further by incorporating contrastive learning
(Oord et al., 2018), e.g. CURL Laskin et al. (2020b). The contrastive loss
enables the model to learn an internal latent representation that is invariant
to the type of data augmentation used. The FERM framework Zhan et al. (2020)
applies this idea to robotic manipulation and is able to learn to perform
simple manipulation tasks directly on physical robotic hardware. The
equivariant models used in this paper are similar to data augmentation in that
the goal is to leverage problem symmetries to accelerate learning. However,
whereas data augmentation and contrastive approaches require the model to
_learn_ an invariant or equivariant encoding, the equivariant model layers
used in this paper _enforce_ equivariance as a prior encoded in the model.
This simplifies the learning task and enables our model to learn faster (see
Section 6).

###  2.3 Grasp detection

In grasp detection, the robot finds grasp configurations directly from visual
or depth data. This is in contrast to classical methods which attempt to
reconstruct object or scene geometry and then do grasp planning. See Platt
for a review on this topic.

2D Grasping: Several methods are designed to detect grasps in 2D, i.e., to
detect the planar position and orientation of grasps in a scene based on top-
down images. A key early example of this was DexNet 2.0, which infers the
quality of a grasp centered and aligned with an oriented image patch Mahler et
al. (2017). Subsequent work proposed fully convolutional architectures,
thereby enabling the model to quickly infer the pose of _all_ grasps in a
(planar) scene Morrison et al. (2018); Satish et al. (2019); Depierre et al.
(2018); Kumra et al. (2019); Zhou et al. (2018) (some of these models infer
the $z$ coordinate of the grasp as well).

3D Grasping: There is much work in 3D grasp detection, i.e., detecting the
full 6-DOF position and orientation of grasps based on truncated signed
distance function (TSDF) or point cloud input. A key early example of this was
GPD ten Pas et al. (2017b) which inferred grasp pose based on point cloud
input. Subsequent work has focused on improving grasp candidate generation in
order to improve efficiency, accuracy, and coverage Huang et al. (2022);
Mousavian et al. (2019); Sundermeyer et al. (2021); Jiang et al. (2021); Fang
et al. (2020); Breyer et al. (2021); Berscheid et al. (2021).

On-robot Grasp Learning: Another important trend has been learning to grasp
directly from physical robotic grasp experiences. Early examples of this
include Pinto and Gupta (2015) who learn to grasp from 50k grasp experiences
collected over 700 hours of robot time and Levine et al. (2018) who learn a
grasp policy from 800k grasp experiences collected over two months. QT-Opt
(Kalashnikov et al., 2018) learns a grasp policy from 580k grasp experiences
collected over 800 hours and James et al. (2019) extends this work by learning
from an additional 28k grasp experiences. Song et al. (2020) learns a grasp
detection model from 8k grasp demonstrations collected via demonstration and
Zeng et al. (2018) learns a online pushing/grasping policy from just 2.5k
grasps.

Transparent objects grasping: Commercial depth sensors that are based on
structured-light or time-of-flight techniques often fail to sense transparent
objects accurately Weng et al. (2020). Pixels in the depth image are often
dropped due to specularities or the object is simply invisible to the sensor
because it is transparent Sajjan et al. (2020). To avoid this type of failure,
RGB or RGB-D sensors are commonly used. Weng et al. (2020) infers grasp pose
from RGBD images. They collect paired RGB and D images on opaque objects and
utilize transfer learning from a trained D modality model to an RGBD modality
model. Sajjan et al. (2020) reconstructs depth images from RGBD images using
CNN and then performs grasp detection. Likewise, Ichnowski et al. (2021) and
Kerr et al.  infer grasp pose from reconstructed depth images, but using
neural radiance field (NeRF) Mildenhall et al. (2021) for reconstruction.
These methods either rely on collecting paired images for training or require
training a NeRF model per grasp during evaluation. In contrast, our method is
trained directly on RGB images without requiring paired images or NeRF models.

Equivariance through canonicalization in grasping: An alternative to modeling
rotational symmetry using equivariant neural network layers is an approach
known as _canonicalization_ where we learn a model over the non-equivariant
variables assuming a single “canonical” group element Wang et al. (2020b);
Kofinas et al. (2021); Gao et al. (2020). Equivariance based on
canonicalization is common in robotic grasping where it is not unusual to
translate and rotate the input image so that it is expressed in the reference
frame of the hand, e.g. Mahler et al. (2017); ten Pas et al. (2017b);
Mousavian et al. (2019). This way, the neural network model need only infer
the quality of a grasp for a single canonical grasp pose rather than over
arbitrary translations and orientations. In this paper, we compare our model-
based approach to equivariance with VPG, a method that obtains rotational
equivariance via canonicalization Song et al. (2020). Our results in Section
6.2 suggest that the model-based approach has a significant advantage.

##  3 Background

###  3.1 Equivariant Neural Network Models

In this paper, we use equivariant neural network layers defined with respect
to a finite group (e.g. a finite group of rotations). Equivariant neural
network Weiler and Cesa (2019); Cohen et al. (2018); Cohen and Welling (2016b,
a) encodes a function $f:X\rightarrow Y$ that satisfy the equivariance
constraint: $gf(x)=f(gx)$, where $g\in G$ is an element of a finite group.
$gx$ is shorthand for the action of $g$ on $x$, e.g. rotation of an image $x$.
Similarly, $g(f(x))$ describes the action of $g$ on $f(x)$. Below, we make
these ideas more precise and summarize how the equivariance constraint is
encoded into a neural network layer.

####  3.1.1 The cyclic group $C_{n}\leq\mathrm{SO}(2)$

We are primarily interested in equivariance with respect to the group of
planar rotations, $\mathrm{SO}(2)$. However, in practice, in order to make our
models computationally tractable, we will use the cyclic subgroup $C_{n}$ of
$\mathrm{SO}(2)$, $C_{n}=\\{2\pi k/n:0\leq k<n\\}$. $C_{n}$ is the group of
discrete rotations by multiples of $2\pi/n$ radians.

####  3.1.2 Representation of a group

The way a group element $g\in G$ acts on $x$ depends on how $x$ is
represented. If $x$ is a point in the plane, then $g$ acts on $x$ via the
_standard representation_ , $\rho_{1}(g)x$, where $\rho_{1}(g)$ is the
standard $2\times 2$ rotation matrix corresponding to $g$. In the hidden
layers of an equivariant neural network model, it is common to encode a
separate feature map for each group element. For example, suppose $G$ is the
order $n$ cyclic group and suppose $x$ is a set of features
$x=(x_{1},x_{2},\dots,x_{n})\in\mathbb{R}^{n\times\lambda}$ that maps the
$k$th group element to a feature $x_{k}$. The _regular representation_ of $g$
acts on $x$ by permuting its elements:
$\rho_{reg}(g)x=(x_{n-m+1},\dots,x_{n},x_{1},x_{2},\dots,x_{n-m})$ where $g$
is the $m$th element in $C_{n}$. Finally, it is sometimes the case that $x$ is
invariant to the action of the group elements. In this case, we have the
_trivial representation_ , $\rho_{0}(g)x=x$.

####  3.1.3 Feature maps of equivariant convolutional layers

An equivariant convolutional layer maps between feature maps which transform
by specified representations $\rho$ of the group. In the hidden layers of an
equivariant model, generally, an extra dimension is added to the feature maps
to encode group elements via the regular representation. So, whereas the
feature map used by a standard convolutional layer is a tensor
$\mathcal{F}\in\mathbb{R}^{m\times h\times w}$, an equivariant convolutional
layer adds an extra dimension: $\mathcal{F}\in\mathbb{R}^{k\times m\times
h\times w}$, where $k$ denotes the dimension of the group representation. This
tensor associates each pixel $(u,v)\in\mathbb{R}^{h\times w}$ with a matrix
$\mathcal{F}(u,v)\in\mathbb{R}^{k\times m}$.

####  3.1.4 Action of the group operator on the feature map

Given a feature map $\mathcal{F}\in\mathbb{R}^{k\times m\times h\times w}$
associated with group $G$ and representation $\rho$, a group element $g\in G$
acts on $\mathcal{F}$ via:

$(g\mathcal{F})(x)=\rho(g)\mathcal{F}(\rho_{1}(g)^{-1}x),$ (1)

where $x\in\mathbb{R}^{2}$ denotes pixel position. The RHS of this equation
applies the group operator in two ways. First, $\rho_{1}(g)^{-1}$ rotates the
pixel position $x$ using the standard representation. Second, $\rho$ applies
the rotation to the feature representation. If the feature is invariant to the
rotation, then we use the trivial representation $\rho_{0}(g)$. However, if
the feature vector changes according to rotation (e.g. the feature denotes
grasp orientation), then it must be transformed as well. This is accomplished
by setting $\rho$ in Equation 1 to be the regular representation that
transforms the feature vector by a circular shift.

####  3.1.5 The equivariant convolutional layer

An equivariant convolutional layer is a function $h$ from $\mathcal{F}_{in}$
to $\mathcal{F}_{out}$ that is constrained to represent only equivariant
functions with respect to a chosen group $G$. The feature maps
$\mathcal{F}_{in}$ and $\mathcal{F}_{out}$ are associated with representations
$\rho_{in}$ and $\rho_{out}$ acting on feature spaces $\mathbb{R}^{k_{in}}$
and $\mathbb{R}^{k_{out}}$ respectively. Then the equivariant constraint for
$h$ is Cohen et al. (2018):

$h(g\mathcal{F}_{in})=gh(\mathcal{F}_{in})=g\mathcal{F}_{out}.$ (2)

This constraint can be implemented by tying kernel weights
$K(y)\in\mathbb{R}^{k_{out}\times k_{in}}$ in such a way as to satisfy the
following constraint (Cohen et al., 2018):

$K(gy)=\rho_{out}(g)K(y)\rho_{in}(g)^{-1}.$ (3)

Please see Appendix.B for an example of equivariant convolustional layer.

When all hidden layers $h$ in a neural network satisfy Equation 2, then by
induction the entire neural network is equivariant Cohen et al. (2018).

###  3.2 Augmented State Representation (ASR)

We will formulate $\mathrm{SE}(2)$ robotic grasping as the problem of learning
a function from an $m$ channel image, $s\in S=\mathbb{R}^{m\times h\times w}$,
to a gripper pose $a\in A=\mathrm{SE}(2)$ from which an object may be grasped.
Since we will use the contextual bandit framework, we need to be able to
represent the $Q$-function, $Q:\mathbb{R}^{m\times h\times
w}\times\mathrm{SE}(2)\rightarrow\mathbb{R}$. However, this is difficult to do
using a single neural network due to the GPU memory limitation. To combat
this, we will use the Augmented State Representation (ASR) (Sharma et al.,
2017; Wang et al., 2020a) to model $Q$ as a pair of functions, $Q_{1}$ and
$Q_{2}$. Another advantage of using ASR is we can use different group order in
$Q_{1},Q_{2}$, as explained in Section 5.1.3.

We follow the ASR framework that factors
$\mathrm{SE}(2)=\mathbb{R}^{2}\times\mathrm{SO}(2)$ into a translational
component $X\subseteq\mathbb{R}^{2}$ and a rotational component
$\Theta\subseteq\mathrm{SO}(2)$. The first function is a mapping
$Q_{1}:\mathbb{R}^{m\times h\times w}\times X\rightarrow\mathbb{R}$ which maps
from the image $s$ and the translational component of action $X$ onto value.
This function is defined to be:
$Q_{1}(s,x)=\max_{\theta\in\Theta}Q(s,(x,\theta))$. The second function is a
mapping $Q_{2}:\mathbb{R}^{m\times h^{\prime}\times
w^{\prime}}\times\Theta\rightarrow\mathbb{R}$ with $h^{\prime}\leq h$ and
$w^{\prime}\leq w$ which maps from an image patch and an orientation onto
value. This function takes as input a cropped version of $s$ centered on a
position $x$, $\mathrm{crop}(s,x)$, and an orientation, $\theta$, and outputs
the corresponding $Q$ value:
$Q_{2}(\mathrm{crop}(s,x),\theta)=Q(s,(x,\theta))$.

Inference is performed on the model by evaluating
$x^{*}=\operatorname*{arg\,max}_{x\in X}Q_{1}(s,x)$ first and then evaluating
$\theta^{*}=\operatorname*{arg\,max}_{\theta}Q_{2}(\mathrm{crop}(s,x^{*}),\theta)$.
Since each of these two models, $Q_{1}$ and $Q_{2}$, are significantly smaller
than $Q$ would be, the inference is much faster. Figure 1 shows an
illustration of this process. The top of the figure shows the action of
$Q_{1}$ while the bottom shows $Q_{2}$. Notice that the semantics of $Q_{2}$
imply that the $\theta$ depends only on $\mathrm{crop}(s,x)$, a local
neighborhood of $x$, rather than on the entire scene. This assumption is
generally true for grasping because grasp orientation typically depends only
on the object geometry near the target grasp point.

Figure 1: Illustration of the ASR representation. $Q_{1}$ selects the
translational component of an action, $Q_{2}$ selects the rotational
component.

##  4 Problem Statement

###  4.1 Planar grasp detection

The _planar grasp detection function_ $\Gamma:\mathbb{R}^{m\times h\times
w}\rightarrow\mathrm{SE}(2)$ maps from a top down image of a scene containing
graspable objects, $s\in S=\mathbb{R}^{m\times h\times w}$, to a planar
gripper pose, $a\in A=\mathrm{SE}(2)$, from which an object can be grasped.
This is similar to the formulations used by Mahler et al. (2017); Morrison et
al. (2018).

###  4.2 Formulation as a Contextual Bandit

We formulate grasp learning as a contextual bandit problem where the state is
an image $s\in S=\mathbb{R}^{m\times h\times w}$ and the action $a\in
A=\mathrm{SE}(2)$ is a grasp pose to which the robot hand will be moved and a
grasp will be attempted, expressed in the reference frame of the image. After
each grasp attempt, the agent receives a binary reward $R$ drawn from a
Bernoulli distribution with unknown probability $r(s,a)$. The true
$Q$-function denotes the expected reward of taking action $a$ from $s$. Since
$R$ is binary, we have that $Q(s,a)=r(s,a)$. This formulation of grasp
learning as a bandit problem is similar to that used by, e.g. Danielczuk et
al. (2020); Kalashnikov et al. (2018); Zeng et al. (2018).

###  4.3 Invariance Assumption

We assume that the (unknown) reward function $r(s,a)$ that denotes the
probability of a successful grasp is invariant to translations and rotations
$g\in\mathrm{SE}(2)$. Let $gs$ denote the image $s$ translated and rotated by
$g$. Similarly, let $ga$ denote the action translated and rotated by $g$.
Therefore, our assumption is:

$r(s,a)=r(gs,ga).$ (4)

Intuitively, when the image of a scene transforms, the grasp poses (located
with respect to the image) transform correspondingly.

##  5 SymGrasp: Symmetric Grasp Learning

###  5.1 Equivariant Learning

We use equivariant neural networks Weiler and Cesa (2019) to model the
$Q$-function that enforces the invariance assumption in Section 4.3. Therefore
once the $Q$ function is fit to a data sample $r(s,a)$, it generalizes to any
$g\in\mathrm{SE}(2)$ transformed data sample $r(gs,ga)$. This generalization
could lead to a significant sample efficiency improvement during training.

####  5.1.1 Invariance properties of $Q_{1}$ and $Q_{2}$

The assumption that the reward function $r$ is invariant to transformations
$g\in\mathrm{SE}(2)$ implies that the optimal $Q$-function is also invariant
to $g$, i.e., $Q(s,a)=Q(gs,ga)$. In the context of the augmented state
representation (ASR, see Section 3.2), this implies separate invariance
properties for $Q_{1}$ and $Q_{2}$:

$\displaystyle Q_{1}(gs,gx)$ $\displaystyle=Q_{1}(s,x)$ (5) $\displaystyle
Q_{2}(g_{\theta}(\mathrm{crop}(s,x)),g_{\theta}+\theta)$
$\displaystyle=Q_{2}(\mathrm{crop}(s,x),\theta),$ (6)

where $g_{\theta}\in\mathrm{SO}(2)$ denotes the rotational component of
$g\in\mathrm{SE}(2)$, $gx$ denotes the rotated and translated vector
$x\in\mathbb{R}^{2}$, and $g_{\theta}(\mathrm{crop}(s,x))$ denotes the cropped
image rotated by $g_{\theta}$.

####  5.1.2 Discrete Approximation of $\mathrm{SE}(2)$

We implement the invariance constraints of Equation 5 and 6 using a discrete
approximation to $\mathrm{SE}(2)$. We constrain the positional component of
the action to be a discrete pair of positive integers $x\in\\{1\dots
h\\}\times\\{1\dots w\\}\subset\mathbb{Z}^{2}$, corresponding to a pixel in
$s$, and constrain the rotational component of the action to be an element of
the finite cyclic group $C_{n}=\\{2\pi k/n:0\leq k<n,i\in\mathbb{Z}\\}$. This
discretized action space will be written
$\hat{\mathrm{SE}}(2)=\mathbb{Z}^{2}\times C_{n}$.

####  5.1.3 Equivariant $Q$-Learning with ASR

In $Q$-Learning with ASR, we model $Q_{1}$ and $Q_{2}$ as neural networks. We
model $Q_{1}$ as a fully convolutional UNet (Ronneberger et al., 2015)
$q_{1}:\mathbb{R}^{m\times h\times w}\to\mathbb{R}^{1\times h\times w}$ that
generates $Q$ value for each discretized translational actions from the input
state image. We model $Q_{2}$ as a standard convolutional network
$q_{2}:\mathbb{R}^{m\times h^{\prime}\times w^{\prime}}\to\mathbb{R}^{n}$ that
evaluates $Q$ value for $n$ discretized rotational actions based on the image
patch. The networks $q_{1}$ and $q_{2}$ thus model the functions $Q_{1}$ and
$Q_{2}$ by partially evaluating at the first argument and returning a function
in the second. As a result, the invariance properties of $Q_{1}$ and $Q_{2}$
(Equation 5 and 6) imply the equivairance of $q_{1}$ and $q_{2}$:

$\displaystyle q_{1}(gs)$ $\displaystyle=gq_{1}(s)$ (7) $\displaystyle
q_{2}(g_{\theta}\mathrm{crop}(s,x))$
$\displaystyle=\rho_{reg}(g_{\theta})q_{2}(\mathrm{crop}(s,x))$ (8)

where $g\in\hat{\mathrm{SE}}(2)$ acts on the output of $q_{1}$ through
rotating and translating the $Q$-map, and $g_{\theta}\in C_{n}$ acts on the
output of $q_{2}$ by performing a circular shift of the output $Q$ values via
the regular representation $\rho_{reg}$.

This is illustrated in Figure 2. In Figure 2a we take an example of a depth
image $s$ in the upper left corner. If we rotate and translate this image by
$g$ (lower left of Figure 2a) and then evaluate $q_{1}$, we arrive at
$q_{1}(gs)$. This corresponds to the LHS of Equation 7. However, because
$q_{1}$ is an equivariant function, we can calculate the same result by first
evaluating $q_{1}(s)$ and _then_ applying the transformation $g$ (RHS of
Equation 7). Figure 2b illustrates the same concept for Equation 8. Here, the
network takes the image patch $\mathrm{crop}(s,x)$ as input. If we rotate the
image patch by $g_{\theta}$ and then evaluate $q_{2}$, we obtain the LHS of
Equation 8, $q_{2}(g_{\theta}\mathrm{crop}(s,x))$. However, because $q_{2}$ is
equivariant, we can obtain the same result by evaluating
$q_{2}(\mathrm{crop}(s,x))$ and circular shifting the resulting vector to
denote the change in orientation by one group element.

(a) Illustration of Equation 7

(b) Illustration of Equation 8.

Figure 2: Equivariance relations expressed by Equation 7 and Equation 8.

####  5.1.4 Model Architecture of Equivariant $q_{1}$

As a fully convolutional network, $q_{1}$ inherits the translational
equivariance property of standard convolutional layers. The challenge is to
encode rotational equivariance so as to satisfy Equation 7. We accomplish this
using equivariant convolutional layers that satisfy the equivariance
constraint of Equation 2 where we assign
$\mathcal{F}_{in}=s\in{\mathbb{R}^{1\times m\times h\times w}}$ to encode the
input state $s$ and $\mathcal{F}_{out}\in\mathbb{R}^{1\times 1\times h\times
w}$ to encode the output $Q$-map. Both feature maps are associated with the
trivial representation $\rho_{0}$ such that the rotation $g$ operates on these
feature maps by rotating pixels without changing their values. We use the
regular representation $\rho_{reg}$ for the hidden layers of the network to
encode more comprehensive information in the intermediate layers. We found we
achieved the best results when we defined $q_{1}$ using the dihedral group
$D_{4}$ which expresses the group generated by rotations of multiples of
$\pi/2$ in combination with vertical reflections.

####  5.1.5 Model Architecture of Equivariant $q_{2}$

Whereas the equivariance constraint in Equation 7 is over
$\hat{\mathrm{SE}}(2)$, the constraint in Equation 8 is over $C_{n}$ only. We
implement Equation 8 using Equation 2 with an input of
$\mathcal{F}_{in}=\mathrm{crop}(s,x)\in\mathbb{R}^{1\times m\times
h^{\prime}\times w^{\prime}}$ as a trivial representation, and an output of
$\mathcal{F}_{out}\in\mathbb{R}^{n\times 1\times 1\times 1}$ as a regular
representation. $q_{2}$ is defined in terms of the group $C_{n}$, assuming the
rotations in the action space are defined to be multiples of $2\pi/n$.

####  5.1.6 $q_{2}$ Symmetry Expressed as a Quotient Group

It turns out that additional symmetries exist when the gripper has a bilateral
symmetry. In particular, it is often the case that rotating a grasp pose by
$\pi$ radians about its forward axis does not affect the probability of grasp
success, i.e.,$r$ is invariant to rotations of the action by $\pi$ radians.
When this symmetry is present, we can model it using the quotient group
$C_{n}/C_{2}\cong\\{2\pi k/n:0\leq k<n/2,k\in\mathbb{Z},0\equiv\pi\\}$ which
pairs orientations separated by $\pi$ radians into the same equivalence
classes.

###  5.2 Other Optimizations

While our use of equivariant models to encode the $Q$-function is responsible
for most of our gains in sample efficiency (Section 6.3), there are several
additional algorithmic details that, taken together, have a meaningful impact
on performance.

####  5.2.1 Loss Function

In the standard ASR loss function, given a one step reward $r(s,a)$, where
$a=(x,\theta)$, $Q_{1}$ and $Q_{2}$ have targets (Wang et al., 2020a):

$\displaystyle\mathcal{L}$ $\displaystyle=\mathcal{L}_{1}+\mathcal{L}_{2}$ (9)
$\displaystyle\mathcal{L}_{1}$
$\displaystyle=\tfrac{1}{2}(Q_{1}(s,x)-\max_{\theta^{\prime}}Q_{2}(\mathrm{crop}(s,x),\theta^{\prime}))^{2}$
(10) $\displaystyle\mathcal{L}_{2}$
$\displaystyle=\tfrac{1}{2}(Q_{2}(\mathrm{crop}(s,x),\theta)-r(s,a))^{2}.$
(11)

In $\mathcal{L}_{1}$ term, however, since the reward $r(s,a)$ is the ground
truth return of $Q_{2}(\mathrm{crop}(s,x),\theta)$, we correct $Q_{2}$ with
$r(s,a)$. Denoted $\hat{Q}_{2}$ as the corrected $Q_{2}$

$\hat{Q}_{2}(\mathrm{crop}(s,x),{\theta^{\prime}})=\begin{cases}Q_{2}(\mathrm{crop}(s,x),{\theta^{\prime}}),&\text{if
}{\theta^{\prime}}\neq\theta\\\ r(s,a),&\text{if
}{\theta^{\prime}}=\theta\end{cases}$ (12)

We then modify $\mathcal{L}_{1}$ to learn from $\hat{Q}_{2}$:

$\mathcal{L}_{1}^{\prime}=\frac{1}{2}\Big{(}Q_{1}(s,x)-\max_{\theta^{\prime}}\left[\hat{Q}_{2}(\mathrm{crop}(s,x),{\theta^{\prime}})\right]\Big{)}^{2}.$
(13)

In addition to the above, we add an off-policy loss term
$\mathcal{L}_{1}^{\prime\prime}$ that is evaluated with respect to an
additional $k$ grasp positions $\bar{X}\subset X$ sampled using a Boltzmann
distribution from $Q_{1}(s)$:

$\begin{split}\mathcal{L}_{1}^{\prime\prime}=&\frac{1}{2k}\sum_{x_{i}\in\bar{X}}\Big{(}Q_{1}(s,x_{i})-\\\
&\max_{\theta^{{}^{\prime}}}\left[Q_{2}(\mathrm{crop}(s,x_{i}),\theta^{{}^{\prime}})\right]\Big{)}^{2},\end{split}$
(14)

where $Q_{2}$ provide targets to train $Q_{1}$. This off-policy loss minimizes
the gap between $Q_{1}$ and $Q_{2}$. Our combined loss function is therefore
$\mathcal{L}=\mathcal{L}_{1}^{\prime}+\mathcal{L}_{1}^{\prime\prime}+\mathcal{L}_{2}$.

####  5.2.2 Prioritizing failure experiences in minibatch sampling

In the contextual bandit setting, we want to avoid the situation where the
agent repeats incorrect actions in a row. This can happen because some failure
grasps left the scene intact, thus the image and the $Q$-map are intact. We
address this problem by prioritizing failure experiences. When a grasp failed,
the experience is included in the sampled minibatch on the next SGD step Zeng
et al. (2018), thereby updating the $Q$-function prior to reevaluating it on
the next time step. The updates in $Q$ reduce the chance of selecting the same
(bad) action.

####  5.2.3 Boltzmann exploration

We find empirically Boltzmann exploration is better compares to
$\epsilon$-greedy exploration in our grasp setting. We use a temperature of
$\tau_{\text{training}}$ during training and a lower temperature of
$\tau_{\text{test}}$ during testing. Using a non-zero temperature at test time
helped reduce the chances of repeatedly sampling a bad action.

####  5.2.4 Data augmentation

Even though we are using equivariant neural networks to encode the
$Q$-function, it can still be helpful to perform data augmentation as well.
This is because the granularity of the rotation group encoded in $q_{1}$
($D_{4}$) is coarser than that of the action space ($C_{n}/C_{2}$). We address
this problem by augmenting the data with translations and rotations sampled
from $\hat{\mathrm{SE}}(2)$. For each experienced transition, we add eight
additional $\hat{\mathrm{SE}}(2)$-transformed images to the replay buffer.

####  5.2.5 Softmax at the output of $q_{1}$ and $q_{2}$

Since we are using a contextual bandit with binary rewards and the reward
function $r(s,a)$ denotes the parameter of a Bernoulli distribution at $s,a$,
we know that $Q_{1}$ and $Q_{2}$ must each take values between zero and one.
We encode this prior using an element-wise softmax layer at the output of each
of the $q_{1}$ and $q_{2}$ networks.

####  5.2.6 Selection of the $z$ coordinate

In order to execute a grasp, we must calculate a full $x,y,\theta,z$ goal
position for the gripper. Since our model only infers a planar grasp pose, we
must calculate a depth along the axis orthogonal to this plane (the $z$ axis)
using other means. In this paper, we calculate $z$ by taking the average depth
over a $5\times 5$ pixel region centered on the grasp point in the input depth
image. The commanded gripper height is set to an offset value from this
calculated height. While executing the motion to this height, we monitor force
feedback from the arm and halt the motion prematurely if a threshold is
exceeded.

###  5.3 Optimizations for transparent object grasping using RGB input

In order to grasp transparent objects using our model, we found it was helpful
to make a few small modifications to our model and setup. Most importantly, we
found it was essential to use RGB rather than depth-only image input to the
model.

Bin color: We found that for transparent objects, our system performed much
better using a black bin color rather than a white or transparent bin color.
Figure 6 illustrates this difference in setup. We believe that the performance
difference is due to the larger contrast between the transparent objects and
the bin in the RGB spectrum.

Dihedral group in $q2$: Another optimization we used in our transparent object
experiments was to implement Equation 8 using dihedral group $D_{n}$ which
expresses the group of multiples of $2\pi/n$ and reflections, rather than
$C_{n}$. As in Section 5.1.6, we use a quotient group that encodes gripper
symmetries. Here, this quotient group becomes $D_{n}/D_{2}$, which pairs
orientations separated by $\pi$ and reflected orientations into the same
equivalent class.

Collision penalty: In our experiments with transparent objects, we found that
collision was a more significant problem than it was with opaque objects. We
believe this was a result of the fact that since the transparent objects did
not completely register in the depth image, standard collision checking
between the object point cloud and the gripper did not suffice to prevent
collisions. Therefore, in our transparent object experiments, we penalized
successful grasps that produced collisions during grasping by awarding those
grasps only 0.8 reward instead of the full 1.0 reward.

##  6 Experiments in Simulation

###  6.1 Setup

(a) 86 GraspNet-1B objects used

(b) Pybullet simulation

(c) Depth image

(d) RGB image

Figure 3: (a) The 86 objects used in our simulation experiments are drawn from
the GraspNet-1Billion dataset Fang et al. (2020). (b) Phybullet simulation.
(c) and (d) are top-down depth and RGB images of the grasping scene.

####  6.1.1 Object Set

All simulation experiments are performed using objects drawn from the
GraspNet-1Billion dataset Fang et al. (2020). This includes 32 objects from
the YCB dataset Calli et al. (2017), 13 adversarial objects used in DexNet 2.0
Mahler et al. (2017), and 43 additional objects unique to GraspNet-1Billion
Fang et al. (2020) (a total of 88 objects). Out of these 88 objects, we
exclude two bowls because they can be stably placed in non-graspable
orientations, i.e., they can be placed upside down and cannot be grasped in
that orientation using standard grippers. Also, we scale these objects so that
they are graspable from any stable object configuration. Lastly, objects are
assigned with random RGB values drawn from uniform distribution
$U((0.6,0.6,0.6),(1,1,1))$. We refer to these 86 mesh models as our simulation
“object set”, shown in Figure 3a.

####  6.1.2 Simulation Details

Our experiments are performed in Pybullet (Coumans and Bai, 2016). The
environment includes a Kuka robot arm and a $0.3\text{m}\times 0.3\text{m}$
tray with inclined walls (Figure 3b). At the beginning of each episode, the
environment is initialized with $15$ objects drawn uniformly at random from
our object set and dropped into the tray from a height of $40$ cm so that they
fall into a random configuration. The state is a depth image (depth modality)
or RGB image (RGB modality) captured from a top-down camera (Figure 3c and d).
On each time step, the agent perceives a state and selects an action to
execute which specifies the planar pose to which to move the gripper. A grasp
is considered to have been successful if the robot is able to lift the object
more than $0.1$m above the table. The environment will be reinitialized when
all objects have been removed from the tray or $30$ grasp attempts have been
made.

###  6.2 Comparison Against Baselines on Depth Modality

####  6.2.1 Baseline Model Architectures

We compare our method against two different model architectures from the
literature: VPG Zeng et al. (2018) and FC-GQ-CNN (Satish et al., 2019). Each
model is evaluated alone and then with two different data augmentation
strategies (soft equ and RAD). In all cases, we use the contextual bandit
formulation described in Section 4.2. The baseline model architectures are:
VPG: Architecture used for grasping in (Zeng et al., 2018). This model is a
fully convolutional network (FCN) with a single-channel output. The $Q$ value
of different gripper orientations is evaluated by rotating the input image. We
ignore the pushing functionality of VPG. FC-GQ-CNN: Model architecture used in
(Satish et al., 2019). This is an FCN with $8$-channel output that associates
each grasp rotation to a channel of the output. During training, our model
uses Boltzmann exploration with a temperature of $\tau=0.01$ while the
baselines use $\epsilon$-greedy exploration starting with $\epsilon=50\%$ and
ending with $\epsilon=10\%$ over 500 grasps (this follows the original
implementation in Zeng et al. (2018)).

####  6.2.2 Data Augmentation Strategies

The data augmentation strategies are: $n\times$ RAD: The method from Laskin et
al. (2020a) that augments each sample in the mini-batch with respect to
Equation 4. Specifically, for each SGD step, we first draw $bs$ (where $bs$ is
the batch size) samples from the replay buffer. Then for each sample, we
augment both the observation and the action using a random $\mathrm{SE}(2)$
transformation, while the reward is unchanged. We perform $n$ SGD steps on the
RAD augmented mini-batch after each grasp sample. $n\times$ soft equ: similar
to $n\times$ RAD except that we produce a mini-batch by drawing $bs/n$
samples, randomly augment those samples $n$ times with respect to equation. 4,
then perform a single SGD step. Details can be found in Appendix C.

####  6.2.3 Results and Discussion

The learning curves of Figure 4 show the grasp success rate versus the number
of grasping attempts on depth modality. Figure 4a shows online learning
performance. Each data point is the average success rate over the last 150
grasps (therefore, the first data point occurs at 150). Figure 4b shows
testing performance by stopping training every 150 grasp attempts and
performing 1000 test grasps and reporting average performance over these 1000
test grasps. Our method tests at a lower test temperature of $\tau=0.002$
while the baselines test pure greedy behavior.

Generally, our proposed equivariant model convincingly outperforms the
baseline methods and data augmentation strategies with depth modality. In
particular, Figure 4b shows that the testing success rate of the equivariant
model after 150 grasp attempts has the same or better performance than all of
the baseline methods after 1500 grasp attempts. Notice that each of the two
data augmentation methods we consider (RAD and soft equ) has a positive effect
on the baseline methods. However, after training for the full 1500 grasp
attempts, our equivariant model converges to the highest grasp success rate
($93.9\pm 0.4\%$). Please see Appendix.D for a comparison with longer training
horizon.

(a) Training

(b) Testing

Figure 4: Comparison with baselines. All lines are an average of four runs.
Shading denotes standard error. (a) shows learning curves as a running average
over the last 150 training grasps. (b) shows the average near-greedy
performance of 1000 validation grasps performed every 150 training steps.

###  6.3 Ablation Study

There are three main parts of SymGrasp as described in this paper: 1) the use
of equivariant convolutional layers instead of standard convolution layers; 2)
the use of the augmented state representation (ASR) instead of a single
network; 3) the various optimizations described in Section 5.2. Here, we
evaluate the performance of the method when ablating each of these three parts
in the depth modality. For additional ablations, see Appendix D.

####  6.3.1 Baselines

In no equ, we replace all equivariant layers with standard convolutional
layers. In no ASR, we replace the equivariant $q_{1}$ and $q_{2}$ models
described in Section 3.2 by a single equivariant network. In no opt, we remove
the optimizations described in Section 5.2. In addition to the above, we also
evaluated rot equ which is the same as no ASR except that we replace ASR with
a U-netRonneberger et al. (2015) and apply $4\times$ RADLaskin et al. (2020a)
augmentation. Detailed network architectures can be found in Appendix A.

####  6.3.2 Results and Discussion

Figure 5a and b shows the results where they are reported exactly in the same
manner as in Section 6.2. no equ does worst, suggesting that our equivariant
model is critical. We can improve on this somewhat by adding data augmentation
(rot equ), but this sill underperforms significantly. The other ablations, no
ASR and no opt demonstrate that those parts of the method are also important.

###  6.4 Effect of background color

This experiment evaluates the effect of background color with RGB modality. We
compare training with different tray colors which have an offset from the mean
RGB value of the objects $(0.8,0.8,0.8)$. For example, in mean$-0.8$, the tray
color would be black $(0,0,0)$. We test four different colors from black
(mean$-0.8$) to white (mean$+0.2$).

####  6.4.1 Results and Discussion

Figure 5c and d show that the contrast between background and object color has
a big impact on grasp learning. In particular, the model has the worst
performance when the background color is the same as the mean of the object
color (mean). The model performs best when the background color has the most
significant contrast from the mean of the object color (mean $-0.8$).

(a) Training

(b) Testing

(c) Training

(d) Testing

Figure 5: Ablation study for depth observation. Lines are an average over $4$
runs. Shading denotes standard error. In the left column, learning curves as a
running average over the last 150 training grasps. In the right column is the
average near-greedy performance of 1000 validation grasps performed every 150
training steps. The first row is in the depth modality while the second row is
in the RGB modality.

(a) Opaque objects grasping

(b) Transparent objects grasping

Figure 6: Setup for self-supervised training on the robot.

##  7 Experiments in Hardware

###  7.1 Setup

####  7.1.1 Robot Environment

Our experimental platform is comprised of a Universal Robots UR5 manipulator
equipped with a Robotiq 2F-85 parallel-jaw gripper. For depth modality, we use
an Occipital Structure Sensor and for RGB modality we use a Kinect Azure
Sensor paired with two black trays. The dual-tray grasping environment is
shown in Figure 6. The workstation is equipped with an Intel Core i7-7800X CPU
and an NVIDIA GeForce GTX 1080 GPU.

####  7.1.2 Self-Supervised Training

The training begins with the 15 training objects (Figure 9a for opaque object
grasping and Figure 10a for transparent object grasping) being dropped into
one of the two trays by the human operator. Then, the robot attempts to pick
objects from one tray and drop them on another. The drop location is sampled
from a Gaussian distribution centered in the middle of the receiving tray. All
grasp attempts are generated by the contextual bandit. When all 15 objects
have been transported in this way, training switches to attempting to grasp
from the other tray and drop into the first. Whether all objects are
transported or not is decided by the heuristic. For opaque objects, we
threshold the depth sensor reading. For transparent objects, we compare the
RGB image of the current scene with an RGB image of an empty tray, similar to
Kalashnikov et al. (2018). During training, the robot performs 600 grasp
attempts in this way (that is 600 grasp _attempts_ , not 600 successful
grasps). A reward $r$ will be set to $1$ if the gripper was blocked by the
grasped object, otherwise $r=0$.

####  7.1.3 In-Motion Computation

We are able to nearly double the speed of robot training by doing all image
processing and model learning while the robotic arm was in motion. This was
implemented in Python as a producer-consumer process using mutexs. As a
result, our robot is constantly in motion during training and the training
speed for our equivariant algorithm is limited by the velocity of robot
motion. This improvement enabled us to increase robot training speed from
approximately 230 grasps per hour to roughly 400 grasps per hour.

####  7.1.4 Evaluation procedure

The evaluation begins with the human operator dropping objects into one tray,
after this, no human interference is allowed. We evaluate the success rate of
robot grasping objects. A key failure mode during testing is repeated failure
grasps. To combat this, we use the procedure of Zeng et al. (2018) to reduce
the chances of repeated grasp failures. The procedure is that after a grasp
failure, we perform multiple SGD steps using that experience to “discourage”
the model from selecting the same action and then use that updated model for
the subsequent grasp. After a successful grasp, we discard these updates and
reload the original network.

All runs are evaluated by freezing the corresponding model and executing 100
greedy (or near greedy) test grasps for each object set in the easy-object
test set (Figure 9b), the hard-object test set (Figure 9c).

####  7.1.5 Model Details

For all methods, prior to training on the robot, model weights are initialized
randomly using an independent seed. No experiences from simulation are used,
i.e.,we train from scratch. For our depth algorithm, the $q_{1}$ network is
defined using $D_{4}$-equivariant layers and the $q_{2}$ network is defined
using $C_{16}/C_{2}$-equivariant layers. For ours RGB algorithm, the $q_{2}$
network is defined using $D_{16}/D_{2}$-equivariant layers. During training,
we use Boltzmann exploration with a temperature of $0.01$. During testing, the
temperature is reduced to $0.002$ (near-greedy). For more details, see
Appendix G.

Baseline |  test set easy |  test set hard |  $t_{\textit{SGD}}$
---|---|---|---
$8\times$ RAD VPG |  61.8 $\pm$3.59 |  52.5 $\pm$5.33 |  12.1s
$8\times$ RAD FC-GQ-CNN |  82.8 $\pm$1.65 |  74.3 $\pm$3.04 |  1.55s
Ours |  95.0 $\pm$1.47 |  87.0 $\pm$1.87 |  1.11s
Table 1: Evaluation success rate ($\%$), standard error, and training time per grasp $t_{\textit{SGD}}$ (in seconds) in the hardware experiments. Results are an average of 100 grasps per training run averaged over four runs, performed on the held out test objects shown in Figure 9b and c. Baseline |  trans. train set |  trans. test set |  $t_{\text{capture}}$
---|---|---|---
Ours RGB |  85.3 $\pm$8.62 |  83.3 $\pm$3.20 |  1.8s
Table 2: Evaluation success rate ($\%$), standard error, and capture time per
grasp (in seconds) in the hardware experiments of transparent objects. Results
are an average of 100 grasps per training run averaged over four runs,
performed on the train and held out test objects shown in Figure 10 a and b.

###  7.2 Opaque object grasping

####  7.2.1 Objects

For opaque objects grasping, all training happens using the 15 objects shown
in Figure 9a. After training, we evaluate grasp performance on both the “easy”
test objects (Figure 9b) and the “hard” test objects (Figure 9c). Note that
both test sets are novel with respect to the training set.

####  7.2.2 Baselines

In our robot experiments for opaque object grasping, we compare our method
against $8\times$ RAD VPG (Zeng et al., 2018) Laskin et al. (2020a) and
$8\times$ RAD FC-GQ-CNN (Satish et al., 2019) Laskin et al. (2020a), the two
baselines we found to perform best in simulation. As before, $8\times$ RAD
VPG, uses a fully convolutional network (FCN) with a single output channel.
The $Q$-map for each gripper orientation is calculated by rotating the input
image. After each grasp, we perform $8\times$ RAD data augmentation (8
optimization steps with a mini-batch containing randomly translated and
rotated image data). $8\times$ RAD FC-GQ-CNN also has an FCN backbone, but
with eight output channels corresponding to each gripper orientation. It uses
$8\times$ RAD data augmentation as well. All exploration is the same as it was
in simulation except that the $\epsilon$-greedy schedule goes from $50\%$ to
$10\%$ over 200 steps rather than over 500 steps.

####  7.2.3 Results and Discussion

(a) Opaque object grasping

(b) Transparent object grasping

Figure 7: Learning curves for (a) the opaque object grasping and (b) the
transparent object grasping hardware experiment, the parenthesis indicates the
color of trays. All curves are averaged over $4$ runs with different random
seeds and random object placement. Each data point in the curve is the average
grasp success over the last 60 grasp attempts. Shading denotes standard error.

(a) Depth observation

(b) RGB observation

(c) $Q_{1}$ from RGB observation

(d) Execute the grasp

Figure 8: Illustrations for transparent object grasping. (a) shows the depth
observation, notice that most transparent objects are not sensed. (b) shows
the same scene with RGB modality, notice that objects are sensed up to
orthographic projection error. (c) shows the $Q_{1}$ map from the observation
in (b) and the selected action in the red dot. (d) shows the executed grasp.

Figure 7a shows the learning curves on opaque object grasping for the three
methods during learning. An important observation is that the results from
training on the physical robot Figure 7a match the simulation training results
Figure 4a. Figure 7a shows for opaque object grasping, our method achieves a
success rate of $>90\%$ after 600 grasp attempts while the baselines are near
$70\%$. Table 1 shows the testing performance and also demonstrates our method
significantly outperforms the baselines during testing. However, performance
is lower on the “hard” test set. We hypothesize that the lower “hard” set
performance is due to a lack of sufficient diversity in the training set. The
other observation is that since each of these 600-grasp training runs takes
approximately 1.5 hours, suggests these methods could efficiently directly
learn from real-world data thus avoiding the simulation-to-real-world gap and
adapting to physical changes in the robot.

###  7.3 Transparent object grasping

####  7.3.1 Objects

For transparent object grasping, all training happens using the 15 objects
shown in Figure 10a. After training, we evaluate grasp performance on the in-
distribution training set and out-of-distribution testing objects (Figure 10a,
b). Note that the test set is novel with respect to the training set.

####  7.3.2 RGB Modality Details

The top-down RGB images are orthographic projections of noisy RGB point
clouds. RGB values are normalized to fit a Gaussian distribution
$\mathcal{N}(0,0.01)$. During training, each mini-batch of images is augmented
in brightness with a $(0.9,1.1)$ range.

###  7.4 Baselines

For transparent object grasping, we compare Ours RGB (black), Ours RGB
(white), and Ours D. Ours RGB (black) is the baseline in our proposed methods,
Ours RGB (white) changes black trays to white trays, see figure 18. Ours D is
the best baseline in physical opaque objects grasping experiments, here we
train and test on transparent objects.

###  7.5 Results and Discussion

Figure 7(b) shows the learning curves for transparent object grasping and
Table 2 shows the grasping performance of ours RGB (black) on the transparent
object set. Figure 8 shows a transparent object grasping process during
evaluation.

Figure 7b shows that for transparent object grasping, ours RGB (black)
produces a significant ($>40\%$) improvement in success rate compared to ours
depth, suggesting that the RGB information is needed to perform well in this
setting. The background color is also important: changing the tray from black
to white leads to a $>30\%$ decrement in success rate. Table 2 summarizes that
ours RGB achieves $>80\%$ success rate on both the training set and testing
set, indicating the method learns a good grasp function of in-distribution
object set and generalizes to novel transparent objects. Also, notice that
from a computational perspective, our method is significantly cheaper ($1.8$
seconds) than NeRF based approaches to transparent object grasping like EVO
Nerf Kerr et al.  and Dex Nerf Ichnowski et al. (2021) which take $9.5$ second
and $16$ seconds, respectively.

(a) Training set

(b) Testing set, easy

(c) Testing set, hard

Figure 9: Object sets used for training and testing. Both training and the
test set easy include 15 objects while the test set hard has 20 objects.
Objects were curated so that they were graspable by the Robotiq 2F-85 parallel
jaw gripper from any configuration and visible to the Occipital Structure
Sensor.

(a) Trans. training set

(b) Trans. testing set

Figure 10: Transparent object sets used for training and testing. The Training
set includes 15 objects and the testing set includes 10 objects. Objects were
curated so that they were graspable by the Robotiq 2F-85 parallel jaw gripper
from any configuration.

##  8 Conclusions and Limitations

Our main contribution is to recognize that planar grasp detection is
$\mathrm{SE}(2)$-equivariant and to leverage this structure using an
$\mathrm{SO}(2)$-equivariant model architecture. This model is significantly
more sample efficient than other grasp-learning methods and can learn a good
grasp function with less than $600$ grasp samples. This increase in sample
efficiency enables us to learn to grasp on a physical robotic system in a
practical amount of time (approximately and hour and a half) without relying
on any sort of pretraining or semi-supervised learning. Training completely on
physical systems is important for two main reasons. First, it eliminates the
need to train in simulation, thereby avoiding the sim2real gap. Second, it
enables our system to adapt to changes or idiosyncrasies of the robot hardware
or the physical environment that would be hard to simulate.

A key limitation in both simulation and the real world is that despite the
fast learning rate, grasp success rates (after training) still seems to be
limited to the low/mid $90\%$ for opaque object and $80\%$ for transparent
objects. This is the same success rate seen in with other grasp detection
methods Mahler et al. (2017); ten Pas et al. (2017b); Mousavian et al. (2019),
but it is disappointing here because one might expect faster adaptation to
lead ultimately to better grasp performance. This could simply be an
indication of the complexity of the grasp function to be learned or it could
be a result of stochasticity in the simulator and on the real robot. Another
limitation of our work is that it is limited to an open-loop control setting
where the model infers only a goal pose. Nevertheless, we expect that the
equivariant model structure leverage in this paper would also be useful in the
closed loop setting, as suggested by the results of Wang et al. (2022).

##  9 Declarations

Ethical Approval Not applicable.

Competing interests The authors declare that they have no relevant financial
or non-financial interests to disclose

Authors’ contributions Xupeng Zhu proposed the method in discussions with Dian
Wang. Xupeng Zhu and Guanang Su conducted the robot experiment. All authors
wrote and reviewed the manuscript.

Funding This work was supported in part by NSF 1724257, NSF 1724191, NSF
1763878, NSF 1750649, NASA 80NSSC19K1474, the Roux Institute, the Harold
Alfond Foundation, NSF grants 2107256, and 2134178.

Availability of data and materials Code is available at
https://github.com/ZXP-S-works/SE2-equivariant-grasp-learning.

## References

  * Berscheid et al. [2021] Lars Berscheid, Christian Friedrich, and Torsten Kröger.  Robot learning of 6 dof grasping using model-based adaptive primitives.  _arXiv preprint arXiv:2103.12810_ , 2021.
  * Breyer et al. [2021] Michel Breyer, Jen Jen Chung, Lionel Ott, Roland Siegwart, and Juan Nieto.  Volumetric grasping network: Real-time 6 dof grasp detection in clutter.  _arXiv preprint arXiv:2101.01132_ , 2021.
  * Calli et al. [2017] Berk Calli, Arjun Singh, James Bruce, Aaron Walsman, Kurt Konolige, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  Yale-cmu-berkeley dataset for robotic manipulation research.  _The International Journal of Robotics Research_ , 36(3):261–268, 2017.  10.1177/0278364917700714.  URL https://doi.org/10.1177/0278364917700714.
  * Cohen and Welling [2016a] Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In _International conference on machine learning_ , pages 2990–2999. PMLR, 2016a.
  * Cohen et al. [2018] Taco Cohen, Mario Geiger, and Maurice Weiler.  A general theory of equivariant cnns on homogeneous spaces.  _arXiv preprint arXiv:1811.02017_ , 2018.
  * Cohen and Welling [2016b] Taco S Cohen and Max Welling.  Steerable cnns.  _arXiv preprint arXiv:1612.08498_ , 2016b.
  * Coumans and Bai [2016] Erwin Coumans and Yunfei Bai.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  _GitHub repository_ , 2016.
  * Danielczuk et al. [2020] Michael Danielczuk, Ashwin Balakrishna, Daniel S Brown, Shivin Devgon, and Ken Goldberg.  Exploratory grasping: Asymptotically optimal algorithms for grasping challenging polyhedral objects.  _arXiv preprint arXiv:2011.05632_ , 2020.
  * Depierre et al. [2018] Amaury Depierre, Emmanuel Dellandréa, and Liming Chen.  Jacquard: A large scale dataset for robotic grasp detection.  _CoRR_ , abs/1803.11469, 2018.  URL http://arxiv.org/abs/1803.11469.
  * Fang et al. [2020] Hao-Shu Fang, Chenxi Wang, Minghao Gou, and Cewu Lu.  Graspnet-1billion: A large-scale benchmark for general object grasping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11444–11453, 2020.
  * Gao et al. [2020] Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid.  Vectornet: Encoding hd maps and agent dynamics from vectorized representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11525–11533, 2020.
  * He et al. [2015] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition, 2015.
  * Huang et al. [2022] Haojie Huang, Dian Wang, Xupeng Zhu, Robin Walters, and Robert Platt.  Edge grasp network: A graph-based se (3)-invariant approach to grasp detection.  _arXiv preprint arXiv:2211.00191_ , 2022.
  * Ichnowski et al. [2021] Jeffrey Ichnowski, Yahav Avigal, Justin Kerr, and Ken Goldberg.  Dex-nerf: Using a neural radiance field to grasp transparent objects.  _arXiv preprint arXiv:2110.14217_ , 2021.
  * James et al. [2019] Stephen James, Paul Wohlhart, Mrinal Kalakrishnan, Dmitry Kalashnikov, Alex Irpan, Julian Ibarz, Sergey Levine, Raia Hadsell, and Konstantinos Bousmalis.  Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12627–12637, 2019.
  * Jiang et al. [2021] Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu.  Synergies between affordance and geometry: 6-dof grasp detection via implicit representations.  _arXiv preprint arXiv:2104.01542_ , 2021.
  * Kalashnikov et al. [2018] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine.  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  _CoRR_ , abs/1806.10293, 2018.  URL http://arxiv.org/abs/1806.10293.
  * [18] Justin Kerr, Letian Fu, Huang Huang, Yahav Avigal, Matthew Tancik, Jeffrey Ichnowski, Angjoo Kanazawa, and Ken Goldberg.  Evo-nerf: Evolving nerf for sequential robot grasping of transparent objects.  In _6th Annual Conference on Robot Learning_.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2015.
  * Kofinas et al. [2021] Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves.  Roto-translated local coordinate frames for interacting dynamical systems.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Kostrikov et al. [2020] Ilya Kostrikov, Denis Yarats, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  _CoRR_ , abs/2004.13649, 2020.  URL https://arxiv.org/abs/2004.13649.
  * Kumra et al. [2019] Sulabh Kumra, Shirin Joshi, and Ferat Sahin.  Antipodal robotic grasping using generative residual convolutional neural network.  _CoRR_ , abs/1909.04810, 2019.  URL http://arxiv.org/abs/1909.04810.
  * Laskin et al. [2020a] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas.  Reinforcement learning with augmented data.  _CoRR_ , abs/2004.14990, 2020a.  URL https://arxiv.org/abs/2004.14990.
  * Laskin et al. [2020b] Michael Laskin, Aravind Srinivas, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning.  In _International Conference on Machine Learning_ , pages 5639–5650. PMLR, 2020b.
  * Levine et al. [2018] Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection.  _The International Journal of Robotics Research_ , 37(4-5):421–436, 2018.
  * Mahler et al. [2017] Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, and Ken Goldberg.  Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics.  _arXiv preprint arXiv:1703.09312_ , 2017.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021\.
  * Mondal et al. [2020] Arnab Kumar Mondal, Pratheeksha Nair, and Kaleem Siddiqi.  Group equivariant deep reinforcement learning.  _arXiv preprint arXiv:2007.03437_ , 2020.
  * Morrison et al. [2018] Douglas Morrison, Peter Corke, and Jürgen Leitner.  Closing the loop for robotic grasping: A real-time, generative grasp synthesis approach.  _arXiv preprint arXiv:1804.05172_ , 2018.
  * Mousavian et al. [2019] Arsalan Mousavian, Clemens Eppner, and Dieter Fox.  6-dof graspnet: Variational grasp generation for object manipulation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , October 2019.
  * Oord et al. [2018] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.
  * Pinto and Gupta [2015] Lerrel Pinto and Abhinav Gupta.  Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours.  _CoRR_ , abs/1509.06825, 2015.  URL http://arxiv.org/abs/1509.06825.
  * [34] Robert Platt.  Grasp learning: Models, methods, and performance.  _Annual Review of Control, Robotics, and Autonomous Systems_ , 6.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  _CoRR_ , abs/1505.04597, 2015.  URL http://arxiv.org/abs/1505.04597.
  * Sajjan et al. [2020] Shreeyak Sajjan, Matthew Moore, Mike Pan, Ganesh Nagaraja, Johnny Lee, Andy Zeng, and Shuran Song.  Clear grasp: 3d shape estimation of transparent objects for manipulation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 3634–3642. IEEE, 2020.
  * Satish et al. [2019] Vishal Satish, Jeffrey Mahler, and Ken Goldberg.  On-policy dataset synthesis for learning robot grasping policies using fully convolutional deep networks.  _IEEE Robotics and Automation Letters_ , 4(2):1357–1364, 2019.  10.1109/LRA.2019.2895878.
  * Sharma et al. [2017] Sahil Sharma, Aravind Suresh, Rahul Ramesh, and Balaraman Ravindran.  Learning to factor policies and action-value functions: Factored action space representations for deep reinforcement learning.  _arXiv preprint arXiv:1705.07269_ , 2017.
  * Song et al. [2020] Shuran Song, Andy Zeng, Johnny Lee, and Thomas Funkhouser.  Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations.  _Robotics and Automation Letters_ , 2020.
  * Sundermeyer et al. [2021] Martin Sundermeyer, Arsalan Mousavian, Rudolph Triebel, and Dieter Fox.  Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes.  _arXiv preprint arXiv:2103.14127_ , 2021.
  * ten Pas et al. [2017a] Andreas ten Pas, Marcus Gualtieri, Kate Saenko, and Robert Platt Jr.  Grasp pose detection in point clouds.  _CoRR_ , abs/1706.09911, 2017a.  URL http://arxiv.org/abs/1706.09911.
  * ten Pas et al. [2017b] Andreas ten Pas, Marcus Gualtieri, Kate Saenko, and Robert Platt.  Grasp pose detection in point clouds.  _The International Journal of Robotics Research_ , 36(13-14):1455–1473, 2017b.
  * van der Pol et al. [2020] Elise van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling.  Mdp homomorphic networks: Group symmetries in reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Walters et al. [2020] Robin Walters, Jinxi Li, and Rose Yu.  Trajectory prediction using equivariant continuous convolution.  _arXiv preprint arXiv:2010.11344_ , 2020.
  * Wang et al. [2020a] Dian Wang, Colin Kohler, and Robert Platt Jr.  Policy learning in SE(3) action spaces.  _CoRR_ , abs/2010.02798, 2020a.  URL https://arxiv.org/abs/2010.02798.
  * Wang et al. [2021] Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt.  Equivariant $q$ learning in spatial action spaces.  In _5th Annual Conference on Robot Learning_ , 2021.  URL https://openreview.net/forum?id=IScz42A3iCI.
  * Wang et al. [2022] Dian Wang, Robin Walters, and Robert Platt.  So(2)-equivariant reinforcement learning.  In _Int’l Conference on Representation Learning_ , 2022.  URL https://openreview.net/forum?id=7F9cOhdvfk_.
  * Wang et al. [2020b] Rui Wang, Robin Walters, and Rose Yu.  Incorporating symmetry into deep dynamics models for improved generalization.  _arXiv preprint arXiv:2002.03061_ , 2020b.
  * Weiler and Cesa [2019] Maurice Weiler and Gabriele Cesa.  General e(2)-equivariant steerable cnns.  _CoRR_ , abs/1911.08251, 2019.  URL http://arxiv.org/abs/1911.08251.
  * Weng et al. [2020] Thomas Weng, Amith Pallankize, Yimin Tang, Oliver Kroemer, and David Held.  Multi-modal transfer learning for grasping transparent and specular objects.  _IEEE Robotics and Automation Letters_ , 5(3):3791–3798, 2020.
  * Zeng et al. [2018] Andy Zeng, Shuran Song, Stefan Welker, Johnny Lee, Alberto Rodriguez, and Thomas A. Funkhouser.  Learning synergies between pushing and grasping with self-supervised deep reinforcement learning.  _CoRR_ , abs/1803.09956, 2018.  URL http://arxiv.org/abs/1803.09956.
  * Zhan et al. [2020] Albert Zhan, Philip Zhao, Lerrel Pinto, Pieter Abbeel, and Michael Laskin.  A framework for efficient robotic manipulation.  _CoRR_ , abs/2012.07975, 2020.  URL https://arxiv.org/abs/2012.07975.
  * Zhou et al. [2018] Xinwen Zhou, Xuguang Lan, Hanbo Zhang, Zhiqiang Tian, Yang Zhang, and Narming Zheng.  Fully convolutional grasp detection network with oriented anchor box.  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7223–7230. IEEE, 2018.

##  Appendix A Neural network architecture

Our network architecture is shown in Figure 12a. The $q1$ network is a fully
convolutional UNet Ronneberger et al. [2015]. The $q2$ network is a residual
neural network He et al. [2015]. These networks are implemented using PyTorch
Paszke et al. [2019], and the equivariant networks are implemented using the
E2CNN library Weiler and Cesa [2019]. Adam optimizer Kingma and Ba [2015] is
used for the SGD step. The ablation no opt has the same architecture as above.
The ablation no asr (Figure 12b) ablated the $q2$ network and is defined with
respect to group $C16$. The ablation no equ (Figure 12c) has a similar network
architecture as ours with approximately the same number of free weights.
However, the equivariant network is replaced with an FCN. The ablation rot equ
(Figure 12d) has a similar network architecture as no asr method with
approximately the same number of free weights. However, the equivariant
network is replaced with an FCN.

##  Appendix B Examples of Equivariant Layer

Figure.11 shows an example of an equivariant layer with $C4$ group that maps
trivial representation to regular representation. In Figure.11. (a) the
equivariant layer will initialize a convolutional filter $f$, then rotates the
filter by each group element to obtain $f,gf,g^{2}f,g^{3}f$. In Figure.11. (b)
since each filter is a rotated copy of $f$ and is correspond to a fiber
channel in the regular representation, rotating image $s$ by $g$ leads to
rotating the $channel_{0}$ spatially by $g$ and shifting the $channel_{0}$
along the fiber channel by 1. There could be more layers following this layer,
as denoted by arrows and ellipsis. When each layer is equivariant, the entire
neural network is equivariant.

(a) With original input

(b) With rotated input

Figure 11: An example of $C4$ equivariant layer that maps 3 channels of
trivial representation to 1 channel of regular representation. $s$: the 3
trivial channel RGB image. $f$: the $3$ convolutional filter. $g\in C4$: the
group action that rotates the signal 90 degrees. $channel_{0}^{i}$: one
regular channel in the neural network with $i=0,1,2,3$ fiber channels for each
group element in $C4$. $*$ stands for standard convolution in CNN. Rotating
image $s$ by $g$ leads to rotating the $channel_{0}$ spatially by $g$ and
shifting the $channel_{0}$ along fiber channel by 1

##  Appendix C Augmentation baseline choices

The data augmentation strategies are: $n\times$ RAD: The method from Laskin et
al. [2020a] where we perform $n$ SGD steps after each grasp sample, where each
SGD step is taken over a mini-batch $bs$ of samples that have been randomly
translated and rotated by $g\in\mathrm{SE}(2)$ according to equation. 4.
Specifically, applying random $\mathrm{SE}(2)$ transformation $g$ for both
observation and action: $(gs,ga,r)$. $n\times$ soft equ: a data augmentation
method Wang et al. [2021] that performs $n$ soft equivariant SGD steps per
grasp, where each SGD step is taken over $n$ times randomly
$\hat{\mathrm{SE}}(2)$ augmented mini-batch. Specifically, we sample $bs/n$
samples ($bs$ is the batch size), augment it $n$ times, and train on this
mini-batch. We perform these SGD steps $n$ times so that $bs$ transitions are
sampled. This augmentation aims at achieving equivariance in the mini-batch.

We apply $n\times$ RAD and $n\times$ soft equ data augmentation to both VPG
and FC-GQ-CNN baselines, with $n={2,4,8}$. The best data augmentation
parameters $n$ are chosen for each baseline in the comparison in Figure 4.

(a) ours and no opt

(b) no asr

(c) no equ

(d) Rot equ

Figure 12: The neural network architecture for ours and ablations. R means
regular representation, T means trivial representation, and Q means quotient
representation.

Figure 13: Action space constraint for action selection. (a) The test set easy
cluttered scene. (b) The state $s$. (c) The action space
$x_{\text{positive}}$, overlays the binary mask $x_{\text{positive}}$ with the
state $s$ for visualization. (d) The $Q$-values within the action space. (e)
Selecting an action. (f) Executing a grasp.

##  Appendix D Additional experiments

To investigate the sample efficiency of SymGrasp, we compare our method with
the other two best methods in Figure.4. We train all the methods with $20$k
grasps to illustrate the sample efficiency. As shown in Figure.14, our method
not only learns faster but also converges to a better success rate, compared
with the other two baselines.

(a) Training

(b) Testing

Figure 14: Additional comparison with baselines for Section.6.2 with depth
modality. Lines are an average over $4$ runs. Shading denotes standard error.
(a) learning curves as a running average over the last 150 training grasps.
(b) average near-greedy performance of 1000 validation grasps performed every
150 training steps.

We then ablate each components in Section.5.2. Baselines are: In ASR loss,
both $q_{1}$ and $q_{2}$ minimize $l2$ loss between prediction and the reward
$r$. In No prioritizing, the mini-batch is uniformly sampled from the replay
buffer. In e greedy, we replace Boltzmann exploration with e greedy
exploration that linearly anneals from 0.5 to 0.1 in 500 grasps. In No data
aug, no data augmentation is performed. In No softmax the pixel-wise softmax
in the last layer of $q_{1}$ and $q_{2}$ is removed.

Figure 15 a, b, shows the learning results. Figure.15(b) shows that ASR loss
affects the performance the most, indicating the importance of the loss
function in Section.5.2.1. Other components No data aug, e greedy slightly
reduces the performance indicating the marginal benefits of data augmentation
and Boltzmann exploration. Lastly, No softmax performs badly at the $600$
grasps and No prioritizing performs badly at the $1500$ grasps, these suggest
softmax and prioritizing failure grasp stabilizes learning.

(a) Training

(b) Testing

Figure 15: Additional ablation study for Section.5.2 with depth modality.
Lines are an average over $4$ runs. Shading denotes standard error. (a)
learning curves as a running average over the last 150 training grasps. (b)
average near-greedy performance of 1000 validation grasps performed every 150
training steps.

We then ablate each component in Section.5.3. Baselines are: In Cyclic group,
we replace $D16$ group with $C16$ group in $q2$. In No collision penalty, we
replace collision penalty by using binary grasp success reward.

(a) Training

(b) Testing

Figure 16: Additional ablation study for Section.5.3 with depth modality.
Lines are an average over $4$ runs. Shading denotes standard error. (a)
learning curves as a running average over the last 150 training grasps. (b)
average near-greedy performance of 1000 validation grasps performed every 150
training steps.

Figure 16 a, b, shows the learning results. Cyclic group shows Cyclic group in
$q_{2}$ leads to slower learning before 300 grasp, compares to Dihedral group
in $q_{2}$; No collision penalty shows collision penalty encourages quicker
and better convergence.

Figure 17: The definition of $\theta$. $x$ is the $x$-axle of the workspace
while $\vec{n}$ is the normal of the gripper.

Figure 18: Transparent object grasping scenes. First row: using black trays.
Second row: using white trays.

##  Appendix E Action space details

The action $\theta$ is defined as the angle between the normal vector
$\vec{n}$ of the gripper and the $x$-axle, see Figure 17.

While the grasping height is selected by heuristic, the grasping aperture is
fixed with the maximum aperture of the gripper.

To prevent the grasps in the empty space where there is no object in $s$, we
constrain the action space to $x_{\text{positive}}\in X$ to exclude the empty
space, see Figure 13c. The constrain $x_{\text{positive}}$ is achieved by
first thresholding the depth image:
$s_{\text{positive}}=s>s_{\text{threshold}}$ ($s_{\text{threshold}}$ is 0.5cm
in simulation and 1.5cm in hardware), then dilating this binary map
$s_{\text{positive}}$ by radius $d_{\text{dilation}}=4$ pixels. The parameter
$s_{\text{threshold}}$ is selected according to sensor noise where
$d_{\text{dilation}}$ is related to the half of the gripper aperture.
Moreover, we constrain the action space within the tray to prevent collisions.

##  Appendix F Success and failure modes

We list typical success and failure modes to evaluate the performance of
SymGrasp.

For success modes, the trained policy of SymGrasp showcases its capability. At
the densely cluttered scene, SymGrasp prefers to grasp the relatively isolated
part of the objects, see Figure 19a, b. At the scene where the objects are
close to each other, SymGrasp can find the grasp pose that is collision free
with other objects, see Figure 19c, d.

For failure modes, we identify several typical scenarios: Bad grasp pose,
either wrong translation or orientation grasp pose (Figure 20a, b, and e)
indicates that there is a clear gap between our method and optimal policy.
This might be caused by the biased dataset collected by the algorithm.
Reasonable grasps failure (Figure 20d, f) means that the agent selects a
reasonable grasp, but it fails due to the stochasticity of the real world,
i.e., limited sensor resolution, contact dynamics, hardware calibration error,
etc. Challenging scenes (Figure 20c, g) is the nature of densely cluttered
objects, it can be alleviated by learning an optimal policy or synthesize
higher DoFs grasps, e.x., predicting grasping high instead of using heuristic.
The sensor distortion (Figure 20h) is caused by an low quality sensor or
occlusion. Among all failure modes, wrong action selection takes the most part
(65% failure in the test set easy and 33% failure in the test set hard). It is
followed by reasonable grasps, challenging scenes, and then sensor distortion.

Figure 19: Success modes in the test set easy, which has 15 hold out objects.

(a) Wrong $q_{1}$ (10/20) (b) Wrong $q_{2}$ (3/20) (c) Challenging scenes
(3/20) (d) Reasonable grasps (3/20) (e) Wrong $q_{1}$ (17/52) (f) Reasonable
grasps (13/52) (g) Challenging scenes (11/52) (h) Sensor distortion (8/52)

Figure 20: Failure modes. The brackets show the failure times divided by the
total number of failures in all four runs. The first row is test in the test
set easy while the second row is test in the test set hard.

##  Appendix G Evaluation details in hardware

The evaluation policy and environment are different from that of training in
the following aspects. First, for all methods, the robot arm moves slower than
that during training in the environment. This helps form stable grasps.
Second, for our method, the evaluation policy uses a lower temperature
($\tau_{\text{test}}=0.002$) than training. After a failure grasp, ours
performs $2$ SGD steps on this failure experience. The network weight will be
reloaded after recovery from the failure Zeng et al. [2018]. For the
baselines, the evaluation policy uses a greedy policy. After a failure grasp,
baselines perform $8$ RAD SGD steps on this failure experience. The network
weight will be reloaded after recovery from the failure Zeng et al. [2018].