# Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for
End-to-End Visual Robotic Manipulation Learning

Hyunwoo Ryu1, Jeong-Hoon Lee2, Hong-in Lee1, and Jongeun Choi3 1H. Ryu and H.
Lee are with the Department of Artificial Intelligence, Yonsei University,
Seoul, Republic of Korea. {tomato1mule<EMAIL_ADDRESS>Lee is
with the School of Mechanical Engineering, Yonsei University, Seoul, Republic
of Korea<EMAIL_ADDRESS>Choi is with the School of Mechanical
Engineering and the Department of Artificial Intelligence, Yonsei University,
Seoul, Republic of Korea<EMAIL_ADDRESS>
###### Abstract

End-to-end learning for visual robotic manipulation is known to suffer from
sample inefficiency, requiring a large number of demonstrations. The spatial
roto-translation equivariance, or the $SE(3)$-equivariance can be exploited to
improve the sample efficiency for learning robotic manipulation. In this
paper, we present fully end-to-end SE(3)-equivariant models for visual robotic
manipulation from a point cloud input. By utilizing the representation theory
of the Lie group, we construct novel SE(3)-equivariant energy-based models
that allow highly sample efficient end-to-end learning. We show that our
models can learn from scratch without prior knowledge yet is highly sample
efficient (${\sim}10$ demonstrations are enough). Furthermore, we show that
the trained models can generalize to tasks with (i) previously unseen target
object poses, (ii) previously unseen target object instances of the category,
and (iii) previously unseen visual distractors. We experiment with 6-DoF
robotic manipulation tasks to validate our models’ sample efficiency and
generalizability. Codes are available at: https://github.com/tomato1mule/edf

Figure 1: Given a small amount (${\sim}10$) of demonstrations, EDFs can learn
to solve highly spatial pick-and-place tasks. The left side of the figure
illustrates an example task demonstration where a mug (red) is picked and
placed on a hanger (green). Note that EDFs work in a fully end-to-end manner
without resorting to auxiliary algorithms such as pose estimation,
segmentation, or self-supervised learning pipelines. We also show that the
trained EDFs can generalize to A) unseen out-of-distribution poses of the
target objects, B) unseen target object instances in the category, and C) the
existence of unseen visual distractors.

##  I INTRODUCTION

End-to-end learning for robotic manipulation often involves a lot of trials
and errors due to explorations [1, 2, 3]. Learning from demonstration (LfD)
[4, 5] methods are advantageous in that they do not involve any trials and
errors since only expert demonstrations are used for training. However, expert
demonstrations are often rare and expensive to collect. Therefore in visual
robotic manipulation, common practices are to incorporate auxiliary pipelines
such as pose estimation [6, 7], object segmentation [8], or pre-trained object
representations [8, 9, 10] to improve the sample efficiency of LfD algorithms.
However, sufficient data for training such pipelines are often unavailable in
practice. Therefore, sample efficient end-to-end visual models are desirable.

Recently, group equivariant models have gained the spotlight for their sample
efficiency in various domains such as protein folding and pose estimation
tasks [11, 12, 13, 14]. Group equivariance is a property that if the input
transforms by some group element, then the output should also transform
accordingly.

For robotic manipulation, the roto-translation equivariance can be exploited
for sample efficient learning. The roto-translation equivariance can be
mathematically formalized as the special Euclidean group $SE(n)$, where $n$
denotes the dimension. Transporter Networks [15] achieve impressive sample
efficiencies for end-to-end learning of planar tasks by exploiting the
$SE(2)$-equivariance, which is the planar roto-translation group. However,
Transporter Networks cannot efficiently solve highly spatial tasks with non-
planar roto-translation symmetries since they are not fully
$SE(3)$-equivariant.

Neural Descriptor Fields (NDFs) [8] have been proposed to incorporate full
$SE(3)$-equivariance in robotic manipulation. NDFs successfully learn to solve
simple pick-and-place tasks with few demonstrations ($5{\sim}10$). In
addition, NDFs can generalize to unseen poses and unseen instances (in the
same object class) of the target objects. However, NDFs cannot be trained end-
to-end. NDFs resort to self-supervised representations of the object’s
geometry, which require hard-to-collect voxel data of the target object
categories for training. In addition, NDFs assume the object point cloud to be
segmented from the background. These assumptions are highly non-trivial in
practice. Therefore, fully $SE(3)$-equivariant end-to-end models for sample
efficient manipulation learning still need to be studied.

To this end, we propose the first fully end-to-end $SE(3)$-equivariant models
for visual robotic manipulation. Our method consists of two models, the
picking model and the placing model. For the equivariance of the models, we
propose Equivariant Descriptor Fields (EDFs), which are the representation-
theoretic generalizations to NDFs [8]. We reformulate the energy optimization
problem of [8] into a distribution learning problem by constructing
representation-theoretic equivariant energy-based models. We also provide
effective sampling strategies and training algorithms for our unnormalized
energy-based models on the $SE(3)$ manifold. We show that our models are fully
end-to-end yet are highly sample efficient (${\sim}10$ demonstrations are
enough). Moreover, We show that our trained models can generalize to
previously unseen out-of-distribution (OoD) poses and unseen instances (in the
same object class) of the target objects (See Fig. 1). We also show that our
models can robustly solve tasks in the presence of unseen visual distractors.

The contributions of our paper are summarized as follows.

  * •

We propose the first fully end-to-end $SE(3)$-equivariant models for visual
robotic manipulation.

  * •

We introduce novel representation-theoretic energy functions that guarantee
$SE(3)$-equivariance.

  * •

We generalize the invariant descriptors of NDFs into the representation-
theoretic equivariant descriptors.

  * •

We reformulate the energy optimization problem of NDFs into a probability
distribution learning problem.

  * •

We provide sampling strategies and a training algorithm for our unnormalized
energy-based models on the SE(3) manifold.

##  II BACKGROUNDS

###  II-A Equivariant Robotic Manipulation Leaning

Equivariant robotic manipulation learning is a branch of robot manipulation
learning that exploits the symmetry of the tasks to improve sample efficiency
in learning.

Consider the problem of finding the pose of the grasp $T_{grasp}$ given an
input $X$111Examples of commonly used inputs are color-depth images, point
clouds, object positions.. Let there be a symmetry group $g\in G$ such that it
acts on $T_{grasp}$ by a transformation $T_{g}$ and acts on $X$ by a
transformation $S_{g}$. For sample efficiency, it would be desirable that the
grasp pose be equivariant to the input. Then the problem can be formally
defined as either learning an equivariant pose regressor $f_{\theta}$

$\begin{split}T_{grasp}&=f_{\theta}(X)\quad\\\ s.t.\qquad
T_{g}T_{grasp}&=f_{\theta}(S_{g}X)\end{split}$ (1)

or learning an equivariant distribution (or policy) $P_{\theta}$

$\begin{split}T_{grasp}&\sim P_{\theta}(T_{grasp}|X)\quad\\\ s.t.\qquad
P_{\theta}(T_{g}T_{grasp}|S_{g}X)&=P_{\theta}(T_{grasp}|X)\end{split}$ (2)

where $\theta$ denotes the learnable parameters.

####  II-A1 Transporter Networks

Transporter Networks [15] are end-to-end models aimed at learning the target
distribution of the grasp pose for pick-and-place tasks. By imposing the
equivariance of the planar roto-translation group, $SE(2$), the models are
capable of learning simple tasks in a few-shot level ($1{\sim}10$
demonstrations). Moreover, the models are capable of learning complex
sequential problems. However, Transporter Networks use group convolutional
models [16] to obtain the equivariance. Group convolutional models require the
target symmetry group to be discretized and thus are prone to inaccuracy
issues and the curse of dimensionality. It is prohibitively expensive to run
group convolutional models for the $SE(3)$ group, which is a 6-dimensional
manifold. Insufficient discretization often results in inaccuracy even for the
$SE(2)$ group. As a result, Transporter networks are inefficient for highly
spatial tasks with the $SE(3)$ symmetry.

####  II-A2 Neural Descriptor Fields

Neural Descriptor Fields (NDFs) [8] have recently been proposed to fully
incorporate the $SE(3)$-equivariance in robotic manipulation. For a point
cloud input $X\in\mathbb{R}^{M\times 3}$, they construct an equivariant
feature descriptor field $f(x|X)$ such that

$f(x|X)=f(Tx|TX)\quad\forall\ T\in SE(3)$ (3)

The descriptor field $f(x|X)$ is trained with self-supervised learning to
encode the geometry of the target object. For the LfD step, they define the
following energy function on the $SE(3)$ manifold

$E(T)=\sum_{x\in\mathcal{X}_{q}}\left\|f(Tx|X)-f(\hat{T}x|\hat{X})\right\|_{1}$
(4)

where
$\mathcal{X}_{q}=\\{x_{1},x_{2},\cdots,x_{N_{q}}\\}\in\mathbb{R}^{N_{q}\times
3}$ is a set of query points. $\hat{T}$ and $\hat{X}$ are the demonstration’s
grasp pose and the point cloud input, respectively. If multiple demonstrations
are given, $f(\hat{T}x|\hat{X})$ are simply averaged over the demonstrations.

NDFs can learn simple pick-and-place tasks with only a few demonstrations
($5{\sim}10$). Moreover, the trained model shows impressive generalization
capabilities for the target objects of unseen poses and unseen instances.
However, the major limitation is that NDFs are not an end-to-end model. First,
NDFs assume the point cloud input to be segmented from the background. In
addition, NDFs assume large datasets of objects in the target object
categories for the self-supervised learning phase. These are not trivial
assumptions in practice.

###  II-B Representation Theory of Lie Group

A representation $D$ of a group $G$ is a map from $G$ to the space of linear
operators acting on some vector space $\mathcal{V}$ that has the following
property for all $g,h\in G$,

$D(g)D(h)=D(g\circ h)$ (5)

Given vector spaces $\mathcal{V}$ and $\mathcal{W}$ and a function
$f:\mathcal{V}\rightarrow\mathcal{W}$, $f$ is said to be equivariant to a
group $G$ if

$D_{\mathcal{W}}(g)f(v)=f(D_{\mathcal{V}}(g)v)\qquad\forall g\in
G,v\in\mathcal{V}$ (6)

where $D_{\mathcal{V}}$ and $D_{\mathcal{W}}$ are some representations of $G$
that linearly acts on $\mathcal{V}$ and $\mathcal{W}$ respectively.

Two different representations $D$ and $D^{\prime}$ are said to be of
equivalence222Not to be confused with equivariance. if there exists some non-
degenerate change of basis $S$ such that

$D^{\prime}(g)=SD(g)S^{-1}\qquad\forall g\in G$ (7)

A representation is said to be reducible if there exists some change of basis
such that the representation can be decomposed (block-diagonalized) into
smaller subspaces. An irreducible representation is a representation that
cannot be reduced anymore.

For the $SO(3)$ group, any representation $D(R)$ for $R\in SO(3)$ can be
reduced into a direct sum of $(2l+1)\times(2l+1)$ dimensional irreducible
representations $D_{l}(R)$ of degree $l\in\\{0,1,2,\cdots\\}$ such that

$D(R)=S\left[\bigoplus_{n=1}^{N}{D_{l_{n}}(R)}\right]S^{-1}\qquad\forall R\in
SO(3)$ (8)

where $\bigoplus$ denotes a direct sum333Direct sum can be intuitively
understood as a concatenation for vectors and a block-concatenation for
matrices.. Although there are infinitely many equivalent representations for
$D_{l}$, a particularly preferred choice is the real basis444Another commonly
used choice of basis is the spherical basis where the representations are
unitary and spherical harmonics are complex.. In this basis, all the
representations $D_{l}$ are orthogonal matrices. These matrices are called the
(real) Wigner D-matrices. Details and the formulas can be found in [17].

$l\in\\{0,1,2,\cdots\\}$ is called the degree of the representation. The
$(2l+1)$ dimensional vectors that are transformed by $D_{l}(R)$ are called
type-l vectors. Type-$l$ vectors are identical to themselves when they are
rotated by $\theta=2\pi/l$. Type-0 vectors, or scalars are invariant to
rotations. Type-1 vectors are the familiar 3-dimensional space vectors.

###  II-C Equivariant Graph Neural Networks

Graph neural networks are often used to model point cloud data [18, 19, 20].
$SE(3)$-equivariant graph neural networks [11, 12, 21] were proposed to
exploit the roto-translation symmetry of graphs with spatial structures. In
this work, we use tensor field networks (TFNs) [11] and the
$SE(3)$-transformers [12] as the backbone networks for our equivariant models.
More detailed descriptions of these networks are provided in Appendix.

###  II-D Equivariant Energy-Based Models

Equivariant EBMs on spaces where the symmetry group acts on (e.g., Euclidean
space or N-body systems) have been investigated by [22, 13, 23]. However, no
work has yet addressed an equivariant EBM on the group manifold itself. In
this paper, we propose equivariant EBMs on the $SE(3)$ manifold.

Figure 2: The architecture for the picking model is illustrated in the figure.
The placing model can be implemented similarly. A) The input to the model is
given as the colored point cloud of a scene with a mug (red) and a hanger
(green). The mug may transform by a rigid body transformation $T=(R,t)\in
SE(3)$. Note that the illustrated transformation is local, in that only the
mug transforms, and the background stays the same. B) The point cloud input is
passed into the EDF model. We illustrate three type-0 scalar fields (left) and
three type-1 vector fields (right) as the model’s outputs. The three fields
are visualized as red, green, and blue, respectively. For clarity, only the
vector fields near the handle are visualized. Note the equivariance of the
output fields by the local $SE(3)$ transformation of the mug. Type-0 scalar
fields are invariant such that the color remains the same after the
transformation. In contrast, type-1 vector fields change their direction
equivariantly according to the transformation. C) The feature fields are
passed into the equivariant energy function. The energy function can be
understood as a type-0 scalar field on the $SE(3)$-manifold. D) The
equivariant probability distribution can be constructed from the equivariant
energy function. The end-effector poses can be sampled from the distribution
by the MCMC algorithms in Section IV.

##  III METHODS

In this section, we present EDFs and the corresponding $SE(3)$-equivariant
energy-based models (EBM). EDFs are the equivariant extensions to NDFs [8]
which are invariant descriptors in the context of the representation theory of
the Lie group. We also reformulate the energy optimization problem of [8] into
the equivariant distribution learning problem for the end-to-end learning. We
illustrate our method in Fig. 2

###  III-A Equivariant Descriptor Field

Consider a colored point cloud input with $M$ points given by
$X=\\{(x_{1},f_{1}),\cdots,(x_{M},f_{M})\\}\in\mathbb{R}^{M\times 6}$ where
$x_{i}\in\mathbb{R}^{3}$ is the position and $f_{i}\in\mathbb{R}^{3}$ is the
color vector of the $i$-th point. Note that $M$ may vary between different
point cloud inputs. Since color vectors are direct sums of three type-$0$
scalar features (red, green, and blue), they are invariant under rigid body
transformations. Therefore, $X$ transforms by a rigid body transformation
$T=(R,t)\in SE(3)$ as

$TX=\\{(Rx_{1}+t,f_{1}),\cdots,(Rx_{M}+t,f_{M})\\}$ (9)

We now define the EDF $f(x|X)$ as the direct sum of irreducible
$SE(3)$-equivariant vector fields whose $n$-th feature vector is a
type-$l_{n}$ vector, which is $(2l_{n}+1)$ dimensional:

$f(x|X)=\bigoplus_{n=1}^{N}f^{(n)}(x|X)$ (10)

By a rigid body transformation $T\in SE(3)$, the EDF transforms equivariantly
as follows:

$f(Tx|TX)=D(R)f(x|X)\quad\forall T\in SE(3)$ (11)

where $D(R)=\bigoplus_{n=1}^{N}D_{l_{n}}(R)$ is the direct sum of the Wigner
D-Matrices of degree $l_{n}$ in the real basis. Note that $D(R)$ is the
orthogonal representation of the $SO(3)$ group such that

$D(R)^{T}=D(R)^{-1}=D(R^{-1})=D(R^{T})$ (12)

###  III-B Equivariant Energy Function on SE(3) Group

In this section, we propose a $SE(3)$-equivariant energy function555Other
variants of the energy function can be found in Appendix.

$E_{\theta}(T|X)=\int{d^{3}x}\rho_{\theta}(x)\|f_{\theta}(Tx|X)-D(R)f_{Q;\theta}(x)\|^{2}$
(13)

where $\rho_{\theta}(x)$ is a learnable query density and $f_{Q;\theta}(x)$ is
a learnable query feature field.

We define the corresponding energy-based model as follows:

$P_{\theta}(T|X)=\frac{\exp{\left[-E_{\theta}(T|X)\right]}}{\int
dT\exp{\left[-E_{\theta}(T|X)\right]}}$ (14)

where $dT$ is the invariant Haar measure such that the integral over $SE(3)$
group is invariant: $\int d(T^{\prime}T)=\int dT$.

Now, we define the equivariance property of probability densities and
functions on the $SE(3)$ manifold. We then show that the energy-based models
with equivariant energy functions are also equivariant.

###### Definition 1

A $X$-conditioned scalar function $h(T|X):SE(3)\times\mathbb{R}^{M\times
6}\rightarrow\mathbb{R}$ is $SE(3)$-equivariant if

$h(T|X)=h(T^{\prime}T|T^{\prime}X)\quad\forall T^{\prime}\in SE(3)$

Since we are dealing with an invariant measure, the equivariance for
probability densities can be defined in a similar manner: A probability
density $P(T|X)$ is $SE(3)$-equivariant if
$P(T|X)=P(T^{\prime}T|T^{\prime}X)\quad\forall T^{\prime}\in SE(3)$.

###### Lemma 1

Consider an energy-based model conditioned by $X$ as follows:

$P(T|X)=\frac{\exp{\left[-E(T|X)\right]}}{\int dT\exp{\left[-E(T|X)\right]}}$
(15)

(15) is then equivariant if the energy function $E(T|X)$ is equivariant, that
is $E(T|X)=E(T^{\prime}T|T^{\prime}X)\ \ \forall T^{\prime}\in SE(3)$.

###### Proof:

$\begin{split}P(T^{\prime}T|T^{\prime}X)&=\frac{\exp{\left[-E(T^{\prime}T|T^{\prime}X)\right]}}{\int
dT\exp{\left[-E(T|T^{\prime}X)\right]}}\\\
&=\frac{\exp{\left[-E(T^{\prime}T|T^{\prime}X)\right]}}{\int
dT\exp{\left[-E(T^{\prime}T|T^{\prime}X)\right]}}\\\
&=\frac{\exp{\left[-E(T|X)\right]}}{\int dT\exp{\left[-E(T|X)\right]}}\\\
&=P(T|X)\end{split}$

∎

Note that the group shift invariance of the integral is used in the second
line, and the equivariance property of the energy function is used in the
third line.

Now, we show that our energy function (13) is indeed $SE(3)$-equivariant.

###### Theorem 1

The following energy function is $SE(3)$-equivariant:

$E_{\theta}(T|X)=\int{d^{3}x}\rho_{\theta}(x)\|f_{\theta}(Tx|X)-D(R)f_{Q;\theta}(x)\|^{2}$

###### Proof:

$\begin{split}&E(T^{\prime}T|T^{\prime}X)\\\
&=\int{d^{3}x}\rho_{\theta}(x)\|f_{\theta}(T^{\prime}Tx|T^{\prime}X)-D(R^{\prime})D(R)f_{Q;\theta}(x)\|^{2}\quad\\\
&=\int{d^{3}x}\rho_{\theta}(x)\|D(R^{\prime})f_{\theta}(Tx|X)-D(R^{\prime})D(R)f_{Q;\theta}(x)\|^{2}\quad\\\
&=\int{d^{3}x}\rho_{\theta}(x)\|f_{\theta}(Tx|X)-D(R)f_{Q;\theta}(x)\|^{2}\\\
&=E(T|X)\end{split}$

∎

Note that (11) is used in the third line, and $D(R)^{T}D(R)=I$ is used in the
fourth line.

###  III-C Implementation

Our method consists of two parts: The picking model and the placing model. The
picking model consists of a single EDF model and the corresponding energy
model. The placing model is similar to the picking model but has an additional
EDF model which replaces the original query feature field. Therefore, the
query feature field for the place model is $f_{Q;\theta}(x|X_{grasp})$ where
$X_{grasp}$ is the point cloud of the gripper with the grasped object. This
architecture is analogous to the transport part of Transporter Networks [15]
where the query feature kernels that encode how the objects are grasped are
template-matched (convolved) to the key feature field.

In practice, we use a weighted sum of the Dirac delta functions $\delta$
centered at query points as the query density.

$\rho_{\theta}(x)=\sum_{i=1}^{N_{q}}w_{i}\delta(x-x_{i})$ (16)

where the query weight $w_{i}$ and the position of the query point $x_{i}$ are
either predefined or learnable parameters. In this case, the integral of (13)
is replaced to a much more tractable summation form as follows:

$E_{\theta}(T|X)=\sum_{i=1}^{N}w_{i}\|f_{\theta}(Tx_{i}|X)-D(R)f_{Q;i}\|^{2}$
(17)

where $f_{Q;i}=f_{Q;\theta}(x_{i})$ and $w_{i}=\rho_{\theta}(x_{i})$.

Note that the $L_{2}$ energy version of (4) can be understood as a special
case for (17), where $w_{i}=1/N_{q}$ and all the field features are the
invariant type-$0$ scalar fields.

For the backbone architectures to the EDF $f_{\theta}(x|X)$ we used
SE(3)-Transformers [12]. However, self-attention cannot be applied to the last
layer because the input points are different from the output transformed query
points $\\{Tx_{i}\\}_{i=1}^{N_{q}}$. Therefore, we used TFNs [11] instead of
SE(3) Transformers for the last layer of EDFs. We used the E3NN [24] package
to implement the equivariant layers. We implemented equivariant layer
normalizations for the stability of the learning.

For the computational efficiency and the locality of the algorithm, we impose
the field values to be zero for the places farther than the field cutoff
distance, which can be set as a hyperparameter. We set infinite energy for
configurations outside the task space, thereby guaranteeing all the samples to
be in the task space. Although not necessary, we also set high energies for
configurations with no neighbor points within the field cutoff distance for
faster sampling (See Sec. IV for the sampling). Details for the
implementations can be found in the code.

##  IV Sampling and Training

Direct sampling from the probability distribution in (14) cannot be done
easily. Moreover, the integral in the denominator of (14) is intractable for
the energy function of (17). Therefore, simple maximum likelihood learning
cannot be used to train our models. In this section, we demonstrate the
sampling strategy for our models using Markov Chain Monte-Carlo (MCMC). We
also demonstrate the contrastive divergence based method for training our
models.

###  IV-A Sampling

For the sampling, we combined the Metropolis-Hastings (MH) [25] algorithm and
the Metropolis-adjusted Langevin algorithm (MALA) [26]. The MH algorithm is a
propose-and-reject algorithm, where a proposal point $x_{p}$ is sampled from
the proposal distribution $Q(x_{p}|x_{t})$. The proposed point $x_{p}$ is
stochastically accepted or rejected by the acceptance ratio
$A=\min{\left[1,\frac{P(x_{p})Q(x_{t}|x_{p})}{P(x_{t})Q(x_{p}|x_{t})}\right]}$.
If the proposal is accepted, the next point is the proposed point, that is
$x_{t+1}=x_{p}$. If rejected, the point remains the same, that is
$x_{t+1}=x_{t}$. It is known that $P_{t}(x_{t})$ converges to $P(x)$ as
$t\rightarrow\infty$ for an arbitrary proposal distribution $Q(x_{p}|x_{t})$.
MALA is a special case of the MH algorithm where the new points are proposed
with the Langevin dynamics.

It is known that typically MALA converges with much fewer iterations than MH
when the variances of the proposal distributions are similar. However, MALA
requires the gradient of the log probability, which is computationally
expensive to calculate. In addition, to maintain the precision of the Langevin
dynamics, the timestep of MALA cannot be arbitrarily high. Therefore, MALA is
inappropriate for rapid initial exploration in broad workspaces. On the other
hand, arbitrarily high variance can be used for the MH proposal. However, the
MH algorithm suffers from a high rejection ratio near the minima. Therefore,
we use the MH algorithm in the beginning for faster exploration. Once the
sample points are settled near the minima, MALA is used to finetune the
samples.

In the following paragraphs, we demonstrate the choice of the proposal
distributions on the $SE(3)$ manifold. Note that the acceptance ratio $A$ is a
coordinate independent quantity. However, special care should be taken for the
sampling to be coordinate independent when dealing with non-Euclidean
manifolds such as the $SE(3)$ manifold.

####  IV-A1 Proposal Distribution for MH

The MH algorithm requires a proposal distribution. We decomposed the proposal
step into 1) orientation proposal and 2) position proposal and used an
isotropic Gaussian for the position proposal. For the orientation proposal,
naive Gaussian sampling in axis-angle parametrization cannot be used when the
variance is large. For a large variance, the tails of the Gaussian start to
fold over, resulting in wrong estimates. Instead, we used
$\mathcal{IG}_{SO(3)}$ which is the normal distribution on $SO(3)$ [27, 28,
29]. In the axis-angle parametrization, the axis is proposed uniformly from a
unit sphere. The proposal distribution for the angle $\omega$ is as follows:

$f_{\epsilon}(\omega)=\left[\sum_{l=0}^{\infty}(2l+1)e^{-\epsilon
l(l+1)}\frac{\sin{((2l+1)\omega/2)}}{\sin{\omega/2}}\right]$ (18)

For the sampling of $\omega$, the Haar measure $(1-\cos{\omega})/\pi$ should
be multiplied to $f_{\epsilon}(\omega)$. We used the inverse CDF sampling from
$(1-\cos{\omega})/\pi\times f_{\epsilon}(\omega)$ as [29].

####  IV-A2 Proposal Distribution for MALA

Let $\mathcal{V}_{i}$ be the $i$-th basis of the Lie algebra of an unimodular
Lie group $G$. Consider the following stochastic process $g(t)\in G$ generated
by a Lie algebra $\delta X(t)=\sum_{i}\delta X_{i}(t)\mathcal{V}_{i}\in
T_{e}G$ such that

$g(t)=g(0)\exp\left[\delta X(0)\right]\exp\left[\delta
X(dt)\right]\cdots\exp\left[\delta X(t-dt)\right]$ (19)

The Langevin dynamics for a Lie group is given as $\delta
X_{i}(t)=-\mathcal{L}_{\mathcal{V}_{i}}\left[E(g)\right]dt+\sqrt{2}dW_{i}\ \
$where $\ dW_{i}\sim\mathcal{N}(\mu=0,\sigma=\sqrt{dt})$ is the white noise of
the standard Wiener process and
$\mathcal{L}_{\mathcal{V}}f=\left.\left(\frac{d}{ds}f(g\exp[s\mathcal{V}])\right)\right|_{s=0}$
is the Lie derivative along $\mathcal{V}$. It is known that this process
converges to $P_{\infty}(g)\propto\exp\left[-E(g)\right]$ when
$t\rightarrow\infty$ [30].

Therefore, for the sampling procedure, one may sample $\delta X_{i}$ from the
$\mathcal{N}(\mu_{i},\sigma)$ where
$\mu_{i}=-\mathcal{L}_{\mathcal{V}_{i}}\left[E(g)\right]dt$ and
$\sigma=\sqrt{2dt}$. The sampled $\delta X_{i}$ is then exponentiated and
applied to $g(t)$ as (19). However, $dt$ should be very small to maintain the
accuracy of the dynamics. For larger timesteps, naive Gaussian sampling tends
to underestimate the variance of the diffusion for the $SO(3)$ part.
Therefore, we approximate the SDE by sampling the noise from
$\mathcal{IG}_{SO(3)}$ instead of Gaussian. Note that $\mathcal{IG}_{SO(3)}$
is a solution to the diffusion process on $SO(3)$ [27]. Therefore, when the
Lie derivatives are sufficiently small, the approximation becomes accurate. In
practice, we sample the noise $R_{\epsilon}$ from
$\mathcal{IG}_{SO(3)}(\epsilon=\frac{\sigma^{2}}{2})$, take a matrix
logarithm, then add to $\mu_{i}$ such that $\delta
X(t)_{i}=\mu_{i}+\left[\log{R_{\epsilon}}\right]_{i}$ for the $SO(3)$
proposal. Instead of (18), we use the following formula which has better
convergence property for small $\epsilon$ [27, 29]:

$\begin{split}f_{\epsilon}(\omega)=&(\sqrt{\pi}/2\sin{(\omega/2)})e^{(\epsilon-\omega^{2}/\epsilon)/4}\epsilon^{-3/2}\\\
&\times\left\\{\omega+\sum_{n=1}^{\infty}(-1)^{n}e^{-\pi^{2}n^{2}/\epsilon}\right.\\\
&\left.\times\left[(\omega-2\pi n)e^{n\pi\omega/\epsilon}+(\omega+2\pi
n)e^{-n\pi\omega/\epsilon}\right]\right\\}\\\ \end{split}$ (20)

Note that for very small $\epsilon$, $f_{\epsilon}(\omega)\propto
e^{-\omega^{2}/4\epsilon}=e^{-\omega^{2}/2\sigma^{2}}$ which is Gaussian.

###  IV-B Training

We use the contrastive divergence (CD) [31, 32] algorithm for training our
energy-based models. For the energy-based model (14), the gradient of the log-
likelihood can be approximated with the CD gradient

$\begin{split}\nabla_{\theta}\log{P_{\theta}(x|X)}\approx&-\mathbb{E}_{P_{0}(x|X)}\left[\nabla_{\theta}E_{\theta}(x|X)\right]\\\
&+\mathbb{E}_{P_{n}(x|X)}\left[\nabla_{\theta}E_{\theta}(x|X)\right]\end{split}$
(21)

where $P_{0}(x|X)$ is the target distribution and $P_{n}(x|X)$ is the n-step
sample distribution of the MCMC that starts from $P_{0}(x|X)$.

##  V Experiment and Result

We evaluate the generalization capability of our models with three criteria:
1) Out-of-distribution (OoD) pose generalization, 2) Unseen instance
generalization, and 3) Robustness to unseen visual distractors. We evaluate
with a pick-and-place task similar to [8], where a mug should be picked and
hung on the hanger’s hook. Ten demonstrations are provided to train the
models. Note that all the demonstrations are given with a single identical mug
with upright poses only.

For the test environment, we use PyBullet [33] simulator for the experiments.
We use the Franka Panda manipulator with a custom end-effector. We use IKFast
[34] with Pybullet-Planning [35] for the inverse kinematics. Three simulated
depth cameras are used to observe the point clouds. Since motion planning is
not in the scope of our work, we assume no collision between the environment
and the robot links except for the hand link. We also allow the robot to
teleport to reach pre-grasp and pre-place poses in order to eliminate the
unnecessary influence of the motion planners. However, we fully simulate the
trajectories of all the task-relevant primitives (e.g., grasping, releasing,
lifting). Since the base of the manipulator is fixed, some end-effector
orientations cannot be reached. In this case, we allow the robot to consider
the opposite orientation along the end-effector axis.

We empirically find that the MCMC algorithm converges after running the MH
algorithm for around 3000 steps and the MALA for around 500 steps. However,
waiting for the algorithm to fully converge takes an impractical amount of
time (more than a minute). Therefore, we prematurely terminate the MCMC
algorithm and take the minimum energy sample instead. We find that this
strategy yields reasonable performance. Specifically, we run the MH algorithm
for 500 steps and then run the MALA for 100 steps for the sampling. We
parallelize the sampling with 100 different seeds. We then take the pose with
the lowest energy from the last 50 steps.

All the experiments were done with CPUs for reproducibility. On average, the
wall-clock inference times are $19$ seconds for the pick tasks and $13$
seconds for the place tasks on an Intel i7-10875H CPU. We also test with an
Nvidia RTX 3090 GPU, but it turns out to be slower than the CPU. We presume
this is due to the non-parallel nature of the MCMC algorithms.

###  V-A Unseen Pose Generalization

TABLE I: unseen pose success rate | Pick | Place | Pick-and-Place
---|---|---|---
Upright poses | 0.95 (95/100) | 0.96 (92/95) | 0.92 (92/100)
Lying (OoD) poses | 0.94 (94/100) | 0.95 (90/94) | 0.90 (90/100)

Figure 3: We evaluate the models’ generalization capability to unseen OoD
poses of the target objects with two distinct poses: Upright poses (Left) and
Lying poses (Right). Only upright poses were given during the training. Lying
poses were never given during the training steps. The planar configurations
($x,y,$ and $\theta$) of the mug and the hanger are randomized among different
seeds.

We evaluate the generalization capability of the trained models with
previously unseen OoD poses of the target object (See Fig. 3). We examine the
performance with 100 samples with the same mug used for training but with
lying poses, which were never given during the training phase. We find no
significant difference in the performance between different poses. The results
are summarized in Table I.

###  V-B Unseen Instance Generalization

TABLE II: unseen instance success rate | Pick | Place | Pick-and-Place
---|---|---|---
Mug 1 | 0.92 (23/25) | 0.78 (18/23) | 0.72 (18/25)
Mug 2 | 0.100 (25/25) | 0.92 (23/25) | 0.92 (23/25)
Mug 3 | 0.88 (22/25) | 0.77 (17/22) | 0.68 (17/25)
Mug 4 | 0.80 (20/25) | 0.70 (14/20) | 0.56 (14/25)
Avg. | 0.90 (90/100) | 0.80 (72/90) | 0.72 (72/100)

Figure 4: We evaluate the models’ generalization capability to unseen target
object instances in the same category. We test with four different instances
of the mug that were never given during the training steps. The planar
configurations ($x,y,$ and $\theta$) of the mugs and the hanger are randomized
among different seeds.

We evaluate the generalization capability of the trained models with
previously unseen target object instances in the same category (See Fig 4). We
examine the performance with four different mug/cup instances that were never
given during the training phase. We examine 100 samples for each mug/cup
instance. The results are summarized in Table II.

###  V-C Unseen Visual Distractor Generalization

TABLE III: unseen visual distractor success rate | Pick | Place | Pick-and-Place
---|---|---|---
Distractors | 0.68 (68/100) | 0.85 (58/68) | 0.58 (58/100)

Figure 5: We evaluate the models’ generalization capability in the presence of
unseen visual distractors. The visual distractors were never given during the
training steps. The planar configurations ($x,y,$ and $\theta$) of the mug,
the distractors, and the hanger are randomized among different seeds.

We evaluate the generalization capability of the trained models in the
presence of previously unseen visual distractors (See Fig. 5). Since our
models do not rely on any segmentation algorithms, robustness in the presence
of previously unseen distractors is an important property. The results are
summarized in Table III.

##  VI Discussion and Conclusion

In this work, we presented fully end-to-end $SE(3)$-equivariant models for
visual robotic manipulation learning. We formulated novel $SE(3)$-equivariant
descriptor fields and energy-based models based on the representation theory
of the Lie group that allow highly sample efficient and generalizable
learning. We also provided working strategies for sampling from our energy-
based models on the $SE(3)$ manifold. The models can learn visual pick-and-
place tasks with only ${\sim}10$ demonstrations in a fully end-to-end manner.
Moreover, we showed that the trained models could generalize to the tasks with
unseen target object poses, unseen target object instances, and unseen visual
distractors.

However, our method has several limitations that should be addressed in future
works. The most severe limitation is the long inference time due to the MCMC
sampling. Faster and more efficient generative models should be developed for
real-time inference.

Another limitation is the strict $SE(3)$-equivariance assertion of the models.
Global $SE(3)$ symmetry cannot be strictly kept in most visual manipulation
problems due to the gravity, occlusions, deformations of objects, and the
background scenes. Therefore, we took special care in the design to make the
models capture only the local $SE(3)$ symmetries. Nevertheless, our models may
perform significantly worse than non-equivariant models when the local $SE(3)$
symmetries are severely broken. Hence, models with relaxed forms of the
$SE(3)$-equivariance would be required to guarantee the minimum performance
for the tasks without the $SE(3)$ symmetry while enjoying the equivariance for
tasks with the $SE(3)$ symmetry.

Although we only demonstrated our models with robotic manipulation tasks, our
models can be generally applied to any $SE(3)$-symmetric problems with a point
cloud as the input and a $SE(3)$ transformation as the output. Therefore, we
expect our novel equivariant energy-based models to be useful in various
domains such as protein folding and pose estimation problems, especially when
the samples are rare.

## APPENDIX

###  VI-A Equivariant Graph Neural Networks

####  VI-A1 Tensor Product and Spherical Harmonics

Given two vectors $u$ and $v$ of type-$l_{1}$ and -$l_{2}$, the tensor product
$u\otimes v$ transforms by a rotation $R\in SO(3)$ as

$u\otimes v\rightarrow(D_{l_{1}}(R)u)\otimes(D_{l_{2}}(R)v)$ (22)

Tensor products are important because they can be used to construct new
vectors of different types. By a change of basis the tensor product $u\otimes
v$ can be decomposed into a direct sum of type-$l$ vectors using the Clebsch-
Gordan coefficients [11, 36, 37] as follows:

$(u\otimes
v)^{(l)}_{m}=\sum_{m_{1}=-l_{1}}^{l_{1}}\sum_{m_{2}=-l_{2}}^{l_{2}}C^{(l,m)}_{(l_{1},m_{1})(l_{2},m_{2})}u^{(l_{1})}_{m_{1}}v^{(l_{2})}_{m_{2}}$
(23)

where the Clebsch-Gordan coefficients $C^{(l,m)}_{(l_{1},m_{1})(l_{2},m_{2})}$
are nonzero for ${|l_{1}-l_{2}|\leq l\leq l_{1}+l_{2}}$ only.

The (real) spherical harmonics $Y_{m}^{(l)}(x/\|x\|)$ are orthonormal
functions that form the complete basis of the Hilbert space on a sphere.
$l\in\\{0,1,2,\cdots\\}$ is called the degree and $m\in\\{-l,\cdots,l\\}$ is
called the order of the spherical harmonic function.

Consider the following $(2l+1)$-dimensional vector field
$Y^{(l)}=(Y^{l}_{m=-l},\cdots,Y^{l}_{m=l})$. By a 3-dimensional rotation $R\in
SO(3)$, $Y^{(l)}$ transforms like a type-$l$ vector field such that

$Y^{(l)}(R(x/\|x\|))=D_{l}(R)Y^{(l)}(x/\|x\|)$ (24)

####  VI-A2 Tensor Field Networks

Tensor field networks (TFNs) [11] are $SE(3)$-equivariant models for
generating representation-theoretic vector fields from a point cloud input.
TFNs construct equivariant output feature vectors from equivariant input
feature vectors and spherical harmonics. Spatial convolutions and tensor
products are used for the equivariance.

Consider a featured point cloud input with $M$ points given by
$X=\\{(x_{1},f_{1}),\cdots,(x_{M},f_{M})\\}$ where $x_{i}\in\mathbb{R}^{3}$ is
the position and $f_{i}$ is the equivariant feature vector of the $i$-th
point. Let $f_{i}$ be decomposed into $N$ vectors such that
$f_{i}=\bigoplus_{n=1}^{N}f_{i}^{(n)}$, where $f_{i}^{(n)}$ is a type-$l_{n}$
vector, which is $(2l_{n}+1)$ dimensional. Then the input feature field
$f_{(in)}(x|X)$ generated by the point cloud input $X$ can be defined as

$f_{(in)}(x|X)=\sum_{j=1}^{M}f_{(in),j}\delta(x-x_{j})$ (25)

where $\delta(x-x_{j})$ is the Dirac delta function centered at $x_{j}$. Note
that $f_{(in)}$ is $SE(3)$ equivariant:

$f_{(in)}(Tx|TX)=D(R)f_{(in)}(x|X)\quad\forall\ T=(R,t)\in SE(3)$

where $R\in SO(3),t\in\mathbb{R}^{3}$ and
$D(R)=\bigoplus_{n=1}^{N}D_{l_{n}}(R)$. Note that $T$ acts on $x$ and $X$ such
that $Tx=Rx+t$ and
$TX=\\{(Rx_{1}+t,D(R)f_{1}),\cdots,(Rx_{M}+t,D(R)f_{M})\\}$.

Now consider the following output feature field of a convolution layer

$f_{(out)}(x|X)=\bigoplus_{n^{\prime}=1}^{N^{\prime}}f_{(out)}^{(n^{\prime})}(x|X)$

with the kernel $W(x-y)\in\mathbb{R}^{dim(f_{(out)})\times dim(f_{(in)})}$
whose $(n^{\prime},n)$-th block
$W^{n^{\prime}n}(x-y)\in\mathbb{R}^{(2l_{n^{\prime}}+1)\times(2l_{n}+1)}$ is

$\begin{split}\left[W^{n^{\prime}n}(x)\right]_{m^{\prime}m}=&\sum_{J=|l_{n^{\prime}}-l_{n}|}^{l_{n^{\prime}}+l_{n}}\phi^{n^{\prime}n}_{J}(\|x\|)\\\
&\times\sum_{k=-J}^{J}C^{(l_{n^{\prime}},m^{\prime})}_{(J,k)(l_{n},m)}Y^{(J)}_{k}\left(x/\|x\|\right)\end{split}$
(26)

Here, $\phi^{n^{\prime}n}_{J}(\|x\|)$ is a learnable radial function,
$Y^{(J)}_{k}$ is the $k$-th component of the real spherical harmonics of
degree $k$, and $C$ is the Clebsch-Gordan coefficients in the real basis. The
output $f_{(out)}(x)$ of the following convolution

$\begin{split}f_{(out)}(x|X)&=\int{d^{3}y}W(x-y)f_{(in)}(x|X)\\\
&=\sum_{j}W(x-x_{j})f_{(in),j}\end{split}$ (27)

is proven to be equivariant [11, 12]. Note that the output equivariant feature
vectors are constructed by the tensor product of input equivariant feature
vectors and the spherical harmonics, which encode the spatial information.

####  VI-A3 SE(3)-Transformers

The SE(3)-Transformers [12] are variants of TFNs with self-attention. Consider
the case that the output field is also of the form

$f_{(out)}(x|X)=\sum_{j=1}^{M}f_{(out),j}\delta(x-x_{j})$ (28)

where $x_{i}$ is the same point as that of the point cloud input $X$. The
SE(3)-Transformers apply type-0 (scalar) self-attention $\alpha_{ij}$ to (27):

$\begin{split}f_{(out),i}=&\bigoplus_{n^{\prime}}^{N^{\prime}}\sum_{n=1}^{N}W_{(S)}^{n^{\prime}n}f_{(in),j}^{(n)}\\\
&+\sum_{j\neq i}\alpha_{ij}W(x-x_{j})f_{(in),j}\end{split}$ (29)

where the $W_{(S)}^{n^{\prime}n}$ term is called the self-interaction [11].
$W_{(S)}^{n^{\prime}n}$ is nonzero only when $l_{n}^{\prime}=l_{n}$. The self-
interaction occurs where $i=j$ such that $W(x_{i}-x_{j})=W(0)$. The self-
interaction term is needed because $W$ is a linear combination of the
spherical harmonics which are not well defined in $x=0$.

###  VI-B Variants of the $SE(3)$-Equivariant Energy Function

In the previous sections, we only introduced the equivariant energy function
with a square error form (13). Although we find that (13) is reasonably good
for our tasks, the square error term may be too strict in some tasks. Thus,
the corresponding energy-based models can be fragile to outliers. Therefore,
we propose two variants of (13), the inner-product variant and the
$L_{1}$-norm variant.

####  VI-B1 Inner Product Form

The following energy function is equivariant

$E_{\theta}(T|X)=\int{d^{3}x}\rho_{\theta}(x)<f_{\theta}(Tx|X),D(R)f_{Q;\theta}(x)>$
(30)

where $<\cdot,\cdot>$ denotes the inner product. The proof for the
equivariance is straightforward using the orthogonality of the real Wigner
D-matrix $D(R)$. One may also consider using a normalized cosine distance
instead of the inner product.

####  VI-B2 $L_{1}$ Form

The $L_{1}$-norm is often preferred over the $L_{2}$-norm for their
robustness. However, unlike the $L_{2}$-norm, the $L_{1}$-norm is not
equivariant by itself. Therefore, we propose the irrep-wise $L_{1}$-norm. For
an equivariant vector $f=\bigoplus_{n=1}^{N}f^{(n)}$, the equivariant irrep-
wise $L_{1}$-norm can be defined as

$\|f\|_{1}^{(I)}=\sum_{n=1}^{N}\|f^{(n)}\|$ (31)

Now, consider an EDF $f_{\theta}(x|X)$ and a query feature field
$f_{Q:\theta}(x)$. The following energy function is then equivariant.

$\begin{split}E_{\theta}(T|X)&=\int{d^{3}x}\rho_{\theta}(x)\|f_{\theta}(Tx|X)-D(R)f_{Q;\theta}(x)\|^{(I)}_{1}\\\
\end{split}$ (32)

## References

  * [1] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, _et al._ , “Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation (2018),” _arXiv preprint arXiv:1806.10293_ , 2018.
  * [2] S. Levine, C. Finn, T. Darrell, and P. Abbeel, “End-to-end training of deep visuomotor policies,” _The Journal of Machine Learning Research_ , vol. 17, no. 1, pp. 1334–1373, 2016.
  * [3] J.-H. Lee and J. Choi, “Attaining interpretability in reinforcement learning via hierarchical primitive composition,” _arXiv preprint arXiv:2110.01833_ , 2021.
  * [4] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 3, pp. 297–330, 2020.
  * [5] B. D. Argall, S. Chernova, M. Veloso, and B. Browning, “A survey of robot learning from demonstration,” _Robotics and autonomous systems_ , vol. 57, no. 5, pp. 469–483, 2009.
  * [6] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao, “Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge,” in _2017 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2017, pp. 1386–1383.
  * [7] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox, “Self-supervised 6d object pose estimation for robot manipulation,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 3665–3671.
  * [8] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann, “Neural descriptor fields: SE(3)-equivariant object representations for manipulation,” _arXiv preprint arXiv:2112.05124_ , 2021\.
  * [9] P. R. Florence, L. Manuelli, and R. Tedrake, “Dense object nets: Learning dense visual object descriptors by and for robotic manipulation,” _arXiv preprint arXiv:1806.08756_ , 2018.
  * [10] T. D. Kulkarni, A. Gupta, C. Ionescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih, “Unsupervised learning of object keypoints for perception and control,” _Advances in neural information processing systems_ , vol. 32, 2019\.
  * [11] N. Thomas, T. Smidt, S. Kearnes, L. Yang, L. Li, K. Kohlhoff, and P. Riley, “Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds,” _arXiv preprint arXiv:1802.08219_ , 2018.
  * [12] F. Fuchs, D. Worrall, V. Fischer, and M. Welling, “SE(3)-transformers: 3d roto-translation equivariant attention networks,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 1970–1981, 2020.
  * [13] J. Wu, T. Shen, H. Lan, Y. Bian, and J. Huang, “SE(3)-equivariant energy-based models for end-to-end protein folding,” _bioRxiv_ , 2021.
  * [14] X. Li, Y. Weng, L. Yi, L. Guibas, A. L. Abbott, S. Song, and H. Wang, “Leveraging SE(3) equivariance for self-supervised category-level object pose estimation,” _arXiv preprint arXiv:2111.00190_ , 2021.
  * [15] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, _et al._ , “Transporter networks: Rearranging the visual world for robotic manipulation,” _arXiv preprint arXiv:2010.14406_ , 2020.
  * [16] T. Cohen and M. Welling, “Group equivariant convolutional networks,” in _International conference on machine learning_. PMLR, 2016, pp. 2990–2999.
  * [17] G. Aubert, “An alternative to wigner d-matrices for rotating real spherical harmonics,” _AIP Advances_ , vol. 3, no. 6, p. 062121, 2013.
  * [18] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph cnn for learning on point clouds,” _Acm Transactions On Graphics (tog)_ , vol. 38, no. 5, pp. 1–12, 2019.
  * [19] G. Te, W. Hu, A. Zheng, and Z. Guo, “Rgcnn: Regularized graph cnn for point cloud segmentation,” in _Proceedings of the 26th ACM international conference on Multimedia_ , 2018, pp. 746–754.
  * [20] W. Shi and R. Rajkumar, “Point-gnn: Graph neural network for 3d object detection in a point cloud,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 1711–1719.
  * [21] C. Deng, O. Litany, Y. Duan, A. Poulenard, A. Tagliasacchi, and L. J. Guibas, “Vector neurons: A general framework for SO(3)-equivariant networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 12 200–12 209.
  * [22] P. Jaini, L. Holdijk, and M. Welling, “Learning equivariant energy based models with equivariant stein variational gradient descent,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 16 727–16 737. [Online]. Available: https://proceedings.neurips.cc/paper/2021/file/8b9e7ab295e87570551db122a04c6f7c-Paper.pdf
  * [23] J. J. Kivinen and C. K. Williams, “Transformation equivariant boltzmann machines,” in _International Conference on Artificial Neural Networks_. Springer, 2011, pp. 1–9.
  * [24] M. Geiger, T. Smidt, A. M., B. K. Miller, W. Boomsma, B. Dice, K. Lapchevskyi, M. Weiler, M. Tyszkiewicz, S. Batzner, D. Madisetti, M. Uhrin, J. Frellsen, N. Jung, S. Sanborn, M. Wen, J. Rackers, M. Rød, and M. Bailey, “Euclidean neural networks: e3nn,” apr 2022. [Online]. Available: https://doi.org/10.5281/zenodo.6459381
  * [25] W. K. Hastings, “Monte carlo sampling methods using markov chains and their applications,” 1970.
  * [26] J. Besag, “Comments on “representations of knowledge in complex systems” by u. grenander and mi miller,” _J. Roy. Statist. Soc. Ser. B_ , vol. 56, no. 591-592, p. 4, 1994.
  * [27] D. I. Nikolayev and T. I. Savyolov, “Normal distribution on the rotation group SO(3),” _Textures and Microstructures_ , vol. 29, 1970.
  * [28] T. I. T. Savyolova, “Normal distributions on SO(3),” in _Programming And Mathematical Techniques In Physics-Proceedings Of The Conference On Programming And Mathematical Methods For Solving Physical Problems_. World Scientific, 1994, p. 220.
  * [29] A. Leach, S. M. Schmon, M. T. Degiacomi, and C. G. Willcocks, “Denoising diffusion probabilistic models on SO(3) for rotational alignment,” in _ICLR 2022 Workshop on Geometrical and Topological Representation Learning_ , 2022.
  * [30] G. S. Chirikjian, _Stochastic models, information theory, and Lie groups, volume 2: Analytic methods and modern applications_. Springer Science & Business Media, 2011, vol. 2.
  * [31] G. E. Hinton, “Training products of experts by minimizing contrastive divergence,” _Neural computation_ , vol. 14, no. 8, pp. 1771–1800, 2002\.
  * [32] M. A. Carreira-Perpinan and G. Hinton, “On contrastive divergence learning,” in _International workshop on artificial intelligence and statistics_. PMLR, 2005, pp. 33–40.
  * [33] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2021.
  * [34] R. Diankov, “Automated construction of robotic manipulation programs,” Ph.D. dissertation, Carnegie Mellon University, Robotics Institute, August 2010. [Online]. Available: http://www.programmingvision.com/rosen˙diankov˙thesis.pdf
  * [35] C. R. Garrett, “Pybullet planning,” https://pypi.org/project/pybullet-planning/, 2018.
  * [36] A. Zee, _Group theory in a nutshell for physicists_. Princeton University Press, 2016, vol. 17.
  * [37] D. J. Griffiths and D. F. Schroeter, _Introduction to quantum mechanics_. Cambridge university press, 2018.