# Differentiable Point-Based Radiance Fields for
Efficient View Synthesis

Qiang Zhang, Seung-Hwan Baek, Szymon Rusinkiewicz, Felix Heide  Princeton
University

###### Abstract.

We propose a differentiable rendering algorithm for efficient novel view
synthesis. By departing from volume-based representations in favor of a
learned point representation, we improve on existing methods more than an
order of magnitude in memory and runtime, both in training and inference. The
method begins with a uniformly-sampled random point cloud and learns per-point
position and view-dependent appearance, using a differentiable splat-based
renderer to evolve the model to match a set of input images. Our method is up
to 300 $\times$ faster than NeRF in both training and inference, with only a
marginal sacrifice in quality, while using less than 10 MB of memory for a
static scene. For dynamic scenes, our method trains two orders of magnitude
faster than STNeRF and renders at near interactive rate, while maintaining
high image quality and temporal coherence even without imposing any temporal-
coherency regularizers.

††submissionid:

##  1\. Introduction

Neural rendering has emerged as an effective tool to model real-world scenes
[Mildenhall et al., 2020], synthesize novel views [Lombardi et al., 2019], and
advance single-shot learning [Yu et al., 2021c]. Specifically, neural
rendering has made it possible to generate photorealistic novel views at
unseen viewpoints from unorganized multi-view training images. Synthesizing
novel views could not only enable after-shot visual effects in consumer
photography and virtual reality but also implicitly/explicitly entails
geometric understanding of a scene. Utilizing an efficient view-synthesis
network in a analysis-by-synthesis framework may further allow these
techniques to become the missing link for inverse-rendering applications in
diverse application domains.

Neural radiance fields (NeRF) [Mildenhall et al., 2020] have spearheaded
neural rendering by modeling a 5D radiance field in a memory-efficient manner
with an implicit scene representation in the form of a multi-layer perceptron
(MLP). A large body of work has built on this method using implicit scene
representations for view synthesis, investigating generalizability [Yu et al.,
2021c] and rendering quality [Wizadwongsa et al., 2021]. However, a drawback
of modern neural rendering methods is computational efficiency. As NeRF relies
on volumetric rendering, training and inference of the model requires many
evaluations and memory accesses of the radiance fields: typically more than
100 evaluations for each ray. As a result, for medium-sized static scenes,
NeRF takes over ten hours of training and 30 seconds of inference for each
scene on a single GPU. The limited computational efficiency of NeRF-based
methods prevents its direct applicability to video applications, where fast
training and inference at a low memory footprint are mandated. Recent methods
have attempted to tackle this problem by pretraining a multi-view stereo model
on a large-scale dataset and fine-tuning it for test inputs [Chen et al.,
2021b], or using an explicit radiance volume [Yu et al., 2021b]. While these
methods outperform NeRF in efficiency, they are not capable of accelerating
both training and inference speed and require a large pretraining dataset as
well as significant storage.

In this work we explore an alternative approach for retaining the highly-
desirable end-to-end differentiability of NeRF, while utilizing an alternate
radiance field representation. Our insight is that, even though the MLP used
by NeRF is compact and relatively efficient to evaluate once, using it for
image synthesis is fundamentally a “backward mapping” operation that requires
multiple evaluations of radiance along a ray. Shifting to a representation
that allows “forward mapping,” i.e., creating an image via a single forward
pass over the representation of the radiance field, enables modest gains in
storage but, more importantly, efficiency gains of multiple orders of
magnitude at both training and rendering time.

Specifically, we use a representation for radiance based on RGB spherical
harmonic coefficients stored at points in 3D. Image synthesis is then
accomplished via a differentiable splat-based rasterizer that operates as a
single forward pass. The rendering pipeline from point primitives and
spherical harmonic coefficients to images is end-to-end differentiable,
allowing for optimizing the position and appearance attributes using a coarse-
to-fine first-order optimization method.

Our strategy yields several benefits: smaller storage, faster training, and
more efficient rendering that avoids volumetric ray marching. The cost is
reduced flexibility in radiance representation, but this is not always a
drawback. Indeed, with few input views, the regularization provided by our
method leads to higher reconstruction quality. However, when many input views
are provided, our method results in slightly lower quality (for a fixed
spherical harmonic degree). The proposed approach learns point clouds with
position and appearance without a neural network or separate geometry
estimation (e.g., via COLMAP) step. In contrast to existing methods, it allows
for efficient inference _and_ training.

Specifically, we make the following contributions in this work:

  * •

We introduce a point-based radiance representation that, together with
differentiable splatting, is 300x faster than neural volumetric methods for
view synthesis in both training and inference. We provide supplementary code
attached to this submission.

  * •

We analyze the proposed point-cloud representation in terms of memory,
inference and training run-time, and show that it outperforms state-of-the-art
neural rendering algorithms [Yu et al., 2021a] in compute, while being two
orders of magnitude smaller in memory.

  * •

We demonstrate that our method is efficient enough in memory, training time,
and rendering time to enable novel-view synthesis for video, even without
establishing explicit frame-to-frame connections. We validate that the
proposed method outperforms existing methods in quality, run time, and memory
footprint.

All code and data required to reproduce the findings from this work is
available here.

#### Limitations

The proposed method requires an object mask to generate the initial point
cloud, and thus can only render the foreground scene components. This
limitation may be potentially solved by replacing the hard multi-view
consistency function with a soft one: setting a threshold to filter based on
projected pixel variance. As popular existing multi-view and video datasets
contain masks, we leave such a potential extension for future work. The
proposed representation cannot handle semi-transparent objects, e.g., objects
with fur.

##  2\. Related Work

#### Neural Rendering

A rapidly growing body of work investigates introducing differentiable modules
into rendering approaches, turning conventional rendering methods into
powerful inverse tools for capturing, reconstructing, and synthesizing real-
world scenes. Among many rendering principles, volumetric rendering has been
actively studied in conjunction with neural networks [Mildenhall et al., 2020]
thanks to its learning-friendly structure. In this context, various network
architectures have been proposed to model scene radiance, including
convolutional neural network (CNN) in Neural Volumes [Lombardi et al., 2019]
and multi-layer perceptron (MLP) in NeRF [Mildenhall et al., 2020]. While the
synthesized images from these methods achieve unprecedented fidelity, one of
the drawbacks is its generalization ability. To tackle this challenge, image
features can be explicitly exploited either from a single image [Yu et al.,
2021c] or multiple images [Wang et al., 2021; Chen et al., 2021b; Liu et al.,
2021]. Another line of neural-rendering methods utilizes a multi-plane image
(MPI) as a scene representation for high-quality view synthesis [Zhou et al.,
2018; Wizadwongsa et al., 2021]. Lastly, using point-clouds has been studied
for neural rendering with iterative filtering [Rosenthal and Linsen, 2008] and
multi-scale approaches [Marroquim et al., 2007; Meshry et al., 2019; Rückert
et al., 2021]. Among the many successful approaches, NeRF-based volumetric
methods in particular have achieved remarkable success.

However, volumetric methods are not without limitations, and one major problem
is their computational efficiency. While occupying only a small memory
footprint, NeRF-based models typically require hours of training in a modern
GPU, and inference takes minutes. This low computational efficiency prevents
NeRF methods from being used in applications that require fast training and
inference. Recently, researchers devised methods to accelerate inference, at
the cost of additional memory incurred: Garbin et al. [2021] caches explicit
volumes, Yu et al. [2021b] proposes an octree structure, Reiser et al. [2021]
replaces a large MLP with multiple small-size MLPs. Although these methods can
speed up the rendering process, the training is still volumetric, making
existing approaches slow. Recently, Lombardi et al. [2021] proposed a hybrid
scene representation with with primitive-based and volumetric components. As
such, this method inherits some of the drawbacks of volumetric methods,
including training time: this work requires 5 days of training time for a
single scene, orders of magnitude more than the proposed method. Yu et al.
[2021a] use explicit volumes instead of an MLP and directly optimizes the
volume parameters. However, this results in gigabytes of memory for each
scene, hence an inevitable explosion with terabytes of memory for multi-frame
video. In this work, we propose a computationally-efficient rendering method
that accelerates both training and inference, achieving 2 $\times$ faster
inference, 3 $\times$ faster training, and 100 $\times$ lower memory footprint
than Yu et al. [2021a].

Figure 2. Learning Point-Based Radiance Fields. The proposed method learns a
point cloud augmented with RGB spherical harmonic coefficients per point.
Image synthesis is accomplished via a differentiable splat-based rasterizer
that operates as a single forward pass, without the need of ray-marching and
hundreds of evaluations of coordinate-based networks as implicit neural scene
representation methods. Specifically, the point rasterizer computes an alpha
value from the ray-point distance with a Gaussian radial basis function
weight. The learnable radiance and the alpha values are rasterized via alpha
blending to render the image. The entire rendering pipeline is differentiable
and we optimize for the point position and spherical harmonic coefficients in
end-to-end. We use a hybrid coarse-to-fine strategy (bottom) to train the
whole model and successively refine the geometry represented by the learned
point cloud.

#### Multi-view Geometry Reconstruction

Our work falls into the category of multi-view geometry reconstruction methods
studied for decades in computer graphics and vision. This direction traces
back to popular multi-view stereo methods using local photometric consistency
and global visibility constraints [Furukawa and Ponce, 2009] and geometric
priors [Schönberger et al., 2016]. Recently, neural networks have accelerated
multi-view geometry acquisition [Yao et al., 2018; Im et al., 2019; Han et
al., 2020]. Notably, Han et al. [2020] uses multi-view projection and exploits
silhouette images to estimate shape, demonstrating state-of-the-art
performance. Recently, Wiles et al. [2020] (SynSin) and Thonat et al. [2021]
propose single-shot video prediction and mesh-based multi-view video methods
with technical approaches orthogonal to ours. Specifically, none of the
existing methods can model the geometry and radiance simultaneously. In
addition, they typically require a separate dataset to pretrain the model for
a geometry prior. The proposed method optimizes a learned point cloud directly
for each scene and does not require additional learned or explicit priors.

#### View-dependent Modeling and Rendering

Modeling and rendering view-dependent scene radiance has been studied
extensively in graphics and vision. Surface light fields [Wood et al., 2000]
are based on a view-dependent level-of-detail rendering for meshes with
subdivision connectivity. Using spherical harmonics in this context has been
proposed as an effective way to represent a view-dependent function, with
broad adoption in rendering and modeling [Cabral et al., 1987; Sillion et al.,
1991; Ramamoorthi and Hanrahan, 2001a, b; Basri and Jacobs, 2003; Sloan et
al., 2003, 2002]. Recent works apply view-dependent models to reflectance
[Chen et al., 2021a; Sztrajman et al., 2021] via implicit BRDF modeling and
use spherical harmonics to describe view-dependent radiance distribution [Yu
et al., 2021b; Wizadwongsa et al., 2021]. The proposed method also uses
spherical harmonics to model view-dependence while maintaining lightweight and
efficient model.

#### Point-based Rendering

Point-based rendering via splatting has been explored for decades, offering
computational efficiency as an elegant scene representation [Pfister et al.,
2000; Zwicker et al., 2001; Zwicker et al., 2004; Zwicker et al., 2002].
Recent works explore combining deep learning with surface splatting, e.g.,
[Yifan et al., 2019] propose a differentiable algorithm for point-based
surface and normal processing and does not have an appearance model and does
not produce radiance outputs. In contrast, the proposed method focuses on
high-quality image-based rendering and novel view synthesis. Aliev et al.
[2020] propose a point-based rendering method which requires a _given point
cloud_ aggregated from RGB-D captures, while the proposed method optimizes for
point clouds from RGB-only inputs. In contrast to our network-free method,
this method also uses a rendering network that requires a day of training
time. Lassner et al. [2021] generate features by rasterization of spheres in a
_deferred neural rendering_ approach using an image-to-image rendering network
(Pix2PixHD). The proposed method synthesizes an RGB image directly by
splatting a point cloud with a per-point appearance model. We report in this
manuscript that this results in an order of magnitude faster training time
while providing better results. Kopanas et al. [2021] use rasterized encoded
point clouds with features, provided by conventional structure-from-motion
methods, i.e., using COLMAP to build a dense point cloud, and SIBR to get the
depth and normal maps. A neural network then translates features into images
for novel views. The feature encoding and decoding are learned, leading to
long training times. Moreover, their inference stage selects image features
from the training set and thus needs to store all these training features,
resulting in storage requirements of multiple gigabytes in contrast to less
than ten megabytes for the proposed method. Similarly, Rückert et al. [2021]
assume _known point clouds_ , recovered by COLMAP. Given a point cloud, their
method employs a deferred neural rendering approach similar to Lassner et al.
[2021] but with a multi-scale feature-to-image transfer network. In contrast
to the proposed method that requires a few minutes of training time, Rückert
et al. train for 12 hours.

##  3\. Differentiable Point Radiance Fields

In this section, we describe the proposed differentiable point-based rendering
method, which is illustrated in Fig. 2. We assume a set of images $I_{j}$,
$j=1\ldots N$ as input, with known camera intrinsics and extrinsics. We learn
a model of the scene as a point cloud with a per-point radiance model based on
spherical harmonics. Learning minimizes the difference between the rendered
scene and the original images, by updating the position and appearance of each
point. The use of a differentiable splat-based rasterizer allows for gradient-
based updates, while being orders of magnitude more efficient than an implicit
volume-based renderer. The point cloud, the proposed method is operating on,
is optimized end-to-end in a coarse-to-fine iterative fashion, starting from a
rejection-sampled initial point cloud. We describe the components of the
proposed method in the following.

Figure 3. Synthetic View Synthesis for Static Scenes. We represent two
synthetic scenes [Mildenhall et al., 2020] with the proposed point-based
radiance model. Trained only in a few minutes on multi-view images for this
static scene, the proposed method achieves rendering quality comparable to
state-of-the-art volumetric rendering approaches such as Plenoxels [Yu et al.,
2021b], while being more than 3 $\times$ faster in training, 2 $\times$ faster
in inference and 100 $\times$ less memory hungry.

###  3.1. Point-Based Scene Radiance

We model the scene as a set of points $P_{i}=(x_{i},y_{i},z_{i})^{\intercal}$,
$i=1,\ldots,n$, with the parameters $H_{i}$ of a view-dependent radiance model
per point. These 3D positions and radiance model parameters serve as the
unknowns that will be updated by the optimization method to reconstruct the
scene.

#### Geometric Model

Each 3D point $P_{i}$ projects to a 2D position $p_{i}^{j}$ in each image
$I_{j}$. To model the projection operation, we assume that we know the
extrinsic rigid-body transform $(R_{j},t_{j})$ and intrinsic matrix $M_{j}$
for each camera, and then compute

(1) $p_{i}^{j}=\left(M_{j}(R_{j}P_{i}+t_{j})\right)^{\downarrow},$

where $\left(x,y,z\right)^{\downarrow}=(x/z,y/z)$ denotes the perspective
divide. Here $P_{i}$ are represented as the learnable neural parameters. Note
that visibility is resolved by the rendering stage (see Section 3.2) via depth
sorting and alpha blending.

#### Photometric Model

We model the view-dependent radiance of each point with spherical harmonics
[Cabral et al., 1987] up to a maximum degree $l_{\textrm{max}}$:

(2) $H_{i}=\\{h_{i,lm}\;|\;0\leq l\leq l_{\textrm{max}},\;-l\leq m\leq l\\}.$

Where $h_{i,lm}$ are represented as the learnable neural parameters. If we
then denote the view direction of point $i$ in image $j$ as

(3) $v_{i}^{j}=\frac{R_{j}P_{i}+t_{j}}{\|R_{j}P_{i}+t_{j}\|},$

we can represent the color of the point in that image as

(4)
$c_{i}^{j}=\sum_{l=0}^{l_{\textrm{max}}}\sum_{m=-l}^{l}h_{i,lm}\,Y_{l}^{m}(v_{i}^{j}).$

#### Point Cloud Initialization

We initialize points randomly within the visual hull defined by the image-
space masks. Specifically, we use a rejection sampling method in which we
uniformly sample random 3D points, project them into each image, and keep them
only if they fall within all the masks. We initialize the point appearance by
sampling the spherical harmonics parameter from the normal distribution with
zero mean and standard deviation of one.

Our method is thus dependent on the availability and quality of the initial
masks. Specifically, we found that the optimization still produces good
results if the masks are inaccurate or there are concavities not modeled by
the visual hull. More details for the initialization and its visualization are
explained in the appendix.

###  3.2. Differentiable Splat Rendering

Given a point cloud model, we render an image via point splatting — see
Zwicker et al. [2001] for background on splatting methods. Specifically, we
use an image-space radial basis function (RBF) kernel to represent the
contribution of each point $p_{i}^{j}$ to each pixel $u$ in image $j$. We use
a Gaussian kernel of radius $r$, computing the opacity as

(5) $\alpha_{i}^{j}(u)=\frac{1}{\sqrt{2\pi
r^{2}}}\,e^{-\frac{\|p_{i}^{j}-u\|^{2}}{2r^{2}}}.$

After obtaining the $\alpha$ map, we sort the points by $z$-distance and
compute the final rendered image $\hat{I}_{j}$ via alpha blending from front
to back. That is, we compute

(6) $\hat{I}_{j}(u)=\sum_{i=1}^{n}A_{i}^{j}(u)\,c_{i}^{j},$

where $A_{i}^{j}(u)$ represents the net contribution of the $i$-th point:

(7)
$A_{i}^{j}(u)=\alpha_{i}^{j}(u)\,\prod\limits_{k=1}^{i-1}\left(1-\alpha_{k}^{j}(u)\right),$

where the points are taken as sorted front-to-back and $c_{i}^{j}$ is the per-
point color computed via Eq. 4. Note that as an optimization we only consider
points within a certain maximum distance of $u$, since otherwise their opacity
will be low.

###  3.3. Point Cloud Training

To train a point cloud model that predicts the image $I_{j}$, we employ a loss
function incorporating MSE loss and total variance loss:

(8) $\mathcal{L}=\sum_{j=1}^{N}||I_{j}-\hat{I}_{j}||_{2}^{2}+\lambda
TV(\hat{I}_{j}).$

The total variation term refers to the conventional anisotropic total
variation penalty $TV(\cdot)=|\nabla_{x}\cdot|_{1}+|\nabla_{y}\cdot|_{1}$,
that is the $\ell_{1}$-norm of the predicted image gradients. This provides
noise reduction via smoothing. Since the loss function, splatting rasterizer,
and point cloud geometric and photometric models are all fully differentiable,
we are able to learn the point positions and their parameters end-to-end using
stochastic gradient descent optimization.

To train the proposed method we propose a non-synchronous learning strategy
for optimizing the point position and harmonics parameters. Specifically, the
initial learning rate for the spherical harmonics radiance model and the point
positions are $3e-3$ and $8e-4$, respectively. Both of them are exponentially
decayed for every epoch (20 epochs in total) with a factor of $0.93$. The TV
weight $\lambda$ is set to $0.01$. The splatting radius $r$ and the
rasterization $n$ are set as $0.008$ and $15$ respectively.  We use
$l_{\textrm{max}}=2$ for all of the results in the paper, since in our
experiments using higher degree resulted in minimal quality gain for
significant extra storage and training time. Section 4.3 contains an ablation
study comparing spherical harmonics to just a single color, which effectively
corresponds to $l_{\textrm{max}}=0$.

#### Hybrid Coarse-to-Fine Updates

To learn an accurate point cloud representation, we repeat the following
several stages of a coarse-to-fine scheme twice, also illustrated in Fig. 2:

  1. (1)

Voxel-based Point Cloud Reduction: We assume a voxel discretization (without
actually voxelizing the scene), and, for each voxel, we aggregate all the
points inside that voxel into one single point by averaging the position and
the spherical harmonics of these points. The motivation behind this step is to
avoid the formation of dense point clusters and improve the representation
efficiency.

  2. (2)

Outlier Removal: For each point, we compute the distance to its neighbors and
the standard deviation of these distances. We then filter out the outliers
whose standard deviation is beyond a threshold, hence removing outlier points.

  3. (3)

Point Generation: In this stage, we insert new points into the point cloud by
generating a single new point for each point in the point cloud. We set all
parameters (including position) to the average of the point’s nearest $K$
neighbors.

We also include Alg. 1 for this coarse-to-fine update in the Appendix. After
these steps, we train the model again on the new point cloud with half of the
original learning rate in a final refinement step.

Synthetic Dataset | Pretraining | Training | Inference | Model Size | Rendering Quality
---|---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
NeRF [Mildenhall et al., 2020] | None | 20 h | 1/12 fps | 14 MB | 31.0 dB | 0.947 | 0.081
IBRNet [Wang et al., 2021] | 1 day | 30 min | 1/25 fps | 15 MB | 28.1 dB | 0.942 | 0.072
MVSNeRF [Chen et al., 2021b] | 20 h | 15 min | 1/14 fps | 14 MB | 27.0 dB | 0.931 | 0.168
Plenoxels [Yu et al., 2021a] | None | 11 min | 15 fps | 1.1 GB | 31.7 dB | 0.958 | 0.050
Plenoxels_s [Yu et al., 2021a] | None | 8.5 min | 18 fps | 234 MB | 28.5 dB | 0.926 | 0.100
Pulsar [Lassner and Zollhofer, 2021] | None | 95 min | 4 fps | 228 MB | 26.9 dB | 0.923 | 0.184
PBNR [Kopanas et al., 2021] | None | 3 h | 4 fps | 2.96 GB | 27.4 dB | 0.932 | 0.164
Ours | None | 3 min | 32 fps | 9 MB | 30.3 dB | 0.945 | 0.078
Table 1. Static Novel View Synthesis Evaluation on the Synthetic Blender
Dataset. We evaluate the proposed method and comparable baseline approaches
for novel view synthesis on the static Blender scenes from [Mildenhall et al.,
2020]. The proposed model does not require an extra dataset for pretraining
and improves on existing methods in training, inference speed, model size, at
cost of only a small reduction in quality. Specifically, although the
concurrent Plenoxels [2021a] achieves better quality, our model is two
magnitudes smaller than theirs. We also compare here to the Plenoxels_s model
from [Yu et al., 2021a] (Plenoxels with smaller volume resolution), which
achieves worse rendering quality with larger model size. Figure 4. Novel View
Synthesis. We synthesize novel views for video timestamps of the STNeRF [2021]
dataset (Frame 58 and Frame70). Comparing to STNeRF [2021] and the NeRF-t
model proposed in the same work, the differential point cloud radiance field
approach produces substantially cleaner object boundaries, finer object detail
and fewer reconstruction artefacts, especially in presence of significant
object motion, as illustrated in the insets.

###  3.4. Video Synthesis

When extending our method from novel image synthesis to novel video synthesis,
we train a model per-frame. This means that the training time grows as the
frame number grows. However, in dynamic video scenes, nearby frames are
temporally correlated. As such, we use the model learned in a given frame to
initialize the next frame, thereby reducing the training time needed til
convergence. Specifically, after we obtain the model for a given frame $1$, an
accurate pointcloud $S_{1}$ with spherical harmonics parameters is obtained
for this frame. Moreover, we can also retrieve a new initialized pointcloud
$S_{2}$ for the next frame $2$, using the same scheme as discussed in Sec.
3.3. We then compute the Chamfer distance $\mathcal{CD}(S_{1},S_{2})$ between
these two point clouds:

(9) $\mathcal{CD}(S_{1},S_{2})=\frac{1}{S_{1}}\sum_{x\in S_{1}}\min_{y\in
S_{2}}{||x-y||_{2}^{2}}+\frac{1}{S_{2}}\sum_{y\in S_{2}}\min_{x\in
S_{1}}{||x-y||_{2}^{2}}.$

By minizing the Chamfer distance via gradient descent (the Adam optimizer)
applied to the point positions in $S_{2}$, we can obtain a more accurate point
cloud initialization for the next frame, in terms of geometry. Then for each
point $p_{i}$ in $S_{2}$, we can compute the top-k closest points in $S_{1}$
and choose the average of their corresponding spherical harmonics parameters
as the initialized parameter of $p_{i}$. This strategy can help achieve about
4 $\times$ training speedup.

##  4\. Assessment

We evaluate the proposed point-based radiance representation for novel view
synthesis task in two scenarios: static scenes and dynamic video scenes. In
both scenarios, we assume that measurements from multiple views are available
as input, either as multi-view images or as videos from multiple views. In
this section, we validate that the proposed method outperforms existing
methods in computational efficiency while allowing for rendering high-fidelity
images and videos from novel views. In addition, we provide extensive
ablations that validate architecture choices we made for the proposed method.
We implemented the proposed method in PyTorch, see Supplementary Code. We use
a single NVIDIA 2080Ti GPU for training in all experiments.

###  4.1. Novel View Synthesis for Static Scenes

We test the proposed method on the rendered multi-view images from the Blender
dataset used in NeRF [Mildenhall et al., 2020]. We compare our method against
state-of-the-art methods for novel-view synthesis: IBRNet [Wang et al., 2021],
MVSNeRF [Chen et al., 2021b], Plenoxels [Yu et al., 2021a], Pulsar [Lassner
and Zollhofer, 2021], and PBNR [Kopanas et al., 2021]. We note that PBNR
requires pre-processed geometry information such as point clouds and point
features retrieved from conventional SfM. In contrast, the proposed method is
_network-free_ and learns both point cloud positions and appearance, without
the need for separate input geometry. Quantitatively, although the full
version of Plenoxel (model size of 1.1GB) performs better in terms of
reconstruction quality than ours, its smaller version Plenoxel_s (model size
of 234MB) is 1.8dB worse than ours in PSNR, while our model size is only 9MB.
Table 1 shows that the proposed method outperforms existing methods in
training speed, inference speed, and model size while maintaining comparable
image quality. Figure 3 provides visualizations of the rendered images which
validate that the synthesized images are on par with state-of-the-art methods.

###  4.2. Video View Synthesis

The computational efficiency and low memory footprint of the proposed point-
based representation, allows it to explicitly represent challenging dynamic
scenes better than implicit methods that rely on networks to produce time-
conditional outputs as ST-NeRF and NeRF-T. Note that such NeRF-based per-frame
modeling of videos results in a heavy computational load: for example, a
1000-frame video corresponding to a 40-seconds clip at 25FPS requires more
than 20K GPU hours (per frame training) to train for a vanilla NeRF model and
more than 1 terabyte storage (when trained per frame) for a Plenoxels [Yu et
al., 2021a] model. Recently, Zhang et al. [2021] proposed NeRF-t and STNeRF as
variants of NeRF that allow to reduce the computational requirements of
modeling videos for view synthesis by conditioning the radiance MLP of NeRF on
the time $t$. While effective in reducing memory requirements, this comes at
the cost of rendering quality compared to frame-by-frame modeling. The
proposed method is light-weight that it allows for frame-by-frame learning
with respect to both training time and storage. We evaluate the method on the
video view synthesis datasets from STNeRF [Zhang et al., 2021] and the human
motion dataset DSC [Vlasic et al., 2009]. We note that the authors of STNeRF
released only a portion of their entire dataset, which we use to perform our
experiments. The dataset features 16 camera views distributed uniformly on a
half circle, from which we use 13 views for training and 3 views for testing.
Table 2 and Figure 4 validate that our method outperforms the existing methods
quantitatively and qualitatively. For experiments on DSC, please refer to the
Appendix.

STNeRF | Train | Render | Model | Rendering Quality
---|---|---|---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
NeRF | 40 h | 1/25 fps | 14 MB | 23.7 dB | 0.853 | 0.304
NeRF-t | 100 h | 1/26 fps | 16 MB | 28.9 dB | 0.913 | 0.259
STNeRF | 50 h | 1/30 fps | 12 MB | 32.1 dB | 0.918 | 0.224
Ours | 30 min | 25 fps | 110 MB | 34.6 dB | 0.927 | 0.207
Table 2. Quantitative Evaluation on the STNeRF Dataset. We report rendering
quality, memory footprint, rendering and training time for novel video view
synthesis on the STNeRF datset[2021]. Compared to STNeRF[2021] and the NeRF
variants suggested in the same work, the training speed and inference speed of
the proposed method is two orders of magnitude higher.

###  4.3. Ablation Experiments

We evaluate each component of our method and validate their effectiveness.
First, we replace the spherical harmonics with learnable directionally
invariant RGB values denoted as ”w/o sh”. Second, we remove the hybrid coarse-
to-fine strategy and only train the point cloud in single stage denoted as
”w/o hybrid”. Third, we remove the consistency filter function for every epoch
during training time, noted as ”w/o filter”.  Table 3 shows that using
spherical harmonics, the coarse-to-fine strategy, and the consistency filter
function jointly achieves the best view synthesis performance. Please refer to
the Appendix for corresponding qualitative results. Finally, we provide an
ablation study for sparse training views on the ficus object from the
Synthetic Blender dataset. The comparison is shown in Figure 5. Although our
method is slightly worse than the NeRF baseline when training on the whole 100
views, the method gradually improves on NeRF when the training images become
sparser. We refer to the Appendix for additional ablation experiments.

Module | Rendering Quality
---|---
PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
w/o sh | 27.1 dB | 0.932 | 0.155
w/o hybrid | 26.6 dB | 0.925 | 0.163
w/o filter | 29.1 dB | 0.941 | 0.103
Ours | 30.3 dB | 0.945 | 0.078

Table 3. Model Ablation Experiments. We evaluate the rendering quality of our
method on the Blender dataset [2020] when gradually removing components from
the rendering pipeline. Specifically, we ablate the spherical harmonics model
per point, the coarse-to-fine strategy, and the filtering function for the
training. The experiments results validate that all components contribute to
the rendering quality.

.

Figure 5. View Ablations. We evaluate the quality of the proposed point-based
method and the volumetric NeRF [2020] on the ’Ficus’ scene [2020] with
gradually reduced number of training views. The proposed sparse representation
allows for reconstructions with reasonable quality even with as few as 20
views, where NeRF drops by more than 10 dB in PSNR.

##  5\. Conclusion

Departing from volumetric representations of scene radiance, extensively
explored in recent neural rendering approaches, we investigate an alternate
radiance field representation based on points without giving up end-to-end
differentiablity. In contrast to coordinate-network representations that
require ray-marching with hundreds of network evaluations per ray, the
proposed representation allows us to render a novel view via differentiable
splat-based rasterization. As a result, the proposed method is two orders of
magnitude faster in training and inference time than volumetric methods, such
as NeRF, while being on par in quality. The method does not require any pre-
training and the representation is highly compact, occupying only 10 Mb per
static scene. Future directions that may build on the approach introduced in
this work could include hybrid point/surface models, and combining the
proposed rendering method with point-based physical simulation methods.

## References

  * [1]
  * Aliev et al. [2020] Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. 2020\.  Neural point-based graphics. In _European Conference on Computer Vision_. Springer, 696–712.
  * Basri and Jacobs [2003] Ronen Basri and David W. Jacobs. 2003.  Lambertian Reflectance and Linear Subspaces.  _IEEE Trans. PAMI_ 25, 2 (2003), 218–233.
  * Cabral et al. [1987] Brian Cabral, Nelson Max, and Rebecca Springmeyer. 1987\.  Bidirectional Reflection Functions from Surface Bump Maps.  _Computer Graphics (Proc. SIGGRAPH)_ 21, 4 (1987), 273–281.
  * Chen et al. [2021b] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. 2021b.  MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo. In _Proc. ICCV_.
  * Chen et al. [2021a] Zhe Chen, Shohei Nobuhara, and Ko Nishino. 2021a.  Invertible Neural BRDF for Object Inverse Rendering.  _IEEE Trans. PAMI_ (2021).
  * Furukawa and Ponce [2009] Yasutaka Furukawa and Jean Ponce. 2009.  Accurate, Dense, and Robust Multiview Stereopsis.  _IEEE Trans. PAMI_ 32, 8 (2009), 1362–1376.
  * Garbin et al. [2021] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. 2021\.  FastNeRF: High-Fidelity Neural Rendering at 200FPS. In _Proc. ICCV_.
  * Han et al. [2020] Zhizhong Han, Chao Chen, Yu-Shen Liu, and Matthias Zwicker. 2020\.  DRWR: A Differentiable Renderer without Rendering for Unsupervised 3D Structure Learning from Silhouette Images. In _Proc. ICML_.
  * Im et al. [2019] Sunghoon Im, Hae-Gon Jeon, Stephen Lin, and In So Kweon. 2019\.  DPSNet: End-to-end Deep Plane Sweep Stereo. In _Proc. ICLR_.
  * Kopanas et al. [2021] Georgios Kopanas, Julien Philip, Thomas Leimkühler, and George Drettakis. 2021.  Point-Based Neural Rendering with Per-View Optimization. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 29–43.
  * Lassner and Zollhofer [2021] Christoph Lassner and Michael Zollhofer. 2021.  Pulsar: Efficient sphere-based neural rendering. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1440–1449.
  * Liu et al. [2021] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. 2021\.  Neural Rays for Occlusion-aware Image-based Rendering.  arXiv:2107.13421.
  * Lombardi et al. [2019] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. 2019\.  Neural Volumes: Learning Dynamic Renderable Volumes from Images.  _ACM Trans. Graphics_ 38, 4, Article 65 (July 2019).
  * Lombardi et al. [2021] Stephen Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. 2021.  Mixture of volumetric primitives for efficient neural rendering.  _ACM Transactions on Graphics (TOG)_ 40, 4 (2021), 1–13.
  * Marroquim et al. [2007] Ricardo Marroquim, Martin Kraus, and Paulo Roma Cavalcanti. 2007\.  Efficient Point-Based Rendering Using Image Reconstruction. In _Proc. PBG_. 101–108.
  * Meshry et al. [2019] Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. 2019.  Neural Rerendering in the Wild. In _Proc. CVPR_. 6878–6887.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2020\.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In _Proc. ECCV_. Springer, 405–421.
  * Pfister et al. [2000] Hanspeter Pfister, Matthias Zwicker, Jeroen Van Baar, and Markus Gross. 2000.  Surfels: Surface Elements as Rendering Primitives. In _Computer Graphics (Proc. SIGGRAPH)_. 335–342.
  * Ramamoorthi and Hanrahan [2001a] Ravi Ramamoorthi and Pat Hanrahan. 2001a.  An Efficient Representation for Irradiance Environment Maps. In _Computer Graphics (Proc. SIGGRAPH)_. 497–500.
  * Ramamoorthi and Hanrahan [2001b] Ravi Ramamoorthi and Pat Hanrahan. 2001b.  On the Relationship Between Radiance and Irradiance: Determining the Illumination from Images of a Convex Lambertian Object.  _JOSA A_ 18, 10 (2001), 2448–2459.
  * Reiser et al. [2021] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. 2021\.  KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs.  arXiv:2103.13744.
  * Rosenthal and Linsen [2008] Paul Rosenthal and Lars Linsen. 2008.  Image-space Point Cloud Rendering. In _Proc. CGI_. 136–143.
  * Rückert et al. [2021] Darius Rückert, Linus Franke, and Marc Stamminger. 2021\.  ADOP: Approximate Differentiable One-Pixel Point Rendering.  arXiv:2110.06635.
  * Schönberger et al. [2016] Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. 2016.  Pixelwise View Selection for Unstructured Multi-View Stereo. In _Proc. ECCV_. Springer, 501–518.
  * Sillion et al. [1991] Françis X Sillion, James R Arvo, Stephen H Westin, and Donald P Greenberg. 1991.  A Global Illumination Solution for General Reflectance Distributions. In _Computer Graphics (Proc. SIGGRAPH)_. 187–196.
  * Sloan et al. [2003] Peter-Pike Sloan, Jesse Hall, John Hart, and John Snyder. 2003\.  Clustered Principal Components for Precomputed Radiance Transfer.  _ACM Trans. Graphics_ 22, 3 (2003), 382–391.
  * Sloan et al. [2002] Peter-Pike Sloan, Jan Kautz, and John Snyder. 2002\.  Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments. In _Computer Graphics (Proc. SIGGRAPH)_. 527–536.
  * Sztrajman et al. [2021] Alejandro Sztrajman, Gilles Rainer, Tobias Ritschel, and Tim Weyrich. 2021.  Neural BRDF Representation and Importance Sampling.  _Computer Graphics Forum_ 40, 6 (2021), 332–346.
  * Thonat et al. [2021] Théo Thonat, Yagiz Aksoy, Miika Aittala, Sylvain Paris, Frédo Durand, and George Drettakis. 2021.  Video-Based Rendering of Dynamic Stationary Environments from Unsynchronized Inputs. In _Computer Graphics Forum_ , Vol. 40. Wiley Online Library, 73–86.
  * Vlasic et al. [2009] Daniel Vlasic, Pieter Peers, Ilya Baran, Paul Debevec, Jovan Popović, Szymon Rusinkiewicz, and Wojciech Matusik. 2009.  Dynamic Shape Capture using Multi-View Photometric Stereo.  _ACM Trans. Graphics_ 28, 5 (Dec. 2009).
  * Wang et al. [2021] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser. 2021.  IBRNet: Learning Multi-View Image-Based Rendering. In _Proc. CVPR_.
  * Wiles et al. [2020] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. 2020.  Synsin: End-to-end view synthesis from a single image. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7467–7477.
  * Wizadwongsa et al. [2021] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn. 2021\.  NeX: Real-time View Synthesis with Neural Basis Expansion. In _Proc. CVPR_.
  * Wood et al. [2000] Daniel N Wood, Daniel I Azuma, Ken Aldinger, Brian Curless, Tom Duchamp, David H Salesin, and Werner Stuetzle. 2000.  Surface Light Fields for 3D Photography. In _Computer Graphics (Proc. SIGGRAPH)_. 287–296.
  * Yao et al. [2018] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. 2018.  MVSNet: Depth Inference for Unstructured Multi-view Stereo. In _Proc. ECCV_. 767–783.
  * Yifan et al. [2019] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019\.  Differentiable Surface Splatting for Point-based Geometry Processing.  _ACM Trans. Graphics_ 38, 6 (2019), 1–14.
  * Yu et al. [2021a] Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. 2021a.  Plenoxels: Radiance Fields without Neural Networks.  arXiv:2112.05131.
  * Yu et al. [2021b] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. 2021b.  PlenOctrees for Real-time Rendering of Neural Radiance Fields. In _Proc. ICCV_.
  * Yu et al. [2021c] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. 2021c.  pixelNeRF: Neural Radiance Fields from One or Few Images. In _Proc. CVPR_. 4578–4587.
  * Zhang et al. [2021] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. 2021.  Editable Free-Viewpoint Video using a Layered Neural Representation.  _ACM Trans. Graphics_ 40, 4 (2021), 1–18.
  * Zhou et al. [2018] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018.  Stereo Magnification: Learning View Synthesis using Multiplane Images.  _ACM Trans. Graphics_ 37, 4, Article 65 (Aug. 2018).
  * Zwicker et al. [2002] Matthias Zwicker, Mark Pauly, Oliver Knoll, and Markus Gross. 2002.  Pointshop 3D: An Interactive System for Point-Based Surface Editing.  _ACM Trans. Graphics_ 21, 3 (2002), 322–329.
  * Zwicker et al. [2001] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. 2001.  Surface Splatting. In _Computer Graphics (Proc. SIGGRAPH)_. 371–378.
  * Zwicker et al. [2004] Matthias Zwicker, Jussi Räsänen, Mario Botsch, Carsten Dachsbacher, and Mark Pauly. 2004\.  Perspective Accurate Splatting. In _Proc. Graphics Interface_. 247–254.

##  6\. Appendix

###  6.1. Additional Evaluation on the Synthetic Dataset

Tab.4 reports the PSNR value for every object for the static novel view
synthesis experiment. Our method is comparable to Plenoxels [Yu et al., 2021a]
and NeRF [Mildenhall et al., 2020] while being resource-efficient in terms of
memory and speed.

| Chair | Drums | Ficus | Hotdog | Lego | Materials | Mic | Ship | Avg
---|---|---|---|---|---|---|---|---|---
NeRF | 33.00 | 25.01 | 30.13 | 36.18 | 32.54 | 29.62 | 32.91 | 28.65 | 31.01
Plenoxels | 33.98 | 25.35 | 31.83 | 36.43 | 34.10 | 29.14 | 33.26 | 29.62 | 31.71
Ours | 32.98 | 25.53 | 29.01 | 34.56 | 31.33 | 28.01 | 33.82 | 27.01 | 30.28

Table 4. Additional Static View Synthesis Evaluation. The proposed method
achieves comparable performance to existing state-of-the-art view synthesis
methods for static scenes baselines while being more efficient in inference
speed and memory usage. Figure 6. Ablating View Dependence. We visualize the
ablations of the learnable spherical harmonics. For the mic object, we observe
noticeable artefacts on the object surface if we remove the view-dependence
module. This is due to the optimization with color unconsistency from
different views.

###  6.2. Additional Visualization on the Blender Dataset

Fig.7 visualizes more results for comparison on the Blender dataset from
[Mildenhall et al., 2020]. The proposed approach achieves substantially better
rendering quality than the two recent point-based rendering baselines: Pulsar
and PBNR.

Figure 7. Additional Comparison to Point-based Methods. We visualize novel
views rendered for the Chair, Lego and Ficus objects from the Blender dataset
from [Mildenhall et al., 2020], validating that our method outperforms
existing point-based representation methods. Figure 8. Analysis of Coarse-to-
Fine Optimization. We analyze rendering quality with and without the hybrid
coarse-to-fine training strategy. We observe noticeable outlier points if we
train the model without the coarse-to-fine training strategy, validating the
proposed approach. Figure 9. Novel View Synthesis on the DSC Dataset. We
synthesize novel views for videos from the DSC dataset[2009] (frame 50 for
every scene). We show the nearest captured frame on the left. We retrained
STNeRF [2021] and the NeRF-t variant suggested by the authors on the same
data. In contrast to the propose method, these existing approaches fail on
highly dynamic scenes (right).

###  6.3. Additional Ablation Studies

We evaluate the impact of the number of points on the final rendering quality
to assess the trade-off between reconstruction accuracy and computational
efficiency, reported Table 6. While a large number of points can better
represent a scene radiance field, at the cost of higher compute and memory,
diminishing returns can be observed at circa 50k points for the given static
scenes we evaluate the method on.

We also assess the proposed method with and without modeling view dependence
with spherical harmonics in Fig.6. Modeling view dependence is essential for
the proposed model to represent specular highlights, as apparent on the brass
symbols and the plastic surface on the microphone object. These experiments
further validate the proposed model that relies on learnable spherical
harmonics.

###  6.4. Results on the DSC Dataset

For the DSC dataset, 8 cameras are distributed in a 360∘ setup and we use all
of them to train the model (due to the low number of cameras views). Here we
should address that this is still a very challenging task since there are only
8 training views. We report quantitative results in Table 5 and qualitative
results in Figure 9 that further corroborate the findings we made for the
STNeRF dataset. In all cases, the proposed method achieves high-quality view
synthesis results with orders of magnitude decreased training time and
improved inference speed.

STNeRF | Train | Render | Model
---|---|---|---
NeRF | 50 h | 1/15 fps | 14 MB
NeRF-t | 75 h | 1/15 fps | 16 MB
STNeRF | 86 h | 1/18 fps | 12 MB
Ours | 1 h | 28 fps | 240 MB
Table 5. Quantitative Evaluation on the DSC Dataset. We report the training
time, rendering speed and the model size required to represent scenes from the
DSC[Vlasic et al., 2009] dataset with the proposed method. Note that we train
a radiance point cloud for each frame here and we report the total memory
consumption.

Point Scale | Train | Render | Model | PSNR$\uparrow$
---|---|---|---|---
15 k | 16 min | 35 fps | 37 MB | 32.2 dB
30 k | 20 min | 32 fps | 73 MB | 33.6 dB
45 k | 30 min | 25 fps | 110 MB | 34.6 dB
60 k | 21 min | 36 fps | 146 MB | 34.8 dB
100 k | 76 min | 10 fps | 244 MB | 34.2 dB

Table 6. Point Cloud Size Ablations. The rendering quality of the proposed
method grows with the density of the point cloud (for a given volume), at the
cost of training time, inference speed and model size. We trade-off these
dimensions and choose 45k points for our experiments.

.

###  6.5. Additional Initialization Details

Next, we provide further details for the point cloud initialization from
initial image masks. We first uniformly sample points from scene volume. For
each 3D point $P^{j}$, we compute the projected 2D point $p_{i}^{j}$ on a
camera $i$ using Eq.1. After each projection, we check whether the
corresponding pixel is a background pixel, in which case we discard the point.
Specifially, we define the following consistency check function

$\mathcal{CC}(p^{j})=\mathbb{I}(\sum_{i=1}^{N}\mathbb{I}(M_{i}(p_{i}^{j})>0)=N),$

where $M_{i}(\cdot)$ is the foreground mask and $\mathbb{I}(\cdot)$ is the
indicator function. We compute the value of this consistency function for
every sample point and only keep the point if the value is true, thus the
initialized point cloud is established.

We also use this consistency function during the training time by filtering
out the point, who become inconsistent after every training epoch.

Algorithm 1 Hybrid Coarse-to-Fine Update Strategy

0: Coarse point cloud $S=\\{P_{i}|P_{i}=(x_{i},y_{i},z_{i},f_{i})\\}$,

0: Hyper-parameters $N,K,\epsilon$

0: New point cloud
$S^{{}^{\prime}}=\\{P_{i}|P_{i}=(x_{i},y_{i},z_{i},f_{i})\\}$

1: $S^{{}^{\prime}}=\emptyset$, Partition the space into $N\times N\times N$
voxels.

2: for each voxel do

3: Fetch all the points $\\{P_{1},P_{2},...,P_{M}\\}$ inside this voxel.

4: Compute the average: $P_{a}=\frac{1}{M}\sum\limits_{i=1}^{M}$.

5: Add $P_{a}$ to $S^{{}^{\prime}}$.

6: end for

7: for $P_{i}\in S^{{}^{\prime}}$ do

8: Fetch neighbor $K$ points $\\{P_{1},P_{2},...,P_{K}\\}$ for $P_{i}$.

9: Compute distances between neighbors and $P_{i}$ to get
$\\{d_{1},d_{2},...,d_{K}\\}$

10: Compute standard deviation $\delta$ for set $\\{d_{1},d_{2},...,d_{K}\\}$

11: if $\delta>\epsilon$ then

12: Remove $P_{i}$ from $S^{{}^{\prime}}$

13: end if

14: end for

15: for $P_{i}\in S^{{}^{\prime}}$ do

16: Fetch neighbor $K$ points $\\{P_{1},P_{2},...,P_{K}\\}$ for $P_{i}$.

17: Compute the average: $P_{a}=\frac{1}{K}\sum\limits_{i=1}^{K}P_{i}$.

18: Add $P_{a}$ to $S^{{}^{\prime}}$.

19: end for

20: return $S^{{}^{\prime}}$

###  6.6. Additional Algorithm Details

In addition to the description in the manuscript, we provide here pseudo-code
of our coarse-to-fine update method, see Algorithm 1. We further note that all
code, models and data required to reproduce the findings in this work, will be
published.

###  6.7. Point Cloud Evolution During Training

We visualize the evolution of the learned point clouds during training in
Fig.10. As the training progresses, we can observe three trends: the point
cloud approximates the target geometry more accurately with more training, the
outliers are eliminated with additional training stages, and the concave
regions are progressively improved with additional training iterations. These
additional visualization qualitatively validate the proposed training scheme.

Figure 10. Point Cloud Visualizations Evolving Throughout Training. As
training progresses, the learned point cloud approximates the object geometry
more accurately, illustrating the effectiveness of the positional updates of
the proposed optimization method.


# ReorientDiff: Diffusion Model based Reorientation for Object Manipulation

Utkarsh A. Mishra and Yongxin Chen Utkarsh A. Mishra and Yongxin Chen are
affiliated to the Institute for Robotics and Intelligent Machines (IRIM),
Georgia Institute of Technology<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The ability to manipulate objects in a desired configurations is a fundamental
requirement for robots to complete various practical applications. While
certain goals can be achieved by picking and placing the objects of interest
directly, object reorientation is needed for precise placement in most of the
tasks. In such scenarios, the object must be reoriented and re-positioned into
intermediate poses that facilitate accurate placement at the target pose. To
this end, we propose a reorientation planning method, ReorientDiff, that
utilizes a diffusion model-based approach. The proposed method employs both
visual inputs from the scene, and goal-specific language prompts to plan
intermediate reorientation poses. Specifically, the scene and language-task
information are mapped into a joint scene-task representation feature space,
which is subsequently leveraged to condition the diffusion model. The
diffusion model samples intermediate poses based on the representation using
classifier-free guidance and then uses gradients of learned feasibility-score
models for implicit iterative pose-refinement. The proposed method is
evaluated using a set of YCB-objects and a suction gripper, demonstrating a
success rate of 96.5% in simulation. Overall, our study presents a promising
approach to address the reorientation challenge in manipulation by learning a
conditional distribution, which is an effective way to move towards more
generalizable object manipulation. For more results, checkout our website:
https://utkarshmishra04.github.io/ReorientDiff.

##  I Introduction

Rearranging objects in a desired pose is an important skill necessary for
daily activities at home as well as for specific arrangement and packing
applications in the industry. Performing such a task requires extracting
object information from visual-sensor data and planning a pick-place sequence
[1, 2]. While a single-step pick-place sequence is a viable solution, placing
the object at a specific position and orientation is not always feasible.
Reorientation is a helpful strategy when successfully changing an object’s
pose allows its placement at the target pose [3]. Reorientation ensures
feasible intermediate transition poses in scenarios where there are no common
grasps between the current pose and an object’s desired placement pose.

In classical approaches, such a problem is usually tackled by using trajectory
planners [4] to plan motion from the current pose to the desired pose via
diverse candidate intermediate poses. Such an exhaustive search is expensive
on time and is limited by choice of the number of intermediate pose options.
Recently, Wada et al. [3] proposed a data-driven sampling-based solution to
reorientation using learned models that predict the feasibility score of an
intermediate pose. While their method significantly improved the success rate
and planning time, the approach relied on the target object’s specification
and placement pose. Lately, with the advances in language descriptor
foundation models like CLIP [5], which projects images and texts to a common
feature space, such specifications can be directly correlated between visual
information and suitable language commands, thus empowering human-robot
interaction. This motivated us to explore grounding the problem statement of
reorientation on language and hence embed semantic knowledge of the task with
the spatial structure of the scene [6].

In this paper, we propose ReorientDiff, a diffusion model-based reorientation
pose generation pipeline for solving the task proposed by [3] for picking
objects from a cluttered pile and placing it in the target pose specified
through language descriptions. The core idea of our approach is to visualize
the feasible intermediate poses as distributions. Such a distribution can be
captured by a diffusion model and will be conditioned on the object’s current
and target pose, or in a more general multi-object scenario, on the pile of
pickable objects and the occupancy of the target location. Note that diffusion
models have also been successfully used for motion planning [7, 8], grasp
planning [9], and object rearrangement [10] applications.

To enable interaction using natural language directly, we use pre-trained CLIP
embeddings with an object segmentation model to structure object selection,
pose prediction, and target object segmentation networks for the task.
Considering the intermediate features as a generic scene and target
representation in reduced dimensionality, the diffusion model samples
reorientation poses conditioned on such features, which are further implicitly
refined by a feasibility-score-based discriminator similar to the models used
by [11, 3]. We combine a generic classifier-free conditional sampling [12]
with classifier-guided sampling [13] to sample from diffusion models. For the
tasks, we consider reorientation of objects in the YCB dataset [14] that are
feasible for suction grippers. For each selected object, we choose target
locations on multiple shelf levels and four possible target orientations. Our
method samples reorientation poses in continuous space without any
discretization or candidate pose selection and reaches a cumulative success
rate of $96.5$% as evaluated on selected individual objects.

Figure 1: Reorientation for precise target placement The above figure
represents the phenomenon of reorientation in which an object from a cluttered
file has to be placed precisely in a shelf (target position shown). As the
object cannot be directly place at the target location, our proposed method,
ReorientDiff, samples a reorientation pose using a learned conditional
distribution by a diffusion model. Such a proposed reorientation pose acts as
a transition for facilitating successful placement. We also consider and take
advantage of the object dynamics, as introduced by Wada et al. [3], by which
we ensure that un-grasping an object in an unstable pose will eventually allow
the object to settle at some favourable pose.

##  II Related Work

###  II-A Object Manipulation: Pick and Place

While traditional methods have tried to solve the pick-and-place task using
grasp planning [15, 16, 17] with known object geometries or using pose
estimation methods [18, 19, 20], the recent literature has focused more on
vision-based object manipulation [1, 21]. Solving single-step pick and place
tasks is typically achieved by planning grasp poses using segmentation and
depth maps of the object, where it is considered that a picked object can be
placed within the region of interest (like in a box) [22, 23]. Recent studies
have also shown object rearrangement planning capabilities [6, 24] where a
target location is sampled based on some user-specified goal. Then the whole-
pipeline for generating a collision free trajectory from current to target
location is planned. Some works have proposed object rearrangement as a long
horizon problem [2] consisting of multiple sequential pick and place actions
to achieve a desired configuration.

###  II-B Language Models for Robotics

Language models like GPT-2 [25] and GPT-3 [26] have proven to be quite
effective in grounding the task’s semantics with the scene’s spatial features
using several foundation models. One such foundation model is CLIP [5] which
encodes visual and language information into common representation space and
has been helpful in learning policies for generalized pick-place tasks in
planar tabletop [6] and 3D [24] manipulation and for control of embodied AI
agents [27, 28]. Further, language models have also been used in language-
conditioned object rearrangement planning [10, 29] and supplying high-level
instructions for long-horizon planning [30].

###  II-C Reorientation and Regrasping

Reorientation is a vital capability required for solving complex manipulation
tasks. Prior research have explored this direction by planning to reorient
objects using extrinsic supports [31, 32], which enables them to re-grasp the
object in a desired way. While [31] proposed a graph neural network structure
for pose sequencing and [32] used an end-to-end point-cloud based model for
predicting reorientation poses, [33] proposed a heuristic based method for
reorienting rock structures in excavation. Recently, ReorientBot [3] was
proposed to solve the reorientation task using learned feasibility prediction
models and rejection sampling.

###  II-D Generative Models for Robotics

Generative models like VAE have been used for planning grasps [11] using
visible point-cloud of objects and for constructing embedding space for high-
level tasks for various downstream planning. Recently, diffusion models have
been used extensively in literature for trajectory planning from imitation
data [7, 8] and for generating target poses for language-conditioned object
rearrangement tasks [10]. With language-guided scene and video generation
applications, such models have been used for generating task-videos for robot
learning [34] and generalizing to unseen scenarios [35].

Figure 2: Joint Embedding Construction We use a pre-trained CLIP-ResNet50 and
BPE-based Tokenizer with CLIP language model for obtaining a semantic
embedding of the tabletop RGB image and instruction prompt, respectively.
While keeping CLIP layers frozen, we train another ResNet50 encoder for
spatial RGB-D observations and combine them with the semantic embeddings to
obtain joint embeddings for visual-language inputs. We train these latent
representations with respect to the object of interest, placement pose, and
object location (segmentation) predictions. It is worth noting that the
predicted object information is also used for predicting the placement pose
and the target object segmentation. Further, the addition of skip-connection
also ensures that the segmentation map construction is accurate while filling
up the embedding vector with only the necessary information. The proposed
pipeline shown above creates a latent space that is consistent with the three
aspects of interest by minimizing information loss.

##  III Preliminary: Diffusion Models

Consider samples $x_{0}$ from an unknown data distribution $q(x_{0})$;
diffusion models [36] learn to estimate the distribution by a parameterized
model $p_{\theta}(x_{0})$ using the given samples. The procedure is completed
in two steps: the forward and the reverse diffusion processes. The former
continuously injects Gaussian noise in $x_{0}$ to create a Markov chain with
latents $x_{1:K}$ following transitions:

$q(x_{1:K}|x_{0})=\prod_{k=1}^{K}q(x_{k}|x_{k-1}),$ (1)

where
$q(x_{k}|x_{k-1})~{}=~{}\mathcal{N}(x_{k};\sqrt{1-\beta_{k}}x_{k-1},\beta_{k}\mathbf{I})$
is the per-step noise injection following variance schedule
$\beta_{1},\dots,\beta_{K}$. This leads to the distribution
$q(x_{k}|x_{0})~{}=~{}\mathcal{N}(x_{k};\sqrt{\bar{\alpha}_{k}}x_{0},~{}(1-\bar{\alpha}_{k})~{}\mathbf{I})$
following notations introduced in [37] as $\alpha_{k}=1-\beta_{k}$ and
$\bar{\alpha}_{k}=\prod_{i=1}^{k}\alpha_{i}$. Note that
$\bar{\alpha}_{K}\approx 0$ and thus $x_{K}\sim\mathcal{N}(0,\mathbf{I})$. The
reverse diffusion learns to denoise the data starting from $x_{K}$ and
following
$p_{\theta}(x_{k-1}|x_{k})~{}=~{}\mathcal{N}(x_{k-1};\mu_{\theta}(x_{k},k),\beta_{k}\mathbf{I})$
where

$\mu_{\theta}(x_{k},k)=\frac{1}{\sqrt{\alpha_{k}}}\Big{(}x_{k}-\frac{\beta_{k}}{\sqrt{1-\bar{\alpha}_{k}}}\epsilon_{\theta}(x_{k},k)\Big{)}.$
(2)

The parameterized model $\epsilon_{\theta}(x_{k},k)$ is called the score-
function, and it is trained to predict the perturbations and the noising
schedule by the score-matching objective [38]

$\arg\min_{\theta}\mathbb{E}_{x_{0}\sim
q,\epsilon\sim\mathcal{N}(0,\mathbf{I})}\Big{[}\|\epsilon-\epsilon_{\theta}(\sqrt{\bar{\alpha}_{k}}x_{0}+\sqrt{1-\bar{\alpha}_{k}}\epsilon)\|^{2}\Big{]}$
(3)

In particular, such a score function represents the gradients of the learned
probability distribution as

$\nabla_{x_{k}}\log
p_{\theta}(x_{k})=-\frac{1}{\sqrt{1-\bar{\alpha}_{k}}}\epsilon_{\theta}(x_{k},k).$
(4)

Figure 3: Forward and Reverse Diffusion Process The above figure shows the
forward diffusion and the reverse denoising and sampling process of
ReorientDiff. As described in Section V, following classifier-free guidance
will result in high-likelihood samples with high-variance in terms of success
feasibility of the samples. Using the feasibility score gradients, we realize
an implicit iterative pose refinement, as marked by the blue box in the
figure. This significantly decrease variance and ensure high success
feasibility of the samples.

##  IV Reorientation

Following the previous environment setup by Wada et al. [3], we construct the
reorientation scenario as a task of i) selecting an object of interest from a
pile of cluttered objects, ii) calculating feasible grasp poses for picking,
iii) calculating grasp poses for placement with prior knowledge of the mesh of
the selected object and iv) finding suitable reorientation poses using our
proposed pipeline based on diffusion models. This section describes the
pipeline for creating a generic scene and task embedding space, followed by
generating grasp poses and training the feasibility score models.

###  IV-A Constructing Generic Scene-Task Representations

We define a scene as the location and occupancy of the place from where a
target object should be picked and a task as the language prompt containing
the descriptions for selecting the target object and deciding placement poses.
A top-down RGB-D camera provides an image $\mathcal{I}\in\mathbb{R}^{H\times
W\times 3}$ and a heightmap $\mathcal{H}\in\mathbb{R}^{H\times W\times 1}$ as
the description of the pile. Motivated from previous work [6, 24] on learning
semantic and spatial embeddings, we use pre-trained CLIP foundation model for
obtaining semantic embeddings from the image $\mathcal{I}$ and language
$\mathcal{L}$, and combine them with spatial embeddings for target object
segmentation to get a joint embedding $\Phi$ as generic scene-task
representation as shown in Fig. 2. The embedding is further used to predict
the target object as a one-hot vector of all the objects of interest and the
final placement pose.

###  IV-B Sampling Grasp Poses

We generate grasp poses by following the classical approach of converting the
heightmap into a point cloud representation and eventually to a point-normal
representation. The predicted target object segmenatation of the scene is then
used to obtain the surface normals of the target object. After performing an
edge masking using the Laplacian of the surface normals, the remaining point-
normals on the surface are feasible grasp poses. While we sample grasp poses
$\eta_{1}$ for picking the object from the pile in the aforementioned manner,
we assume that we have the mesh of the selected object for sampling grasp
poses $\eta_{2}$ for placing the object at the predicted pose.

###  IV-C Training Feasibility Score Models

Following prior works [11, 3, 29], a feasibility prediction model is important
for early-evaluation and rejection of unfavourable samples. Such a feasibility
model predicts the probability of success of a given grasp pose in
successfully grasping an object in some candidate pose for a specified scene
representation. The phenomenon of grasp success evaluation in dynamic
reorientation pose, as addressed by [3], is of particular interest for our
setup. Modelling dynamics for every object is indeed non-trivial and adds to
the complexity; hence the feasibility model implicitly takes care of the
dynamics of the object after deactivating the grasp. For checking feasibility
or the probability of success ($y$) of sampled grasps for candidate
reorientation poses $\mathbf{q}$, we train two models:

  * •

For predicting success of reorientation from the current pose in a pile to a
candidate pose given pick grasp poses ($\eta_{1}$) and scene representation,
denoted as $\mathcal{M}_{1}(y|\eta_{1},\mathbf{q},\Phi)$

  * •

For predicting success of post-grasp deactivation pose from the candidate pose
and placement grasp poses ($\eta_{2}$), denoted as
$\mathcal{M}_{2}(y|\eta_{2},\mathbf{q},\Phi)$

##  V ReorientDiff: Diffusion for Reorientation

We aim to generate intermediate reorientation poses for the target object,
which enables successive placement at the desired pose and is reachable from
the current pose. We introduce a diffusion model based approach to sample most
probable successful reorientation poses ($\mathbf{q}$) conditioned on the
scene representation priors ($\Phi$), denoted as $p(\mathbf{q}|\Phi)$, which
already contains the spatial and semantic information about the scene and the
task. The denoising process can be further flexibly conditioned by sampling
from modified distributions of the form

$p_{h}(\mathbf{q})\propto p(\mathbf{q}|\Phi)h(\mathbf{q},\Phi),$ (5)

where $h(\mathbf{q},\Phi)$ can represent several grasp success probability
heuristics. By separating the grasp success from reorientation candidate
sampling, the diffusion model trained for reorientation poses can be reused
for varied selection of picking ($\eta_{1}$) and placement grasp poses
($\eta_{2}$).

###  V-A Classifier-free Conditional Pose Generation

Following the distribution defined in (5), we use classifier-free guidance
[12] to sample high-likelihood reorientation poses for a particular scene-task
representation. We train a score-network [38],
$\epsilon_{\theta}(\mathbf{q}_{k},\Phi)\propto\nabla_{\mathbf{q}_{k}}\log
p(\mathbf{q}_{k}|\Phi)$ , to denoise from
$\mathbf{q}_{K}\sim\mathcal{N}(\bf{0},\mathbf{I})$ to possible reorientation
poses $\mathbf{q}_{0}$ from a $K$-step reverse diffusion denoising process.
For each step, we calculate $\tilde{\epsilon}_{k}$ as

$\tilde{\epsilon}_{k}=\epsilon_{\theta}(\mathbf{q}_{k},\Phi)+w_{c}\Big{(}\epsilon_{\theta}(\mathbf{q}_{k},\Phi)-\epsilon_{\theta}(\mathbf{q}_{k},\o)\Big{)}$
(6)

The scalar $w_{c}$ implicitly guides the reverse-diffusion towards poses that
best satisfy the scene-task representations. Further, we calculate the
successive samples for the next $(k-1)^{th}$ step using the DDIM [37] sampling
strategy and $\tilde{\epsilon}_{k}$ as follows:

$\displaystyle\tilde{\mathbf{q}}_{k-1}$
$\displaystyle\longleftarrow\sqrt{\bar{\alpha}_{k-1}}\Big{(}\frac{\mathbf{q}_{k}-\sqrt{1-\bar{\alpha}_{k}}\;\;\tilde{\epsilon}_{k}}{\sqrt{\bar{\alpha}_{k}}}\Big{)}+\sqrt{1-\bar{\alpha}_{k-1}}\;\;\tilde{\epsilon}_{k}$
(7)

where, $\bar{\alpha}_{k}$ is as described in Section III.

###  V-B Feasibility Guided Pose Refinement

We use the two feasibility-score prediction models ($\mathcal{M}_{1}$ and
$\mathcal{M}_{2}$), which are pre-trained for predicting grasp feasibility for
picking grasp, reorientation pose pairs and placement grasp, reorientation
pose pairs, respectively. In such a case, the scores can be converted into
probability distributions for each heuristic, defined as, for each $i=1,2$,

$h_{i}\equiv\
p(y=1|\eta_{i},\mathbf{q},\Phi)|_{\mathcal{M}_{i}}=\text{exp}\Big{(}-(1-\mathcal{M}_{i}(y|\eta_{i},\mathbf{q},\Phi))^{2}\Big{)}$

Following classifier-based guidance [13] formulation for the heuristics, the
reverse diffusion can be formulated as:

$p_{h}(\mathbf{q}_{k}|\mathbf{q}_{k+1},y,\Phi)\propto\\\
p(\mathbf{q}_{k}|\mathbf{q}_{k+1},\Phi)\;\;p(y|\eta_{1},\hat{\mathbf{q}}_{0}^{k},\Phi)|_{\mathcal{M}_{1}}\
p(y|\eta_{2},\hat{\mathbf{q}}_{0}^{k},\Phi)|_{\mathcal{M}_{2}}$ (8)

where, $\hat{\mathbf{q}}_{0}^{k}$ is the sample proposed at diffusion step $k$
and defined as:

$\hat{\mathbf{q}}_{0}^{k}=\frac{\mathbf{q}_{k}-\sqrt{1-\bar{\alpha}_{k}}\;\;\tilde{\epsilon}_{k}}{\sqrt{\bar{\alpha}_{k}}}$
(9)

Considering Taylor first order approximations for heuristics and standard
reverse process Gaussian
$(\mu_{\theta}(\mathbf{q}_{k},k,\Phi),\beta_{k}\mathbf{I})$ as described in
Section III, we get the new mean ($\mu_{\theta,h}(\mathbf{q}_{k},k,\Phi)$) for
the distribution $p_{h}(\mathbf{q}_{k}|\mathbf{q}_{k+1},y,\Phi)$ in (8) as:

$\displaystyle\mu_{\theta,h}(\mathbf{q}_{k},k,\Phi)$
$\displaystyle=\mu_{\theta}(\mathbf{q}_{k},k,\Phi)+\beta_{k}\sum_{i=1}^{2}w_{i}\nabla_{\mathbf{q}_{k}}\log
p(y|\eta_{i},\mathbf{q}_{k},\Phi)|_{\mathcal{M}_{i}}$
$\displaystyle=\mu_{\theta}(\mathbf{q}_{k},k,\Phi)-\beta_{k}\sum_{i=1}^{2}w_{i}\nabla_{\mathbf{q}_{k}}\Big{[}1-\mathcal{M}_{i}(y|\eta_{i},\hat{\mathbf{q}}_{0}^{k},\Phi)\Big{]}^{2}.$

In view of (2), we then obtain the modified score

$\epsilon_{k}\longleftarrow\tilde{\epsilon}_{k}-\sqrt{1-\bar{\alpha}_{k}}\
g_{k}$

where
$g_{k}=-\beta_{k}\sum_{i=1}^{2}w_{i}\nabla_{\mathbf{q}_{k}}\Big{[}1-\mathcal{M}_{i}(y|\eta_{i},\hat{\mathbf{q}}_{0}^{k},\Phi)\Big{]}^{2}$.
We notice that injecting noise to $g_{k}$, as in stochastic DDIM, can slightly
improve the performance. We calculate the final $\mathbf{q}_{k-1}$ using the
refined $\epsilon_{k}$ in (7). A visual clarification of the forward and
reverse diffusion is shown in Fig. 3.

Figure 4: Visual Analysis of Scene-Task Network Performance The scene-task
network maps the visual (row $2$) image of the pile (row $1$) and language
(bottom row) inputs to a feature space which is used to predict the placement
location (row $4$) and target object segmentation (row $3$). Figure 5: Reverse
Diffusion for Reorientation Pose Generation The reverse sampling process for
$4$ $k$-steps at $k=100,64,32,0$ for $K=256$ in four different scene-task
scenarios comprising of the Cracker Box, Mustard Bottle and Sugar Box in
different target orientations are shown above. The scenes are shown in the
left-side of every sub-figure and consists of the pile with the target object
and the predicted placement location on the shelf. The language prompt
defining each of the tasks is mentioned below each sub-figure. It consists of
either the absolute (the object’s name) or the relative (heaviest/lightest)
reference to the object and details about the target placement.

##  VI Results: Simulation

Based on the environment setup as discussed in Section IV, we create datasets,
train diffusion and feasibility score models and evaluate them in simulation
for proper placement conditions.

###  VI-A Dataset Generation and Training

We use PyBullet [39] and an OMPL [40] based motion planner to solve for
collision-free path between current pose and a candidate reorientation pose
and from the reorientation pose to the ground-truth placement pose for diverse
set of YCB-objects and target locations. We sampled approximately $40000$
candidate poses following Wada et al. [3]. The goal properties were converted
into modular language instructions, and the success of pick and place for both
the steps was recorded. The scene and task properties were used to construct
the joint visual-language embedding space, which was further used to train the
feasibility score models using binary success labels. Eventually, we train a
conditional diffusion model using only the successful reorientation poses.
Such a diffusion model is reusable for diverse set of grasp poses based on the
feasibility score models.

###  VI-B Performance Evaluation: Scene-Task Representation

To evaluate the quality of the scene-task embedding network, we analyze the
accuracy of the object selection and placement pose prediction along with the
error in the predicted segmentation. We show a visual analysis in Fig. 4 where
the output segmentation and the predicted placement pose in the shelf are
shown for three scenes and tasks. For accurate shelf-level estimation, we
round each object’s predicted height to the nearest shelf-level height, and a
similar post-processing is conducted for the object orientation. To add
complexity, although we consider only four orientations: front, back, left and
right, we discretize the possible orientations into $8$ possible options and
round the predicted orientation to the nearest option.

Numerically, the object selection network was $100\%$ accurate, and the number
of pixels wrongly classified was about $1\%$ of the complete image on average
over $100$ random samples. The average error in predicting the height of the
target placement after post-processing is around $8$ mm, and the mean error in
the yaw angle of the predicted pose is $0.3$ rad.

###  VI-C Performance Evaluation: Diffusion with Guidance

The trained classifier-free conditional diffusion model and the score
feasibility models are used to perform the reverse diffusion using the
classifier-free guidance with and without feasibility score guidance.
Experiments comparing the performance of both the methods are shown in Fig. 5
for a set of YCB Objects [14] and different scene-task scenarios where only
$50$ candidate poses are sampled and top $10$ high-likelihood poses are
selected. The comparison shows that while the classifier-free guidance is good
enough to sample high-likelihood reorientation poses, the primary purpose of
the feasibility score gradients is to reduce the variance in the pose
generation and ensure high success probability. A numerical analysis of the
overall success is shown and compared with the rejection sampling based
baseline [3] in Table I.

TABLE I: Success evaluation of the proposed method as compared to the rejection sampling based baseline. The ReorientDiff algorithm was tested for more than 300 different scene task settings consisting of equal distribution of the selected objects and all the orientations. A task is considered a success if it is completed at-least once in 4 random seeds. Method |  | Success (%)
---
Reorient
| Success (%)
---
Place
| Success (%)
---
Overall
ReorientBot | 97.9 | 95.1 | 93.2
| ReorientDiff
---
(w/o Guide)
97.4 | 86.3 | 85.8
ReorientDiff | 98.9 | 96.5 | 94.8

The reorientation success percentage holds different relevance as compared to
the baseline. The baseline does two step reverse rejection sampling where
reorientation search is conducted over candidates which are feasible for
placement, so there might be a scenario where there is no solution possible.
For the case of ReorientDiff, the reorientation success measures the
capability of the diffusion model to generalize to poses which ensure
reorientability and scope for future placement. Higher reorientation success
and lower placement success is an indication that the model is short-sighted
and is giving importance to a single step success metric. From Table I, we
ensure high reorientability success along with better placement success, even
without any candidate pose discretization. The overall success is based on the
accurate placement of the object from the reoriented pose, and it represents
the successful completion of a task. The metric is measured by calculating the
difference between the desired and the pose after final placement.

TABLE II: Success evaluation with different levels of discretization while sampling using ReorientDiff. ReorientDiff K |  | Success (%)
---
Reorient
| Success (%)
---
Place
| Success (%)
---
Overall
$K=256$ | 98.9 | 96.5 | 94.8
$K=100$ | 99.3 | 92.4 | 91.5
$K=50$ | 97.5 | 91.1 | 88.6

###  VI-D Performance Evaluation: K-Step Reverse Diffusion

Sampling from a trained diffusion models is flexible and can be achieved using
different levels of discretization between
$x_{K}\sim\mathcal{N}(0,\mathbf{I})$ to meaningful reorientation poses. We
perform the complete analysis for multiple values of the number of reverse
denoising steps $K$ as shown in Table II. It is evident that minimizing
resolution degrades overall performance, but even with much fewer resolutions,
ReorientDiff can reach a descent performance.

Following our analysis on performance, we explored the time taken for overall
planning of a successful reorientation pose from a given scene and
corresponding task information. We provide the recorded timings for all of our
ablations as well as the baseline in Table III.

TABLE III: Computational analysis of the planning time for finding a suitable reorientation pose for the proposed method, ReorientDiff, along with the baseline and all conducted ablations. Method |  | Planning
---
Time (sec)
ReorientBot | 2.5
ReorientDiff (w/o Guide) | 2.7
ReorientDiff @ $K=256$ | 5.3
ReorientDiff @ $K=100$ | 2.5
ReorientDiff @ $K=50$ | 1.5

Our findings show that ReorientDiff is computationally heavy due to gradient
calculations for reverse denoising steps. Without using the guidance from the
feasibility-score models, classifier-free guidance requires similar time as
the baseline, ReorientBot. However, as we decrease the discretization
resolution, the planning time decreases significantly with some trade-off in
performance, as shown in Table II. We believe that using higher-order solvers
such as one proposed in [41], similar level of performance as ReorientDiff (w/
$K=256$) can be achieved at the computation cost of $K=50$. However, such an
analysis is out of scope of the proposed methodology. Hence, from all of our
visual and empirical analysis, ReorientDiff successfully proves that
formulating the problem of reorientation as learning a conditional
distribution is an effective way to move towards more generalizable object
manipulation.

##  VII Conclusion

Diffusion models are powerful generative models capable of modeling
(conditional) distributions. In the proposed method, ReorientDiff exploits the
capabilities of such models to predict reorientation poses conditioned on a
compact scene-task representation embedding containing information about the
target object and its placement location. Further, the samples are refined
using learned feasibility-score models to reduce uncertainty and ensure
success of the planned intermediate poses. Considering as little as $10$
reorientation poses, we achieved an overall success rate of $96.5$% across
variety of objects. We consider incorporating more efficient sampling schemes
and better generalization performance for unseen objects and placement goals
as a potential future work.

## References

  * [1] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, et al., “Transporter networks: Rearranging the visual world for robotic manipulation,” in Conference on Robot Learning, pp. 726–747, PMLR, 2021.
  * [2] B. Tang and G. S. Sukhatme, “Selective object rearrangement in clutter,” in 6th Annual Conference on Robot Learning, 2022.
  * [3] K. Wada, S. James, and A. J. Davison, “Reorientbot: Learning object reorientation for specific-posed placement,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 8252–8258, IEEE, 2022.
  * [4] J. J. Kuffner and S. M. LaValle, “Rrt-connect: An efficient approach to single-query path planning,” in Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), vol. 2, pp. 995–1001, IEEE, 2000\.
  * [5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning, pp. 8748–8763, PMLR, 2021.
  * [6] M. Shridhar, L. Manuelli, and D. Fox, “Cliport: What and where pathways for robotic manipulation,” in Conference on Robot Learning, pp. 894–906, PMLR, 2022.
  * [7] M. Janner, Y. Du, J. Tenenbaum, and S. Levine, “Planning with diffusion for flexible behavior synthesis,” in International Conference on Machine Learning, 2022.
  * [8] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal, “Is conditional generative modeling all you need for decision-making?,” arXiv preprint arXiv:2211.15657, 2022.
  * [9] J. Urain, N. Funk, G. Chalvatzaki, and J. Peters, “Se (3)-diffusionfields: Learning cost functions for joint grasp and motion optimization through diffusion,” arXiv preprint arXiv:2209.03855, 2022.
  * [10] W. Liu, T. Hermans, S. Chernova, and C. Paxton, “Structdiffusion: Object-centric diffusion for semantic rearrangement of novel objects,” arXiv preprint arXiv:2211.04604, 2022.
  * [11] A. Mousavian, C. Eppner, and D. Fox, “6-dof graspnet: Variational grasp generation for object manipulation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2901–2910, 2019.
  * [12] J. Ho and T. Salimans, “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022.
  * [13] P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,” Advances in Neural Information Processing Systems, vol. 34, pp. 8780–8794, 2021.
  * [14] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The ycb object and model set: Towards common benchmarks for manipulation research,” in 2015 international conference on advanced robotics (ICAR), pp. 510–517, IEEE, 2015.
  * [15] J. Mahler, J. Liang, S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, “Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics,” arXiv preprint arXiv:1703.09312, 2017.
  * [16] J. Mahler, M. Matl, X. Liu, A. Li, D. Gealy, and K. Goldberg, “Dex-net 3.0: Computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning,” in 2018 IEEE International Conference on robotics and automation (ICRA), pp. 5620–5627, IEEE, 2018.
  * [17] J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg, “Learning ambidextrous robot grasping policies,” Science Robotics, vol. 4, no. 26, p. eaau4984, 2019.
  * [18] M. Zhu, K. G. Derpanis, Y. Yang, S. Brahmbhatt, M. Zhang, C. Phillips, M. Lecce, and K. Daniilidis, “Single image 3d object detection and pose estimation for grasping,” in 2014 IEEE International Conference on Robotics and Automation (ICRA), pp. 3936–3943, IEEE, 2014.
  * [19] A. Zeng, K.-T. Yu, S. Song, D. Suo, E. Walker, A. Rodriguez, and J. Xiao, “Multi-view self-supervised deep learning for 6d pose estimation in the amazon picking challenge,” in 2017 IEEE international conference on robotics and automation (ICRA), pp. 1386–1383, IEEE, 2017.
  * [20] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox, “Self-supervised 6d object pose estimation for robot manipulation,” in 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3665–3671, IEEE, 2020.
  * [21] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng, “Learning to rearrange deformable cables, fabrics, and bags with goal-conditioned transporter networks,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4568–4575, IEEE, 2021.
  * [22] L. Pinto and A. Gupta, “Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours,” in 2016 IEEE international conference on robotics and automation (ICRA), pp. 3406–3413, IEEE, 2016.
  * [23] A. Zeng, S. Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, et al., “Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching,” The International Journal of Robotics Research, vol. 41, no. 7, pp. 690–705, 2022.
  * [24] M. Shridhar, L. Manuelli, and D. Fox, “Perceiver-actor: A multi-task transformer for robotic manipulation,” arXiv preprint arXiv:2209.05451, 2022.
  * [25] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, vol. 1, no. 8, p. 9, 2019.
  * [26] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * [27] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi, “Simple but effective: Clip embeddings for embodied ai,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14829–14838, 2022\.
  * [28] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans, “Objectnav revisited: On evaluation of embodied agents navigating to objects,” arXiv preprint arXiv:2006.13171, 2020.
  * [29] W. Liu, C. Paxton, T. Hermans, and D. Fox, “Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects,” in 2022 International Conference on Robotics and Automation (ICRA), pp. 6322–6329, IEEE, 2022.
  * [30] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al., “Do as i can, not as i say: Grounding language in robotic affordances,” arXiv preprint arXiv:2204.01691, 2022.
  * [31] S. Cheng, K. Mo, and L. Shao, “Learning to regrasp by learning to place,” in 5th Annual Conference on Robot Learning, 2021.
  * [32] P. Xu, Z. Chen, J. Wang, and M. Q.-H. Meng, “Planar manipulation via learning regrasping,” arXiv preprint arXiv:2210.05349, 2022.
  * [33] M. Wermelinger, R. Johns, F. Gramazio, M. Kohler, and M. Hutter, “Grasping and object reorientation for autonomous construction of stone structures,” IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 5105–5112, 2021.
  * [34] Y. Dai, M. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and P. Abbeel, “Learning universal policies via text-guided video generation,” arXiv preprint arXiv:2302.00111, 2023.
  * [35] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al., “Scaling robot learning with semantically imagined experience,” arXiv preprint arXiv:2302.11550, 2023\.
  * [36] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851, 2020.
  * [37] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” arXiv preprint arXiv:2010.02502, 2020.
  * [38] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” arXiv preprint arXiv:2011.13456, 2020.
  * [39] E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning.” http://pybullet.org, 2016–2021.
  * [40] I. A. Şucan, M. Moll, and L. E. Kavraki, “The Open Motion Planning Library,” IEEE Robotics & Automation Magazine, vol. 19, pp. 72–82, December 2012.  https://ompl.kavrakilab.org.
  * [41] Q. Zhang and Y. Chen, “Fast sampling of diffusion models with exponential integrator,” arXiv preprint arXiv:2204.13902, 2022.