#  Points2NeRF: Generating Neural Radiance Fields from 3D point cloud

D. Zimny, T. Trzciński, P. Spurek
Faculty of Mathematics and Computer Science,
Jagiellonian University, Kraków, Poland
<EMAIL_ADDRESS>

###### Abstract

Contemporary registration devices for 3D visual information, such as LIDARs
and various depth cameras, capture data as 3D point clouds. In turn, such
clouds are challenging to be processed due to their size and complexity.
Existing methods address this problem by fitting a mesh to the point cloud and
rendering it instead. This approach, however, leads to the reduced fidelity of
the resulting visualization and misses color information of the objects
crucial in computer graphics applications. In this work, we propose to
mitigate this challenge by representing 3D objects as Neural Radiance Fields
(NeRFs). We leverage a hypernetwork paradigm and train the model to take a 3D
point cloud with the associated color values and return a NeRF network’s
weights that reconstruct 3D objects from input 2D images. Our method provides
efficient 3D object representation and offers several advantages over the
existing approaches, including the ability to condition NeRFs and improved
generalization beyond objects seen in training. The latter we also confirmed
in the results of our empirical evaluation.

##  1 Introduction

3D registration devices, such as LIDARs and depth cameras, have become
ubiquitous across a multitude of contemporary computer vision applications,
including autonomous driving Yang et al. (2018a) or robotic manipulation Kehoe
et al. (2015). They capture the surrounding 3D visual data as point clouds -
structures that require large memory and computational costs to be processed
and rendered. To increase the efficiency of processing the point clouds,
current approaches transform them into regular 3D voxel grids or collections
of images Su et al. (2015); Wu et al. (2015). This, however, increases the
memory footprint of object representations and leads to reduced fidelity of
the resulting visualization by omitting color information required , e.g. for
computer graphics applications.

Recently introduced approaches attempt to mitigate those challenges by
representing 3D objects as meshes Spurek et al. (2020, 2021c); Wang et al.
(2018a); Wen et al. (2022) or voxels Chen and Zhang (2019); Proszewska et al.
(2021), rather than 3D point clouds, which makes them easier to be rendered.
Unfortunately, the former approaches assume that the resulting meshes are
topologically coherent with spheres which reduces the generality of this
approach. In contrast, the latter methods assume voxelized training datasets
and are hard to generalize beyond objects provided during training.

Figure 1:  Our Points2NeRF approach takes a 3D point cloud with the associated
color values and returns the weights of a NeRF network that reconstructs 3D
objects with high fidelity and coherent coloring.

This paper addresses all the above problems by representing objects as Neural
Radiance Fields (NeRF) Mildenhall et al. (2020). NeRF represents a static
scene as a continuous 5D function that outputs the radiance emitted in each
direction $(\theta,\phi)$ at each point $(x,y,z)$ in space, and a density at
each point which acts like a differential opacity controlling how much
radiance is accumulated by a ray passing through $(x,y,z)$. The vanilla NeRF
method optimizes a deep fully-connected neural network to represent this
function by regressing from a single 5D coordinate $(x,y,z,\theta,\phi)$ to a
single volume density and view-dependent RGB color. In this work, we introduce
a Points2NeRF111We make our implementation available at
https://github.com/gmum/points2nerf.  method that leverages a hypernetworks
architecture Ha et al. (2016) to produce a NeRF representation of a 3D point
cloud, as Fig. 1 shows. The main intuition behind our approach is to train a
hypernetwork that takes a 3D point cloud with colors as an input and returns
the parameters of the target network – NeRF. As a result, we obtain a set of
network parameters that can be interpreted as a continuous parametrization of
the 3D objects. Furthermore, the resulting NeRFs are easy to render and can be
converted to meshes with colors. Last but not least, this formulation allows
for conditioning NeRF models and offers improved generalization beyond 3D
objects seen during training.

To summarize, our contributions of this work are the following:

  * •

We propose a new method dubbed Points2NeRF which adapts a hypernetwork
framework to the NeRF architecture, and hence allows to produce Radiance
Fields from 3D point cloud.

  * •

Our approach enables conditioning NeRFs, which, in turn, allows to generalize
the model beyond 3D objects seen in training.

  * •

Lastly, our method offers a generative model that can represent 3D objects as
NeRF parameters in a continuous manner, enabling interpolation within the
object space.

##  2 Related works

3D objects can be represented by using various techniques, including voxel
grids (Choy et al., 2016; Girdhar et al., 2016; Liao et al., 2018; Wu et al.,
2016), octrees (Häne et al., 2017; Riegler et al., 2017; Tatarchenko et al.,
2017; Wang et al., 2018b), multi-view images (Arsalan Soltani et al., 2017;
Lin et al., 2018; Su et al., 2015), point clouds (Achlioptas et al., 2018; Fan
et al., 2017; Qi et al., 2017a, b; Yang et al., 2018b), geometry images (Sinha
et al., 2016, 2017), deformable meshes (Girdhar et al., 2016; Sinha et al.,
2017; Wang et al., 2018a; Yang et al., 2018b), and part-based structural
graphs (Li et al., 2017; Zhu et al., 2018).

Mentioned above representations are discreet, which is a substantial
limitation in real-life applications. Alternatively, we can represent 3D
objects as a continuous function (Dupont et al., 2021). In practice implicit
occupancy (Chen and Zhang, 2019; Mescheder et al., 2019; Peng et al., 2020),
distance field (Michalkiewicz et al., 2019; Park et al., 2019) and surface
parametrization (Yang et al., 2019; Spurek et al., 2020, 2021c; Cai et al.,
2020) models use a neural network to parameterize a 3D object. We do not have
a fixed number of voxels, points, or vertices in such a case, but we represent
shapes as a continuous function.

In Mescheder et al. (2019) the occupancy networks represent the 3D surface as
a continuous decision boundary of a deep classifier. In (Chen and Zhang, 2019;
Peng et al., 2020) the authors propose an implicit decoder (IM-NET) which uses
a binary classifier that takes a point coordinate concatenated with a feature
vector encoding a shape and outputs an inside or outside label.

DeepSDF Park et al. (2019) represents the surface of the object by a
continuous volumetric field. Points represent the distance to the surface
boundary, and the sign indicates whether the region is inside or outside. Such
representation parameterizes a shape’s border as a classification boundary.

In (Spurek et al., 2020, 2021c) the authors propose HyperCloud model that uses
a hypernetwork to output weights of a generative network to create 3D point
clouds instead of generating a fixed size reconstruction. One neural network
is trained to produce a continuous representation of an object. In (Yang et
al., 2019) authors propose to use a conditioning mechanism to produce a flow
model which transfers gaussian noise into a 3D object.

These models are limited by their requirement of access to ground truth 3D
geometry. Recently works relaxed this requirement of ground truth 3D shapes by
using only 2D images. In Niemeyer et al. (2020) authors present 3D occupancy
fields. The numerical method is used to find the surface intersection for each
ray. In Sitzmann et al. (2019) propose neural network which produces feature
vector and RGB color at each continuous 3D coordinate, and propose a
differentiable rendering function consisting of a recurrent neural network.

Above models are limited to simple shapes with low geometric complexity,
resulting in over smoothed renderings. To solve such a problem NeRF Mildenhall
et al. (2020) model was proposed. NeRF represents a static scene as a
continuous 5D function that outputs the radiance emitted in each direction at
each point and a density at each point which acts like a differential opacity
controlling how much radiance is accumulated by a ray passing through the
point.

The NeRF method is a state-of-the-art solution for representing 3D objects.
However, there are many different generalizations of the model for static
Mildenhall et al. (2020) as well as dynamic scene Kania et al. (2021).

Information from point clouds was used in NeRF across different applications.
In Xu et al. (2022) authors present a novel neural scene representation Point-
NeRF that models a volumetric radiance field with a neural point cloud. In
NeuS Wang et al. (2021), authors propose to represent a surface as the zero-
level set of a signed distance function (SDF) and develop a new volume
rendering method to train a neural SDF representation. In consequence, we
obtain a novel neural surface reconstruction method. In Ost et al. (2021)
authors introduce Neural Point Light Fields that represent scenes implicitly
with a light field living on a sparse point cloud. In our work, we use point
cloud as a conditional for producing NeRF representation.

Most of such models are trained on a single scene. NeRF-VAE Kosiorek et al.
(2021) is a 3D scene generative model. In contrast to NeRF, such a model
considers shared structure across scenes. Unfortunately, the model was trained
only on simple scenes containing geometric figures. In our paper, we present
Points2NeRF, which is trained on an extensive data set and can transform a 3D
point cloud to NeRF representation.

##  3 Points2NeRF: generating Neural Radiance Fields from 3D point cloud

In this section, we present our Points2NeRF model for building NeRF
representations of 3D point clouds. To that end, we leverage three main
components, described below: the auto-encoder architecture, the NeRF
representation of a 3D static scene, and the hypernetwork training paradigm.
The main intuition behind our approach is the construction of an auto-encoder,
which takes as an input 3D point cloud and generates weight of the target
network – NeRF. For effective NeRF training, our model requires a set of 2D
images, on top of the 3D point clouds, all of which is captured by 3D
registration devices.

### Hypernetwork

Hypernetworks, introduced in Ha et al. (2016), are defined as neural models
that generate weights for a separate target network solving a specific task.
The authors aim to reduce the number of trainable parameters by designing a
Hypernetwork with a smaller number of parameters. Making an analogy between
Hypernetworks and generative models, the authors of Sheikh et al. (2017), use
this mechanism to generate a diverse set of target networks approximating the
same function.

In the context of 3D objects, various methods make use of a hypernetwork to
produce a continuous representation of objects Proszewska et al. (2021);
Spurek et al. (2021a, 2020, b, c). HyperCloud (Spurek et al., 2020) represent
3D point cloud as a classical NLP while in Spurek et al. (2021c) by a
Continuous Normalizing Flow Grathwohl et al. (2018). In the case of Proszewska
et al. (2021) authors model voxel representation by hypernetwork architecture.

### Auto-encoders for 3D Point Clouds

In our approach, we use hypernetwork paradigm to aggregate information from 3D
point cloud representation and produce the weight of NeRF architecture.
Moreover, such a solution allows us to create a high-resolution model of the
3D point cloud.

In Points2NeRF hypernetwork is an auto-encoder type architecture for the 3D
point cloud. Let
$\mathcal{X}=\\{X_{i}\\}_{i=1,\ldots,n}=\\{(x_{i},y_{i},z_{i},r_{i},g_{i},b_{i})\\}_{i=1,\ldots,n}$
be a given dataset containing point clouds with colors. The first three
elements encode the position while the last three encode an RGB color. The
basic aim of autoencoder is to transport the data through a typically, but not
necessarily, lower-dimensional latent space
$\mathcal{Z}\subseteq\mathbb{R}^{D}$ while minimizing the reconstruction
error. Thus, we search for an encoder $\mathcal{E}:X\to Z$ and decoder
$\mathcal{D}:Z\to X$ functions, which minimizes the reconstruction error
between $X_{i}$ and its reconstructions $\mathcal{D}(\mathcal{E}X_{i})$. We
use a permutation invariant encoder that is based on PointNet architecture Qi
et al. (2017a) and a modified decoder to produce weight instead of row points.
For point cloud representation, the crucial step is to define proper
reconstruction loss that can be used in the auto encoding framework. In our
approach, we use NeRF cost function.

Figure 2: Reconstruction of object obtained by Points2NeRF.

### Neural Radiance Fields (NeRF)

NeRF Mildenhall et al. (2020) achieves state-of-the-art results for
synthesizing novel views of complex scenes. The algorithm represents a scene
using a fully-connected architecture. On the input NeRF takes 5D coordinate
(spatial location ${\bf x}=(x,y,z)$ and viewing direction $(\theta,\psi)$) and
whose output is an emitted color ${\bf c}=(r,g,b)$ and volume density
$\sigma$. Since volume rendering is naturally differentiable, the only input
required to optimize NeRF is a set of images with known camera poses.

In practice, NeRF express direction as a 3D Cartesian unit vector ${\bf d}$.
We approximate this continuous 5D scene representation with an MLP network
$F_{\Theta}:({\bf x},{\bf d})\to({\bf c},\sigma)$ and optimize its weights
$\Theta$ to map from each input 5D coordinate to its corresponding volume
density and directional emitted color. The MLP $F_{\Theta}$ first processes
the input 3D coordinate ${\bf x}$ with $8$ fully-connected layers, and outputs
$\sigma$ and a $256$-dimensional feature vector. This feature vector is then
concatenated with the camera ray’s viewing direction and passed to one
additional fully-connected layer that outputs the view-dependent RGB color.
The architecture of NeRF is relatively simple but uses many weights. Therefore
Hypernetwork must generate many parameters. Therefore we use the chunking
mechanism Von Oswald et al. (2019). In practice, we generate weight to a
single layer by conditioning our hypernetwork by the index of layers we would
like to produce.

The cost function of NeRF is inspired by classical volume rendering Kajiya and
Von Herzen (1984). We render the color of any ray passing through the scene.
The volume density $\sigma({\bf x})$ can be interpreted as the differential
probability of a ray terminating at an infinitesimal particle at location
${\bf x}$. The expected color $C({\bf r})$ of camera ray ${\bf r}(t)={\bf
o}+t{\bf d}$ with near and far bounds $t_{n}$ and $t_{f}$ is:

$C({\bf r})=\int_{t_{n}}^{t_{f}}T(t)\sigma({\bf r}(t)){\bf c}({\bf
r}(t),d)dt,\mbox{ where }T(t)=\exp(-\int_{t_{n}}^{t}\sigma({\bf r}(s))ds).$

The function $T(t)$ denotes the accumulated transmittance along the ray from
$t_{n}$ to $t$, i.e., the probability that the ray travels from $t_{n}$ to $t$
without hitting any other particle. In practice, this continuous integral is
numerically estimated using quadrature. We use a stratified sampling approach
where we partition $[t_{n},t_{f}]$ into $N$ evenly-spaced bins and then draw
one sample uniformly at random from within each bin:

$t_{i}\sim\mathcal{U}[t_{n}+\frac{i-1}{N}(t_{f}-t_{n}),t_{n}+\frac{i}{N}(t_{f}-t_{n})].$

We use these samples to estimate $C({\bf r})$ with the quadrature rule
discussed in the volume rendering review by Max Max (1995):

$\hat{C}({\bf r})=\sum_{i=1}^{N}T_{i}(1-\exp(-\sigma_{i}\delta_{i})){\bf
c}_{i},\mbox{ where
}T(t)=\exp\left(-\sum_{j=1}^{i-1}\sigma_{i}\delta_{i}\right).$

where $\delta_{i}=t_{i+1}-t_{i}$ is the distance between adjacent samples.
This function for calculating $\hat{C}({\bf r})$ from the set of
$(c_{i},\sigma_{i})$ values trivially differentiable.

We then use the volume rendering procedure to render the color of each ray
from both sets of samples. Our loss is simply the total squared error between
the rendered and true pixel colors

$\mathcal{L}=\sum_{{\bf r}\in R}\|\hat{C}({\bf r})-C(r)\|_{2}^{2}$ (1)

where $R$ is the set of rays in each batch, and $C({\bf r})$, $\hat{C}({\bf
r})$ are the ground truth and predicted RGB colors for ray r  respectively.
Contrary to the original paper of NeRF, we use only single architecture.

###  Points2NeRF

In our model we use such three components: hypernetwork, autoencoder and NeRF.
The Points2NeRF model uses Hypernetwork to output weights of generative
network to create NeRF representation from 3D point cloud. More specifically,
we present parameterization of the 3D objects as a function
$F_{\Theta}:\mathbb{R}^{5}\to\mathbb{R}^{4}$, which given location $(x,y,z)$
and viewing direction $(\theta,\psi)$) returns a color ${\bf c}=(r,g,b)$ and
volume density $\sigma$. Roughly speaking, instead of producing 3D objects, we
would like to produce many neural networks (a different neural network for
each object) that model them.

Figure 3: From NeRF representation, we can extract 3D mesh with color. Mesh
representation of objects produced by Points2NeRF. Figure 4: Interpolations
between elements produce by Generative Points2NeRF.

We have one neural network architecture that uses different weights for each
3D object in practice. The target network (NeRF) is not trained directly. We
use a Hypernetwork $\begin{array}[]{c}H_{\Phi}:\mathbb{R}^{3}\supset
X\to\Theta,\end{array}$ which for an point-cloud $X\subset\mathbb{R}^{3}$
returns weights $\Theta$ to the corresponding target network (NeRF)
$F_{\Theta}$. Thus, a point cloud $X$ is represented by a function

$\begin{array}[]{c}F((x,y,x,\theta,\psi);\Theta)=F((x,y,x,\theta,\psi);H_{\Phi}(X)).\end{array}$

To use the above model, we need to train the weights $\Phi$ of the
hypernetwork. For this purpose, we minimize the NeRF cost function over the
training set consisting of pairs: point clouds and 2D images of the object.
More precisely, we take an input point cloud $X\subset\mathbb{R}^{6}$ (the
first three elements encode the position while the last three encode an RGB
color) and pass it to $H_{\Phi}$. As a result, the hypernetwork returns
weights $\Theta$ to the target network $F_{\Theta}$. Next, the set of 2D
images is compared to the renderings generated by the target network
$F_{\Theta}$. As a hypernetwork, we use a permutation invariant encoder that
is based on PointNet architecture Qi et al. (2017a) and a modified decoder to
produce weight instead of raw points. The architecture of $H_{\Phi}$ consists
of: an encoder ($\mathcal{E}$) which is a PointNet-like network that
transports the data to lower-dimensional latent space
$\mathcal{Z}\in\mathbb{R}^{D}$ and a decoder ($\mathcal{D}$) (fully-connected
network), which transfers latent space to the vector of weights for the target
network. In our framework hypernetwork $H_{\Phi}(X)$ represents our
autoencoder structure $\mathcal{D}(\mathcal{E}X)$. Assuming
$H_{\Phi}(X)=\mathcal{D}(\mathcal{E}X)$, we train our model by minimizing the
cost function given by equation (1).

We only train a single neural model (hypernetwork), which allows us to produce
various functions at test time.

### Mesh representation

In the paper, we represent 3D objects as Neural Radiance Fields. Such
representation has few advantages over the classical one. In particular, we
can obtain 3D mesh with colors.

Thanks to volume density $\sigma$, we obtain voxel representation. We can
predict an inside/outside category for points from grid $(x,y,z)$. Then we can
render objects via the iso-surface extraction method such as Marching Cubes.
When we have mesh representation, we can predict the color for all vertices.
By using colors in vertices of mesh, we can add colors to the faces of the
graph. We present 3D objects with colors in Fig. 3.

Figure 5: Object generated by Generative Points2NeRF.

### Generative model

In our model, we use autoencoder architecture in hypernetwork. Therefore it is
easy to construct a generative model.

Autoencoder-based generative model is a classical auto-encoder model with a
modified cost function, which forces the model to be generative, i.e., ensures
that the data transported to the latent space comes from the prior
distribution (typically Gaussian) Kingma and Welling (2013); Tolstikhin et al.
(2017); Knop et al. (2020). Thus, to construct a generative auto-encoder
model, we add to its cost function to measure the distance of a given sample
from the prior distribution.

Variational Auto-encoders (VAE) are generative models that are capable of
learning approximated data distribution by applying variational inference
Kingma and Welling (2013). To ensure that the data transported to latent space
$\mathcal{Z}$ are distributed according to standard normal density. We add the
distance from standard multivariate normal density. By adding Kullback–Leibler
divergence to our cost function to obtain a generative model, which we cold
Generative Points2NeRF. In Fig. 5 we present samples an in Fig. 4
interpolations obtained by Generative Points2NeRF.

| Ours, test set | Ours, train set | NeRF
---|---|---|---
planes | 20.45 | 24.83 | 27.66
cars | 20.86 | 28.14
chairs | 17.17 | 23.90
Figure 6: Comparison between our model trained on ShapeNet data and classical
NeRF trained on 8 different objects. Five random images per object from set
were taken and mean was calculated. While NeRF used 25 test images.

##  4 Experiments

In this section, we describe the experimental results of the proposed model.
Up to our knowledge, it is the first model which obtains translation from 3D
point cloud to Neural Radiance Fields therefore it is hard to compare our
results to other algorithms. In the first subsection, we show that our model
produces high-quality NeRF representations of the objects by comparing our
model with classical NeRF dedicated to static scenes. In the second one, we
compare our model by using voxel representation obtained from NeRF.

### Methodology

We used ShapeNet dataset to train our model. First, we sampled $2048$ colored
points from each object from three categories: cars, chairs, planes. For each
object: fifty $200$x$200$ transparent background images, from random camera
positions taken from upper hemisphere were rendered.

### Point to NeRF evaluation

We compare metric reported by NeRF called PSNR (peak signal-to-noise ratio)
which is used to measure image reconstruction effectiveness. In practice, this
comparison can be seen as unfair since NeRF introduces two networks fine and
coarseMildenhall et al. (2020) which highly improves rendering of a small
detailed elements whereas we only use one coarse model with taking $256$
points along each ray.

Additionally, notice that NeRF authors report PSNR only on 8 objects from
different dataset and measure it on unseen views from objects that were
trained. In our method we compare Points2NeRF reconstructions from training
set objects as well as objects from test set. To compute that metric, we take
$5$ random images related to each cloud of point for that object and compare
it with its’ reconstruction, see Tab. 6.

We achieve comparable results only for one category but as it was previously
mentioned, Points2NeRF could compress many NeRF networks with relatively small
loss of information. It is also worth to remember, that our model could reach
those PSNR values after three weeks of training for a few thousands of object,
where one classical NeRF for one object can take up to two days.

### Voxel representation

NeRF networks, which are output of our hypernetwork can describe occupancy of
a given point in 3D space. With use of marching cubes algorithm we can obtain
mesh reconstruction for a given point cloud.

To compare reconstruction with original mesh we use Chamfer Distance defined
as distance between two cloud of points $P_{1}$ and $P_{2}$ such that:

$CD(P_{1},P_{2})=\frac{1}{2|P1|}\sum_{p_{1}\in P1}\max_{p_{2}\in
P_{2}}d(p_{1},p_{2})+\frac{1}{2|P2|}\sum_{p_{2}\in P2}\max_{p_{1}\in
P_{1}}d(p_{1},p_{2})$ (2)

Additionally, we use F-Score metric between two cloud of points with some
threshold $t$ defined as:

$\textrm{F-Score}(P_{1},P_{2},t)=\frac{2\textrm{Recall
Precision}}{\textrm{Recall + Precision}}.$ (3)

For F-Score and Chamfer Distance calculation, we sample random $3000$ points
from both original and reconstructed mesh, and we used threshold $t=0.01$ to
find matching points for F-Score.

Category | PointConvPeng et al. (2020) | ONetMescheder et al. (2019) | ConvONetTang et al. (2021) | POCOBoulch and Marlet (2022) | Ours
---|---|---|---|---|---
cars | 0.577 | 0.747 | 0.849 | 0.946 | 0.995
planes | 0.562 | 0.829 | 0.965 | 0.994 | 0.952
chairs | 0.618 | 0.730 | 0.939 | 0.985 | 0.992
Table 1: F-Score comparison between our model and other models. We trained our
model on $2048$ colored points and sampled on $3000$ while other methods used
$3000$ to train and test.

Even though our loss function was not directly related to mesh but to image
reconstruction, we were able to achieve competitive results, see Tab 1 and Tab
2.

Category | PointConvPeng et al. (2020) | ONetMescheder et al. (2019) | ConvONetTang et al. (2021) | POCOBoulch and Marlet (2022) | Ours
---|---|---|---|---|---
cars | 1.49 | 1.04 | 0.75 | 0.41 | 0.16
planes | 1.40 | 0.64 | 0.34 | 0.23 | 0.91
chairs | 1.29 | 0.95 | 0.46 | 0.33 | 0.15
Table 2: Chamfer Distance (multipied by $10^{2}$) comparison between our model
and other methods. We trained our model on $2048$ colored points and sampled
on $3000$ while other methods used $3000$ to train and test.

##  5 Conclusions

In this work, we presented a novel approach to generating NeRF representation
from 3D point clouds. Our model leverages a hypernetwork paradigm and NeRF
representation of the 3D scene. Points2NeRF take a 3D point cloud with the
associated color values and return the weights of a NeRF network that
reconstructs 3D objects from 2D images. Such representation gives several
advantages over the existing approaches. First of all, we can add a
conditioning mechanism to NeRFs that allows to control the process of creating
a generative model. Secondly, we can quickly obtain mesh representation with
colors, which is a challenging task in 3D object rendering.

### Limitations

The main limitation of Points2NeRF is the computational cost required to build
and store NeRF architecture. In practice, hypernetwork must produce many
output weights, and consequently, the training time takes up significant chunk
of time.

## References

  * Achlioptas et al. [2018] P. Achlioptas, O. Diamanti, I. Mitliagkas, and L. Guibas.  Learning representations and generative models for 3d point clouds.  In _International conference on machine learning_ , pages 40–49. PMLR, 2018.
  * Arsalan Soltani et al. [2017] A. Arsalan Soltani, H. Huang, J. Wu, T. D. Kulkarni, and J. B. Tenenbaum.  Synthesizing 3d shapes via modeling multi-view depth maps and silhouettes with deep generative networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1511–1519, 2017.
  * Boulch and Marlet [2022] A. Boulch and R. Marlet.  Poco: Point convolution for surface reconstruction.  _arXiv:2201.01831_ , 2022.
  * Cai et al. [2020] R. Cai, G. Yang, H. Averbuch-Elor, Z. Hao, S. Belongie, N. Snavely, and B. Hariharan.  Learning gradient fields for shape generation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 364–381. Springer, 2020.
  * Chen and Zhang [2019] Z. Chen and H. Zhang.  Learning implicit fields for generative shape modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5939–5948, 2019.
  * Choy et al. [2016] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese.  3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.  In _European conference on computer vision_ , pages 628–644. Springer, 2016.
  * Dupont et al. [2021] E. Dupont, Y. W. Teh, and A. Doucet.  Generative models as distributions of functions.  _arXiv preprint arXiv:2102.04776_ , 2021.
  * Fan et al. [2017] H. Fan, H. Su, and L. J. Guibas.  A point set generation network for 3d object reconstruction from a single image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 605–613, 2017.
  * Girdhar et al. [2016] R. Girdhar, D. F. Fouhey, M. Rodriguez, and A. Gupta.  Learning a predictable and generative vector representation for objects.  In _European Conference on Computer Vision_ , pages 484–499. Springer, 2016.
  * Grathwohl et al. [2018] W. Grathwohl, R. T. Chen, J. Betterncourt, I. Sutskever, and D. Duvenaud.  Ffjord: Free-form continuous dynamics for scalable reversible generative models.  _arXiv preprint arXiv:1810.01367_ , 2018.
  * Ha et al. [2016] D. Ha, A. Dai, and Q. V. Le.  Hypernetworks.  _arXiv preprint arXiv:1609.09106_ , 2016.
  * Häne et al. [2017] C. Häne, S. Tulsiani, and J. Malik.  Hierarchical surface prediction for 3d object reconstruction.  In _2017 International Conference on 3D Vision (3DV)_ , pages 412–420. IEEE, 2017.
  * Kajiya and Von Herzen [1984] J. T. Kajiya and B. P. Von Herzen.  Ray tracing volume densities.  _ACM SIGGRAPH computer graphics_ , 18(3):165–174, 1984.
  * Kania et al. [2021] K. Kania, K. M. Yi, M. Kowalski, T. Trzciński, and A. Tagliasacchi.  Conerf: Controllable neural radiance fields.  _arXiv preprint arXiv:2112.01983_ , 2021.
  * Kehoe et al. [2015] B. Kehoe, S. Patil, P. Abbeel, and K. Goldberg.  A survey of research on cloud robotics and automation.  _IEEE Transactions on automation science and engineering_ , 12(2):398–409, 2015.
  * Kingma and Welling [2013] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Knop et al. [2020] S. Knop, P. Spurek, J. Tabor, I. Podolak, M. Mazur, and S. Jastrzębski.  Cramer-wold auto-encoder.  _Journal of Machine Learning Research_ , 21, 2020.
  * Kosiorek et al. [2021] A. R. Kosiorek, H. Strathmann, D. Zoran, P. Moreno, R. Schneider, S. Mokrá, and D. J. Rezende.  Nerf-vae: A geometry aware 3d scene generative model.  In _International Conference on Machine Learning_ , pages 5742–5752. PMLR, 2021.
  * Li et al. [2017] J. Li, K. Xu, S. Chaudhuri, E. Yumer, H. Zhang, and L. Guibas.  Grass: Generative recursive autoencoders for shape structures.  _ACM Transactions on Graphics (TOG)_ , 36(4):1–14, 2017.
  * Liao et al. [2018] Y. Liao, S. Donne, and A. Geiger.  Deep marching cubes: Learning explicit surface representations.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 2916–2925, 2018.
  * Lin et al. [2018] C.-H. Lin, C. Kong, and S. Lucey.  Learning efficient point cloud generation for dense 3d object reconstruction.  In _proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Max [1995] N. Max.  Optical models for direct volume rendering.  _IEEE Transactions on Visualization and Computer Graphics_ , 1(2):99–108, 1995.
  * Mescheder et al. [2019] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4460–4470, 2019.
  * Michalkiewicz et al. [2019] M. Michalkiewicz, J. K. Pontes, D. Jack, M. Baktashmotlagh, and A. Eriksson.  Implicit surface representations as layers in neural networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4743–4752, 2019.
  * Mildenhall et al. [2020] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In _European conference on computer vision_ , pages 405–421. Springer, 2020.
  * Niemeyer et al. [2020] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger.  Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3504–3515, 2020.
  * Ost et al. [2021] J. Ost, I. Laradji, A. Newell, Y. Bahat, and F. Heide.  Neural point light fields.  _arXiv preprint arXiv:2112.01473_ , 2021.
  * Park et al. [2019] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 165–174, 2019.
  * Peng et al. [2020] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger.  Convolutional occupancy networks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 523–540. Springer, 2020.
  * Proszewska et al. [2021] M. Proszewska, M. Mazur, T. Trzciński, and P. Spurek.  Hypercube: Implicit field representations of voxelized 3d models.  _arXiv preprint arXiv:2110.05770_ , 2021.
  * Qi et al. [2017a] C. R. Qi, H. Su, K. Mo, and L. J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 652–660, 2017a.
  * Qi et al. [2017b] C. R. Qi, L. Yi, H. Su, and L. J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _arXiv preprint arXiv:1706.02413_ , 2017b.
  * Riegler et al. [2017] G. Riegler, A. O. Ulusoy, H. Bischof, and A. Geiger.  Octnetfusion: Learning depth fusion from data.  In _2017 International Conference on 3D Vision (3DV)_ , pages 57–66. IEEE, 2017.
  * Sheikh et al. [2017] A.-S. Sheikh, K. Rasul, A. Merentitis, and U. Bergmann.  Stochastic maximum likelihood optimization via hypernetworks.  _arXiv preprint arXiv:1712.01141_ , 2017.
  * Sinha et al. [2016] A. Sinha, J. Bai, and K. Ramani.  Deep learning 3d shape surfaces using geometry images.  In _European Conference on Computer Vision_ , pages 223–240. Springer, 2016.
  * Sinha et al. [2017] A. Sinha, A. Unmesh, Q. Huang, and K. Ramani.  Surfnet: Generating 3d shape surfaces using deep residual networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6040–6049, 2017.
  * Sitzmann et al. [2019] V. Sitzmann, M. Zollhöfer, and G. Wetzstein.  Scene representation networks: Continuous 3d-structure-aware neural scene representations.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Spurek et al. [2020] P. Spurek, S. Winczowski, J. Tabor, M. Zamorski, M. Zieba, and T. Trzcinski.  Hypernetwork approach to generating point clouds.  In _International Conference on Machine Learning_ , pages 9099–9108. PMLR, 2020.
  * Spurek et al. [2021a] P. Spurek, A. Kasymov, M. Mazur, D. Janik, S. Tadeja, u. Struski, J. Tabor, and T. Trzciński.  Hyperpocket: Generative point cloud completion.  _arXiv preprint arXiv:2102.05973_ , 2021a.
  * Spurek et al. [2021b] P. Spurek, S. Winczowski, M. Zięba, T. Trzciński, and K. Kania.  Modeling 3d surface manifolds with a locally conditioned atlas.  _arXiv preprint arXiv:2102.05984_ , 2021b.
  * Spurek et al. [2021c] P. Spurek, M. Zieba, J. Tabor, and T. Trzcinski.  General hypernetwork framework for creating 3d point clouds.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021c.
  * Su et al. [2015] H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller.  Multi-view convolutional neural networks for 3d shape recognition.  In _Proceedings of the IEEE international conference on computer vision_ , pages 945–953, 2015.
  * Tang et al. [2021] J. Tang, J. Lei, D. Xu, F. Ma, K. Jia, and L. Zhang.  Sa-convonet: Sign-agnostic optimization of convolutional occupancy networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6504–6513, 2021.
  * Tatarchenko et al. [2017] M. Tatarchenko, A. Dosovitskiy, and T. Brox.  Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2088–2096, 2017.
  * Tolstikhin et al. [2017] I. Tolstikhin, O. Bousquet, S. Gelly, and B. Schoelkopf.  Wasserstein auto-encoders.  _arXiv preprint arXiv:1711.01558_ , 2017.
  * Von Oswald et al. [2019] J. Von Oswald, C. Henning, J. Sacramento, and B. F. Grewe.  Continual learning with hypernetworks.  _arXiv preprint arXiv:1906.00695_ , 2019.
  * Wang et al. [2018a] N. Wang, Y. Zhang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang.  Pixel2mesh: Generating 3d mesh models from single rgb images.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 52–67, 2018a.
  * Wang et al. [2021] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021.
  * Wang et al. [2018b] P.-S. Wang, C.-Y. Sun, Y. Liu, and X. Tong.  Adaptive o-cnn: A patch-based deep representation of 3d shapes.  _ACM Transactions on Graphics (TOG)_ , 37(6):1–11, 2018b.
  * Wen et al. [2022] C. Wen, Y. Zhang, C. Cao, Z. Li, X. Xue, and Y. Fu.  Pixel2mesh++: 3d mesh generation and refinement from multi-view images.  _arXiv preprint arXiv:2204.09866_ , 2022.
  * Wu et al. [2016] J. Wu, C. Zhang, T. Xue, W. T. Freeman, and J. B. Tenenbaum.  Learning a probabilistic latent space of object shapes via 3d generative-adversarial modeling.  In _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , pages 82–90, 2016.
  * Wu et al. [2015] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao.  3d shapenets: A deep representation for volumetric shapes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1912–1920, 2015.
  * Xu et al. [2022] Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann.  Point-nerf: Point-based neural radiance fields.  _arXiv preprint arXiv:2201.08845_ , 2022.
  * Yang et al. [2018a] B. Yang, W. Luo, and R. Urtasun.  Pixor: Real-time 3d object detection from point clouds.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 7652–7660, 2018a.
  * Yang et al. [2019] G. Yang, X. Huang, Z. Hao, M.-Y. Liu, S. Belongie, and B. Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4541–4550, 2019.
  * Yang et al. [2018b] Y. Yang, C. Feng, Y. Shen, and D. Tian.  Foldingnet: Point cloud auto-encoder via deep grid deformation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 206–215, 2018b.
  * Zhu et al. [2018] C. Zhu, K. Xu, S. Chaudhuri, R. Yi, and H. Zhang.  Scores: Shape composition with recursive substructure priors.  _ACM Transactions on Graphics (TOG)_ , 37(6):1–14, 2018.