# Voint Cloud: Multi-View Point Cloud Representation for 3D Understanding

Abdullah Hamdi Silvio Giancola Bernard Ghanem
King Abdullah University of Science and Technology (KAUST), Thuwal, Saudi
Arabia
{abdullah.hamdi, silvio.giancola<EMAIL_ADDRESS>
###### Abstract

Multi-view projection methods have demonstrated promising performance on 3D
understanding tasks like 3D classification and segmentation. However, it
remains unclear how to combine such multi-view methods with the widely
available 3D point clouds. Previous methods use unlearned heuristics to
combine features at the point level. To this end, we introduce the concept of
the multi-view point cloud (Voint cloud), representing each 3D point as a set
of features extracted from several view-points. This novel 3D Voint cloud
representation combines the compactness of 3D point cloud representation with
the natural view-awareness of multi-view representation. Naturally, we can
equip this new representation with convolutional and pooling operations. We
deploy a Voint neural network (VointNet) with a theoretically established
functional form to learn representations in the Voint space. Our novel
representation achieves state-of-the-art performance on 3D classification and
retrieval on ScanObjectNN, ModelNet40, and ShapeNet Core55. Additionally, we
achieve competitive performance for 3D semantic segmentation on ShapeNet
Parts. Further analysis shows that VointNet improves the robustness to
rotation and occlusion compared to other methods.

##  1 Introduction

A fundamental question in 3D computer vision and computer graphics is how to
represent 3D data [41, 45, 42, 9, 48, 39, 44]. This question becomes
particularly vital given how the success of deep learning in 2D computer
vision has pushed for wide adoption of deep learning in 3D vision and
graphics. In fact, deep networks already achieve impressive results in 3D
classification [17], 3D segmentation [23], 3D detection [35], 3D
reconstruction [41, 45], and novel view synthesis [42]. 3D computer vision
networks either rely on direct 3D representations, indirect 2D projection on
images, or a mixture of both. Direct approaches operate on 3D data commonly
represented with point clouds [48, 49, 58], meshes [12, 19], or voxels [39, 6,
16]. In contrast, indirect approaches commonly render multiple 2D views of
objects or scenes [53, 17], and process each image with a traditional 2D
image-based architecture. The human visual system is closer to such a multi-
view indirect approach for 3D understanding, as it receives streams of
rendered images rather than explicit 3D representations.

Figure 1.1: 3D Voint Clouds. We propose the multi-view point cloud (Voint
cloud), a novel 3D representation that is compact and naturally descriptive of
view projections of a 3D point cloud. Each point in the 3D cloud is tagged
with a Voint, which accumulates view-features for that point. Note that not
all 3D points are visible from all views. The set of Voints constructs a Voint
cloud.

3D Representation | Explicitness | View-Dependent | Main Use | Memory |  3D Descriptiveness
---|---|---|---|---|---
3D Point Cloud | Explicit | ✗ | 3D Understanding | Low | Medium
Multi-View Projections | Implicit | ✓ | 3D Understanding | Medium | Low
Voxels | Explicit | ✗ | 3D Understanding | High | Medium
Mesh | Explicit | ✗ | 3D Modeling | Low | High
Surface Implicit ([41, 45]) | Implicit | ✗ | 3D Modeling | Medium | High
Volume Implicit (NeRFs [42]) | Implicit | ✓ | Novel View Synthesis | High | Medium
3D Voint Cloud (ours) | Explicit | ✓ | 3D Understanding | Low | Medium

Table 1.1: Comparison of Different 3D Representations. We compare some of the
widely used 3D representations to our proposed Voint cloud. Note that our
Voint cloud shares the view-dependency of NeRFs [42] while inheriting the
merits of 3D point clouds.

Tackling 3D vision tasks with indirect approaches has two main advantages: (i)
mature and transferable 2D computer vision models (CNNs, Transformers,
_etc_.), and (ii) large and diverse labeled image datasets for pre-training
(_e.g_. ImageNet [51]). Multi-view approaches achieve impressive performance
in 3D shape classification and segmentation [28, 59, 29, 27, 8]. However, the
challenge with the multi-view representation (especially for 3D segmentation)
lies in properly aggregating the per-view features with 3D point clouds. The
appropriate aggregation is necessary to obtain representative 3D point clouds
with a single feature per point suitable for typical point cloud processing
pipelines. Previous multi-view works rely on heuristics (_e.g_. average or
label mode pooling) after mapping pixels to points [29], or multi-view fusion
with voxels [8]. Such setups might not be optimal for a few reasons. (i) Such
heuristics may aggregate information of misleading projections that are
obtained from arbitrary view-points. For example, looking at an object from
the bottom and processing that view independently can carry wrong information
about the object’s content when combined with other views. (ii) The views lack
geometric 3D information. To this end, we propose a new hybrid 3D data
structure that inherits the merits of point clouds (_i.e_. compactness,
flexibility, and 3D descriptiveness) and leverages the benefits of rich
perceptual features of multi-view projections. We call this new representation
multi-view point cloud (or Voint cloud) and illustrate it in Figure 1.1. A
Voint cloud is a set of Voints, where each Voint is a set of view-dependent
features (view-features) that correspond to the same point in the 3D point
cloud. The cardinality of these view-features may differ from one Voint to
another.

Likewise, Neural Radiance Fields (NeRFs) [42] extend 3D implicit
representations [45, 41] to characterize the 3D volume with a learned function
that depends on the coordinates and the viewing angles. In our Voint cloud
representation, we assume that the implicit 3D function defining the surface
3D points to be continuous spherical functions, _i.e_. their features depend
on the view-point from which they are observed. Voint clouds inherit the
characteristics of the parent explicit 3D point clouds, which facilitates
learning Voint representations for a variety of vision applications (_e.g_.
classification and segmentation). In Table 1.1, we compare some of the widely
used 3D representations and our Voint cloud representation.

To deploy deep learning on the new Voint space, we define basic operations on
Voints, such as pooling and convolution. We define a theoretical and practical
way of building Voint neural networks that we dub VointNet. VointNet takes a
Voint cloud and outputs point cloud features for 3D point cloud processing
pipelines. We show how learning this Voint cloud representation leads to
strong performance and gained robustness for the tasks of 3D classification,
3D object retrieval, and 3D part segmentation on standard benchmarks like
ModelNet [61], ScanObjectNN [55], and ShapeNet [2].

Contributions: (i) We propose a novel multi-view 3D point cloud representation
(denoted as Voint cloud), which represents each point (namely a Voint) as a
set of features from different view-points. (ii) We define pooling and
convolutional operations at the Voint level to construct a theoretically-
grounded formulation (VointNet) capable of learning to aggregate information
from multiple views in the Voint space. (iii) Our VointNet reaches state-of-
the-art performance on several 3D understanding tasks, including 3D shape
classification and retrieval. Further, VointNet achieves competitive
performance on part segmentation, with substantial robustness improvement to
rotation and occlusion compared to other methods, with negligible added
memory.

##  2 Related Work

Learning from 3D Data. Different from 2D data that deals with images of
structured pixels, there are multiple ways to represent 3D data, which mostly
depends on the application. These 3D representations include 3D point clouds
[48, 49, 70], 3D volumetric voxel representations [39, 34], 3D meshes [19,
12], and 3D implicit functions [41, 45]. 3D point clouds are widely used for
3D representation in computer vision due to their compactness, flexibility,
and because they can be obtained naturally from sensors like LiDAR and RGBD
cameras. PointNet [48] paved the way as the first deep learning algorithm to
operate directly on 3D point clouds. PointNet computes point features
independently and aggregates them using an order-invariant function like max-
pooling. Subsequent works focused on finding neighborhoods of points to define
point convolutional operations [49, 58, 32, 31, 30, 57]. Voxel-based deep
networks explicitly use 3D CNNs on a voxel representation that suffers from
cubic memory complexity [39]. Several recent works combine point cloud
representations with other 3D modalities like voxels [34, 67] or multi-view
images [25]. This paper proposes a novel Voint cloud representation for 3D
shapes and investigates novel architectures that aggregate view-dependent
features at the 3D point level.

Multi-View Applications. The idea of using 2D images to understand the 3D
world was initially proposed in 1994 by Bradski  _et al_. [1]. This intuitive
multi-view approach was combined with deep learning for 3D understanding in
MVCNN [53]. A line of works continued developing multi-view approaches for
classification and retrieval by improving the aggregation of the view-features
from each image view [28, 11, 7, 59, 17]. Another relevant task is 3D shape
retrieval, which is traditionally conducted by extracting descriptors of 3D
shapes to match them. Several multi-view methods have been proposed to extract
these visual signatures describing 3D objects, including MVCNN [53], view-
group CNN [13], triplet-center loss [21], and Multi-Loop View CNN [26]. In the
3D object detection task, using multiple views proved crucial in many of the
recent methods that use Bird-Eye View (BEV) or projective view, in combination
with 3D point clouds and voxels [5, 71].

For 3D generative tasks like 3D reconstruction, classical Multi-View Stereo
(MVS) [44, 14, 43] and more contemporary learning-based reconstruction from
multiple images [64, 65, 69, 24, 4] are commonly followed. Recently, NeRFs
[42, 68, 38] use multi-view images to implicitly learn a neural volume
function for the task of novel view synthesis. In this work, we fuse the
concept of multi-view into the 3D structure itself, such that every 3D point
would have an independent set of view-features according to the view-points
available in the setup. Our Voints are aligned with the sampled 3D point
cloud, offering a compact representation that allows for efficient computation
and memory usage while maintaining the view-dependent component that
facilitates view-based learning for vision.

Hybrid Multi-View for 3D Segmentation. On the task of 3D semantic
segmentation, a smaller number of works tried to follow the multi-view
approach [8, 29, 56, 27, 25]. A problem arises when combining view features to
represent local points/voxels while preserving local geometric features. These
methods tend to average the view-features [29, 27], propagate the labels only
[56], learn from reconstructed points in the neighborhood [25], or combine the
multi-view features with 3D voxel features [8, 22]. To this end, our proposed
VointNet operates on the Voint cloud space while preserving the compactness
and 3D descriptiveness of the original point cloud. VointNet leverages the
power of multi-view features with learned aggregation on the view-features
applied to each point independently.

Figure 2.1: A Toy 2D Example of Voints. Voints assume view-dependency for
every 3D point. Here, we look at a single 2D point at the center with a
circular function $g(u)=\text{sign}\left(\cos u\right)$ from five arbitrary
view-points $\\{u_{j}\\}_{j=1}^{5}$. Trying to reduce $g$ to a single value
based on $u_{j}$ projections undermines the underlying structure of $g$. We
take the full set $\\{\left(u_{j},g(u_{j})\right)\\}_{j=1}^{5}$ as a
representation of $g$ and learn a set function $f$ on these view-features for
a more informative manner of representation aggregation. Figure 2.2: Learning
from Voint Clouds. To construct a 3D Voint cloud $\widehat{\mathcal{X}}$, a
renderer $\mathbf{R}$ renders the point cloud $\mathcal{X}$ from view-points
$\mathcal{U}$ and image features are extracted from the generated images via a
2D backbone $\mathbf{C}$. The image features are then unprojected to the Voint
cloud by $\Phi_{\mathbf{B}}$ and passed to VointNet $\widehat{\mathbf{F}}$. To
learn both $\mathbf{C}$ and $\widehat{\mathbf{F}}$, a 3D loss on the output
points is used with an optional auxiliary 2D loss on $\mathbf{C}$.

##  3 Methodology

The primary assumption in our work is that surface 3D points are spherical
functions, _i.e_. their representations depend on the viewing angles observing
them. This condition contrasts most 3D point cloud processing pipelines that
assume a view-independent representation of 3D point clouds [48, 49, 54]. In
the toy 2D example in Figure 2.1, the center point (represented by a circular
function $g$) is viewed from various view-points $u_{j}$ that are agnostic to
the underlying function itself. In many applications, it is desired to have a
single feature representing each point in the point cloud. When the projected
values of $g$ from these $u_{j}$ view-points are aggregated together (_e.g_.
by max/mean pool) to get a constant representation of that point, the
underlying properties of $g$ are lost. We build our Voint representation to
keep the structure of $g$ intact by taking the full set
$\\{\left(u_{j},g(u_{j})\right)\\}_{j=1}^{5}$ in learning the aggregations.
This section introduces the concept of Voint clouds and presents a theoretical
and practical framework on how to learn functions on this Voint space. The
full pipeline is illustrated in Figure 2.2.

###  3.1 3D Voint Cloud

From Point Clouds to Voint Clouds. A 3D point cloud is a compact 3D
representation composed of sampled points on the surface of a 3D object or a
scene and can be obtained by different sensors like LiDAR [5, 71] or as a
result of reconstruction [44, 14]. Formally, we define the coordinate function
for the surface $g_{\text{s}}(\mathbf{x}):\mathbb{R}^{3}\rightarrow\mathbb{R}$
as the Sign Distance Function (SDF) in the continuous Euclidean space [45,
41]. The 3D iso-surface is then defined as the set of all points $\mathbf{x}$
that satisfy the condition $g_{\text{s}}(\mathbf{x})=0$. We define a surface
3D point cloud $\mathcal{X}\in\mathbb{R}^{N\times 3}$ as a set of $N$ 3D
points, where each point $\mathbf{x}_{i}\in\mathbb{R}^{3}$ is represented by
its 3D coordinates $(x_{i},y_{i},z_{i})$ and satisfies the iso-surface
condition as follows:
$\mathcal{X}=\left\\{\mathbf{x}_{i}\in\mathbb{R}^{3}\leavevmode\nobreak\
\leavevmode\nobreak\ |\leavevmode\nobreak\ \leavevmode\nobreak\
g_{\text{s}}(\mathbf{x}_{i})=0\right\\}_{i=1}^{N}$.

In contrast, NeRFs [42, 38, 68] learn an implicit volume function
$\mathbf{g}_{\text{v}}(\mathbf{x},\mathbf{u}):\mathbb{R}^{5}\rightarrow\mathbb{R}^{4}$
that defines the density and RGB value in the five dimensional joint space
comprising the continuous Euclidean space $\mathbf{x}\in\mathbb{R}^{3}$ and
the viewing direction space $\mathbf{u}\in\mathbb{R}^{2}$.

In this work, we aim to extend the view-dependency to 3D discriminative tasks
by defining Voint clouds. Inspired by NeRF [42], we assume that surface points
also depend on the view direction from which they are being observed.
Specifically, there exists a continuous implicit spherical function
$\mathbf{g}(\mathbf{x},\mathbf{u}):\mathbb{R}^{5}\rightarrow\mathbb{R}^{d}$
that defines the features of each point $\mathbf{x}$ depending on the view-
point direction $\mathbf{u}$. Given a set of $M$ view-point directions
$\mathcal{U}\in\mathbb{R}^{M\times 2}$, a Voint
$\widehat{\mathbf{x}}\in\mathbb{R}^{M\times d}$ is a set of $M$ view-dependent
features of size $d$ for the sphere centered at point $\mathbf{x}$ as follows.

$\displaystyle\widehat{\mathbf{x}}_{i}=\left\\{\mathbf{g}\left({\mathbf{x}_{i}},\mathbf{u}_{j}\right)\in\mathbb{R}^{d}\leavevmode\nobreak\
\leavevmode\nobreak\ |\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbf{x}_{i}\in\mathcal{X}\right\\}_{j=1}^{M}$ (3.1)

The Voint cloud $\widehat{\mathcal{X}}\in\mathbb{R}^{N\times M\times
d}=\left\\{\widehat{\mathbf{x}}_{i}\right\\}_{i=1}^{N}$ is the set of all $N$
Voints $\widehat{\mathbf{x}}_{i}$ corresponding to the parent point cloud
$\mathcal{X}$. Note that we typically do not have access to the underlying
implicit function $\mathbf{g}$ and we approximate it with a camera projection,
image feature extraction, and camera unprojection.

1- Multi-View Projection. As mentioned earlier, a Voint combines multiple
view-features of the same 3D point. These view-features come from a multi-view
projection of the points by a point cloud renderer
$\mathbf{R}:\mathbb{R}^{N\times 3}\rightarrow\mathbb{R}^{M\times H\times
W\times 3}$ that renders the point cloud $\mathcal{X}$ from multiple view-
points $\mathcal{U}$ into $M$ images of size $H\times W\times 3$. In addition
to projecting the point cloud into the image space, $\mathbf{R}$ defines the
mapping $\mathbf{B}\in\\{0,..,N\\}^{M\times H\times W}$ between each pixel to
the N points and background it renders. Also, $\mathbf{R}$ outputs the
visibility binary matrix $\mathbf{V}\in\\{0,1\\}^{N\times M}$ for each point
from each view. Since not all points appear in all the views due to pixel
discretization, the visibility score $\mathbf{V}_{i,j}$ defines if the Voint
$\widehat{\mathbf{x}_{i}}$ is visible in the view $\mathbf{u}_{j}$. The matrix
$\mathbf{B}$ is crucial for unprojection, while $\mathbf{V}$ is needed for
defining meaningful operations on Voints.

2- Multi-View Feature Extraction. The rendered images are processed by a
function $\mathbf{C}:\mathbb{R}^{M\times H\times W\times
3}\rightarrow\mathbb{R}^{M\times H\times W\times d}$ that extracts image
features, as shown in Figure 2.2. If $\mathbf{C}$ is the identity function,
all the view-features would be identical for each Voint (typically the RGB
value of the corresponding point). However, the $\mathbf{C}$ function can be a
2D network dedicated to the downstream task and can extract useful global and
local features about each view.

3- Multi-View Unprojection. We propose a module
$\Phi_{\mathbf{B}}:\mathbb{R}^{M\times H\times W\times
d}\rightarrow\mathbb{R}^{N\times M\times d}$ that unprojects the 2D features
from each pixel to be 3D view-features at the corresponding voint. This is
performed by using the mapping $\mathbf{B}$ created by the renderer to form
the Voint cloud features $\widehat{\mathcal{X}}$. Note that the points are not
necessarily visible from all the views, and some Voints that are not visible
from any of the $M$ views will not receive any features. We post-process these
empty points ($\sim$ 0.5% of points during inference) to be filled with
nearest 3D neighbors features.

To summarize, the output Voint cloud is described by Eq (3.1), where
$\mathbf{g}\left({\mathbf{x}_{i}},\mathbf{u}_{j}\right)=\Phi_{\mathbf{B}}\big{(}\mathbf{C}\left(\mathbf{R}\left({\mathcal{X}},\mathbf{u}_{j}\right)\right)\big{)}_{i}$
and the features are only defined for a view $j$ of Voint
$\widehat{\mathbf{x}_{i}}$ if $\mathbf{V}_{i,j}=1$.

###  3.2 Operations on 3D Voint Clouds

Functional Form. We can look at a simplified setup to decide on the functional
form of the deep neural network that operates in the Voint space. In this
simplified setup, we consider a 2D example (instead of 3D Voints) and assume
that a circular function describes a point at the center (as in Figure 2.1).
The center point will assume its value according to the angle $u$. The
following Theorem 3.1 proves that for any continuous set function $f$ that
operates on any set of $M$ angles $\\{u_{1},...,u_{M}\\}$, there exists an
equivalent composite function consisting of transformed max-pooled individual
view-features. This composition is the functional form we describe later for
Voint neural networks

###### Theorem 3.1

Suppose $f:\mathcal{S}\rightarrow\mathbb{R}$ is a continuous set function
operating on an angles set $\mathcal{S}=\left\\{u\leavevmode\nobreak\
\leavevmode\nobreak\ |\leavevmode\nobreak\ \leavevmode\nobreak\
u\in[0,2\pi]\right\\}$. The continuity of $f$ is based on the Hausdorff
distance $d_{H}$ between two sets of angles, where
$d_{H}(\mathcal{S},\mathcal{S}^{\prime})=\max_{u_{i}^{\prime}\in\mathcal{S}^{\prime}}\min_{u_{i}\in\mathcal{S}}d_{A}(u_{i},u_{i}^{\prime})$,
and $d_{A}$ is the smallest positive angle between two angles:
$d_{A}(u,u^{\prime})=\min(|u-u^{\prime}|,2\pi-|u-u^{\prime}|)$. Then, for
every $\epsilon>0$, and $\\{u_{1},...,u_{M}\\}\subset\mathcal{S}$, there
exists a continuous function $\mathbf{h}$ and a symmetric function
$g(u_{1},...,u_{M})=\gamma\circ\operatorname{MAX}$, such that:

$\displaystyle\left|f(\\{u_{1},...,u_{M}\\})-\gamma\Big{(}\operatorname{MAX}\big{(}\mathbf{h}\left(u_{1}\right),\ldots,\mathbf{h}\left(u_{M}\right)\big{)}\Big{)}\right|<\epsilon,$

where $\gamma$ is a continuous function, and $\operatorname{MAX}$ is an
element-wise vector max operator.

Proof. The full proof is provided in Appendix A. In summary, it utilizes the
property of continuity in the set space under Hausdorff distance to divide the
range $[0,2\pi]$ into $K$ intervals that bounds the deviation of the function
$f$. The max operation would not affect the behavior of $f$ within those
intervals, and hence the equivalence between $f$ and its composition can be
established. Extending Theorem 3.1 to 3D is straightforward once the Hausdorff
distance is adapted to the azimuth and elevation angles $\mathbf{u}$ used in
Voints in Eq (3.1). This theorem extends PointNet’s theorem of point cloud
functional composition [48] and its Universal Approximation to spherical
functions underlying Voints.

VointMax. We define $\operatorname{VointMax}$ as max-pooling on the visible
view-features along the views dimension of the voint $\widehat{\mathbf{x}}$.
For all $i\in 1,2,...,N$ and $j\in 1,2,...,M$,

$\displaystyle\operatorname{VointMax}(\widehat{\mathbf{x}_{i}})=\max_{j}\leavevmode\nobreak\
\widehat{\mathbf{x}}_{i,j},\leavevmode\nobreak\ \leavevmode\nobreak\
\text{s.t.}\leavevmode\nobreak\ \leavevmode\nobreak\ \mathbf{V}_{i,j}=1$ (3.2)

VointConv. We define the convolution operation
$h_{\text{V}}:\leavevmode\nobreak\ \mathbb{R}^{N\times M\times
d}\rightarrow\mathbb{R}^{N\times M\times d^{{}^{\prime}}}$ as any learnable
function that operates on the Voint space with shared weights on all the
Voints and has the view-features input size $d$ and outputs view-features of
size $d^{{}^{\prime}}$ and consists of $l_{V}$ layers. Examples of this
VointConv operation include the following operations applied only on the
visible view-features: a shared MLP, a graph convolution, and a graph
attention. We detail these operations later in Section 4.2, which result in
different non-exhaustive variants of VointNet.

###  3.3 Learning on 3D Voint Clouds

VointNet. Typical 3D point cloud classifiers are defined as a composition of
learned operations on individual points with shared weights followed by max-
pooling to obtain a global feature for the shape. On the other hand, the goal
of the VointNet model is to obtain multi-view point cloud features that can be
subsequently used by any point cloud processing pipeline. The VointNet module
$\widehat{\mathbf{F}}:\mathbb{R}^{N\times M\times
d}\rightarrow\mathbb{R}^{N\times d}$ is defined as follows.

$\widehat{\mathbf{F}}(\widehat{\mathcal{X})}=h_{\text{P}}\left(\operatorname{VointMax}\left(h_{\text{V}}\big{(}\widehat{\mathcal{X}}\big{)}\right)\right),$
(3.3)

where $h_{\text{P}}$ is any point convolutional operation (_e.g_. shared MLP
or EdgeConv). The VointNet module transforms the individual view-features
using the learned VointConv $h_{\text{V}}$ before VointMax is applied on the
view-features to obtain point features. Theorem 3.1 highlights this
composition.

VointNet Pipeline for 3D Point Cloud Processing. The full pipeline is
described in Figure 2.2. The loss for this pipeline can be described as
follows:

$\displaystyle\operatorname*{arg\,min}_{\boldsymbol{\theta}_{\mathbf{C}},\boldsymbol{\theta}_{\widehat{\mathbf{F}}}}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
\sum_{i}^{N}L\leavevmode\nobreak\
\Big{(}\widehat{\mathbf{F}}\big{(}\Phi_{\mathbf{B}}\big{(}\mathbf{C}\left(\mathbf{R}\left({\mathcal{X}},\mathcal{U}\right)\right)\big{)}\big{)}_{i}\leavevmode\nobreak\
,\leavevmode\nobreak\ \mathbf{y}_{i}\Big{)},$ (3.4)

where $L$ is a Cross-Entropy (CE) loss defined on all the training points
$\mathcal{X}$, and $\\{y_{i}\\}_{i=1}^{N}$ defines the labels of these points.
The other components $(\mathbf{R},\Phi_{\mathbf{B}},\mathcal{U},\mathbf{C})$
are all defined before. The weights to be jointly learned are those of the 2D
backbone ($\boldsymbol{\theta}_{\mathbf{C}}$) and those of the VointNet
($\boldsymbol{\theta}_{\widehat{\mathbf{F}}}$) using the same 3D loss. An
auxiliary 2D loss on $\boldsymbol{\theta}_{\mathbf{C}}$ can be optionally
added for supervision at the image level. For classification, the entire
object can be treated as a single Voint, and the global features of each view
would be the view-features of that Voint. We analyze different setups in
detail in Section 6.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. We benchmark VointNet on the challenging and realistic ScanObjectNN
dataset for 3D point cloud classification [55]. The dataset has three
variants, includes background and occlusion, and has 15 categories and 2,902
point clouds. Also, we test for 3D classification on ModelNet40 [61], which is
composed of 40 classes and 12,311 3D objects. We follow the common practice
for tests on ModelNet40 [48, 49, 58, 70]. For the shape retrieval task, we
benchmark on ShapeNet Core55 as a subset of ShapeNet [2]. The dataset consists
of 51,162 3D mesh objects labeled with 55 object classes. We follow the MVTN’s
setup [17] in sampling 5,000 points from each mesh object to obtain point
clouds. On the other hand, for the task of shape part segmentation, we test on
ShapeNet Parts [66], a subset of ShapeNet [2] that consists of 16,872 point
cloud objects from 16 categories and 50 parts.

Metrics. For 3D point cloud classification, we report the overall accuracy,
while shape retrieval is evaluated using mean Average Precision (mAP) over
test queries [17]. On the other hand, 3D semantic segmentation is evaluated
using mean Intersection over Union (mIoU) on points. For part segmentation, we
report two types of mIoUs: Instance-averaged mIoU (Ins. mIoU) and Category-
averaged mIoU (Cat. mIoU) as followed commonly in the literature [48, 49, 70].

Baselines. We include PointNet [48], PointNet++ [49], DGCNN [58], KPConv [54],
Point Transformer [70] and CurveNet [62] as baselines that use point clouds.
We also compare against multi-view classification approaches like MVCNN [53]
and MVTN [17] as baselines for classification and retrieval and adopt some of
the multi-view segmentation baselines (_e.g_. Label Fusion [56] and Mean
Fusion [29]) for part segmentation to work in the Voint space.

###  4.2 VointNet Variants

VointNet in Eq (3.3) relies on the VointConv operation $h_{\text{V}}$ as the
basic building block. Here, we briefly describe three examples of
$h_{\text{V}}$ operations used in our work.

Shared Multi-Layer Perceptron (MLP). It is the most basic VointConv
formulation. For a layer $l$, the features of Voint $i$ at view $j$ are
updated to layer $l+1$ as:
$\mathbf{h}_{i,j}^{l+1}=\rho\left(\mathbf{h}_{i,j}^{l}\mathcal{W}_{\rho}\right)$,
where $\rho$ is the shared MLP with weights $\mathcal{W}_{\rho}$ followed by
normalization and a nonlinear function (_e.g_. ReLU). This operation is
applied on all Voints independently and only involves the visible views-
features for each Voint. This formulation extends the shared MLP formulation
for PointNet [48] to work on Voints’ view-features.

Graph Convolution (GCN). We define a fully connected graph for each Voint by
creating a virtual center node connected to all the view-features to aggregate
their information (similar to ”cls" token in ViT [10]). Then, the graph
convolution can be defined as the shared MLP (as described above) but on the
edge features between all view features, followed by a max pool on the graph
neighbors. An additional shared MLP is used before the final output.

Graph Attention (GAT). A graph attention operation can be defined just like
the GCN operation above but with learned attention weights on the graph
neighbor’s features before averaging them. A shared MLP computes these
weights.

| Classification Overall Accuracy
---|---
Method | OBJ_BG | OBJ_ONLY | Hardest
PointNet [48] | 73.3 | 79.2 | 68.0
SpiderCNN [63] | 77.1 | 79.5 | 73.7
PointNet ++ [49] | 82.3 | 84.3 | 77.9
PointCNN [32] | 86.1 | 85.5 | 78.5
DGCNN [58] | 82.8 | 86.2 | 78.1
SimpleView [15] | - | - | 79.5
BGA-DGCNN [55] | - | - | 79.7
BGA-PN++ [55] | - | - | 80.2
MVTN [17] | 92.6 | 92.3 | 82.8
VointNet (ours) | 93.7 | 94.0 | 85.4

Table 4.1: 3D Point Cloud Classification on ScanObjectNN. We report the
accuracy of VointNet in 3D point cloud classification on three different
variants of ScanObjectNN [55]. Bold denotes the best result in its setup.

|  | Classification | Shape Retrieval
---|---|---|---
Method | Data Type | ModelNet40 | ShapeNet Core
PointNet [48] | Points | 89.2 | -
PointNet++ [49] | Points | 91.9 | -
DGCNN [58] | Points | 92.2 | -
KPConv[54] | Points | 92.9 | -
CurveNet[62] | Points | 93.8 | -
ReVGG [52] | M-View | - | 74.9
MVCNN [53] | M-View | 90.1 | 73.5
ViewGCN [59] | M-View | 93.3 | 78.4
MVTN [17] | M-View | 93.8 | 82.9
VointNet (ours) | Voints | 92.8 | 83.3

Table 4.2: 3D Shape Classification and Retrieval. We report VointNet’s
classification accuracy on ModelNet40 [61] and its 3D shape retrieval mAP on
ShapeNet Core55 [2, 52]. Baseline results are reported from [17, 70, 62].

###  4.3 Implementation Details

Rendering and Unprojection. We choose the differentiable point cloud renderer
$\mathbf{R}$ from Pytorch3D [50] in our pipeline for its speed and
compatibility with Pytorch libraries [46]. We render point clouds on multi-
view images with size $224\times 224\times 3$. We color the points by their
normals’ values or keep them white if the normals are not available. Following
a similar procedure to [59, 17], the view-points setup is randomized during
training (using $M=8$ views) and fixed to spherical views in testing (using
$M=12$ views).

Architectures. For the 2D backbone $\mathbf{C}$, we use ViT-B [10] (with
pretrained weights from TIMM library [60]) for classification and DeepLabV3
[3] for segmentation. We use the 3D CE loss on the 3D point cloud output and
the 2D CE loss when the loss is defined on the pixels. The feature dimension
of the VointNet architectures is $d=64$, and the depth is $l_{V}=4$ layers in
$h_{\text{V}}$. The main results are based on VointNet (MLP), unless otherwise
specified as in Section 6, where we study in details the effect of VointConv
$h_{\text{V}}$ and $\mathbf{C}$.

Training Setup. We train our pipeline in two stages, where we start by
training the 2D backbone on the 2D projected labels of the points, then train
the entire pipeline end-to-end while focusing the training on the VointNet
part. We use the AdamW optimizer [37] with an initial learning rate of
$0.0005$ and a step learning rate schedule of $33.3\%$ every $12$ epochs for
$40$ epochs. The pipeline is trained with one NVIDIA Tesla V100 GPU. We do not
use any data augmentation. More details about the training setup (loss and
rendering), VointNet, and the 2D backbone architectures can be found in
Appendix B.

##  5 Results

The main test results of our Voint formulations are summarized in Tables
4.1,4.2, 5.1, 5.2, and 5.3. We achieve state-of-the-art performance in the
task of 3D classification and retrieval, and competitive performance on 3D
part segmentation. More importantly, under the realistic rotated setup, we
improve the segmentation mIoU by 25% compared to baselines, while achieving
higher robustness to occlusion compared to other 3D methods. Following common
practice [17], we report the best results out of four runs in benchmark
tables, but detailed results are provided in Appendix C.

###  5.1 3D Shape Classification and Retrieval

Table 4.1 reports the classification accuracy on the 3D point cloud
classification task on ScanObjectNN [55]. It benchmarks VointNet against other
recent and strong baselines [17, 15, 17]. VointNet demonstrates state-of-the-
art results on all the variants, including the challenging Hardest (PB_T50_RS)
variant that includes challenging scenarios of rotated and translated objects.
The increase in performance (+2.4%) is significant on this variant, which
highlights the benefits of Voints on challenging scenarios, with further
affirming results in Section 5.3. Table 4.2 benchmarks the 3D classification
accuracy on ModelNet40 [61] and the retrieval mAP on ShapeNet Core55 [2].
VointNet achieves state-of-the-art performance on ShapeNet Core55 retrieval
tasks and competitive performance on ModelNet40. Unlike the realistic
ScanObjectNN, ModelNet40 is based on CAD 3D models and has saturated in the
recent years in terms of performance [70, 62, 17, 59].

|  | Part Segmentation
---|---|---
Method | Data Type | (Cat. mIoU) | (Ins. mIoU)
PointNet++ [49] | Points | 81.9 | 85.1
DGCNN [58] | Points | 82.3 | 85.1
Spider CNN[63] | Points | 81.7 | 85.3
KPConv[54] | Points | 85.1 | 86.4
PTransformer[70] | Points | 83.7 | 86.6
PointNet [48]∗ | Points | 75.5 | 80.1
Label Fuse [56]∗ | Voints | 78.9 | 80.0
Mean Fuse [29]∗ | Voints | 79.1 | 77.5
VointNet (ours)∗ | Voints | 80.3 | 81.2

Table 5.1: 3D Part Segmentation on ShapeNet Parts. We compare VointNet against
other methods in 3D segmentation on ShapeNet Parts [66]. ∗ indicates test
results we reproduce by our experiments, while other results are reported from
[70] (no-voting results).

###  5.2 3D Part Segmentation

Table 5.1 reports the Category-averaged and Instance-averaged segmentation
mIoU of VointNet compared with recent methods on ShapeNet Parts [66]. We
achieve comparative performance to strong point cloud networks [54, 70, 62].
However, when we compare our VointNet to other multi-view segmentation
baselines (that work on the Voint space and use the same 2D backbone) like
Label Fuse [56] or Mean Fuse [29], our VointNet formulation achieves better
performance. The reason for the discrepancy between the Voint results and the
results of some point cloud methods is that Voints heavily depend on the
underlying 2D backbone and inherit all its biases, especially those from
pretraining. Hence, the 2D backbone limits what the performance can reach with
VointNet. With future advancements in 2D computer vision, we expect a
proportional improvement in Voint performance on 3D tasks. We study the effect
of the backbone in detail in Section 6. Figure 5.1 shows qualitative 3D
segmentation results for VointNet and Mean Fuse [29] as compared to the ground
truth.

###  5.3 Model Robustness

Rotation Robustness. We follow the previous 3D literature by testing the
robustness of trained models to perturbations at test time [33, 17, 18]. We
perturb the shapes in ShapeNet Parts with random rotations at test time (ten
runs) and report Ins. mIoU in Table 5.2. PointNet [48] and DGCNN [58] are used
as point-based baselines, given their wide adoption in 3D vision. We expect
similar behaviors from other point networks with a significant margin for the
robustness to rotation in favor of VointNet. Note how our VointNet performance
largely exceeds the baselines in this realistic unaligned scenario.

Ground Truth | VointNet (ours) | Mean Fuse [29]
---|---|---
|  |
|  |

Figure 5.1: Qualitative Comparison for 3D Part Segmentation. We compare our
VointNet 3D segmentation prediction to Mean Fuse [29] that is using the same
trained 2D backbone. Note how VointNet distinguishes detailed parts (_e.g_.
the car window frame).

| Segmentation Under Rotation
---|---
Method | Unrotated | Rotated
PointNet [48] | 80.1 | 36.6 $\pm$0.2
DGCNN [58] | 80.1 | 37.1 $\pm$0.2
VointNet | 81.2 |  61.5 $\pm$0.2

Table 5.2: Rotation Robustness for 3D Part Segmentation. At test time, we
randomly rotate objects from ShapeNet Parts [66] and report the Ins. mIoUs of
our VointNet compared to trained PointNet [48] and DGCNN [58]. Note how
VointNet’s performance largely exceeds the baselines in realistic unaligned
scenarios, highlighting the benefit of view dependency.

Occlusion Robustness. One of the aspects of the robustness of 3D
classification models that have been recently studied is their robustness to
occlusion, as detailed in MVTN [17]. These simulated occlusions are introduced
at test time, and the average test accuracy is reported on each cropping
ratio. We benchmark our VointNet against recent baselines in Table 5.3.
PointNet [48] and DGCNN [58] are used as point-based baselines, and MVTN [17]
as a multi-view baseline. We expect similar behaviors from other point and
multi-view approaches compared to VointNet’s robustness.

| Occlusion Ratio
---|---
Method | 0 | 0.1 | 0.2 | 0.3 | 0.5
PointNet [48] | 89.1 | 88.2 | 86.1 | 81.6 | 53.5
DGCNN [58] | 92.1 | 77.1 | 74.5 | 71.2 | 30.1
MVTN [17] | 92.3 | 90.3 | 89.9 | 88.3 | 67.1
VointNet (ours) | 92.8 | 91.6 | 91.2 | 89.1 | 66.1

Table 5.3: Occlusion Robustness for 3D Classification. We report the test
accuracy on ModelNet40 for different occlusion ratios of the data to measure
occlusion robustness of different 3D methods. VointNet exhibits significantly
more robustness to occlusion than point cloud methods [48, 58].

2D Backbone | VointConv | mIoU
---|---|---
FCN [36] | MLP |  78.8 $\pm$ 0.2
GCN |  77.6 $\pm$ 0.2
GAT |  77.1 $\pm$ 0.2
DeepLabV3 [3] | MLP |  80.6 $\pm$ 0.1
GCN |  77.2 $\pm$ 0.4
GAT |  80.4 $\pm$ 0.2

Table 5.4: Ablation Study for 3D Segmentation. We ablate different components
of VointNet (2D backbone and VointConv choice), and report Ins. mIoU
performance on ShapeNet Parts [66].

##  6 Analysis and Insights

Choice of Backbones. We ablate the choice of the 2D backbone and the VointConv
operation used in VointNet and report the segmentation Ins. mIoU results in
Table 5.4. Note how the 2D backbone greatly affects performance, while the
VointConv operation type does not. This ablation highlights the importance of
the 2D backbone in VointNet pipeline and motivates the use of the simplest
variant of VointNet (MLP) in all of our main results.

Number of Views. We study the effect of the number of views $M$ on the
performance of 3D part segmentation using multiple views. We compare Mean Fuse
[29] and Label Fuse [56] to our VointNet when all of them have the same
trained 2D backbone. The views are randomly picked, and the experiments are
repeated four times. Instance-averaged mIoU with confidence intervals are
shown in Figure 6.1. We observe a consistent improvement with VointNet over
the other two baselines across different numbers of views.

Figure 6.1: Effect of the Number of Views. We plot Ins. mIoU of 3D
segmentation _vs_. the number of views ($M$) used in inference on ShapeNet
Parts [66]. Note VointNet’s consistent improvement over Mean Fuse [29] and
Label Fuse [56]. Both baselines use the same trained 2D backbone as VointNet.

Time and Memory Requirements. To assess the contribution of the Voint module,
we take a macroscopic look at the time and memory requirements of each
component in the pipeline. We record the number of floating-point operations
(GFLOPs) and the time of a forward pass for a single input sample. In Table
6.1, the VointNet module contributes negligibly to the memory requirements
compared to multi-view and point networks.

Other Factors Affecting VointNet. We study the effect of dimension $d$ of the
view-features and the effect of model size $l_{V}$ on the performance of
VointNet. Further, we look into the impact of rotation and other factors
related to the renderings (like image size and the distance to the object) and
3D classification ablations. We provide a detailed study of these factors in
Appendix C.3.

Network | GFLOPs | Time (ms) | Parameters # (M)
---|---|---|---
ViT-B [10] | 33.70 | 12.46 | 86.57
DeeplabV3 [3] | 92.61 | 20.62 | 58.64
PointNet [48] | 1.78 | 4.24 | 3.50
DGCNN [58] | 10.42 | 0.95 | 16.35
MVTN [17] | 1.78 | 4.24 | 3.50
VointNet (MLP) | 1.90 | 2.90 | 0.04

Table 6.1: Time and Memory Requirements. We assess the contribution of the
Voint module to the time and memory requirements in the multi-view and point
cloud pipeline. Note that VointNet (shared MLP) is almost 100 times smaller
than PointNet [48].

##  7 Limitations and Future Work

This work introduces the Voint cloud representation that inherits the merits
of point clouds and the rich visual features of multi-view projections,
leading to enhanced multi-view aggregation and a strong performance on many 3D
vision tasks. One crucial aspect limiting the performance of Voints is how
well-trained the 2D backbone is for the downstream 3D task. In most cases, the
2D backbone must be pretrained with enough data to learn meaningful
information for VointNet. Another aspect that limits the capability of the
Voint cloud is how to properly select the view-points for segmentation,
leading to suboptimal performance. We follow the literature in randomizing the
views while training [29]. Addressing these limitations is an important
direction for future work. Also, extending Voint learning on more 3D tasks
like 3D scene segmentation and 3D object detection is left for future work.
The code for this work will be released publicly upon publication to encourage
future works in this direction.

## References

  * [1] Gary Bradski and Stephen Grossberg.  Recognition of 3-d objects from multiple 2-d views by a self-organizing neural architecture.  In From Statistics to Neural Networks, pages 349–375. Springer, 1994.
  * [2] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [3] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [4] Rui Chen, Songfang Han, Jing Xu, and Hao Su.  Point-based multi-view stereo network.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1538–1547, 2019.
  * [5] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.  Multi-view 3d object detection network for autonomous driving.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1907–1915, 2017.
  * [6] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3075–3084, 2019.
  * [7] Taco Cohen and Max Welling.  Group equivariant convolutional networks.  In International conference on machine learning, pages 2990–2999, 2016.
  * [8] Angela Dai and Matthias Nießner.  3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 452–468, 2018.
  * [9] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.  Cvxnet: Learnable convex decomposition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 31–44, 2020.
  * [10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [11] Carlos Esteves, Yinshuang Xu, Christine Allen-Blanchette, and Kostas Daniilidis.  Equivariant multi-view networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1568–1577, 2019.
  * [12] Yutong Feng, Yifan Feng, Haoxuan You, Xibin Zhao, and Yue Gao.  Meshnet: Mesh neural network for 3d shape representation.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8279–8286, 2019.
  * [13] Yifan Feng, Zizhao Zhang, Xibin Zhao, Rongrong Ji, and Yue Gao.  Gvcnn: Group-view convolutional neural networks for 3d shape recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 264–272, 2018.
  * [14] Yasutaka Furukawa and Jean Ponce.  Accurate, dense, and robust multi-view stereopsis (pmvs).  In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 2, 2007.
  * [15] Ankit Goyal, Hei Law, Bowei Liu, Alejandro Newell, and Jia Deng.  Revisiting point cloud shape classification with a simple and effective baseline.  In ICML, 2021.
  * [16] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten.  3d semantic segmentation with submanifold sparse convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9224–9232, 2018.
  * [17] Abdullah Hamdi, Silvio Giancola, and Bernard Ghanem.  Mvtn: Multi-view transformation network for 3d shape recognition.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1–11, October 2021.
  * [18] Abdullah Hamdi, Sara Rojas, Ali Thabet, and Bernard Ghanem.  Advpc: Transferable adversarial perturbations on 3d point clouds.  In Computer Vision – ECCV 2020, pages 241–257, Cham, 2020. Springer International Publishing.
  * [19] Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or.  Meshcnn: a network with an edge.  ACM Transactions on Graphics (TOG), 38(4):1–12, 2019.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [21] Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai.  Triplet-center loss for multi-view 3d object retrieval.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1945–1954, 2018.
  * [22] Ji Hou, Angela Dai, and Matthias Nießner.  3d-sis: 3d semantic instance segmentation of rgb-d scans.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4421–4430, 2019.
  * [23] Wenbo Hu, Hengshuang Zhao, Li Jiang, Jiaya Jia, and Tien-Tsin Wong.  Bidirectional projection network for cross dimension scene understanding.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14373–14382, 2021.
  * [24] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang.  Deepmvs: Learning multi-view stereopsis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2821–2830, 2018.
  * [25] Maximilian Jaritz, Jiayuan Gu, and Hao Su.  Multi-view pointnet for 3d scene understanding.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [26] Jianwen Jiang, Di Bao, Ziqiang Chen, Xibin Zhao, and Yue Gao.  Mlvcnn: Multi-loop-view convolutional neural network for 3d shape retrieval.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8513–8520, 2019.
  * [27] Evangelos Kalogerakis, Melinos Averkiou, Subhransu Maji, and Siddhartha Chaudhuri.  3d shape segmentation with projective convolutional networks.  In proceedings of the IEEE conference on computer vision and pattern recognition, pages 3779–3788, 2017.
  * [28] Asako Kanezaki, Yasuyuki Matsushita, and Yoshifumi Nishida.  Rotationnet: Joint object categorization and pose estimation using multiviews from unsupervised viewpoints.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5010–5019, 2018.
  * [29] Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David Ross, Brian Brewington, Thomas Funkhouser, and Caroline Pantofaru.  Virtual multi-view fusion for 3d semantic segmentation.  In European Conference on Computer Vision (ECCV), pages 518–535. Springer, 2020.
  * [30] Loic Landrieu and Mohamed Boussaha.  Point cloud oversegmentation with graph-structured deep metric learning.  pages 7440–7449, 2019.
  * [31] Loic Landrieu and Martin Simonovsky.  Large-scale point cloud semantic segmentation with superpoint graphs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4558–4567, 2018.
  * [32] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.  Pointcnn: Convolution on x-transformed points.  In Advances in neural information processing systems (NIPS), pages 820–830, 2018.
  * [33] Yongcheng Liu, Bin Fan, Shiming Xiang, and Chunhong Pan.  Relation-shape convolutional neural network for point cloud analysis.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8895–8904, 2019.
  * [34] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han.  Point-voxel cnn for efficient 3d deep learning.  In Advances in Neural Information Processing Systems, pages 965–975, 2019.
  * [35] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong.  Group-free 3d object detection via transformers.  arXiv preprint arXiv:2104.00678, 2021.
  * [36] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [37] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [38] Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth.  Nerf in the wild: Neural radiance fields for unconstrained photo collections.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7210–7219, 2021.
  * [39] Daniel Maturana and Sebastian Scherer.  Voxnet: A 3d convolutional neural network for real-time object recognition.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922–928. IEEE, 2015.
  * [40] Leonard McMillan and Gary Bishop.  Plenoptic modeling: An image-based rendering system.  In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 39–46, 1995.
  * [41] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460–4470, 2019.
  * [42] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In European conference on computer vision, pages 405–421. Springer, 2020.
  * [43] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In 2011 10th IEEE international symposium on mixed and augmented reality, pages 127–136. IEEE, 2011.
  * [44] Masatoshi Okutomi and Takeo Kanade.  A multiple-baseline stereo.  IEEE Transactions on pattern analysis and machine intelligence, 15(4):353–363, 1993.
  * [45] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165–174, 2019.
  * [46] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [47] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [48] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 652–660, 2017.
  * [49] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Advances in neural information processing systems (NIPS), pages 5099–5108, 2017.
  * [50] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  arXiv:2007.08501, 2020.
  * [51] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li.  Imagenet large scale visual recognition challenge.  CoRR, abs/1409.0575, 2014.
  * [52] Konstantinos Sfikas, Theoharis Theoharis, and Ioannis Pratikakis.  Exploiting the PANORAMA Representation for Convolutional Neural Network Classification and Retrieval.  In Ioannis Pratikakis, Florent Dupont, and Maks Ovsjanikov, editors, Eurographics Workshop on 3D Object Retrieval, pages 1–7. The Eurographics Association, 2017.
  * [53] Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller.  Multi-view convolutional neural networks for 3d shape recognition.  In Proceedings of the IEEE international conference on computer vision, pages 945–953, 2015.
  * [54] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6411–6420, 2019.
  * [55] Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Duc Thanh Nguyen, and Sai-Kit Yeung.  Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data.  In International Conference on Computer Vision (ICCV), 2019.
  * [56] Brian H Wang, Wei-Lun Chao, Yan Wang, Bharath Hariharan, Kilian Q Weinberger, and Mark Campbell.  Ldls: 3-d object segmentation through label diffusion from 2-d images.  IEEE Robotics and Automation Letters, 4(3):2902–2909, 2019.
  * [57] Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann.  Sgpn: Similarity group proposal network for 3d point cloud instance segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2569–2578, 2018.
  * [58] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon.  Dynamic graph cnn for learning on point clouds.  ACM Transactions on Graphics (TOG), 2019.
  * [59] Xin Wei, Ruixuan Yu, and Jian Sun.  View-gcn: View-based graph convolutional network for 3d shape analysis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1850–1859, 2020.
  * [60] Ross Wightman.  Pytorch image models.  https://github.com/rwightman/pytorch-image-models, 2019.
  * [61] Zhirong Wu, S. Song, A. Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and J. Xiao.  3d shapenets: A deep representation for volumetric shapes.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1912–1920, 2015.
  * [62] Tiange Xiang, Chaoyi Zhang, Yang Song, Jianhui Yu, and Weidong Cai.  Walk in the cloud: Learning curves for point clouds shape analysis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 915–924, October 2021.
  * [63] Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao.  Spidercnn: Deep learning on point sets with parameterized convolutional filters.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 87–102, 2018.
  * [64] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.  Mvsnet: Depth inference for unstructured multi-view stereo.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 767–783, 2018.
  * [65] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan.  Recurrent mvsnet for high-resolution multi-view stereo depth inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5525–5534, 2019.
  * [66] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas.  A scalable active framework for region annotation in 3d shape collections.  ACM Transactions on Graphics (ToG), 35(6):1–12, 2016.
  * [67] Haoxuan You, Yifan Feng, Rongrong Ji, and Yue Gao.  Pvnet: A joint convolutional network of point cloud and multi-view for 3d shape recognition.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1310–1318, 2018.
  * [68] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In ICCV, 2021.
  * [69] Zehao Yu and Shenghua Gao.  Fast-mvsnet: Sparse-to-dense multi-view stereo with learned propagation and gauss-newton refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1949–1958, 2020.
  * [70] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun.  Point transformer.  arXiv preprint arXiv:2012.09164, 2020.
  * [71] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan.  End-to-end multi-view fusion for 3d object detection in lidar point clouds.  In Conference on Robot Learning, pages 923–932. PMLR, 2020.

##  Appendix A Detailed Formulations

###  A.1 Proof of Theorem 3.1

###### Theorem A.1

Suppose $f:\mathcal{S}\rightarrow\mathbb{R}$ is a continuous set function
operating on an angles set $\mathcal{S}=\left\\{u\leavevmode\nobreak\
\leavevmode\nobreak\ |\leavevmode\nobreak\ \leavevmode\nobreak\
u\in[0,2\pi]\right\\}$. The continuity of $f$ is based on the Hausdorff
distance $d_{H}$ between two sets of angles, where
$d_{H}(\mathcal{S},\mathcal{S}^{\prime})=\max_{u_{i}^{\prime}\in\mathcal{S}^{\prime}}\min_{u_{i}\in\mathcal{S}}d_{A}(u_{i},u_{i}^{\prime})$,
and $d_{A}$ is the smallest positive angle between two angles
$d_{A}(u,u^{\prime})=\min(|u-u^{\prime}|,2\pi-|u-u^{\prime}|)$. Then, for
every $\epsilon>0$, and $\mathcal{U}=\\{u_{1},...,u_{M}\\}\subset\mathcal{S}$,
there exists a continuous function $\mathbf{h}$ and a symmetric function
$g(u_{1},...,u_{M})=\gamma\circ\operatorname{MAX}$, such that:

$\displaystyle\left|f(\mathcal{U})-\gamma\Big{(}\operatorname{MAX}\big{(}\mathbf{h}\left(u_{1}\right),\ldots,\mathbf{h}\left(u_{M}\right)\big{)}\Big{)}\right|<\epsilon,$
(A.1)

where $\gamma$ is a continuous function, and $\operatorname{MAX}$ is an
element-wise vector max operator.

Proof. By the continuity of $f$, we take $\delta_{\epsilon}$ so that
$|f(\mathcal{U})-f(\mathcal{U}^{\prime})|<\epsilon$ for any
$\mathcal{U},\mathcal{U}^{\prime}\subset\mathcal{S}$ if
$d_{H}(\mathcal{U},\mathcal{U}^{\prime})<\delta_{\epsilon}$. Define
$K=[2\pi/\delta_{\epsilon}]$, which split $[0,2\pi]$ into $K$ intervals evenly
and define an auxiliary function that maps an angle to the beginning of the
interval it lies in:

$\displaystyle\sigma(u)=\frac{\lfloor Ku\rfloor}{K}$

Let $\tilde{\mathcal{U}}={\sigma(u):u\in\mathcal{U}}$, then

$\displaystyle|f(\mathcal{U})-f(\tilde{\mathcal{U}})|<\epsilon$ (A.2)

Let $h_{k}(u)=e^{-d\left(u,\left[\frac{k-1}{K},\frac{k}{K}\right]\right)}$ be
a soft indicator function where
$d\left(u,\left[\frac{k-1}{K},\frac{k}{K}\right]\right)=\min\left(d_{A}\left(u,\frac{k-1}{K}\right)\leavevmode\nobreak\
,\leavevmode\nobreak\ d_{A}\left(u,\frac{k}{K}\right)\right)$ is the distance
between angle $u$ to interval $\left[\frac{k-1}{K},\frac{k}{K}\right]$. Let
$\mathbf{h}(u)=\left[h_{1}(u);\ldots;h_{K}(u)\right]$, then
$\mathbf{h}:\mathbb{R}\rightarrow\mathbb{R}^{K}$

Let
$q_{j}\left(u_{1},\ldots,u_{M}\right)=\max\left\\{h_{j}\left(u_{1}\right),\ldots,h_{j}\left(u_{M}\right)\right\\}$,
indicating the occupancy of the $j$-th interval by angles in $\mathcal{U}$.
Let $\mathbf{q}=\left[q_{1};\ldots;q_{K}\right]$, then
$\mathbf{q}:[0,2\pi]^{M}\rightarrow\\{0,1\\}^{K}$ is a symmetric function,
indicating the occupancy of each interval by angles in $\mathcal{U}$.

Define $\zeta:\\{0,1\\}^{K}\rightarrow\mathcal{S}$ as
$\zeta(\mathbf{q})=\left\\{\frac{k-1}{K}:q_{k}\geq 1\right\\}$ which maps the
occupancy vector to a set which contains the left end of each angle interval.
It is straightforward to show:

$\displaystyle\zeta\left(\mathbf{q}\left(\mathcal{U}\right)\right)\equiv\tilde{\mathcal{U}}$
(A.3)

Let $\gamma:\mathbb{R}^{K}\rightarrow\mathbb{R}$ be a continuous function such
that $\gamma(\mathbf{q})=f(\zeta(\mathbf{q}))$ for
$\mathbf{q}\in\\{0,1\\}^{K}$. Then from Eq (A.2) and Eq (A.3),

$\displaystyle\left|\gamma\left(\mathbf{q}\left(\mathcal{U}\right)\right)-f(\mathcal{U})\right|$
(A.4) $\displaystyle=$
$\displaystyle\left|f\left(\zeta\left(\mathbf{q}\left(\mathcal{U}\right)\right)\right)-f(\mathcal{U})\right|<\epsilon$

Note that $\gamma\left(\mathbf{q}\left(\mathcal{U}\right)\right)$ can be
rewritten as follows:

$\displaystyle\gamma\left(\mathbf{q}\left(\mathcal{U}\right)\right)$
$\displaystyle=\gamma\left(\mathbf{q}\left(u_{1},\ldots,u_{M}\right)\right)$
(A.5)
$\displaystyle=\gamma\left(\operatorname{MAX}\left(\mathbf{h}\left(u_{1}\right),\ldots,\mathbf{h}\left(u_{M}\right)\right)\right)$
$\displaystyle=(\gamma\circ\operatorname{MAX})\left(\mathbf{h}\left(u_{1}\right),\ldots,\mathbf{h}\left(u_{M}\right)\right)$

Since $\gamma\circ$ $\operatorname{MAX}$ is a symmetric function and from Eq
(A.4) and Eq (A.5), we reach to the main result in Eq (A.1). This concludes
the proof. $\square$

###  A.2 3D Voint Cloud

Plenoptic and Spherical Coordinate Functions. The Plenoptic function was first
introduced by McMillan and Bishop [40] in 1995 as a general function that
describes the visible world. The Plenoptic function $P$ is a continuous
spherical function that describes the visibility at any Euclidean 3D point in
space $(V_{x},V_{y},V_{x})$ when looking into any direction $(\theta,\phi)$
across wavelength $\lambda$ at time $t$ . It is defined as
$p=P\left(\theta,\phi,\lambda,V_{x},V_{y},V_{x},t\right)$. Such a remarkable
and compact formulation covers all the images observed as just samples of the
function $P$. For fixed time and wavelength, the reduced Plenoptic function
$P$ becomes $p=P\left(\theta,\phi,V_{x},V_{y},V_{x},\right)$ which can
describe any field in 3D space. This shortened formulation is what Neural
Radiance Fields (NeRFs) [42, 47, 38] try to learn with MLPs to describe the
radiance and RGB values in the continuous Euclidean space with a dependency on
the view direction $(\theta,\phi)$. In the same spirit of the Plenoptic
function and NeRFs, the Voint cloud representation relies on the viewing
angles $(\theta,\phi)$ to define the view-features. The problem with the
plenoptic functions $P$, and subsequently NeRFs, is that they are very high
dimensional, and any attempt to densely represent the scene with discrete and
fixed data will cause memory and compute issues [68, 47]. Unlike NERFs [42]
that define dense 3D volumes, we focus only on the surface of the 3D shapes
with our Voint clouds representation. Our Voints are in the order of the
sampled point cloud, offering a compact representation that allows for
efficient computation and memory while maintaining the view-dependent
component that facilitates view-based learning.

From Point Clouds to Voint Clouds. Implicit representation of 3D surfaces
typically aims to learn an implicit function
$g_{\text{s}}(\mathbf{x}):\mathbb{R}^{3}\rightarrow\mathbb{R}$ that define the
Sign Distance Function (SDF) or the occupancy in the continuous Euclidean
space [45, 41]. The 3D iso-surface is then defined as the set of all points
$\mathbf{x}$ that satisfy the condition $g_{\text{s}}(\mathbf{x})=0$ (assuming
$g_{\text{s}}(\mathbf{x})$ as SDF hereafter). We define a surface 3D point
cloud $\mathcal{X}\in\mathbb{R}^{N\times 3}$, as a set of $N$ 3D points, where
each point $\mathbf{x}_{i}\in\mathbb{R}^{3}$ is represented by its 3D
coordinates $(x_{i},y_{i},z_{i})$ and satisfy the iso-surface condition as
follows.

$\displaystyle\mathcal{X}=\left\\{\mathbf{x}_{i}\in\mathbb{R}^{3}|g_{\text{s}}(\mathbf{x}_{i})=0\right\\}_{i=1}^{N}$
(A.6)

Here, we assume that surface points also depend on the view direction from
which they are being observed. Specifically, there exists a continuous
implicit spherical function
$\mathbf{g}(\mathbf{x},\mathbf{u}):\mathbb{R}^{5}\rightarrow\mathbb{R}^{d}$
that defines the features at each point $\mathbf{x}$ depending on the view
direction $\mathbf{u}$. Given a set of $M$ view-point directions
$\mathcal{U}\in\mathbb{R}^{M\times 2}$, a Voint
$\widehat{\mathbf{x}}\in\mathbb{R}^{M\times d}$ is a set of $M$ view-dependent
features of size $d$ for the sphere centered at point $\mathbf{x}$. The Voint
cloud $\widehat{\mathcal{X}}\in\mathbb{R}^{N\times M\times d}$ is the set of
all $N$ Voints $\widehat{\mathbf{x}}$.

$\displaystyle\widehat{\mathbf{x}}_{i}$
$\displaystyle=\left\\{\mathbf{g}\left({\mathbf{x}_{i}},\mathbf{u}_{j}\right)\in\mathbb{R}^{d}\leavevmode\nobreak\
\leavevmode\nobreak\ |\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbf{x}_{i}\in\mathcal{X}\right\\}_{j=1}^{M}$ (A.7)
$\displaystyle\widehat{\mathcal{X}}$
$\displaystyle=\left\\{\widehat{\mathbf{x}}_{i}\in\mathbb{R}^{M\times
d}\right\\}_{i=1}^{N}$

Note that we typically do not have access to the underlying implicit function
$\mathbf{g}$ and we approximate it by 2D projection, feature extraction, and
then un-projection as we show next.

1- Multi-View Projection. As mentioned earlier, a Voint combines multiple
view-features of the same 3D point. These view-features come from a multi-view
projection of the points by a point cloud renderer
$\mathbf{R}:\mathbb{R}^{N\times 3}\rightarrow\mathbb{R}^{M\times H\times
W\times 3}$ that renders the point cloud $\mathcal{X}$ from multiple view-
points $\mathcal{U}$ into $M$ images of size $H\times W\times 3$. In addition
to projecting the point cloud into the image space, $\mathbf{R}$ defines the
mapping $\mathbf{B}\in\\{0,..,N\\}^{M\times H\times W}$ between each pixel to
the N points and background it renders. Also, $\mathbf{R}$ outputs the
visibility binary matrix $\mathbf{V}\in\\{0,1\\}^{N\times M}$ for each point
from each view. Since not all points appear in all the views due to pixel
discretization, the visibility score $\mathbf{V}_{i,j}$ defines if the Voint
$\widehat{\mathbf{x}_{i}}$ is visible in the view $\mathbf{u}_{j}$. The matrix
$\mathbf{B}$ is crucial for unprojection, while $\mathbf{V}$ is needed for
defining meaningful operations on Voints.

2- Multi-View Feature Extraction. The rendered images are processed by a
function $\mathbf{C}:\mathbb{R}^{M\times H\times W\times
3}\rightarrow\mathbb{R}^{M\times H\times W\times d}$ that extracts image
features. If $\mathbf{C}$ is the identity function, all the view-features
would be identical for each Voint (typically the RGB value of the
corresponding point). However, the $\mathbf{C}$ function can be a 2D network
dedicated to the downstream task and can extract useful global and local
features about each view.

3- Multi-View Unprojection. We propose a module
$\Phi_{\mathbf{B}}:\mathbb{R}^{M\times H\times W\times
d}\rightarrow\mathbb{R}^{N\times M\times d}$ that unprojects the 2D features
from each pixel to be 3D view-features at the corresponding Voint. This is
performed by using the mapping $\mathbf{B}$ created by the renderer to form
the Voint cloud features $\widehat{\mathcal{X}}$. Note that the points are not
necessarily visible from all the views, and some Voints that are not visible
from any of the $M$ views will not receive any features. We post-process these
empty points ($\sim$ 0.5% of points during inference) to be filled with
nearest 3D neighbors features. The output Voint cloud features would be
described as follows.

$\displaystyle\widehat{\mathbf{x}}_{i}$
$\displaystyle=\left\\{\mathbf{g}_{i,j,:}\in\mathbb{R}^{d}\leavevmode\nobreak\
\leavevmode\nobreak\ |\leavevmode\nobreak\ \leavevmode\nobreak\
\mathbf{x}_{i}\in\mathcal{X}\leavevmode\nobreak\ ,\leavevmode\nobreak\
\mathbf{V}_{i,j}=1\right\\}_{j=1}^{M}$ (A.8) $\displaystyle\mathbf{g}_{:,j}$
$\displaystyle=\Phi_{\mathbf{B}}\left(\mathbf{C}\left(\mathbf{R}\left({\mathcal{X}},\mathbf{u}_{j}\right)\right),\mathbf{B}\right)$
$\displaystyle\widehat{\mathcal{X}}$
$\displaystyle=\left\\{\widehat{\mathbf{x}}_{i}\in\mathbb{R}^{M\times
d}\right\\}_{i=1}^{N}$

###  A.3 Voint Operations

VointMax. In order to learn a neural network in the Voint space in the form
dictated by Theorem A.1, we need to define some basic differentiable
operations on the Voint space. The max operation on the Voint cloud can be
defined as follows.

$\displaystyle\operatorname{VointMax}(\widehat{\mathbf{x}})=\max_{j}\widehat{\mathbf{x}}_{i,j},\leavevmode\nobreak\
\forall i,j$ (A.9) $\displaystyle\text{s.t.}\leavevmode\nobreak\
\leavevmode\nobreak\ i\in 1,2,...,N\leavevmode\nobreak\ ,\leavevmode\nobreak\
j\in 1,2,...,M\leavevmode\nobreak\ \leavevmode\nobreak\ ,\mathbf{V}_{i,j}=1$

Equivalently,
$\operatorname{VointMax}(\widehat{\mathbf{x}})=\max_{j}\left(\widehat{\mathbf{x}}_{:,j}-\infty\overline{\mathbf{V}}_{:,j}\right)$,
where $\overline{\mathbf{V}}$ is the complement of $\mathbf{V}$.

VointConv. We define the convolution operation
$h_{\text{V}}:\leavevmode\nobreak\ \mathbb{R}^{N\times M\times
d}\rightarrow\mathbb{R}^{N\times M\times d^{{}^{\prime}}}$ as any learnable
function that operates on the Voint space with shared weights on all the
Voints and has the view-features input size $d$ and outputs view-features of
size $d^{{}^{\prime}}$ and consists of $l_{V}$ layers. Examples of this
VointConv operation include the following operations applied only on the
visible view-features: a shared MLP, a graph convolution, and a graph
attention. We detail these operations later in Section A.5, which result in
different non-exhaustive variants of VointNet.

Figure A.1: VointNet Variants. We propose three variants of VointNet that use
three different examples of VointConv operation $h_{v}$: shared MLP (MLP),
Graph Convolution (GCN), and Graph Attention (GAT). Here we highlight the main
difference between VointNet (MLP) that shares the MLP on all the view-features
and VointNet (GCN) that creates a fully connected graph on the view-features
and learn an MLP on the edge view-features. VointNet (GAT) is similar to
VointNet (GCN) in addition to learning attention weights for each view-feature
in weighted average aggregation.

###  A.4 Learning on 3D Voint Clouds

VointNet. Typical 3D point cloud classifiers with a feature max pooling layer
work as in Eq (A.10), where $h_{\text{mlp}}$ and $h_{\text{Pconv}}$ are the
MLP and point Convolutional ($1\times 1$ or edge) layers, respectively. This
produces a K-class classifier $\mathbf{F}$.

$\mathbf{F}(\mathcal{X})=h_{\text{mlp}}(\max_{\mathbf{x}_{i}\in\mathcal{X}}\left\\{h_{\text{Pconv}}\left(\mathbf{x}_{i}\right)\right\\})$
(A.10)

Here, $\mathbf{F}:\leavevmode\nobreak\ \mathbb{R}^{N\times
3}\rightarrow\mathbb{R}^{K}$ produces the logits layer of the classifier with
size $K$. On the other hand, the goal of the VointNet model is to get multi-
view point cloud features that can be used after which by any point cloud
processing pipeline. The VointNet module
$\widehat{\mathbf{F}}:\mathbb{R}^{N\times M\times
d}\rightarrow\mathbb{R}^{N\times d}$ as follows.

$\widehat{\mathbf{F}}(\widehat{\mathcal{X})}=h_{\text{P}}\left(\operatorname{VointMax}\left(h_{\text{V}}\big{(}\widehat{\mathcal{X}}\big{)}\right)\right),$
(A.11)

###  A.5 VointNet Variants

We define the convolution operation $h_{\text{V}}:\leavevmode\nobreak\
\mathbb{R}^{N\times M\times d}\rightarrow\mathbb{R}^{N\times M\times
d^{{}^{\prime}}}$ in VointNet from Eq (A.11) as any learnable function that
operates on the Voint space with shared weights on all the Voints and has the
view-features input size $d$ and outputs view-features of size
$d^{{}^{\prime}}$ and consists of $l_{V}$ layers. Examples of this VointConv
operation include the following:

Shared MLP. It is the most basic Voint neural network. For layer $l$, the
features of Voint i at view j is updated as follows to layer $l+1$

$\displaystyle\mathbf{h}_{i,j}^{l+1}$
$\displaystyle=\rho\left(\mathbf{h}_{i,j}^{l}\mathcal{W}_{\rho}\right),\leavevmode\nobreak\
\forall i,j$ (A.12) $\displaystyle\text{s.t.}\leavevmode\nobreak\
\leavevmode\nobreak\ i\in 1,2,...,N\leavevmode\nobreak\ ,\leavevmode\nobreak\
j\in 1,2,...,M\leavevmode\nobreak\ \leavevmode\nobreak\ ,\mathbf{V}_{i,j}=1$

where $\rho$ is the shared MLP with weights $\mathcal{W}_{\rho}$ followed by
normalization and nonlinear function ( _e.g_. ReLU) applied on all Voints
independently at the visible views features for each Voint. This formulation
extends the shared MLP formulation for PointNet [48] to make the MLP shared
across the Voints and the views-features.

Graph Convolution (GCN). Just like how DGCNN [58] extended PointNet [48] by
taking the neighborhood information and extract edge features, we extend the
basic VointNet formulation in Eq (A.11). We define a fully connected graph for
each Voint along the views dimension by creating a center virtual node
connected to all the view features ( similar to the classification token in
ViT [10]). This center virtual view-feature would be assigned the index $j=0$
and can be initilized with zeros as the “cls" token in ViT [10]. Then, Voint
graph convolution operation can be defined as follows to update the
activations from layer $l$ to $l+1$

$\displaystyle\mathbf{h}_{i,j}^{l+1}=\rho\left(\Big{(}\max_{k}\psi\left((\mathbf{h}_{i,j}^{l},\mathbf{h}_{i,k}^{l})\mathcal{W}_{\psi}\right)\Big{)}\mathcal{W}_{\rho}\right)$
(A.13) $\displaystyle\forall i,j,k\leavevmode\nobreak\ \leavevmode\nobreak\
\text{s.t.}\leavevmode\nobreak\ \leavevmode\nobreak\ i\in
1,2,...,N\leavevmode\nobreak\ ,\leavevmode\nobreak\ j\in 0,1,...,M$
$\displaystyle\quad\quad\quad\quad k\in 0,1,...,M\leavevmode\nobreak\
\leavevmode\nobreak\ ,k\neq j\leavevmode\nobreak\ \leavevmode\nobreak\
,\leavevmode\nobreak\ \mathbf{V}_{i,j}=1$

where $\rho,\psi$ are two different shared MLPs as in Eq (A.12). The
difference between VointNet (MLP) and VointNet (GCN) is highlighted in Figure
A.1.

Graph Attention (GAT). Similar to how Point Transformer [70] extended the
graph convolution by adding attention to DGCNN [58], we extend the basic Voint
GraphConv formulation in Eq (A.13). Voint graph attention operation can be
defined as follows to update the activations from layer $l$ to $l+1$

$\displaystyle\mathbf{h}_{i,j}^{l+1}=\rho\left(\Bigg{(}\sum_{k=0,k\neq
j}^{M}\eta_{k}\psi\left((\mathbf{h}_{i,j}^{l},\mathbf{h}_{i,k}^{l})\mathcal{W}_{\psi}\right)\Bigg{)}\mathcal{W}_{\rho}\right)$
(A.14) $\displaystyle\forall i,j\leavevmode\nobreak\ \leavevmode\nobreak\
\text{s.t.}\leavevmode\nobreak\ \leavevmode\nobreak\ i\in
1,2,...,N\leavevmode\nobreak\ ,\leavevmode\nobreak\ j\in 0,1,...,M$
$\displaystyle\quad\quad\quad\quad\eta_{k}=\zeta\left(\mathbf{h}_{i,k}^{l}\mathcal{W}_{\zeta}\right)\leavevmode\nobreak\
\leavevmode\nobreak\ ,\leavevmode\nobreak\ \mathbf{V}_{i,j}=1$

where $\rho,\psi,\zeta$ are three different shared MLPs as in Eq (A.12), and
$\eta_{k}$ are the learned attention weights for each neighbor view-feature.

##  Appendix B Detailed Experimental Setup

###  B.1 Datasets

ScanObjectNN: 3D Point Cloud Classification. We follow the literature [15, 17]
on testing 3D classification in the challenging ScanObjectNN [55] point cloud
dataset, since it includes background and considers occlusions. The dataset is
composed of 2902 point clouds divided into 15 object categories. We use 2048
sampled points per object for Voint learning. We benchmark on its variants:
Object only, Object with Background, and the Hardest perturbed variant
(PB_T50_RS variant). Visualization is provided in Figure B.1 of some of the
renderings used in training the 2D backbone in our pipeline.

ModelNet40: 3D Shape Classification. ModelNet40 [61] is composed of 12,311 3D
objects (9,843/2,468 in training/testing) labelled with 40 object classes. We
sample 2048 points clouds from the objects following previous works [49, 70].
Visualization is provided in Figure B.2 of some of the renderings used in
training the 2D backbone in our pipeline.

ShapeNet Core55: 3D Shape Retrieval. The shape retrieval challenge SHREC [52]
uses ShapeNet Core55 is a subset of ShapeNet [2] for benchmarking. The dataset
consists of 51,162 3D mesh objects labeled with 55 object classes. The
training, validation, and test sets consist of 35764, 5133, and 10265 shapes.
We create a dataset of point clouds by sampling 5000 points from each mesh
object as in MVTN [17]. Visualization is provided in Figure B.3 of some of the
renderings used in training the 2D backbone in our pipeline.

ShapeNet Parts: 3D Part Segmentation. ShapeNet Parts is a subset of ShapeNet
[2] that consists of 13,998 point cloud objects for train and 2,874 objects
for the test from 16 categories and 50 parts. It is designed for the part
segmentation task [66]. Visualization is provided in Figure B.4 of some of the
renderings used in training the 2D backbone in our pipeline colored with the
ground truth segmentation labels.

###  B.2 Metrics

Classification Accuracy. The standard evaluation metric in 3D classification
is accuracy. We report overall accuracy (percentage of correctly classified
test samples) and average per-class accuracy (mean of all true class
accuracies).

Retrieval mAP. Shape retrieval is evaluated by mean Average Precision (mAP)
over test queries. For every query shape $\mathbf{S}_{q}$ from the test set,
AP is defined as
$AP=\frac{1}{\text{GTP}}\sum_{n}^{N}\frac{\mathbbm{1}(\mathbf{S}_{n})}{n}$,
where $GTP$ is the number of ground truth positives, $N$ is the size of the
ordered training set, and $\mathbbm{1}(\mathbf{S}_{n})=1$ if the shape
$\mathbf{S}_{n}$ is from the same class label of query $\mathbf{S}_{q}$. We
average the retrieval AP over the test set to measure retrieval mAP.

Segmentation mIoU. Semantic Segmentation is evaluated by mean Intersection
over Union (mIoU) over pixels or points. For every class label, measure the
size of the intersection mask between the ground truth points of that label
and the predicted points as that label. Then, divide by the size of the union
mask of the same label to get IoU. This procedure is repeated over all the
labels, and averaging the IoUs gives mIoU. We report two types of mIoUs:
Instance-averaged mIoU (averages all mIoUs across all objects ) and Category-
averaged mIoU (averages all mIoU from shapes of the same category, and then
average those across object categories).

###  B.3 Baselines

Point Cloud Networks. We include PointNet [48], PointNet++ [49], DGCNN [58],
PVNet [67], and KPConv [54], Point Transformer [70] and CurveNet [62] as
baselines that use point clouds. These methods leverage different convolution
operators on point clouds by aggregating local and global point information.

Multi-View Networks. We also compare against multi-view classification
approaches like MVCNN [53] and MVTN [17] as baselines for classification and
retrieval. Since there is no available multi-view pipeline for 3D part
segmentation, we adopt some of the multi-view segmentation baselines (_e.g_.
Label Fusion [56] and Mean Fusion [29]) for part segmentation to work in the
Voint space.

###  B.4 Implementation Details

Rendering and Un-Projection. We choose the differentiable point cloud renderer
$\mathbf{R}$ from Pytorch3D [50] in our pipeline for its speed and
compatibility with Pytorch libraries [46]. We render multi-view images with
size $224\times 224\times 3$. We color the points by their normals’ values or
keep them white if the normals are not available. Following a similar
procedure to [59, 17], the view-point setup is randomized during training
(using $M=8$ views) and fixed to spherical views in testing (using $M=12$
views).

Architectures. For the 2D backbone, we use ViT [10] (with pretrained weights
from TIMM library [60]) for classification and DeepLabV3 [3] for segmentation.
We used parallel heads for each object category for part segmentation since
the task is solely focused on parts. We use the 3D cross-entropy loss on the
3D point cloud output and the 2D cross-entropy loss when the loss is defined
on the pixels. When used, the linear tradeoff coefficient of the 2D loss term
is set to $0.003$. To balance the frequency of objects in part segmentation,
we multiply the loss by the frequency of the object class of each object we
segment. The feature dimension of the VointNet architectures is $d=64$, and
the depth is $l_{V}=4$ layers in $h_{\text{V}}$. The main results are based on
VointNet (MLP) variant unless otherwise specified. The coordinates
$\mathbf{x}$ can be optionally appended to the input view-features
$\widehat{\mathbf{x}}$, which can improve the performance but reduce the
rotation robustness as we show later in Section C.2 and Table C.3.

Training Setup. We train our pipeline in two stages, where we start by
training the 2D backbone on the 2D projected labels of the points, then train
the full pipeline end-to-end while focusing the training on the VointNet part.
We use the AdamW optimizer [37] with an initial learning rate of $0.0005$ and
a step learning rate schedule of $33.3\%$ every $12$ epochs for 40 epochs. The
pipeline is trained with one NVIDIA Tesla V100 GPU. We do not use any data
augmentation.

Object Only |
---|---
With Background |
PB_T50_RS (Hardest) |

Figure B.1: ScanObjectNN Variants. We show examples of point cloud renderings
of different variants of the ScanObjectNN [55]. These renderings are used in
training VointNet for 3D point cloud classification.

Figure B.2: ModelNet40. We show some examples of point cloud renderings of
ModelNet40 [61] used for 3D shape classification in our setup.

Figure B.3: ShapeNet Core55. We show some examples of point cloud renderings
of ShapeNet Core55 [2] used for 3D shape retrieval in our setup.

Figure B.4: ShapeNet Parts. We show some examples of point cloud renderings of
ShapeNet Parts [66] colored with ground truth segmentation labels. We use
these renderings as 2D ground truth to pre-train the 2D backbone $\mathbf{C}$
for 2D segmentation before training VointNet’s pipeline for 3D segmentation.

##  Appendix C Additional Results

###  C.1 3D Part Segmentation

Table C.1 reports the Category-averaged and Instance-averaged segmentation
mIoU of VointNet compared with recent methods on ShapeNet Parts [66]. The
discrepancy between the Voint results and the results of some point cloud
methods is that Voints heavily depend on the underlying 2D backbone and
inherit all its biases, especially those from pretraining. Hence, the 2D
backbone limits what the performance can reach with VointNet. We study the
effect of the backbone in detail in Section C.3. Figure C.1 shows qualitative
3D segmentation results for VointNet and Mean Fuse [29] as compared to the
ground truth.

###  C.2 Model Robustness

Rotation Robustness for 3D Classification. We follow the standard practice in
3D shape classification literature by testing the robustness of trained models
to perturbations at test time [33, 17]. We perturb the shapes with random
rotations around the Y-axis (gravity-axis) contained within $\pm 90^{\circ}$
and $\pm 180^{\circ}$ and report the test accuracy over ten runs in Table C.2.

Rotation Robustness for 3D Segmentation. We follow the previous 3D literature
by testing the robustness of trained models to perturbations at test time [33,
17, 18]. We perturb the shapes in ShapeNet Parts with random rotations in
$SO(3)$ at test time (ten runs) and report Ins. mIoU in Table C.3. Note how
our VointNet performance largely exceeds the baselines in this realistic
unaligned scenario. We can augment the training with rotated objects for the
baselines, which improves their robustness, but loses performance on the
unrated setup. Adding $xyz$ coordinates to the view-features of VointNet
improves the performance on an unrotated setup but negatively affects the
robustness to rotations.

|  | Part Segmentation
---|---|---
Method | Data Type | (Cat. mIoU) | (Ins. mIoU)
PointNet++ [49] | Points | 81.9 | 85.1
DGCNN [58] | Points | 82.3 | 85.1
Spider CNN[63] | Points | 81.7 | 85.3
PointCNN [32] | Points | 84.6 | 86.1
KPConv[54] | Points | 85.1 | 86.4
PTransformer[70] | Points | 83.7 | 86.6
CurveNet[62] | Points | - | 86.6
PointNet [48]∗ | Points | 75.5 | 80.1
DGCNN [58]∗ | Points | 74.5 | 80.1
Label Fuse [56]∗ | Voints | 78.9 | 80.0
Mean Fuse [29]∗ | Voints | 79.1 | 77.5
VointNet (MLP)∗ | Voints | 80.3 | 80.7
VointNet (GCN)∗ | Voints | 76.8 | 76.6
VointNet (GAT)∗ | Voints | 79.9 | 81.2

Table C.1: 3D Part Segmentation on ShapeNet Parts. We compare VointNet against
other methods in 3D segmentation on ShapeNet Parts [66]. ∗ indicates test
results we reproduce by our experiments, while other results are reported from
[70] (no-voting results).

Ground Truth | VointNet (ours) | Mean Fuse [29]
---|---|---
|  |
|  |
|  |
|  |
|  |

Figure C.1: Qualitative Comparison for 3D Part Segmentation. We compare our
VointNet 3D segmentation prediction to Mean Fuse [29] that is using the same
trained 2D backbone. Note how VointNet distinguishes detailed parts (_e.g_.
the car window frame). Beware that visualization colors can shift if an extra
label is predicted (_e.g_. the motorbike labels are correct).

| Rotation Perturbations Range
---|---
Method | $0^{\circ}$ | $\pm 90^{\circ}$ | $\pm 180^{\circ}$
PointNet [48] | 88.7 | 42.5 | 38.6
PointNet ++ [49] | 88.2 | 47.9 | 39.7
RSCNN [33] | 90.3 | 90.3 | 90.3
MVTN [17] | 91.7 | 90.8 | 91.2
VointNet (ours) | 91.5 | 90.9 | 91.1

Table C.2: Rotation Robustness for 3D Classification. At test time, we
randomly rotate objects in ModelNet40 [61] around the Y-axis (gravity) with
different ranges and report the overall accuracy.

| Segmentation Under Rotation
---|---
Method | Unrotated | Rotated
PointNet [48] | 80.1 | 36.6 $\pm$0.2
DGCNN [58] | 80.1 | 37.1 $\pm$0.2
PointNet + Aug. | 65.8 | 65.8 $\pm$0.1
DGCNN + Aug. | 60.7 | 60.7 $\pm$0.2
Mean Fuse [29] | 79.1 | 61.6 $\pm$0.1
Label Fuse [56] | 78.9 | 61.0 $\pm$ 0.1
VointNet (w/o $xyz$) | 79.6 | 65.4 $\pm$0.1
VointNet (w/o $xyz$) + Aug. | 68.0 |  68.5 $\pm$ 0.1
VointNet (w/ $xyz$) | 81.2 | 61.5 $\pm$0.2

Table C.3: Rotation Robustness for 3D Part Segmentation. At test time, we
randomly rotate objects from ShapeNet Parts [66] and report the Ins. mIoUs of
our VointNet compared to trained PointNet [48] and DGCNN [58]. Note how
VointNet’s performance largely exceeds the baselines in realistic unaligned
scenarios, highlighting the benefit of view dependency. If we use rotation
augmentation in training for the baselines, the rotated performance improves,
but the unrotated performance drops.

###  C.3 Detailed Analysis

Classification Backbone. We study the effect of ablating the 2D backbone
$\mathbf{C}$ for 3D classification on ModelNet40. We show in Table C.4 the
performance of VointNet (MLP) when Vit-B [10] and ResNet-18 [20] are used
_vs_. using the same backbones but in a naive multi-view setup [53]. We also
show that following the per-point classification setup instead of the per-
shape for 3D shape classification leads to worse performance for VointNet and
the naive multi-view. This is why we used the per-shape approach when adopting
VointNet for 3D classification (using one Voint for the entire shape).

Feature Size ($d$). We study the effect of the feature size $d$ on the
performance of VointNet (MLP) in 3D part segmentation on ShapeNet Parts [66]
and plot the results ( with confidence intervals) in Figure C.2. We note that
the performance peaks at $d=128$, but it is close to what we use in the main
results ($d=64$).

Model Depth ($l_{v}$). We study the effect of the model depth $l_{v}$ on the
performance of VointNet (MLP) in 3D part segmentation on ShapeNet Parts [66]
and plot the results ( with confidence intervals) in Figure C.3. We note that
model depth of VointNet does not enhance the performance significantly. Our
choice of $l_{v}=4$ balances the performance and the memory/computations
requirements of VointNet (MLP).

Distance to the Object. We study the effect of distance to the object in
rendering as in Figure C.6 to the performance of VointNet (MLP) in 3D part
segmentation on ShapeNet Parts [66] and plot the results ( with confidence
intervals) in Figure C.4. We note that our default choice of 1.0 is actually
reasonable. This choice of distance shows the object entirely ( as illustrated
in Figure C.6), but also cover the details needed for small parts segmentation
(see Figure C.1).

Image Size ($H,W$). We study the effect of the image size $H\&W$ on the
performance of Mean Fuse [29] baseline when training the 2D backbone for 3D
part segmentation. We plot the results ( with confidence intervals) in Figure
C.5.

Time and Memory Requirements. To assess the contribution of the Voint module,
we take a macroscopic look at the time and memory requirements of each
component in the pipeline. We record the number of floating-point operations
(GFLOPs) and the time of a forward pass for a single input sample. In Table
C.5, the VointNet module contributes negligibly to the memory requirements
compared to multi-view and point networks.

###  C.4 Visualizations

In Figure C.6, we visualize the multi-view renderings of the point clouds
along with the 2D learned features based on the DeepLabV3 [3] backbone. These
features are then unprojected and transformed by VointNet to obtain 3D
semantic labels.

View Aggregation | 2D Backbone
---|---
| ResNet18 | ViT-B | DeepLabV3
| (per-shape) | (per-shape) | (per-point)
Naive Multi-View | 91.2 | 92.5 | 89.0
VointNet | 91.2 | 92.8 | 10.2

Table C.4: Ablation Study for 3D Classification. We study the effect of
different 2D backbone for ModelNet40 3D classification task. We compare
VointNet’s performance to naive multi-view (_e.g_. MVCNN [53] or Mean Fuse
[29]) using the same 2D backbone. Note that using the per-point classification
setup instead of the per-shape for 3D shape classification leads to worse
performance for VointNet and the naive multi-view.  Figure C.2: The Effect of
Feature Size $d$. We plot Ins. mIoU of 3D segmentation _vs_. the feature size
$d$ used in training on ShapeNet Parts [66]. We note that the performance
peaks at $d=128$, but it is close to what we use in the main results ($d=64$).
Figure C.3: The Effect of Model Depth $l_{v}$. We plot Ins. mIoU of 3D
segmentation _vs_. the model depth $l_{v}$ used in training on ShapeNet Parts
[66]. We note that model depth of VointNet does not enhance the performance
significantly. Our choice of $l_{v}=4$ balances the performance and the
memory/computations requirements of VointNet (MLP). Figure C.4: The Effect of
Distance to the Object. We plot Ins. mIoU of 3D segmentation _vs_. the
distance to the object used in inference on ShapeNet Parts [66]. We note that
our default choice of 1.0 is actually reasonable. This choice of distance
shows the object entirely ( as illustrated in Figure C.6), but also cover the
details needed for small parts segmentation (see Figure C.1). Figure C.5: The
Effect of Image Size $H,W$. We plot Ins. mIoU of 3D segmentation _vs_. the
image size used in inference on ShapeNet Parts [66].

Network | GFLOPs | Time (ms) | Parameters # (M)
---|---|---|---
MVCNN [53] | 43.72 | 39.89 | 11.20
ViewGCN [59] | 44.19 | 26.06 | 23.56
ResNet 18 [20] | 3.64 | 3.70 | 11.20
ResNet 50 [20] | 8.24 | 9.42 | 23.59
ViT-B [10] | 33.70 | 12.46 | 86.57
ViT-L [10] | 119.30 | 29.28 | 304.33
FCN [36] | 53.13 | 10.34 | 32.97
DeeplabV3 [3] | 92.61 | 20.62 | 58.64
PointNet [48] | 1.78 | 4.24 | 3.50
DGCNN [58] | 10.42 | 0.95 | 16.350
MVTN [17] | 1.78 | 4.24 | 3.5
VointNet (MLP) | 1.90 | 2.90 | 0.04
VointNet (GCN) | 16.18 | 32.10 | 0.05
VointNet (GAT) | 32.05 | 68.71 | 0.07

Table C.5: Time and Memory Requirements. We assess the contribution of the Voint module to the time and memory requirements in the multi-view and point cloud pipeline. Note that VointNet (shared MLP) is almost 100 times smaller than PointNet [48]. (INPUT) |
---|---
(PRED 2D) |
(PRED 3D) |
(GT) |
(INPUT) |
(PRED 2D) |
(PRED 3D) |
(GT) |
Figure C.6: Multi-view Projected Segmentation. We show how, after rendering
points, we can segment in the image space. For each example, we show (INPUT):
the projections of the points (colored with normals) used in training with
random view-points. (PRED 2D): the segmentation prediction of the 2D backbone
(DeepLabV3) [3]). (PRED 3D): the unprojected 3D segmentation prediction. (GT):
the 3D segmentation ground truth.