Representing Shape Collections With Alignment-Aware Linear Models

Romain Loiseau1, 2

Tom Monnier1

Mathieu Aubry1

Loïc Landrieu2

1LIGM, Ecole des Ponts, Univ Gustave Eiffel, CNRS, France

2LASTIG, Univ. Gustave Eiffel, ENSG, IGN, F-94160 Saint-Mande, France

In this paper, we revisit the classical representation of 3D point clouds as linear shape models.
Our key insight is to leverage deep learning to represent a collection of shapes as affine transformations of low-dimensional linear shape models.
Each linear model is characterized by a shape prototype, a low-dimensional shape basis and two neural networks. The networks take as input a point cloud and predict the coordinates of a shape in the linear basis and the affine transformation which best approximate the input.
Both linear models and neural networks are learned end-to-end using a single reconstruction loss.
The main advantage of our approach is that, in contrast to many recent deep approaches which learn feature-based complex shape representations, our model is explicit and every operation occurs in 3D space.
As a result, our linear shape models can be easily visualized and annotated, and failure cases can be visually understood. While our main goal is to introduce a compact and interpretable representation of shape collections, we show it leads to state of the art results for few-shot segmentation.
Code and data are available at: <https://romainloiseau.github.io/deep-linear-shapes>

§ INTRODUCTION

Discovered linear models. Our approach discovers without supervision  linear shape models from large collections of shapes. We show two examples of two-dimensional families and eight additional prototypes discovered for ABC [34] (top) and ShapeNet [8] (bottom).

Method overview.
Given an input point cloud $x$, we predict for each shape model $\cR^k$ the element that best reconstructs the input:
the projection network $\cPk$ outputs the coordinates $a$ of a shape in a linear family, and the alignment network $\cAk$ predicts the parameters of an affine transformation $\cA^k(x)$ which is applied to the selected shape. The input point cloud is then assigned to the shape model that best reconstructs it, here highlighted in green.

Picture a company acquiring thousands of 3D scans of technical components; how to leverage, organize, or even simply visualize these 3D models?
Deep shape analysis techniques have flourished over the last years [26] but, even when motivated by geometric intuitions, these methods and their results remain hard to interpret and interact with.
Moreover, they are often limited by the availability of domain and application-specific annotations.
Instead of pushing for even more complex architectures, we
operate directly in 3D space and revisit the simple linear shape model with a deep learning perspective. As illustrated in fig:teaser, we model a collection of 3D shapes with a set of low-dimensional linear shape models. Each linear model is defined by a prototype 3D point cloud and a set of vector basis that can be interpreted as fields of translation vectors for each point of the prototype. By adding a linear combination of this basis vector to the prototype, one can continuously move in a low-dimensional subspace of the shape space.

We face three key challenges when trying to represent 3D shape collections with such linear models. First, comparing shapes using Chamfer or Earth Mover distances has strong limitations for shape analysis, since they are impacted by simple rigid or affine shape transformations, which cannot be easily represented by linear models. Transformation-invariant distances such as the Gromov-Hausdorff Distance [39] can be defined to overcome this problem, but they are typically very hard to work with. Second, finding the coordinates in the shape basis that best reconstruct a sample according to a given similarity measure is a difficult non-convex problem.
Third, operations as simple as averaging are non-trivial for point clouds, and dimensionality reduction techniques such as Principal Component Analysis [59] do not directly apply.

In this work, we present an unsupervised approach that learns small sets of linear shape models to explain large collections of point clouds. We propose to solve this task with a clustering formulation directly in 3D space, where clusters are associated to linear shape families, each modeled as a reference prototype point cloud and a set of basis vectors that can be interpreted as displacement fields.
We explore two ways of defining such displacement fields - either using a pointwise parametrization or an implicit one based on parametric differentiable functions of 3D space - and analyze their benefits. In addition, to predict the coordinates of a point cloud in the linear basis and account for shape transformations, we extend the idea from the work of Monnier  [41] on transformation-invariant image clustering to the setting of 3D shape alignment. By jointly learning linear shape families and parametric functions predicting both shape basis coordinates and alignment parameters, our approach is able to discover rich and meaningful shape models from a collection of point clouds without any supervision.

We believe that our method has strong advantages compared to recent unsupervised 3D shape analysis approaches. First, by manipulating objects directly in 3D space, our results are
easy to interpret and visualize.
Second, our linear shape models can serve as a mean to explore large collections of raw 3D point clouds. Finally, we show that despite its simplicity, our model yields competitive results for shape clustering and state-of-the-art results for few-shot shape segmentation.

Our contributions can be summarized as follows:

* we present an unsupervised method to represent large point cloud collections with a small set of linear families of shapes;

* we extend the DTI clustering framework to learn linear shape models by introducing projection networks;

* we analyse two different representations for linear shape modeling and show the benefits of representing them with continuous functions of space rather than pointwise displacements;

* we demonstrate qualitative results for visualizing the large unstructured ABC dataset [34] and obtain state-of-the-art few-shot segmentation performances on the standard ShapeNetPart dataset [8].

§ RELATED WORK

Point cloud distances and alignment.
Classical similarity measures between point clouds include the Chamfer and Earth mover distances. These distances are however not invariant to rigid transformations.  The Gromov-Hausdorff distance [39] provides a nice framework to define a transformation-invariance distance, but is difficult to use in practice.
Transformation-Invariant distances were also defined for images and used for clustering by Frey and Jojic [18, 20, 21, 19].
The Transformed Component Analysis (TCA) approach  [19] would be particularly relevant, although operating with a discrete set of transformations may be too limiting for aligning 3D shapes. Applying them to 3D point clouds would require to align them.
This is classicaly done using the the Iterative Closest Point (ICP) algorithm  [3]. Instead, we take inspiration from the Deep Transformation-Invariant framework [41] and use neural networks to predict alignment and define similarity.

Linear Shape Modeling. The idea of representing a collection of images using a low-dimensional image basis was first developed for face images [49]. Popularized by the classical eigenfaces model [55], linear models have since been applied to diverse computer vision problems and data. A linear 3D face model was designed in [4] and applied to new view synthesis. [12] demonstrated applications to medical data.
Non-rigid surface-from-motion can also benefit from linear shape basis decomposition to recover 3D shapes [6, 54, 13]. An application of linear modeling to unsupervised 3D keypoint discovery was recently demonstrated in [17].

These linear models are typically learned from a set of examples by principal component analysis, factorization techniques [53], or defined manually [58]. Additionally, some recent works propose to analyse shape collections through implicit representations [32, 64, 14]. In contrast, we propose a learning-based approach to model arbitrary unregistered shapes from large collections of examples, and we use several low-dimensional linear families.

Deep Learning for 3D Analysis.
Neural networks successfully tackled numerous challenges in 3D shape analysis. The main approaches can be broadly classified depending on the representation of 3D data they leverage,  voxels [11], point clouds [44], graphs [35], surfaces [7, 24], structured models [37, 23], or more recently implicit volumetric models [43, 10, 40]. They have been successfully applied to tasks as diverse as classification [38], segmentation [52], shape generation [24, 51], matching [42], denoising [29] and compression [30]. In this paper, we focus on 3D point clouds but our approach is general and could be extended to other representations.

The key idea to use Multi-Layer Perceptron (MLP) on point clouds was initially proposed for shape classification and segmentation by Qi  [44] with an architecture called PointNet, and for 3D point cloud generation in [16]. To answer the difficulty of annotating 3D data, new approaches are able to perform self-supervised and unsupervised feature learning [62, 46, 28] and low-shot segmentation [25, 57, 22]. Especially related to ours is the recent 3D capsule approach [63] that explicitly tries to design a shape representation invariant to 3D transformations and can be applied to many tasks. However, the latent representations learned with 3D capsules and the associated generation process are difficult to interpret.
Also related to ours is the approach of Deprelle  [15] which proposes a 3D shape reconstruction model obtained by combining and transforming learned elementary structures. This method shares similarities with ours as it allows us to learn prototypes of parts of shapes. However, it focuses on reconstruction accuracy, uses a single prototype per part and mainly follows the black-box AtlasNet [24] deformation framework.

§ MODELING SHAPE COLLECTIONS

Our goal is to explain a collection of $N$ point clouds $\xn$ with a small set of $K$ shape models. For simplicity, we assume that all point clouds have the same number $M$ of points. We propose to solve this task with a clustering formulation described in sec:clustering. We then describe how we model alignment (sec:align_aware) and linear shape families (sec:linearmodels) resulting in our final modeling. Finally, we present how we parametrize our linear shape models and give some training details (sec:param).

§.§ Method overview

We build a set of $K$ shape models $\cR=\{\cR^1,\ldots,\cR^K\}$.
Each $\cRk$ maps a sample point cloud $x$ to a reconstructed point cloud $\cRk(x)$ which can be interpreted as the approximation of $x$ by the corresponding model. We denote by $d$ a distance between point clouds which measures the quality of a reconstruction. We use the Chamfer distance in all our experiments.
We learn the shape models $\cR$ by minimizing the loss

\begin{align}
\cL(\cR)
&=\sum_{x\in \xn} \min_{k=1}^K
\left(x,\cRk(x)\right)~,
\label{eq:loss}
\end{align}

which can be interpreted as a clustering objective defined as the sum of the reconstruction errors with optimal cluster assignment.

Prototype model.
The simplest form of $\cRk$ is a constant function:  $\cRk_\text{proto}(x)=\ck\in\bR^{M \times 3}$ where each $\ck$ can be seen as a prototype point cloud. Such prototype point clouds can be learned by minimizing $\cL$ with batch Stochastic Gradient Descent (SGD). This
amounts to performing stochastic K-means [5] for $3$D point clouds. Note that this is a weak reconstruction model, however, the goal of this paper is not to learn the most faithful reconstruction, but rather to summarize the collection.

§.§ Alignment-Aware Model

A clear limitation of the prototype model is that it does not take into account simple geometric transformations of the point clouds, such as rigid transformations.
For example, point clouds can be close to a model's prototype $\ck$ according to the distance $d$, while a rotated or translated version of the same point cloud is far away.
We would like both point clouds to be associated with the same shape model.
To address this issue, we incorporate in each model $\cRk$ an affine alignment component. In practice, we use neural networks $\cAk$ - which we refer to as alignment networks - whose goal is to predict an affine transformation $\cAk(x)$
aligning the prototype $\ck$ with a target point cloud $x$. This results in an alignment-aware model $\cRk_\text{align}$ defined by:

\begin{equation}
\cRk_\text{align}(x)=\cAk(x)\left[c^{k}\right]~,
\label{eq:Rdti}
\end{equation}

where the affine transformation $\cAk(x)$ is applied to each point of the prototype point cloud $\ck$. The alignment networks $\cA^{1},\ldots,\cA^{K}$ can be trained alongside the prototypes $c^{1},\ldots,c^{ K}$ by minimizing (<ref>). This model can be seen as an extension of the recent Deep Transformation-Invariant (DTI) clustering framework [41] developed for images to point clouds. Indeed, our alignment models can be understood as defining an approximation of an affine-invariant version of the distance $d$ according to which the clustering is performed. In this paper, we rather view these networks as an integral part of the shape models.

Note that different transformation models could be considered. In our experimental analysis, we study variations of the model using weaker transformations, such as rigid transformations or scaling, and show the benefits of the affine model. On the contrary, one could consider complex deformations parametrized by deep networks, such as the ones used in FoldingNet [62] or AtlasNet [24], which would surely lead to higher accuracy reconstructions. However, such transformations completely change the geometry of a point cloud and are hard to interpret.

§.§ Linear Shape Modeling

Our goal in this section is to model changes in objects more subtle than those that can be modeled by affine transformations, such as the angle of the wings of an airplane, while maintaining the model intepretability. We propose to associate a linear shape family to each prototype point cloud.

Linear shape families.

For each model $k$, we define a linear shape family as a pair formed by
(i) a prototype point cloud $\ck$ in $\bR^{M \times 3}$ and
(ii) a set $\vk$ of $D$ basis vectors $\vk=\{\vk_1,\ldots,\vk_D\}$, where each $\vk_i \in \bR^{M \times 3}$ associates to each point of the prototype a 3D vector and can be interpreted as displacement fields. Each $(\ck,\vk)$ defines a continuous collection of shapes covered by translating the points of $\ck$ along the directions defined by $\vk$. Each element $u$ of the linear family $(\ck, \vk)$ is characterized by a vector $a$ in $\bR^D$ defining its coordinates in the linear shape family:

\begin{equation}
u = \ck
+  \sum_{i=1}^D a_i\,\vk_{i}~.
\label{eq:linear}
\end{equation}

The vector $a$ can be interpreted as the set of amplitudes to apply to the displacement fields $\{\vk_1,\ldots,\vk_D\}$. Note this formally describes an affine space but we follow the convention of previous works and refer to it as linear.[An analogy can be made with the face reconstruction model EigenFace [56]: $c$ is equivalent to the mean face, and $v$ to the eigenfaces.] Also note that we do not explicitly enforce linear independence between basis vectors, but their high dimensionality ($M\times 3$) leads to such independence in practice.

Projection networks. If we had access to ordered point clouds, lists of $M$ points in $\bR^3$ where the $i$-th points are in correspondence, we would be able to use the $L_2$ distance to measure point clouds similarity. In this case, computing the coordinates of the element of the linear family closest to a target point cloud would simply amount to performing Euclidean projection. This is however not the case for unordered point clouds, for which the notion of distance is more complicated. For common point cloud similarity measures such as the Chamfer distance, finding the closest point cloud in a linear family is a difficult non-convex optimization problem. This task is made even harder by the fact that we
use our alignment networks to transform the elements of the family before comparing them with the input cloud.

Therefore, we propose to leverage deep learning to estimate which element of a linear family is the closest to a target point cloud after alignment. More specifically, we associate to each linear family $(\ck,\vk)$ a neural network $\cPk$ which aims at associating to a given input sample the coordinates of the element in the linear family minimizing the distance $d$. The output of the network $\cPk(x)\in\bR^D$ is interpreted as the coordinates $a$ of the point cloud defined in (<ref>). By analogy with the $L_2$ distance case, we refer to these networks as projection networks.

Full model. We define our final shape model $\cR$ as a collection of models $\cRk_\text{full}$ each composed of a linear family $(\ck,\vk)$, an alignment network $\cAk$ and a projection network $\cPk$.
Given a target point cloud $x$, our model reconstructs it by
(i) selecting an element of the linear family $(\ck,\vk)$ through the projection network $\cPk$, and
(ii) aligning it with the target using the transformation predicted by the alignment network $\cAk$. More formally, we write each shape model as:

\begin{equation}
\cRk_\text{full}(x)=\cAk(x) \left[c^{k}+ \sum_{i=1}^D \left[\cPk(x)\right]_i \vk_i\right],
\label{eq:reconstruction}
\end{equation}

where $\left[\cPk(x)\right]_i$ refers to the i-th component of $\cPk(x)$ and the affine transformation $\cAk(x)$ is applied to each point of the point cloud independently. Again, we optimize jointly the $\ck, \vk, \cAk$ and $\cPk$ to minimize the reconstruction loss defined in Equation (<ref>).

Learned prototypes and comparisons. We compare the prototypes from our different shape modeling discovered in ABC [34] (left, $5$ shape models out of $10$) and ShapeNetCore [8] (right, $5$ shape models out of $55$). Note how sharp the prototypes become when the shape modeling complexity increases, respectively with alignment-awareness and 5-dimensional linear families.

§.§ Parameterization and training details

We first describe how we parametrize the linear families, then provide implementation details such as networks architecture and our curriculum learning strategy.

Linear family parametrization.
While the prototypical point cloud  $\ck$ is modeled directly using learnable parameters in $\mathbb{R}^{M\times 3}$, the basis vectors $\vk_i$ can be parametrized in two different ways:

* Pointwise parametrization: for each model $k$, we represent $\vk$ as vectors of learnable parameters of size $D\times(M \times 3)$ that can directly be interpreted as $D$ pointwise displacement vectors of the prototype $\ck$.

* Implicit parametrization: we use implicit parametric functions of the 3D space modeled as neural networks to define the displacement fields. More precisely, for each model $k$ and basis dimension $i$, we learn a parametric function $\mathcal{V}_i^k:\bR^3 \mapsto \bR^3$ mapping any point in the 3D space to a displacement direction.
Writting $[\ck]_p$ the 3D coordinates of the $p$-th point of prototype $\ck$, the 3D coordinates $[\vk_i]_p$ of the $i$-th basis vector associated to the point $p$ are  $[\vk_i]_p=\mathcal{V}_i^k([\ck]_p)$.

Intuitively, the pointwise parametrization seems better suited for modeling complex and discontinuous transformations within a shape family such as the appearance/disappearance of object parts. On the contrary, the transformations learned with implicit parametrizations are derived from continuous functions of the 3D space and can be expected to be more regular.

We compare both settings in sec:quant, and show that pointwise parametrizations provide better shape reconstructions, but that implicit parametrization yields more interpretable transformations preserving semantic correspondences. Thus, unless specified otherwise, we use the implicit parametrization of the basis in the rest of the paper.

Architecture. For each model $k$, the alignment network $\cAk$ takes as input a point cloud and outputs a vector in $\bR^{12}$ corresponding to a linear 3D operator and a translation vector applied to each point of the model. The projection network $\cPk$ also takes a point cloud as input and outputs a vector in $\bR^{D}$ that is interpreted as coordinates in the linear family $(\ck,\vk)$. These networks share a common PointNet [44] backbone encoder which acts as a global feature extractor. This shared encoder starts with a sequence of three linear layers with batch normalization [31] and ReLU activation acting on points independently and sequentially generating representations of size 64, 128 and 1024, and ends with a max-pooling over all points.
This encoder is then followed by $2\times K$ Multi-Layer Perceptrons (MLPs)
corresponding to each prediction task (alignment or projection) and each shape model. Each MLP has one hidden layer of size 128. The implicit parametrizations $\cV^k_i:\bR^3 \mapsto \bR^3$ are MLPs with 2 hidden layers of size 128.

Curriculum learning. Inspired by the curriculum learning of [41], we propose to learn our models by gradually increasing the models complexity. We first learn raw prototype models ($\cRk_\text{proto}$), an optimization which corresponds to performing a gradient-based K-means algorithm in the 3D space. Second, we augment each model with alignment awareness ($\cRk_\text{align}$). Finally, we gradually increase the linear families dimension up to the desired one, resulting in our final shape model ($\cRk_\text{full}$).

Implementation details.
Our implementation - which will be released upon publication - uses PyTorch, Torch-Points3D [9], and an efficient CUDA implementation of the Chamfer distance which significantly speeds up training.
With $K=10$ prototypes and $D=5$, our model has $4.6M$ parameters. For comparison, the reconstruction models proposed by Wang [57] and Groueix [25] have respectively $2.6M$ and $10.0M$ parameters.
See our supplementary material for additional details.

§ EXPERIMENTS

In this section, we analyze the benefits of our method to represent shape collections, first qualitatively (Section <ref>) then quantitatively  (Section <ref>).
Finally, we demonstrate that it leads to results on par with state of the art for few-shot and low shot  shape segmentation (Section <ref>).

§ CONCLUSION

We presented a new take on linear shape models with deep learning, representing large un-annotated collections of 3D shapes. Our alignment-aware model produces concise, expressive and interpretable overviews of unaligned point clouds collections. We
show that our method
leads to state-of-the-art results for few-shot segmentation.

*AcknowledgementsThis work was supported in part by ANR project READY3D ANR-19-CE23-0007 and HPC resources from GENCI-IDRIS (Grant 2020-AD011012096). We thank François Darmon, Damien Robert, Vivien Sainte Fare Garnot and Yang Xiao for inspiring discussions and valuable feedback.

*Supplementary material

§ IMPLEMENTATION DETAILS

10 42 43 41 2 8 54 15 17 38 30
full $D=5$last_epochshapenet0
1 35 4 24 49 51 5 19 31 44 52
full $D=5$last_epochshapenet0
45 37 48 12 25 50 47 36 6 21 40
full $D=5$last_epochshapenet0
3 14 33 0 29 28 16 20 18 22 32
full $D=5$last_epochshapenet0
11 9 13 23 7 22 27 34 46 39 26
full $D=5$last_epochshapenet0
Modeling ShapeNet.
Prototypes from all $55$ linear shape models learned on ShapeNet [8], with our final $5$-dimensional model “Ours, full $D=5$".
In this figure, the prototypes have been manually rearranged with respect to their semantics. Note that some diverse classes such as tables or chairs are modeled by several models.

Our implementation uses PyTorch, Torch-Points3D [9], and an efficient CUDA implementation of the Chamfer distance which significantly speeds up training. Code and data are available at: <https://romainloiseau.github.io/deep-linear-shapes>

Training strategy. We use the Adam optimizer [33] with a learning rate of $0.001$, a batch size of 64, and neither weight decay nor data augmentation. Our model takes point clouds in $\bR^{1024\times3}$ as input for all experiments, except for the few-shot segmentation task that takes point clouds in $\bR^{2048\times3}$ as input.

Curriculum learning.Inspired by the curriculum learning strategy of [41], we propose to learn our models by gradually increasing the models complexity. We first learn raw prototype models ($\cRk_\text{proto}$), an optimization which corresponds to performing a gradient-based K-means algorithm in the 3D space. Second, we augment each model with alignment awareness ($\cRk_\text{align}$). Finally, we gradually increase the linear families dimension up to the desired one, resulting in our final shape model ($\cRk_\text{full}$). Curriculum learning allows the model to choose the number of displacement fields $D$ according to the complexity of the studied dataset. Early stopping occurs when the benefit of adding a new degree of liberty (increasing $D$ by one) does not meet a criterion on the loss or on a validation task, see Figure <ref>.

Alignment networks and basis vectors are initially set to identity and zero, respectively. When unfreezing a new module (alignment or a dimension of projection), the learning rate for the new weights is initially set to a tenth of the learning rate applied for the rest of the network, and gradually increased over $50$ epochs to the global learning rate. This “warm-up" heuristic helps the network learn more smoothly from one step of the curriculum to the next.

Influence of $D$ on ModelNet. The reconstruction error (CD) decreases with added degrees of freedom. In contrast, the clustering Accuracy stops increasing when $D>=3$, hinting that we have reached a sufficient level of complexity.

Initialization strategy.As it is the case for many clustering algorithms, initialization can be critical. In our case, we initialize the prototype point clouds with samples of the training set chosen according to a k-means++ strategy [2] with respect to the Chamfer distance.

Cluster reassignment.To prevent empty clusters, we reassign at the end of each epoch any cluster that was selected fewer times than $20\%$ of the expected size of clusters ($N/K$) in the evenly distributed cluster assignment of Equation 1. Clusters are reassigned by selecting and duplicating another cluster. The duplicated cluster is chosen with a probability proportional to the mean of its reconstruction error over the last epoch. To break the symmetry, we add Gaussian noise with variance $10^{-4}$ to both its prototype and vector basis. The alignment and projection networks are copied without adding noise. We decrease the reassignment threshold tenfold after each curriculum step in order to preserve less populated but expressive clusters.

Low-shot supervised segmentation results on ShapeNetPart. We report the IoU averaged over all classes.

Training      SONet
3D-PointCapsNet     Ours

data      [36]
[63]
full $D=5$
$1\%$     $64$     $67$     $68$

$5\%$     $69$     $70$     $72$

Memory and Speed.With $K=10$ prototypes and $D=5$, our model has $4.6M$ parameters. For comparison, the reconstruction models proposed by Wang  [57] and Groueix  [25] have respectively $2.6M$ and $10.0M$ parameters. Our model can be trained on a single NVIDIA GeForce RTX 2080Ti within a few hours on the $3\,991$ samples of ModelNet10, and in less than a day on ShapeNetCore. Inference on all samples from ShapeNetCore ($\approx 50k$ shapes) takes less than $4$ minutes.

Choice of $K$.
The number of models can be automatically selected through usual model selection heuristics such as the Bayesian Information Criterion (BIC), as we show in Figure <ref>.
Being entirely unsupervized, there is no restriction on how linear families relate to classes: complex classes can be represented by several models, and similar classes by a single family. However, as demonstrated in our clustering experiments, when the number of classes and models are the same, linear families and classes tend to be assigned on a one-to-one basis

Model Selection. We can select the number of clusters $K$ using the BIC. We obtain $7$ clusters for ModelNet and $30$ for ShapeNet, which is consistent with the shapes' diversity.

§ LOW-SHOT SETTING

§ SELF-SUPERVISED CLASSIFICATION

Self-supervised classification.
To assess the capacity of our approach to extract relevant information from 3D models, we evaluate it in a standard self-supervised feature learning setup. We train our method on ShapeNetCore with $55$ 5-dimensional families, extract features from our results, and train and evaluate a linear SVM on the ModelNet10/40 train-test split following standard practices. We define three features that can be extracted from our model. A first type of features is defined as the soft-minimum of the distance between an input point and the reconstruction predicted by each linear shape model, with a temperature taken here as $100$ (“Distances"). These first features are complemented by concatenating the coordinates predicted by the projection networks (“Distances and coordinates"). Finally, we directly use the features from our point cloud encoder (“Embedding").

We present the results obtained with these different features in tab:classification, and compare to approaches specifically designed for self-supervised classification. While our method's performance is below most of these dedicated approaches, our results are still promising. Interestingly, we can see that adding shape coordinates to the distances significantly boosts the results, and even outperforms the latent embedding learned by the encoder, which outlines that our learned shape spaces are informative and meaningful.

Results of the self-supervised classification task. We report the accuracy of a linear SVM trained on the training set of ModelNet using as input feature the reconstruction error to $55$ linear shape families of dimension $5$ augmented or not by the predicted coordinates, or the latent vector outputted by the point cloud encoder. In parenthesis, we report the name of the backbone network used (PointNet [44], PointNet++ [45], or VGG19 [48]).

$N_\text{feat.}$    MN40     MN10
Ours, full $D=5$ (PointNet)

Distances   $55$     $70.5$      $86.2$

Distances and coordinates     $330$     $86.8$      $90.9$

Embedding     $512$    $86.2$       $89.6$
3D-GAN [60]    $7168$     $83.3$      $91.0$

VIP-GAN [27] (VGG19)   $512$     $90.2$     $92.2$

FoldingNet [62]     $512$     $88.4$     $94.4$

Latent-GAN [1] (PointNet)   $512$     $84.5$     $95.4$

Rec-Space [46] (PointNet)     $512$     $87.3$     $91.6$

Multi-Task [28]     $512$     $89.1$     —

Label-Efficient [22] (PointNet++)     $512$     $89.8$   —

§ LEARNED LINEAR SHAPE MODELS

In this section, we show qualitative examples of learned linear shape models. In fig:lsm_sn, we represent all $55$ models learned on ShapeNet [8], with our final $5$-dimensional model “Ours, full $D=5$". This illustrates how our model can be used to represent concisely a diverse and complex dataset such as ShapeNet without any supervision. In fig:lsm_abc, we  show the $10$ models learned on our subset of ABC [34] for the three steps of our curriculum strategy, illustrating the benefit of both alignment networks and linear families to learn such a diverse shape dataset. We also display the vector basis for all $5$ learned dimensions, representing the richness of each linear shape family.

Lastly, we represent in fig:lsm_mn10 the models learned on ModelNet with random rotations. We observe that when alignment networks are used, the obtained prototypes are similar to the ones obtained on the aligned version of the dataset. This shows that our approach can be used successfully on raw, un-aligned datasets.

0 1 2 3 4 5 6 7 8 9
Modeling ABC.
$10$ prototype from linear shape models learned on the ABC dataset [34].
Note that the prototypes are smoother and sharper when using alignment networks and $5$-dimensional linear families.
On the right-most columns, we illustrate the $5$ dimensions of the linear family of each shape model. Each linear family spans a rich subset of the space of shapes.

0 1 2 3 4 6 7 8 9 5

$10$ prototypes of the linear shape models learned from the aligned ModelNet10 dataset.

6 2 4 8 9 3 0 1 7 5

$10$ prototypes of the linear shape models learned from the rotated ModelNet10 dataset.

Modeling ModelNet10.
Prototype learned on ModelNet's [61] aligned version (fig:modelnet:aligned) and with random $z$-axis rotations (fig:modelnet:rotated). In this figure, the models are manually rearranged to be in correspondence across the two experiments.
Note how our model without alignment networks (“Ours, proto") is unable to learn meaningful prototypes on un-aligned data. In contrast, our models with alignment networks learn sharp and informative prototypes despite the rotations. This shows that alignment networks allow our model to handle a raw, un-aligned dataset to produce a compact overview of its shape diversity.

§ RECONSTRUCTION RESULTS

We show some reconstruction results in fig:rec_abc and fig:rec_sn for ABC and ShapeNet respectively.
For each model, we represent some sample shapes for which the model provides the reconstruction with the lowest error. Viewing our approach in terms of clustering, this amounts to showing elements from the clusters associated with each model.
Note that in fig:rec_sn, our linear models are associated with rich subsets of shapes which remain mostly semantically homogeneous.

0 5 6 10 14 16
1 2 3 6 8 17
6 7 8 10 16 13
0 3 4 8 12 15
0 1 2 3 4 5
0 1 5 10 13 14
0 5 7 12 13 14
0 2 4 8 12 14
6 7 8 9 11 13
0 1 6 8 9 13
Visualizing Reconstructions on ABC.
The left-most columns represent prototypes from all 10 linear models learned on the ABC dataset. For each prototype, we select $6$ samples for which this model gives the best reconstruction (“Input", top line). We then represent the associated reconstruction provided by the model (“Output", top line).
Each family represents a wide variety of morphologically homogeneous shapes: round rings, square rings, bent archs, cylinders, etc...
Looking at the prototypes gives us a concise overview of the shape diversity.

0 1 2 3 5 16
0 1 2 5 15 16
3 6 9 10 11 14
2 5 6 10 13 17
0 1 4 6 8 15
0 1 3 7 9 14
0 1 2 3 4 5
0 3 5 10 11 13
0 1 10 11 12 14
2 7 9 10 13 14
0 2 6 7 10 11
1 5 10 11 14 16
2 3 5 6 7 8
1 2 4 5 6 9
0 1 2 3 4 5
0 3 6 11 14 17
0 1 4 6 7 13
0 2 4 5 9 10
Visualizing Reconstructions on ShapeNet.
The left-most columns represent prototypes from some of the $55$ linear models learned on the ShapeNet dataset. For each prototype, we select $6$ samples for which this model gives the best reconstruction (“Input", top line). We then represent the associated reconstruction provided by the model (“Output", top line).
We observe that the samples associated with a given model are for the most part semantically homogeneous, and well represented by their prototype.

[1]
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.
Learning representations and generative models for 3D point clouds.
In ICCV, 2018.

[2]
David Arthur and Sergei Vassilvitskii.
k-means++: The advantages of careful seeding.
Technical report, Stanford, 2006.

[3]
Paul J Besl and Neil D McKay.
Method for registration of 3-d shapes.
In Sensor fusion IV: control paradigms and data structures.
International Society for Optics and Photonics, 1992.

[4]
Volker Blanz and Thomas Vetter.
A morphable model for the synthesis of 3D faces.
In SIGGRAPH, 1999.

[5]
Leon Bottou and Yoshua Bengio.
Convergence properties of the k-means algorithms.
In NeurIPS, 1995.

[6]
Christoph Bregler, Aaron Hertzmann, and Henning Biermann.
Recovering non-rigid 3D shape from image streams.
In CVPR, 2000.

[7]
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre
Geometric deep learning: going beyond euclidean data.
Signal Processing Magazine, 2017.

[8]
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang,
Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao,
Li Yi, and Fisher Yu.
ShapeNet: An Information-Rich 3D Model Repository.
Technical Report arXiv:1512.03012, Stanford University — Princeton
University — Toyota Technological Institute at Chicago, 2015.

[9]
Thomas Chaton, Chaulet Nicolas, Sofiane Horache, and Loic Landrieu.
Torch-points3D: A modular multi-task frameworkfor reproducible deep
learning on 3D point clouds.
In 3DV, 2020.

[10]
Zhiqin Chen and Hao Zhang.
Learning implicit fields for generative shape modeling.
In CVPR, 2019.

[11]
Christopher Choy, JunYoung Gwak, and Silvio Savarese.
4D spatio-temporal convnets: Minkowski convolutional neural
In CVPR, 2019.

[12]
Tim F Cootes and Christopher J Taylor.
Statistical models of appearance for medical image analysis and
computer vision.
In Medical Imaging : Image Processing. International Society
for Optics and Photonics, 2001.

[13]
Yuchao Dai, Hongdong Li, and Mingyi He.
A simple prior-free method for non-rigid structure-from-motion
International Journal of Computer Vision, 2014.

[14]
Yu Deng, Jiaolong Yang, and Xin Tong.
Deformed implicit field: Modeling 3d shapes with learned dense
In CVPR, pages 10286–10296, 2021.

[15]
Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C
Russell, and Mathieu Aubry.
Learning elementary structures for 3D shape generation and
In NeurIPS, 2019.

[16]
Haoqiang Fan, Hao Su, and Leonidas J Guibas.
A point set generation network for 3D object reconstruction from a
single image.
In CVPR, 2017.

[17]
Clara Fernandez-Labrador, Ajad Chhatkuli, Danda Pani Paudel, Jose J Guerrero,
Cédric Demonceaux, and Luc Van Gool.
Unsupervised learning of category-specific symmetric 3d keypoints
from point sets.
ECCV, 2020.

[18]
Brendan J Frey and Nebojsa Jojic.
Estimating Mixture Models of Images and Inferring Spatial
Transformations Using the EM Algorithm.
In CVPR, 1999.

[19]
Brendan J Frey and Nebojsa Jojic.
Transformed component analysis: Joint estimation of spatial
transformations and image components.
In ICCV, 1999.

[20]
Brendan J Frey and Nebojsa Jojic.
Fast, large-scale transformation-invariant clustering.
In NeurIPS, 2002.

[21]
Brendan J Frey and Nebojsa Jojic.
Transformation-Invariant Clustering Using the EM Algorithm.
Transactions on Pattern Analysis and Machine Intelligence,

[22]
Matheus Gadelha, Aruni RoyChowdhury, Gopal Sharma, Evangelos Kalogerakis,
Liangliang Cao, Erik Learned-Miller, Rui Wang, and Subhransu Maji.
Label-efficient learning on point clouds using approximate convex
In ECCV, 2020.

[23]
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T Freeman, and
Thomas Funkhouser.
Learning shape templates with structured implicit functions.
In ICCV, 2019.

[24]
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu
A papier-mâché approach to learning 3D surface generation.
In CVPR, 2018.

[25]
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu
Unsupervised cycle-consistent deformation for shape matching.
In Computer Graphics Forum, 2019.

[26]
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun.
Deep learning for 3D point clouds: A survey.
Transactions on Pattern Analysis and Machine Intelligence,

[27]
Zhizhong Han, Mingyang Shang, Yu-Shen Liu, and Matthias Zwicker.
View inter-prediction gan: Unsupervised representation learning for
3D shapes by learning global shape memories to support local view
In AAAI, 2019.

[28]
Kaveh Hassani and Mike Haley.
Unsupervised multi-task feature learning on point clouds.
In CVPR, 2020.

[29]
Pedro Hermosilla, Tobias Ritschel, and Timo Ropinski.
Total denoising: Unsupervised learning of 3D point cloud cleaning.
In CVPR, 2019.

[30]
Lila Huang, Shenlong Wang, Kelvin Wong, Jerry Liu, and Raquel Urtasun.
Octsqueeze: Octree-structured entropy model for lidar compression.
In CVPR, 2020.

[31]
Sergey Ioffe and Christian Szegedy.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
In ICML, 2015.

[32]
Chiyu Jiang, Jingwei Huang, Andrea Tagliasacchi, and Leonidas Guibas.
Shapeflow: Learnable deformations among 3d shapes.
In NeurIPS, 2020.

[33]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
ICLR, 2015.

[34]
Sebastian Koch, Albert Matveev, Zhongshi Jiang, Francis Williams, Alexey
Artemov, Evgeny Burnaev, Marc Alexa, Denis Zorin, and Daniele Panozzo.
Abc: A big cad model dataset for geometric deep learning.
In CVPR, 2019.

[35]
Loic Landrieu and Martin Simonovsky.
Large-scale point cloud semantic segmentation with superpoint graphs.
In CVPR, 2018.

[36]
Jiaxin Li, Ben M Chen, and Gim Hee Lee.
So-net: Self-organizing network for point cloud analysis.
In CVPR, 2018.

[37]
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas
Grass: Generative recursive autoencoders for shape structures.
Transactions on Graphics, 2017.

[38]
Franco Manessi, Alessandro Rozza, and Mario Manzo.
Dynamic graph convolutional networks.
Pattern Recognition, 2020.

[39]
Facundo Mémoli and Guillermo Sapiro.
A theoretical and computational framework for isometry invariant
recognition of point cloud data.
Foundations of Computational Mathematics, 2005.

[40]
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and
Andreas Geiger.
Occupancy networks: Learning 3D reconstruction in function space.
In CVPR, 2019.

[41]
Tom Monnier, Thibault Groueix, and Mathieu Aubry.
Deep Transformation-Invariant Clustering.
In NeurIPS, 2020.

[42]
G Dias Pais, Srikumar Ramalingam, Venu Madhav Govindu, Jacinto C Nascimento,
Rama Chellappa, and Pedro Miraldo.
3DRegNet: A deep neural network for 3D point registration.
In CVPR, 2020.

[43]
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven
Deepsdf: Learning continuous signed distance functions for shape
In CVPR, 2019.

[44]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.
Pointnet: Deep learning on point sets for 3D classification and
In CVPR, 2017.

[45]
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.
Pointnet++: Deep hierarchical feature learning on point sets in a
metric space.
In NeurIPS, 2017.

[46]
Jonathan Sauder and Bjarne Sievers.
Self-supervised deep learning on point clouds by reconstructing
In NeurIPS, 2019.

[47]
Manolis Savva, Angel X Chang, and Pat Hanrahan.
Semantically-enriched 3D models for common-sense knowledge.
In CVPR Workshops, 2015.

[48]
Karen Simonyan and Andrew Zisserman.
Very deep convolutional networks for large-scale image recognition.
In ICLR, 2015.

[49]
Lawrence Sirovich and Michael Kirby.
Low-dimensional procedure for the characterization of human faces.
Journal of the Optical Society of America, 1987.

[50]
An Tao.
Unsupervised point cloud reconstruction for classific feature
<https://github.com/AnTao97/UnsupervisedPointCloudReconstruction>, 2020.

[51]
Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen
Koltun, and Thomas Brox.
What do single-view 3D reconstruction networks learn?
In CVPR, 2019.

[52]
Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui,
François Goulette, and Leonidas J Guibas.
Kpconv: Flexible and deformable convolution for point clouds.
In ICCV, 2019.

[53]
Carlo Tomasi and Takeo Kanade.
Shape and motion from image streams under orthography: a
factorization method.
International Journal of Computer Vision, 1992.

[54]
Lorenzo Torresani, Aaron Hertzmann, and Chris Bregler.
Nonrigid structure-from-motion: Estimating shape and motion with
hierarchical priors.
Transactions on Pattern Analysis and Machine Intelligence,

[55]
Matthew Turk and Alex Pentland.
Eigenfaces for recognition.
Journal of Cognitive Neuroscience, 1991.

[56]
Matthew A Turk and Alex P Pentland.
Face recognition using eigenfaces.
In CVPR, 1991.

[57]
Lingjing Wang, Xiang Li, and Yi Fang.
Few-shot learning of part-specific probability space for 3D shape
In CVPR, 2020.

[58]
Yu Wang, Alec Jacobson, Jernej Barbič, and Ladislav Kavan.
Linear subspace design for real-time shape deformation.
Transactions on Graphics, 2015.

[59]
Svante Wold, Kim Esbensen, and Paul Geladi.
Principal component analysis.
Chemometrics and Intelligent Laboratory Systems, 1987.

[60]
Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum.
Learning a probabilistic latent space of object shapes via 3D
generative-adversarial modeling.
In NeurIPS, 2016.

[61]
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang,
and Jianxiong Xiao.
3D shapenets: A deep representation for volumetric shapes.
In CVPR, 2015.

[62]
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian.
Foldingnet: Point cloud auto-encoder via deep grid deformation.
In CVPR, 2018.

[63]
Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari.
3D point capsule networks.
In CVPR, 2019.

[64]
Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu.
Deep implicit templates for 3d shape representation.
In CVPR, 2021.