11institutetext: Department of Electronic and Telecommunication Engineering,
University of Moratuwa, Sri Lanka 22institutetext: The University of Sydney

# 3DLatNav: Navigating Generative Latent Spaces for Semantic-Aware 3D Object
Manipulation

Amaya Dharmasiri 11 Dinithi Dissanayake 11 Mohamed Afham 11 Isuru Dissanayake
11 Ranga Rodrigo 11 Kanchana Thilakarathna 22

###### Abstract

3D generative models have been recently successful in generating realistic 3D
objects in the form of point clouds. However, most models do not offer
controllability to manipulate the shape semantics of component object parts
without extensive semantic attribute labels or other reference point clouds.
Moreover, beyond the ability to perform simple latent vector arithmetic or
interpolations, there is a lack of understanding on how part-level semantics
of 3D shapes are encoded in their corresponding generative latent spaces. In
this paper, we propose 3DLatNav; a novel approach to navigate pretrained
generative latent spaces to enable controlled part-level semantic manipulation
of 3D objects. First, we propose a part-level weakly supervised shape
semantics identification mechanism using latent representations of 3D shapes.
Then, we transfer that knowledge to a pretrained 3D object generative latent
space to unravel disentangled embeddings to represent different shape
semantics of component parts of an object in the form of linear subspaces,
despite the unavailability of part-level labels during the training. Finally,
we utilize those identified subspaces to show that controllable 3D object part
manipulation can be achieved by applying the proposed framework on any
pretrained 3D generative model. With two novel quantitative metrics to
evaluate the consistency and localization accuracy of part-level
manipulations, we show that 3DLatNav outperforms existing unsupervised latent
disentanglement methods in identifying latent directions that encode part-
level shape semantics of 3D objects. With multiple ablation studies and
testing on state-of-the-art generative models, we show that 3DLatNav can
implement controlled part-level semantic manipulations on an input point cloud
while preserving other features and the realistic nature of the object.

###### Keywords:

Latent Navigation, Latent Disentanglement, 3D Semantic Manipulation,
Controlled 3D regeneration

Figure 1: Versatile interpretable latent directions discovered by 3DLatNav in
3DAAE[60] generative latent space. Part semantic attributes of sample objects
are manipulated by translating their latent codes to the required latent space
direction. The changes are limited to the intended part while the features of
other parts are preserved.

##  1 Introduction

The increasing popularity of applications that rely on 3D vision in domains
such as virtual or augmented reality, gaming and entertainment leads the way
to true realization of metaverse in the near future. The advancements of deep
learning and the availability of large-scale 3D point cloud datasets have
enabled fast progress in the 3D vision tasks such as classification [35, 51,
22, 46, 54, 63, 62, 2], segmentation [63, 35, 46, 21, 15, 62], 3D
reconstruction [11, 17, 25, 55], and shape and view synthesis [33, 18, 26,
30]. Controlled generation or editing of 3D point cloud objects is of growing
interest due to its ability to develop interactive 3D vision applications [31,
16, 28]. 3D generative models have been successful in learning overall
representations of specific object classes [1, 56, 60, 24, 7, 10]. However,
these models cannot enable controllable manipulation of part-level semantics
in point clouds, unless fine granular semantic labels and definitions or
reference target point clouds are provided. This significantly limits the
utility of existing methods. We hypothesise that this is also due to the lack
of understanding and prior work of how semantic features are embedded in 3D
generative latent spaces, although there is a large body of work in 2D
generative latent spaces [41, 36, 5, 42, 14, 12, 6, 3]. (Object parts as
refered in the rest of the paper has the obvious meaning. part-Semantics or
part-level-shape-semantics are different features/types of the said part. (Eg:
Part- legs. Par-semantic- swivel legs)).

To this end, we raise and address the following questions in this paper;

  * •

How can we define and identify part-level semantic features for 3D point cloud
objects when such annotations are unavailable?

  * •

How does a latent code of an object embed different parts and different
semantic attributes of each component part?

  * •

How can we manipulate shape-semantics of a given point cloud when we do not
have a reference point cloud to extract the target features from?

Hence, we formulate 3DLatNav: a novel simple framework to part-semantic aware
manipulation or generation of objects through 3D generative Latent space
Navigation. 3DLatNav can independently manipulate part-semantics of an object.
In contrast to prior work, our approach does not require part-level semantic
feature annotations, a reference point cloud to extract features of expected
end result, or specialized post-processing. First, we generate latent
representations for independent object parts. (Eg: Backrests) We implement
agglomerative clustering in this latent space to find different shape-
semantics within the specific object part. (Eg: Curved, reclined etc.
Backrests). Next, we utilize the identified part-level semantic categories to
find linear subspaces in a pre-trained object-level generative latent space.
As such, we unravel that 3D object generative models learn disentangled
representations for the semantics of object parts, despite the unavailability
of part-level labels during training. Finally, we demonstrate that one or a
combination of identified part-semantics can be manipulated on a query object
by simply translating its latent vector to a weighted combination of
corresponding linear subspaces within the generative latent space. (Unlike
approaches that swap parts between different objects[31, 9], the extent of
manipulation of an added shape semantic in 3DLatNav is controlled by tuning a
weight parameter). We validate the performance of 3DLatNav on multiple object
classes from ShapeNet dataset [8] with state-of-the-art 3D generative models
[60, 56, 24].

The paper presents the following contributions:

  * •

We propose a method to extract independent shape semantics from different 3D
object parts in a weakly supervised manner and prove that they can be used
learn disentangled latent directions that encode part-level shape semantics
for any pre-trained 3D generative model.

  * •

We present comprehensive analysis and ablation studies about how part-level
semantics behave and are encoded 3D generative latent spaces.

  * •

We show that such behaviors can be used to controllably manipulate object
parts, hence we introduce 3DLatNav; a framework to manipulate one or many
independent part semantics of 3D point clouds in a preferred way, purely by
navigating latent spaces of a pre-trained object generators, while preserving
the object’s quality and realistic nature.

  * •

We introduce two quantitative metrics to measure the localization ability and
consistency of part semantics controlled/ manipulated by 3DLatNav and show
that it outperforms existing unsupervised latent disentanglement methods in
identifying meaningful directions.

##  2 Related Work

###  2.1 3D Generative models

Point Cloud Generation: There have been plethora of work in 3D point cloud
generative models These generative frameworks vary from tree structure based
graph convolutions [10, 43, 49] to progressive point embeddings [52] and
attention based modeling[44]. Autoencoders [60, 58], flow based models [56,
34] and energy based models [53, 24] are increasingly becoming popular. 3D
Shape modeling and generations by learning disentangled representations [45,
64, 20, 57] also produce interesting results. _The focus of our paper is to
understand the properties of these generative latent spaces and propose
methods to introduce model agnostic semantic control for point cloud
generation, because none of these existing generative models offer such
control without the prior knowledge of the end point cloud._
Part Based Modeling: Several of the 3D generative models [19, 40] use part-
based decomposing techniques to generate new point cloud objects. [9] uses a
decomposer and a composer for part-aware shape embedding and synthesis,
permitting for swapping and randomly assembling of the parts. These have
several drawbacks centered on the symmetry and the connectivity of the edited
3D objects. [27, 39] use hierarchical aspects to generate new structures.
_However, these methods need extensive fine grained semantic labeling and
specialised training, whereas we propose a framework that is scalable and
applicable to any pre-trained generative model._ Instead of generating parts
separately, [31] introduces an autoencoder model that modifies and generates
3D objects with respect to their parts. Such methods require specialized post
processing to perform the editing. _In contrast, 3DLatNav does not require any
post processing beyond the pre-trained generator_

###  2.2 Understanding Generative Latent Spaces

2D Generative Latent Spaces: Varying input latent codes of generative model,
has been explored as a way of manipulating the generated output. The unit
based specialization in synthesizing visual concepts [5] and the vector
arithmetic properties [36, 47] of Generative Adversarial Networks(GANs) were
explored in 2D domain for controlled generations. More recently, InterfaceGAN
[41] followed by [42] explains how semantics are encoded in the latent space
of 2D GANs, providing a face editing framework. However, these work rely on
annotated semantic labels in 2D datasets. Due to the unavailability of such
semantic labels and definitions in 3D objects, understanding of 3D generative
latent spaces has not been investigated thoroughly in literature. The above
frameworks, have not been generalised or tested for adaptability to 3D use
cases.
3D Generative Latent Spaces: There are a few unsupervised methods for
manipulating 3D generative latent spaces. ShapeVAE [29] measures the activity
of each latent unit and demonstrates that high activity latent units tend to
encode significant shape changes. [38] tries to understand the activations in
3D point cloud AEs, and visualizes the features by clustering in the feature
space. These changes or features neither localise into one part of the object
nor comprehend the expected manipulation. Latent space interpolation and
latent space arithmetic is commonly used by 3D generative and disentanglement
methods[1, 11, 24, 32, 34, 29, 50, 60, 37, 4] to manipulate the semantics.
This also needs a source object and a reference object which has the intended
semantic we want to change, to either interpolate or perform latent
arithmetic. _In this paper, using the available part annotations for 3D point
cloud objects, we introduce a novel definition for part based semantics. Using
these semantic labels we unravel the latent spaces of generative models to
enable independently controlling object parts by editing the latent code,
without the need of a reference point cloud._

Figure 2: Architecture of the proposed framework. Fragmented latent space
encodes the shapes of independent object parts and enable unsupervised part-
level shape semantic identification. Identified semantics are combined with
the contextual knowledge of object part connections from generative latent
space to identify latent-space directions. Finally, latent codes of generator
are edited by translating them in latent-space directions to generate objects
with a required semantic attribute

##  3 The Framework

In this section, we elaborate on our novel procedure to identify part-level
shape-semantics using an object-part autoencoder latent space, and how we
utilize those features to identify linear subspaces that encode part-level
semantic features in generative latent spaces. Next, we illustrate how
3DLatNav can translate generative latent codes towards a preferred semantic
direction to enable part-level semantic-control in object generation. The
architecture of the proposed framework is presented in Fig. 2.

###  3.1 Identifying part-level semantic features and corresponding generative
latent subspaces

#### Part-level semantic features.

3D object datasets usually provide part annotations for objects (e.g.
Backrest, seat etc.), and occasionally subclass labels within a superclass
(e.g. Lounge chair, Armchair etc.). But there’s a lack of annotated labels on
different shape semantics of each part (e.g. Swivel legs, Reclined backrests
etc.). To mitigate this, we utilize a weakly supervised approach to extract
shape-semantics of object parts. (3DLatNav requires part-annotations for the
object to train the part segmentation model, but it does not require any
annotations for the shape semantic.)

Let $S_{i}$ be the $i^{th}$ point cloud of class $C$, consisting of $n$
different object parts. $S_{i}=\bigcup_{j=0}^{n}S_{ij}$ where $S_{ij}$ is the
subset of points that represent the $j^{th}$ part category (Eg; Armrests of a
chair). We leverage a 3D shape auto-encoder consisting of an Encoder $E_{p}$
and a Decoder $D_{p}$ to extract a latent space $L_{p}$.
$z_{ij}=E_{p}(S_{ij}),\bar{S_{ij}}=D_{p}(z_{ij})$ ,
$L_{p}=\bigcup_{i,j}z_{ij}$. The latent space $L_{p}$ contains information
about the shapes, sizes, and rotation information for different individual
object parts. Although this autoencoder is not trained with part-labels, the
resulting $L_{p}$ is fragmented with clear segregation between parts (e.g.
Seats, Backrests, etc), indicating the structural knowledge acquired by the
latent space. _(We experiment with training separate latent spaces for
separate parts, but the improvement was only marginal at the cost of added
complexity and loss of scalability.)_

We perform a hierarchical agglomerative clustering of part latent codes within
$L_{p}$ separately for each object part $j$ to identify different part-level
semantics. (The exact hyper-parameters of the clustering method is available
along with the code implementation). A resulting cluster $A_{jk}$ contains the
part-level latent codes of the $j^{th}$ part (Eg:Legs) that represent its
$k^{th}$ unique semantic attribute (Eg:Swivel leg). (Note that all instances
of one part category are treated as an ensemble; ie, all legs of the chair.)
The set $\\{S_{i}|z_{ij}\in A_{jk}\\}$ contains a set of point cloud objects
that posses a specific semantic attribute $k$ localized in its $j^{th}$ part
category _(E.g- The set of chairs whose legs are swivel)_ , irrespective of
the geometries in the rest of the parts _(Backrest, Seat, Armrest)_ as shown
in Fig. 2. Since we have independently encoded the structural information of
each part, these clusters show a diversity of features in the rest of the
parts.

#### Generative latent subspaces for part-semantics.

Point cloud object generators are trained to map a latent code $Z_{i}$ to a
point cloud object $\widehat{S_{i}}=G_{o}(Z_{i})$. These models are also often
accompanied by a representation encoder $E_{o}$ to calculate the latent code
$Z_{i}$ from a given point cloud object $S_{i}$, such that
$Z_{i}=E_{o}(S_{i})$. Using this encoder, we form a set of latent codes within
$L_{o}$ containing object-level embeddings for point clouds objects in our
dataset (Shapenet Chair). Using the clusters $A_{jk}$ identified from the
part-latent-space $L_{p}$, we find sets of positive examples (of object latent
codes) from $L_{o}$ for each $k$th semantic attribute localized in the $j$th
object part as $Q_{jk}=\\{Z_{i}|z_{ij}\in A_{jk}\\}$. Note that a latent code
$Z_{i}$ could be a positive example for multiple semantic attributes, but in
different object parts. Unlike for datasets like 2D face images [23] for which
positive and negative semantic attributes can be easily defined (e.g. Smile-
frown, Old-young etc.), assigning negative meanings for a shape semantic is
difficult and ambiguous. Therefore, for every set of positive examples
$Q_{jk}$, we define negative examples as
$Q_{jk}^{\prime}=\\{Z_{i}|z_{ij}\notin A_{jk}\\}$, that simply include all the
latent codes $Z_{i}$ except those included in $Q_{jk}$.

Table 1: Average classification accuracy for linear SVMs fitted for each part-level semantic attribute in the object generative latent space. _High accuracies confirm the existence of disentangled linear subspaces that encode part-level semantics._ Chair |  | Backrest | Seat | Legs | Armrest
---|---|---|---|---|---
3DAAE | 0.957 | 0.975 | 0.967 | 0.917
Pointflow | 0.956 | 0.975 | 0.974 | 0.918
Diffusion P. | 0.957 | 0.981 | 0.98 | 0.938
Airplane |  | Body | Wings | Tail | Engine
3DAAE | 0.88 | 0.854 | 0.868 | 0.854
Car |  | Hood | Bonnet | Wheels | Body
3DAAE | 0.911 | 0.872 | 0.936 | 0.826

Next, we fit an independent linear SVM in the _object generative latent space_
$L_{o}$ for each identified semantic attribute using $Q_{jk}$ and
$Q_{jk}^{\prime}$ as positive and negative examples. We computed the
classification accuracy for linear SVMs fitted for three generative latent
spaces – 3DAAE [60], Pointflow [56], and Diffusion P. [24] – leveraging
ShapeNet [8] dataset. Tab.1 shows high classification accuracy irrespective of
the generative model. Thus, we validate the hypothesis that _there are linear
subspaces in 3D generative latent spaces that embed disentangled
representations for individual object part semantics_. While a similar
observation had been made about the 2D binary facial semantics in [41], this
is the first analysis and experimental validation of the existences of non-
binary part-level semantics in 3D generative latent spaces. _(We also observed
that creating clusters in the object generative space $L_{o}$ leads to heavy
part-level semantic entanglements due to missing part knowledge. Hence, we
utilize two different latent spaces; $L_{p}$ and $L_{o}$.)_

Figure 3: Given an object point cloud, our framework can manipulate either
single or multiple semantics, specific to different object parts, through
simple latent code translations. Latent codes are translated perpendicular to
the hyperplanes fitted for the two part-semantics.

###  3.2 Semantic aware navigation in generative latent spaces

The unit normal vector $\bar{q}_{jk}$ to the hyperplane fitted by the SVM
separating $Q_{jk}$ and $Q_{jk}^{\prime}$ indicates a direction in the object
generative latent space. We illustrate that we can manipulate the extent of
the particular semantic ($k^{th}$ semantic localized in the $j^{th}$ object
part) in the generated point cloud $\widehat{S_{i}}^{\prime}$ by translating a
latent code $Z$ in the direction of $\bar{q_{jk}}$,
$Z^{\prime}=Z+\alpha\bar{q}_{jk}$ where $\alpha$ is a tunable scalar parameter
that can be used to control the level of semantic _(E.g- the level of
reclination of the chair’s backrest)_. Moreover, this linear operation can be
extended to combine multiple directions to simultaneously edit semantics in
multiple object parts as shown in Fig. 3 and Eqn. 1.

$Z_{i}^{\prime}=Z_{i}+\alpha_{0}\overline{q}_{o,k_{0}}+\alpha_{1}\overline{q}_{1,k_{1}}+...+\alpha_{n}\overline{q}_{n,k_{n}}$
(1)

The positive example sets $Q_{jk}$ are not mutually exclusive for different
object part categories $j$, hence, occasional entanglements across semantics
in different object parts are possible. In the forthcoming sections, we show
that such entanglements are caused by biases in the dataset, and sometimes are
required to preserve the realistic nature of generated objects. Furthermore,
the translated latent code $Z^{\prime}$ is within $L_{o}$ itself, hence the
same mapping function $G_{o}$ can be used for controlled object regeneration
without any additional post-processing. The different part-level semantic
features identified as clusters in the fragmented latent space corresponds to
different features (like swivel legs, cantilever legs, straight legs etc.)
within a particular part, (leg). The number of different semantic features
identified for each part depends on the dataset. If the dataset has
significant diversity in semantics, clustering will identify more unique
semantic features, hence more latent-space directions and therefore, finer-
grained semantic control could be enabled for object regeneration. However, we
observe that 3DLatNav can successfully identify latent directions with as few
as 50 objects representing the particular part-semantic. (Clusters for part-
semantics contain between 7% to 50% of all samples)

##  4 Experiments

In this section, we quantitatively and qualitatively evaluate the performance
of 3DLatNav, and compare We introduce two novel metrics to quantify
_consistency_ and _ability to localize_ semantic manipulations to object parts
and show that 3DLatNav outperforms the existing disentanglement methods.

###  4.1 Evaluation setup

Dataset. We use 3D models of ShapeNet [8] dataset to obtain a pointcloud
representation of $2048\times 3$ for each object and use them to train the
object generative models. Moreover, we utilize the point-level part
annotations provided by ShapeNet-part[59] dataset to train a PointNet[35]
based object segmentation model to create part decompositions required for the
3DLatNav pipeline.

Metrics.

(i). Semantic Localization Score (SLS). To ensure controllability of part-
level object manipulation, it is required that the change added to the object
is restricted to the intended part. We introduce the ratio of structural
change added to the intended part and to the rest of the object parts as a
novel metric, Semantic Localization Score (refer Eqn. 2). _A higher value of
SLS indicates better controllability of part level semantic manipulations._

$\small\scriptstyle
SLS_{k,a}=E\left[\frac{CD\left(\widehat{S_{ia}},\widehat{S_{ia}}^{\prime}\right)}{CD\left(\bigcup_{j\neq
a}\widehat{S_{ij}},\bigcup_{j\neq
a}\widehat{S_{ij}}^{\prime}\right)}\right]\normalsize$ (2)

$CD$ is the Chamfer distance [48] between original pointcloud
$\widehat{S_{i}}$ and manipulated pointcloud $\widehat{S_{i}}^{\prime}$ by
translating latent code $Z_{i}$ towards a particular latent-space direction;
$Z^{\prime}=Z+\alpha\bar{q}$. $\widehat{S_{ij}}$ is the set of points that
represent the $j^{th}$ part of the object. Expected value $E$ is calculated by
translating 1000 random latent samples by an empirically selected distance
$\alpha$ towards a particular latent direction.

(ii). Semantic Consistency Score (SCS). Consistency of part-level object
manipulation is achieved if all different latent codes translated towards a
particular direction is consistently added with the same semantic change. We
quantify the consistency by using the confidence score of a semantic
classifier model for the transformed object. For a given part semantic (e.g.,
swivel leg), we manually label the objects with the given part semantic as
positive labels and the objects with other semantic attributes (e.g., straight
leg, cantilever leg) as negative labels. Then we train a set of binary
semantic classifier models (Eg: Straight leg or not a straight leg) with a
PointNet [35] feature extractor. The model is expected to classify the
transformed object to the corresponding semantic category. For example, a
model trained using swivel legs as positive labels should classify a latent
navigation towards swivel legs direction as positive outputs and that of
cantilever legs and straight legs directions as negative outputs. We use a
well-validated PointNet model to perform this evaluation which yields the
consistency of semantic manipulations quantitatively.

Comparison. We compare the performance of 3DLatNav in identifying latent
directions with other unsupervised latent disentanglement methods. Closed-form
[42] proposes a closed-form factorization based method to find semantically
meaningful directions in a GAN latent spaces in a fully unsupervised manner by
directly decomposing the pre-trained weights of the generator. GANSpace [13]
proposes a technique to analyze GANs and create interpretable controls in
terms of latent directions based on a Principal Component Analysis (PCA). Both
have only been applied to 2D image synthesis tasks by the original authors. We
adapt these approaches to 3D object synthesis using a 3DAAE[60] model and
compare its performance with 3DLatNav in terms of identifying disentangled
latent directions for part semantic control. Both approaches yield a large
number of directions that mostly added either the same semantic change or no
change at all. To compare 3DLatNav with their upperbound performances, we
matched the latent directions from these methods (out of 100+ suggested
directions) with those of 3DLatNav after manual observation.

The part-editing performance of 3DLatNav is limited by the representation
capability of the selected generative model. Hence, we cannot fairly compare
the part-editing performance with specialized models such as [9, 28]. We focus
on the performance of latent disentanglement of 3DLatNav where our key
contribution is.

###  4.2 Results

Figure 4: Comparison of part-level object manipulation results of 3DLatNav
with unsupervised latent disentanglement methods; Closed-form[42] and
GANSpace[13].

Visual qualitative results. Fig. 1 illustrates the quality of semantic
manipulations of 3DLatNav for different object parts and semantic features.
Fig.4 shows results of part level semantic manipulation using 3DLatNav,
GANSpace [13] and Closed-form [42]. Unlike in 3DLatNav, for a majority of the
cases both of the other methods add unintended changes to the other parts of
the objects during object manipulation. (E.g- _Curved backrest and Stuffed
seat in GANSpace_ adds changes legs, _Connected armrest in GANSpace_ adds
changes to seat, _Removing armrest in Closed-form_ adds changes to backrest,
etc.) Although _GANSpace_ has been more successful than _Closed-form_ in
identifying directions for common semantics like _swivel legs_ , specific
semantics like _cantilever legs_ were not present/clearly distinguishable
among directions yielded by either GANSpace or Closed-form. 3DLatNav can be
used on the generative latent spaces of any pretrained generator. We present
the visual results of applying our method on different generators [60, 56, 24]
in supplementary materials.

Table 2: Semantic localization score (Eqn. 2) to quantify the relative structural change added to the specific object part by manipulating a semantic attribute. | Disentanglem
---
-ent method
| Generative
---
model
| Object
---
class
Part1 | Part2 | Part3 | Part4 | Average
3DLatNav | 3DAAE | airplane | Body | Wings | Tail | Engine |
0.61 | 0.7513 | 2.5914 | 5.3587 | 5.4342
car | Hood | Bonnet | Wheels | Body |
6.372 | 3.3651 | 2.6592 | 0.4635 | 3.6267
3DLatNav |  |  | Backrest | Seat | Legs | Armrest |
Pointflow | chair | 3.885 | 0.329 | 7.129 | 10.113 | 5.403
Diffusion | chair | 1.826 | 1.121 | 6.425 | 7.774 | 4.263
3DAAE | chair | 1.6943 | 0.4703 | 8.7261 | 7.8573 | 4.7617
GANSpace | 3DAAE | chair | 1.5274 | 0.595 | 6.3166 | 3.1231 | 2.8905
Closed form | 3DAAE | chair | 1.403 | 0.474 | 4.453 | 6.9471 | 3.3192

Semantic localization. The average Semantic Localization Scores (SLS) greater
than one for of the object parts in Tab.2 show that the structural changes
added to the intended part significantly exceeds those added to the rest of
the object. There are also certain cases where the SLS is less than 1, but
they account to the scenarios where a structural change to one part affects
the rest of the parts, E.g- changing the seat of a chair requires a change in
backrests, armrests, and legs. As shown in Table 2, 3DLatNav outperforms both
GANSpace and Closed-form in restricting the semantic changes to a required
part, confirming its superior performance in disentangling part-level semantic
controls. These results are agreeing with the visual results in Fig.4. We also
present results for implementing 3DLatNav on Pointflow[56] and Diffusion
Probablistic models[24] generative latent spaces and confirm that the
discovered latent directions can successfully control the localization of
part-level object manipulations with different generative models.

Table 3: SCS to evaluate consistency of semantic manipulation. Transformed objects are tested with the trained binary classifier model. 3DLatNav consistently outperforms the existing approaches in both object-level and part-level. | Object-Level
---|---
| Backrest | Seat | Legs | Armrest
|  curved |  reclined |  stuffed |  narrow |  stuffed |  wide |  swivel |  straight |  cantilever |  connected |  disconnected |  none
| GANSpace
---
0.705 | 0.939 | 0.793 | 0.620 | 0.612 | 0.655 | 0.933 | 0.661 | 0.590 | 0.604 | 0.654 | 0.919
| Closedform
---
0.493 | 0.683 | 0.468 | 0.414 | 0.665 | 0.663 | 0.671 | 0.573 | 0.684 | 0.698 | 0.656 | 0.775
| 3DLatNav
---
0.943 | 0.980 | 0.923 | 0.661 | 0.854 | 0.764 | 0.851 | 0.888 | 0.954 | 0.927 | 0.952 | 0.989
| Part-Level
| GANSpace
---
0.756 | 0.906 | 0.734 | 0.452 | 0.702 | 0.673 | 0.908 | 0.682 | 0.643 | 0.410 | 0.500 | 0.820
| Closed form
---
0.580 | 0.677 | 0.409 | 0.380 | 0.664 | 0.658 | 0.679 | 0.540 | 0.674 | 0.540 | 0.440 | 0.853
| 3DLatNav
---
0.948 | 0.989 | 0.960 | 0.692 | 0.869 | 0.719 | 0.834 | 0.737 | 0.923 | 0.896 | 0.912 | 0.998

Semantic consistency. The results of semantic consistency score (SCS) are
reported in Tab. 3. We compute object-level scores (Tab. 3 top) where the
whole object is assessed with the trained model and the part-level scores
(Tab. 3 bottom) where only the specific object parts are assessed with a
trained part-level semantic classifier model. 3DLatNav consistently
outperforms the previous other latent disentanglement and navigation
approaches[13, 42] in most semantic manipulations. These are supported by
visual results in Fig. 4 where 3DLatNav shows consistency in the semantic
manipulation across all semantics compared to the prior approaches and the
visual results in Fig. 1 where 3DLatNav shows consistency in semantic
manipulations for different objects for a given latent direction.

##  5 Analysis and discussion

We present an in-depth analysis of the properties associated with part-
semantic representations in generative latent spaces and their manipulations
using the discovered latent directions from 3DLatNav. Unless mentioned
otherwise, 3DLatNav is implemented with two 3D Adversarial Autoencoders
(3DAAE), one as the object part shape autoencoder, and other as the 3D object
generator. 111We will present specific details of implementation, experimental
setups, and additional visual results in Supplementary materials.

###  5.1 Semantic entanglement of features

We calculate Cosine similarity between identified $n$-dimensional latent space
directions that encode different part-semantic attributes. Fig.5 illustrates
that entanglement of features across different object parts are very low with
overall low Cosine similarity. Occasional entanglements such as between Narrow
seats and Armrests absent preserve the realistic nature of the objects
enforced by the ShapeNet[8] dataset. Negative correlations are sometimes
observed between semantically contrasting attributes of the same part, _(E.g.
highest negative correlation is between availability and absence of armrest.)_

Figure 5: Cosine similarities between latent space directions each pair of
selected part-semantics for Chair class. Figure 6: Manipulating multiple
semantics by translating latent codes to a combination of latent-space
directions.

###  5.2 Part manipulation by latent space navigation

Manipulating multiple semantics. The linearity of the subspaces that encode
part-level semantics enable us to manipulate/edit a query object with any
linear combinations of identified semantics as formulated in Eqn. 1. For
example, we can translate a latent code $Z$ in multiple directions
$\bar{q}_{0,1}$. and $\bar{q}_{1,1}$ with a controllable extent of the added
semantic using the tunable parameters $\alpha_{1}$ and $\alpha_{2}$, such that
$Z^{\prime}=Z+\alpha_{1}\bar{q}_{0,1}+\alpha_{2}\bar{q}_{1,1}$. The first two
rows of Fig.6 show individual semantic manipulation and the bottom row shows
the linear combination of the two semantics while preserving the other
features to a large extent.

Quality of generated and transformed objects. We demonstrate the robustness of
3DLatNav by conducting semantic manipulation for different object classes in
Fig. 7 The results show that 3DLatNav semantic manipulation performs similar
to the chair class in other two classes of airplane and car. Unlike 3D object
part manipulation/editing frameworks that detach the object parts to
interchange or interpolate between different objects and subsequently combine
them to form the complete object, 3DLatNav generates semantically edited
objects from the same generator with acquired contextual knowledge of object
part connections [9, 31]. Therefore, 3DLatNav does not produce disconnected
object parts or unrealistic object shapes. The edited point clouds are
guaranteed to have the quality offered by the generative model.

Figure 7: 3DLatNav on other object classes. Semantics can be controlled for
different parts independently. Feature entanglements are present where
realistic nature of the object needs to be preserved. e.g., changing the
airplane tail when engine is moved from wings to the body.

###  5.3 Further analysis

A generator trained on a certain dataset may have entangled part-semantic
features due to correlations in the dataset. To ablate this effect, we trained
a generator on an extended ShapeNet dataset including additional sets of
examples synthesized by naively combining objects parts from different
objects. We show that the directions found using 3DLatNav are even more
disentangled when the generative model has been trained with such synthetic
data as opposed to limiting to real-life objects. We present these results and
the implementation details in supplementary materials.

##  6 Conclusion and future work

We introduced 3DLatNav, a framework to identify meaningful directions in 3D
generative latent spaces that independently encode semantics of different
object parts. Using comprehensive qualitative and quantitative measurements,
we show that 3DLatNav outperform existing unsupervised latent disentanglement
methods in identifying meaningful latent directions. 3DLatNav can successfully
be used to manipulate part level semantics of 3D objects using any pretrained
generative model, without any post processing or reference objects to extract
the target features from. _We contribute to widening of understanding of how
3D part semantics behave in generative latent spaces by presenting extensive
analysis and ablation studies._ Further research directions will be measuring
the utility of 3DLatNav as an editing framework (using a user-study) to create
a 3D semantic manipulation framework deployable practically in interactive
mixed reality or gaming applications, and extending the framework to other
types of 3D data representations and to larger 3D scenes.

##  7 Code and Demo

We have made the project repository and an interactive demo publicly
available. 222Repository- https://github.com/theamaya/3DLatNav, Demo-
https://colab.research.google.com/drive/1w1QoyfIkhWFQG1wvGgpGRhVivn4-LNu9?usp=sharing

##  8 Implementation Details

###  8.1 3DLatNav framework

We used the ShapeNet [8] dataset throughout this work. First to extract part-
level semantic features, we part-segmented the point clouds of Shapenet
dataset using a PointNet [35] part-segmentation model trained on ShapeNet-Part
dataset [59]. To obtain the object-part latent space $L_{p}$ for these
segmented parts, we used the 3D Adversarial Autoencoder architecture [61]. We
train a 3DAAE model with different parts of a particular object class using
input point clouds of 400 points to generate a 400-dimensional latent space
$L_{p}$. Each individual object part was either up-sampled or down-sampled to
yield a representation of 400 points. The unsupervised semantic feature
identification explained in Sec. 3 of the main paper is done on this latent
space.

We verified 3DLatNav for multiple generative models and their corresponding
generative latent spaces. Baseline results were generated for a 3DAAE model
trained for a specific object class on the original ShapeNet dataset. The
input point clouds are generated by sampling 2048 points from each object. The
model is trained to regenerate 2048 points using a 2048-dimensional latent
space $L_{o}$. Correspondingly other generative models [56, 24] were also
trained on the same train dataset. The object generative latent space $L_{o}$
is generated by extracting the encodings of each object from the Shapenet
dataset using their corresponding representation encoders. The dimensions of
each latent space varies as per defaults set by original authors.

Fitting SVMs to find linear subspaces, deriving of latent directions and point
cloud part-editing by translating latent codes as mentioned in Sec.3 of the
main paper is carried out on this latent space $L_{o}$.

###  8.2 Quantitative evaluation

In this work we introduced two quantitative metrics to evaluate the
performance of part-level semantic manipulation by navigating object
generative latent spaces. The two metrics quantify two properties of the
resulting manipulated regenerations.

Semantic localization score (SLS). As shown in Eqn.2 of the original
submission, we use Semantic Localization Score (SLS) to measure the level of
localization of the intended manipulation to the specific object part as
compared to the rest of the object. To calculate SLS for a single object
manipulation, we part-segment both the original object and the manipulated
object, and create two point sets each; points representing the object part
that we plan to manipulate, and the points representing the rest of the
object. Next we calculate the Chamfer distance[48] between the corresponding
pointsets from the original and manipulated objects, and calculate their ratio
to obtain the SLS. The average results reported in Tab.2 of the original
submission is calculated by translating 1000 random objects to each meaningful
direction discovered by the particular framework, and averaging the results
over all objects and all directions corresponding to each object part. The
$\alpha$ (distance translated in the latent space $L_{o}$) is selected
empirically for each generative latent space.

Semantic consistency score (SCS). As reported in Tab.3 of our main paper, we
calculated semantic consistency scores to evaluate the consistency of 3DLatNav
transformations on different objects. We train 12 binary semantic classifier
models for object-level and part-level evaluations each. (3 most prominent
semantics for each part- Legs- swivel, cantilever, straight, Armrests-
connected, disconnected, none, Backrest- narrow, stuffed, reclined, Seats-
narrow, wide, stuffed). The binary classifiers were trained using manually
annotated objects from the shapenet dataset with the above part-level
semantics. (Eg- For a swivel leg binary classifier, shapenet chairs with
swivel legs were taken as positive examples while those with straight and
cantilever legs were taken as negative examples.) We transformed a set of 1000
randomly selected chairs to each of these 12 directions to create the test
set. The average results for these 1000 objects are reported in Tab.3. Similar
to the evaluation of Semantic Localization Score, the magnitude of
transformation $\alpha$ was selected empirically for different generative
latent spaces.

##  9 Further Results

Emergence of negative attributes. The latent space directions are calculated
based on hyperplanes that separate a set of positive examples $Q_{jk}$ of a
selected part-level shape semantic against the rest of the training dataset,
$Q_{jk}^{\prime}$. It should be noted that the set $Q_{jk}^{\prime}$ does not
exclusively contain examples with an opposite semantic attribute, but includes
all the objects in which the considered semantic attribute is not present.
Nonetheless, upon translating a latent code to the negative direction of an
identified latent-space direction, a corresponding interpretable negative
semantic attribute is added to the generated objects as shown in Fig.8. It
shows that latent spaces of generative models have embedded information about
matching negative semantics for a specified part-level semantics, _although
specific negative examples were not provided during training the SVMs_.

Figure 8: Emergence of negative attributes when translating latent codes to
the negative directions.

Model robustness on other object classes The proposed framework can
successfully identify and manipulate part-level semantics in different object
classes. Fig.9 includes visual results of implementing the framework on
Shapenet Airplane and Car classes. Figure illustrates random objects and their
manipulated results by translating the latent code towards the identified
latent space directions using the 3DLatNav. These results confirms the
robustness and adaptability of 3DLatNav in identifying weakly-supervised part-
semantics and navigating in object level latent spaces for different object
classes.

Figure 9: Manipulating single semantic attributes by translating a latent code
to the identiﬁed latent space directions in Airplane and Car classes. The
part-level features change while preserving the other parts.

Quality of generated and transformed objects. Unlike 3D object part
manipulation/editing frameworks that detach the object parts to interchange or
interpolate between different objects and subsequently combine them to form
the complete object, 3DLatNav generates semantically edited objects from the
same generator with acquired contextual knowledge of object part connections
[9, 31]. Therefore, 3DLatNav does not produce disconnected object parts or
unrealistic object shapes. Since we leverage existing state-of-the-art 3D
point cloud regeneration models, the edited point clouds will depict the same
qualitative results of the used regeneration model. However, as shown in
Fig.10, if the latent code moves to the far periphery of the latent space,
there can be a reduction in the regeneration quality, and an increase in the
unintended changes added.

However, as shown in Fig.10, if the latent code moves to the far periphery of
the latent space, there can be a reduction in the regeneration quality, and an
increase in the unintended changes added.

Figure 10: Controlling the extent of manipulation. The original refers to a
generation close to a hyperplane. Moving the latent code farther away from the
hyperplane adds unintended changes , and reduce regeneration quality.

###  9.1 Ablation analysis

Training with synthetic data. A generator trained on a certain dataset
sometimes learn entangled part-level semantic features due to the
entanglements introduced by the dataset. To investigate the ablation of this
effect, we trained our generator on an extended ShapeNet dataset formed by
introducing a set of additional examples synthesized by naively combining
objects parts from different objects. The synthetic objects were created by
randomly combining different segmented parts from different objects in a naive
fashion. No post-processing was done to enforce a connectivity or realistic
nature on these synthetic chairs. Fig.11 shows a few examples of such
synthetic chairs.

Figure 11: Examples of synthetic chairs generated by naively combining random
object parts of chair class.

Nevertheless as shown in Fig.12, we see that editing the same part-level
semantics in a generator trained on this extended dataset produces objects
with more disentangled part-level features, e.g. reclining the backrest
without changing the seats or legs. Moreover, exposure to diverse structures
in the extended dataset enables the generator to produce objects with better
quality for previously unseen feature combinations, e.g. widening the seats.

Figure 12: Manipulating the same part-level semantic in two generators trained
on ShapeNet [8] dataset and Shapenet+Synthetic objects dataset respectively.
Additional training on synthetic data results in further part-level semantic
disentanglement, and limits the unintended changes added to the other parts.

Using subclasses as a proxy for part semantics. The ShapeNet dataset provides
subclass labels for its 3D objects, e.g.: ‘Chair’ superclass has subclasses
such as Tulip chair, Cantilever chair, and Lounge chair etc. In this ablation
study, we compare the part-level semantics identified in the 3DLatNav
framework with Shapenet object subclasses taken as a proxy for their implied
part-level semantics. E.g- Cantilever chair - for cantilevel legs, Straight
chair - for straight legs, straight backrests, Armchair - for connected
armrests, etc.

We project the segmented objects parts from corresponding subclasses to the
part-level latent space $L_{p}$ to evaluate the performance in part-level
semantic identification. The Silhouette score calculates the ratio between
mean intra-cluster distance, and the mean nearest-cluster distance. The
negative values in Tab. 4 shows that the taking subclass annotations as a
proxy for part-semantics result in overlapping clusters in the part-level
latent space, indicating poor performance in distinguishing part-based
semantic identification. Whereas near-zero positive values for the clusters
yielded by 3DLatNav indicate far superior identification of unique part-
semantics.

Table 4: Silhouette scores for the identified part-level latent clusters in $L_{p}$ and for Shapenet subclass labels taken as a proxy for part-level semantics. Part-based semantic identification using 3DLatNav method deliver better clustering results than using the available subclass labels, indicating more accurate identification of unique part-level semantics. | Backrest | Seat | Legs | Armrest
---|---|---|---|---
3DLatNav labels | 0.024 | 0.141 | 0.085 | 0.618
ShapeNet subclass labels | -0.254 | -0.174 | -0.475 | -0.892

Next we replaced part-level semantics identified by 3DLatNav with these
inclusive labels of object-level subclasses and fitted SVMs in the generative
latent space $L_{o}$ by taking each subclass label as positive examples and
transformed random query latent codes normal to the identified hyperplanes. A
subclass comprises of chairs with a specific combination of shape semantics in
each of its constituent parts. Therefore, translating along such latent
directions tend to change multiple part semantics at once, consequently
decreasing the extent of controlability of the shape manipulation. Moreover,
some of these subclass labels are based on the objects’ functionality, and
does not necessarily reflect a similarity in their shapes or structures. One
such example is the Lounge chair. The latent directions corresponding to such
subclasses tend to have negligible net effect on the transformed query object.

Fig.13 include visualizations from this ablation Study. Row 1 shows a success
case where fitting SVM and translating perpendicular to its the hyperplane for
cantilever chair subclass adds cantilever legs to a test chair. In Row 2
(Failure case), latent direction derived using Arm chair subclass tend to
change multiple part semantics of the test object at once, decreasing the
controlability of the shape editing. In Row 3 (Failure case), the subclass
label Recliners is based on the objects’ functionality, and does not
necessarily reflect a similarity in shapes or structures. Therefore the
corresponding latent direction tend to have no net effect on the transformed
test object.

Figure 13: Left- examples of Shapenet chairs with the particular subclass
label, Right- Examples of translating a random object towards the semantic
direction derived using the subclass as positive examples. Row 1- (Success
case) Cantilever legs added to normal chair, Row 2- (Failure case) Multiple
unintended parts changed in the test chair, Row 3- (Failure case) No
significant semantic change added to test chair

Using 3DLatNav on different generative models. 3DLatNav can be used on the
generative latent spaces of any pretrained generator. Fig. 14 presents the
visual results of applying our method on different generators ; Diffusion
Probablistic Models[24], Pointflow[56], and 3DAAE[60]. We show that the
proposed method can be applied on any pretrained object generative latent
space, and each generative latent space may have slight differences in terms
of the added changes. These changes are caused by the differences in model
architectures and levels of training.

Figure 14: Implementing the 3DLatNav on different pretrained 3D object
generators; Diffusion Probablistic Models[24], Pointflow[56], and 3DAAE[60]

## References

  * [1] Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.J.: Learning representations and generative models for 3D point clouds. In: 6th International Conference on Learning Representations(ICLR) (2018)
  * [2] Afham, M., Dissanayake, I., Dissanayake, D., Dharmasiri, A., Thilakarathna, K., Rodrigo, R.: Crosspoint: Self-supervised cross-modal contrastive learning for 3d point cloud understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9902–9912 (2022)
  * [3] Ansari, A.F., Soh, H.: Hyperprior induced unsupervised disentanglement of latent representations. In: AAAI (2019)
  * [4] Aumentado-Armstrong, T., Tsogkas, S., Jepson, A., Dickinson, S.: Geometric disentanglement for generative latent shape models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
  * [5] Bau, D., Zhu, J.Y., Strobelt, H., Zhou, B., Tenenbaum, J.B., Freeman, W.T., Torralba, A.: Gan dissection: Visualizing and understanding generative adversarial networks. In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)
  * [6] Bouchacourt, D., Tomioka, R., Nowozin, S.: Multi-level variational autoencoder: Learning disentangled representations from grouped observations. In: McIlraith, S.A., Weinberger, K.Q. (eds.) Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. pp. 2095–2102. AAAI Press (Feb 2018)
  * [7] Cai, R., Yang, G., Averbuch-Elor, H., Hao, Z., Belongie, S., Snavely, N., Hariharan, B.: Learning gradient fields for shape generation. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
  * [8] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: An Information-Rich 3D Model Repository. Tech. Rep. arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago (2015)
  * [9] Dubrovina, A., Xia, F., Achlioptas, P., Shalah, M., Groscot, R., Guibas, L.J.: Composite shape modeling via latent space factorization. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 8139–8148. IEEE (2019)
  * [10] Gadelha, M., Wang, R., Maji, S.: Multiresolution tree networks for 3D point cloud processing. In: Proceedings of the European Conference on Computer Vision (ECCV) (September 2018)
  * [11] Groueix, T., Fisher, M., Kim, V.G., Russell, B., Aubry, M.: AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2018)
  * [12] Hadad, N., Wolf, L., Shahar, M.: A two-step disentanglement method. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [13] Härkönen, E., Hertzmann, A., Lehtinen, J., Paris, S.: Ganspace: Discovering interpretable GAN controls. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual (2020)
  * [14] Hosoya, H.: Group-based learning of disentangled representations with generalizability for novel contents. In: IJCAI (2019)
  * [15] Huang, Q., Wang, W., Neumann, U.: Recurrent slice networks for 3d segmentation of point clouds. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition pp. 2626–2635 (2018)
  * [16] Jiang, Z.H., Wu, Q., Chen, K., Zhang, J.: Disentangled representation learning for 3d face shape. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 11949–11958 (2019)
  * [17] Kar, A., Häne, C., Malik, J.: Learning a multi-view stereo machine. In: NIPS (2017)
  * [18] Kwon, Y., Petrangeli, S., Kim, D., Wang, H., Park, E., Swaminathan, V., Fuchs, H.: Rotationally-temporally consistent novel view synthesis of human performance video. In: ECCV (2020)
  * [19] Li, J., Niu, C., Xu, K.: Learning part generation and assembly for structure-aware shape synthesis. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence. pp. 11362–11369. AAAI Press (2020), https://aaai.org/ojs/index.php/AAAI/article/view/6798
  * [20] Li, R., Li, X., Hui, K.H., Fu, C.W.: Sp-gan: Sphere-guided 3d shape generation and manipulation 40(4) (2021). https://doi.org/10.1145/3450626.3459766
  * [21] Li, Y., Bu, R., Sun, M., Wu, W., Di, X., Chen, B.: Pointcnn: Convolution on x-transformed points. In: NeurIPS (2018)
  * [22] Liu, Y., Fan, B., Xiang, S., Pan, C.: Relation-shape convolutional neural network for point cloud analysis. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 8887–8896 (2019)
  * [23] Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: Proceedings of International Conference on Computer Vision (ICCV) (December 2015)
  * [24] Luo, S., Hu, W.: Diffusion probabilistic models for 3D point cloud generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2837–2845 (June 2021)
  * [25] Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: Learning 3d reconstruction in function space. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) (2019)
  * [26] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
  * [27] Mo, K., Guerrero, P., Yi, L., Su, H., Wonka, P., Mitra, N., Guibas, L.: Structurenet: Hierarchical graph networks for 3d shape generation. ACM Transactions on Graphics (TOG), Siggraph Asia 2019 38(6), Article 242 (2019)
  * [28] Mo, K., Guerrero, P., Yi, L., Su, H., Wonka, P., Mitra, N.J., Guibas, L.J.: Structedit: Learning structural shape variations. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [29] Nash, C., Williams, C.K.I.: The shape variational autoencoder: A deep generative model of part-segmented 3D objects. Computer Graphics Forum 36 (2017)
  * [30] Niemeyer, M., Mescheder, L.M., Oechsle, M., Geiger, A.: Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3501–3512 (2020)
  * [31] Öngün, C., Temizel, A.: Lpmnet: Latent part modification and generation for 3D point clouds. Comput. Graph. 96, 1–13 (2021)
  * [32] Park, J.J., Florence, P.R., Straub, J., Newcombe, R.A., Lovegrove, S.: Deepsdf: Learning continuous signed distance functions for shape representation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 165–174 (2019)
  * [33] Peng, S., Zhang, Y., Xu, Y., Wang, Q., Shuai, Q., Bao, H., Zhou, X.: Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In: CVPR (2021)
  * [34] Pumarola, A., Popov, S., Moreno-Noguer, F., Ferrari, V.: C-flow: Conditional generative flow models for images and 3d point clouds. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [35] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (July 2017)
  * [36] Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks. In: Bengio, Y., LeCun, Y. (eds.) 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (2016), http://arxiv.org/abs/1511.06434
  * [37] Rakotosaona, M.J., Ovsjanikov, M.: Intrinsic point cloud interpolation via dual latent space navigation. In: ECCV (2020)
  * [38] Rios, T., van Stein, B., Menzel, S., Back, T., Sendhoff, B., Wollstadt, P.: Feature visualization for 3d point cloud autoencoders. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–9 (2020)
  * [39] Roberts, D., Danielyan, A., Chu, H., Golparvar-Fard, M., Forsyth, D.: Lsd-structurenet: Modeling levels of structural detail in 3d part hierarchies. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 5836–5845 (October 2021)
  * [40] Schor, N., Katzir, O., Zhang, H., Cohen-Or, D.: Componet: Learning to generate the unseen by part synthesis and composition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
  * [41] Shen, Y., Gu, J., Tang, X., Zhou, B.: Interpreting the latent space of gans for semantic face editing. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [42] Shen, Y., Zhou, B.: Closed-form factorization of latent semantics in gans. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 1532–1540 (2021)
  * [43] Shu, D.W., Park, S.W., Kwon, J.: 3D point cloud generative adversarial network based on tree structured graph convolutions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
  * [44] Sun, Y., Wang, Y., Liu, Z., Siegel, J.E., Sarma, S.E.: Pointgrow: Autoregressively learned point cloud generation with self-attention. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) pp. 61–70 (2020)
  * [45] Tewari, A., Pan, X., Fried, O., Agrawala, M., Theobalt, C., et al.: Disentangled3d: Learning a 3d generative model with disentangled geometry and appearance from monocular images. arXiv preprint arXiv:2203.15926 (2022)
  * [46] Thomas, H., Qi, C., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: Kpconv: Flexible and deformable convolution for point clouds. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 6410–6419 (2019)
  * [47] Upchurch, P., Gardner, J., Pleiss, G., Pless, R., Snavely, N., Bala, K., Weinberger, K.: Deep Feature Interpolation for image content changes. In: Computer Vision and Pattern Recognition (CVPR) (2017)
  * [48] Urbach, D., Ben-Shabat, Y., Lindenbaum, M.: Dpdist: Comparing point clouds using deep point cloud distance. In: Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI. vol. 12356, pp. 545–560. Springer (2020)
  * [49] Valsesia, D., Fracastoro, G., Magli, E.: Learning localized generative models for 3d point clouds via graph convolution. In: ICLR (2019)
  * [50] Wang, H., Schor, N., Hu, R., Huang, H., Cohen-Or, D., Huang, H.: Global-to-local generative model for 3D shapes. ACM Transactions on Graphics (TOG) 37, 1 – 10 (2018)
  * [51] Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG) (2019)
  * [52] Wen, C., Yu, B., Tao, D.: Learning progressive point embeddings for 3d point cloud generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 10266–10275 (June 2021)
  * [53] Xie, J., Xu, Y., Zheng, Z., Zhu, S.C., Wu, Y.N.: Generative pointnet: Deep energy-based learning on unordered point sets for 3d generation, reconstruction and classification. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 14971–14980 (2021)
  * [54] Xu, M., Ding, R., Zhao, H., Qi, X.: Paconv: Position adaptive convolution with dynamic kernel assembling on point clouds. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) pp. 3172–3181 (2021)
  * [55] Xu, Q., Wang, W., Ceylan, D., Mech, R., Neumann, U.: Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. In: NeurIPS (2019)
  * [56] Yang, G., Huang, X., Hao, Z., Liu, M.Y., Belongie, S., Hariharan, B.: Pointflow: 3D point cloud generation with continuous normalizing flows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (October 2019)
  * [57] Yang, J., Mo, K., Lai, Y.K., Guibas, L.J., Gao, L.: Dsg-net: Learning disentangled structure and geometry for 3d shape generation. ACM Trans. Graph. 42(1) (aug 2022). https://doi.org/10.1145/3526212
  * [58] Yang, Y., Feng, C., Shen, Y., Tian, D.: Foldingnet: Point cloud auto-encoder via deep grid deformation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
  * [59] Yi, L., Kim, V.G., Ceylan, D., Shen, I.C., Yan, M., Su, H., Lu, C., Huang, Q., Sheffer, A., Guibas, L.: A scalable active framework for region annotation in 3d shape collections. SIGGRAPH Asia (2016)
  * [60] Zamorski, M., Zieba, M., Klukowski, P., Nowak, R., Kurach, K., Stokowiec, W., Trzciński, T.: Adversarial autoencoders for compact representations of 3D point clouds. Computer Vision and Image Understanding 193 (April 2020)
  * [61] Zamorski, M., Zieba, M., Klukowski, P., Nowak, R., Kurach, K., Stokowiec, W., Trzciński, T.: Adversarial autoencoders for compact representations of 3D point clouds. Computer Vision and Image Understanding 193 (April 2020)
  * [62] Zhang, Z., Hua, B.S., Yeung, S.K.: Shellnet: Efficient point cloud convolutional neural networks using concentric shells statistics. 2019 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 1607–1616 (2019)
  * [63] Zhao, H., Jiang, L., Jia, J., Torr, P.H., Koltun, V.: Point transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 16259–16268 (October 2021)
  * [64] Zhu, J.Y., Zhang, Z., Zhang, C., Wu, J., Torralba, A., Tenenbaum, J., Freeman, B.: Visual object networks: Image generation with disentangled 3d representations. Advances in neural information processing systems 31 (2018)