Giving machines the ability to imagine possible new objects or scenes from linguistic descriptions and produce their realistic renderings is arguably one of the most challenging problems in computer vision. Recent advances in deep generative models have led to new approaches that give promising results towards this goal. In this paper, we introduce a new method called DiCoMoGAN for manipulating videos with natural language, aiming to perform local and semantic edits on a video clip to alter the appearances of an object of interest. Our GAN architecture allows for better utilization of multiple observations by disentangling content and motion to enable controllable semantic edits. To this end, we introduce two tightly coupled networks: (i) a representation network for constructing a concise understanding of motion dynamics and temporally invariant content, and (ii) a translation network that exploits the extracted latent content representation to actuate the manipulation according to the target description. Our qualitative and quantitative evaluations demonstrate that DiCoMoGAN significantly outperforms existing frame-based methods, producing temporally coherent and semantically more meaningful results.

§ INTRODUCTION

Making desired edits on an image or video using tools like Adobe Photoshop, Adobe Premiere Pro and Apple Final Cut Pro is quite challenging and requires extensive training and experience. Thanks to the proliferation of deep learning, some user-friendly solutions are proposed for editing images [Li et al., 2020]. Yet, democratizing the video editing process to improve accessibility and empower the non-experts still requires rethinking modern architectures.

Towards this end, we set off to ask: Can we learn to semantically manipulate videos through natural language descriptions in a temporally consistent way? ( <ref>)
The existing literature approaches this problem on a frame-by-frame basis applying minimal necessary modifications specified by the input text, disjointly to the each and every input frame. Almost all of the image editing methods use encoder-decoder architectures [Reed et al., 2016, Dong et al., 2017, Nam et al., 2018, Li et al., 2020, Li et al., 2020, Xia et al., 2021, Patashnik et al., 2021] and employ adversarial learning strategies [Isola et al., 2017, Karras et al., 2019] to provide the agreement between the resulting images and the target text and to generate photo-realistic outcomes. However, performing language-driven edits on videos requires models to not only understand the frame content but also be aware of the global video context and its temporal unidirectionality. Moreover, a harmonious editing demands the target descriptions to be related not only to static frames but to the entire video to achieve good gestalt.

Text-based video manipulation. Given a video sequence and a target , our DiCoMoGAN model generates a temporally coherent output sequence, carrying out the necessary structural changes while preserving the attributes not referred in the text ( hair style, identity), and does this without any extra guidance like semantic layout information.

To achieve all these, we propose a new data-driven text based video manipulation model called DiCoMoGAN. The key to our approach is a unified network model consisting of a representation network (RepNet) and a translation network (TraNet), which jointly learn to disentangle video content and motion dynamics and to perform the text-specified edits on a given video sequence. Under the assumption that textual description is strongly related to appearance, we create a structured latent space composed of text relevant, text irrelevant and dynamic subspaces ( <ref>). To ensure the former, we steer the latent subspace to be shared between global video descriptor and the text features, encoded by CLIP [Patashnik et al., 2021]. We then use the features from this structured latent space along with text features to condition multi-feature modulation (MFMOD) blocks. We train this integrated architecture via a multi-task loss function in an end to end manner to encode scene specific transformations effectively while capturing the relationships between the spatiotemporal data and the text input.
Our experiments on the standard 3D Shapes benchmark [Burgess and Kim, 2018] as well as on our new dataset Fashion Videos demonstrate that  can produce high quality, temporally consistent videos faithfully reflecting the intentions stated in the target descriptions. In summary, our contributions are as follows: (1) Our representation network, RepNet, implements a neural architecture that explicitly enforces the separation of static and dynamic features via a set-based $\beta$-VAE model [Higgins et al., 2017] equipped with a Latent ODE [Rubanova et al., 2019]. (2) Our translation network, TraNet, follows an encoder-decoder architecture which is guided by the representation network through a novel multi-feature modulation block called  where the residual activation maps are modulated based on both the given textual description and the disentangled content code. (3) To test the capabilities of our model in a more realistic setting, we collect a new dataset containing Fashion Videos with the related textual descriptions.

§ RELATED WORK

Disentangled representations
The aim of unsupervised disentangled representation learning is to discover underlying factors of variation in a training data, in which each dimension encodes a unique and semantically meaningful aspect of the data [Bengio et al., 2013]. To this end, most of the existing approaches are based on VAEs [Kingma and Welling, 2014, Rezende et al., 2014] with slight modifications in the VAE objective, such as $\beta$-VAE [Higgins et al., 2017], FactorVAE [Kim and Mnih, 2018]. Similarly, some prior work tweak the objective of GANs [Goodfellow et al., 2014] to achieve disentanglement, InfoGAN [Chen et al., 2016], IB-GAN [Jeon et al., 2021]. Locatello  [Locatello et al., 2019] showed that unsupervised learning of disentangled generative factors can not be achieved without strong inductive biases on both the models and the data, which can be alleviated using weak supervision or a few labeled training samples [Träuble et al., 2021]. Key to the success of our model, in our work we especially focus on disentangling motion and content, which has been previously studied in a fully unsupervised setting [Denton and Birodkar, 2017], or using action/attribute labels [He et al., 2018]. But we instead utilize natural language descriptions as weak supervision.

VAE-GAN hybrids GANs are superior to VAEs in terms of visual quality, but VAEs provide better disentangled representations. There is a line of research that explores combining VAE and GAN frameworks, ALI [Dumoulin et al., 2017], BiGAN [Donahue et al., 2016], IntroVAE [Huang et al., 2018], to name a few. The promise of these so-called hybrid approaches is to combine the advantages of both models, while providing a much stable training and improved diversity in the generated samples.

Video synthesis
Video synthesis aims at
generating temporally coherent video clips either from scratch [Mathieu et al., 2016, Finn et al., 2016, Xue et al., 2016, Liang et al., 2017, Denton and Birodkar, 2017, Weissenborn et al., 2020], or according to a single image or a short sequence of images [Vondrick et al., 2016, Saito et al., 2017, Tulyakov et al., 2018, Denton and Fergus, 2018, Akan et al., 2021]. Motivated by these works, there are also some efforts to allow the users to control the video generation process by introducing natural text [Marwah et al., 2017, Pan et al., 2017, Li et al., 2018, Balaji et al., 2019] or spoken language [Zhou et al., 2019, Das et al., 2020, Thies et al., 2020, Zhou et al., 2020, Chen et al., 2021] as an additional input.

Language based image manipulation
In text-to-image synthesis, the goal is to generate an image with a natural language description [Reed et al., 2016, Zhang et al., 2017, Xu et al., 2018, Zhu et al., 2019]. On the other hand, semantic image manipulation aka language guided image editing models [Dong et al., 2017, Nam et al., 2018, Li et al., 2020, Li et al., 2020] aim at modifying a source image according to a given textual description summarizing the desired object characteristics. SISGAN [Dong et al., 2017] involves a text-conditioned encoder-decoder architecture. TAGAN [Nam et al., 2018] learns to disentangle different semantic attributes of the target object during training by considering a text-adaptive discriminator. ManiGAN [Li et al., 2020] and LightweightGAN [Li et al., 2020], on the other hand, utilize text-image combination modules, which are used to match semantic attributes with certain words in the given descriptions, along with explicit word-level discriminators to improve the quality of the results. The recently proposed TediGAN [Xia et al., 2021], Latent Transformer [Yao et al., 2021],  and StyleCLIP [Patashnik et al., 2021] models are also capable of performing language-driven edits on a given image, but they all require a StyleGAN model pre-trained for a specific domain (faces), which is hard to train for less structured domains like full body images.
Recently, Jiang et al. [Jiang et al., 2021] propose a new language-guided editing model specifically designed for performing global edits such as changing brightness or color tone of an image.

Language based video manipulation
Our task of video-editing using natural language descriptions is a relatively new one. There are two studies worth mentioning which are concurrent to this work: (i) [Fu et al., 2023] tackles a similar problem by proposing a transformer-based architecture, but they did not make their implementation freely available; (ii) [Alaluf et al., 2023] presents a StyleGAN3-based video-editing framework, but, it only considers manipulations based on a single attribute. The latter belongs to the family of GAN-inversion based methods which uses the latent space of a pretrained StyleGAN model for editing purposes, where the existing methods focus on distinct domains such as aligned faces, as the style-based generators do not work well on unstructured datasets. Note that while inversion constitutes a sensible research direction for GANs, inverting diffusion models without significant distortion remains a challenge. Exploring these directions is future work. Nonetheless, our approach and concept of disentangled video editing can be used regardless of the backbone architectures.

§ DICOMOGAN

Problem setting
We consider the problem of manipulating a given input video according to a provided textual description. Inputs to our text-based video manipulation approach, called DiCoMoGAN, are a short video clip of a single object and a target text description summarizing the object's new look. We represent the source video as an image sequence denoted by $X = (\mathbf{x}_i\in\mathbb{R}^{3\times H\times W})_{i=1}^N$,  with $i$ being the frame index and $N$ indicating the total number of frames. Our goal is to perform seamless and semantically meaningful edits on each video frame $\mathbf{x}_i$ to reflect what is being described in the target text $\mathbf{desc}$, and accordingly generate an output sequence $Y = (\mathbf{y}_i)_{i=1}^N$ of the same spatial dimensions as the input – with the desired look. Our model carries this out by:

* a representation network to learn a disentangled latent space in which static and dynamic semantic scene characteristics are encoded independently,

* a translation network capable of transferring the target look stated in the textual description to the source video in a truthful and temporally coherent manner, and

* unifying (1) and (2) with a combined neural architecture where the two networks are trained simultaneously in an end-to-end manner.

In what follows, we describe DiCoMoGAN in detail, following the structure shown in <ref>.

§ EXPERIMENTS

First, we use the 3D Shapes dataset [Burgess and Kim, 2018] which is proposed for learning and assessing factors of variation from data. This dataset has 480K images of $64\times64$ resolution. There are $6$ ground truth independent latent factors. They are floor color, wall color, object color, scale, shape and orientation. For our purpose, we build simple text descriptions which covers object related latent factors object color, scale and shape, “There is a big blue capsule.”. To prevent scale ambiguity, we remove two elements of the scale factor which is of length $8$, originally. In that case, “small”, “medium” and “big” in the descriptions correspond to the first two, middle two and the last two values, respectively. Moreover, we consider the orientation factor as a dynamic dimension taking $15$ different values. We have 19.2K train and 4.8K test videos with $15$ frames and simple text descriptions for each video.

Second, to explore how well our model generalizes to more challenging datasets,  we collected a new video dataset, Fashion Videos, from an online shopping site, containing short video clips of individuals wearing different kinds of garments. Each clip includes full-body images of a single person moving around a scene, showing how the clothing looks from different angles. Moreover, the clip is endowed with a textual product description of the garment, detailing its visual features (color, material properties, and design details) as well as its category (dress, jumpsuits, trousers, jumper, skirt, pant). After pre-processing, we obtained 3178 video clips (109K frames), out of which 2579 are used for training and 598 for testing. More details are provided in the supplementary material and Fashion Videos will be made publicly available.

Evaluation metrics
We evaluate the results via Inception Score (IS)[Salimans et al., 2016], Fréchet Inception Distance (FID) [Heusel et al., 2017], and Fréchet Video Distance (FVD) [Unterthiner et al., 2018]. Moreover, we modify and use the manipulative precision ($\text{MP}$) metric suggested in [Li et al., 2020] to assess the manipulation performance of the models according to the target natural language descriptions. Our version ($\text{MP}_{\text{CLIP}}$) measures the similarity between the manipulated video frames and their corresponding target texts through the cosine similarity in CLIP embedding space [Radford et al., 2021].
Further details on the precise definitions of these metrics are found in our supplementary.

Baselines As, to the best of our knowledge, the literature lacks a strong language-driven video manipulation model, we compare
against SISGAN [Dong et al., 2017], TAGAN [Nam et al., 2018] and ManiGAN [Li et al., 2020][We exclude LightweightGAN [Li et al., 2020] from our analysis as it requires part-of-speech (POS) tagging, and our analysis revealed that existing POS taggers do not give satisfactory results on domain-specific fashion descriptions.]. For video editing, we conduct a frame-by-frame translation.

§.§ Evaluations

Implementation details For 3D Shapes, we use a 6-dim latent code for the frames in which the first three encode the text relevant static features, the next two the text-irrelevant static features, and the last one the dynamic feature. We set $\beta=32$. For Fashion Videos, we do not have access to the ground truth factors of variation. Thus, we consider a $16$-dim latent code in which the first eight encode the static text relevant features and the next eight the static text-irrelevant ones. The last four are reserved for the dynamic features. We set $\beta=1$.

Qualitative results on 3D Shapes [Burgess and Kim, 2018] and Fashion Videos datasets. As compared to SISGAN [Dong et al., 2017], TAGAN [Nam et al., 2018] and ManiGAN [Li et al., 2020], DiCoMoGAN gives sharper images faithful to the target descriptions while preserving inherent features not mentioned in the text, wall and floor colors, identity, hair style, much better than the competing approaches.

Quantitative results. Our approach outperforms the existing frame-based methods by a large margin in terms of all evaluation measures.

Model     IS ($\uparrow$)     FID ($\downarrow$)     FVD ($\downarrow$)     $\text{MP}_{\text{CLIP}}$ ($\uparrow$)

[t]3mm4*[origin=c]903D Shapes     SISGAN [Dong et al., 2017]     2.29     138.78    1185.48    0.18

TAGAN [Nam et al., 2018]     2.34     88.95     974.59     0.19

ManiGAN [Li et al., 2020]     2.71     26.90     753.89     0.18

DiCoMoGAN     2.76     9.08     69.30     0.26

Model     IS ($\uparrow$)     FID ($\downarrow$)     FVD ($\downarrow$)     $\text{MP}_{\text{CLIP}}$ ($\uparrow$)

[t]3mm4*[origin=c]90Fashion Videos    SISGAN [Dong et al., 2017]     2.13     80.15     2274.69     0.19

TAGAN [Nam et al., 2018]     2.24     87.60     1294.72     0.24

ManiGAN [Li et al., 2020]     2.76     37.22     392.59     0.22

DiCoMoGAN     2.96     15.34     53.75     0.25

Manipulation results
In <ref>, we provide our quantitative analysis on the 3D Shapes and the Fashion datasets. As compared to the state-of-the-art, our method gives the best results in terms of all of the evaluation metrics. In particular, our method achieves much better FVD values on both datasets. The qualitative results in <ref> indicate that our model can produce high quality results as compared to the existing models. SISGAN and TAGAN fail to preserve the text irrelevant parts like the wall color or the identity of the person. ManiGAN tends to keep the original structure intact and fails to produce the necessary structural changes. Our method, on the other hand, performs more relevant edits on the input video sequences according to the target descriptions, altering only the necessary parts of the frames while keeping what is not mentioned in text unchanged. Please refer to the supplementary material for additional higher resolution results. Our main goal is to analyze disentangling factors in videos for text-based manipulation, and there is definitely room for improvement for visual quality. In the supplementary material, we also show that better results can be obtained when we train a local enhancer network on top of TraNet in a similar vein to pix2pixHD [Wang et al., 2018].

Latent traversals.  DiCoMoGAN learns latent variables depicting highly interpretable concepts decomposed into text relevant, text irrelevant static, and dynamic features. Note that wall and floor colors are not mentioned in the descriptions during training.

Measuring disentanglement.
For disentanglement performance, we carry out experiments on 3D Shapes, where we have ground truth factors of variation, by considering the latent space discovered by RepNet's image encoder. <ref> shows sample traversals in the latent dimensions learned by our method. These traversals clearly depict interpretable
properties of the images exist in the 3D Shapes dataset. While the last latent dimension steered by the Latent ODE encodes the orientation of the camera (camera movement), all the others encode static features of the scene and the object. In fact, while the first three static dims. are about text relevant features like object color, shape and size of the object, the last two encode wall and floor colors – successfully identified without even referred in the provided textual descriptions. We provide further quantitative analysis in the supplementary.

Ablation study
In <ref>, we show the results of our ablation study in which we examine the contribution of certain components of our method on the performance. Latent ODE within RepNet plays a key role in achieving high-quality manipulation results, which can be attributed to its ability of effectively disentangling motion and content. In the supplementary material, we also analyze the effect of different loss functions and the MFMOD block.

Ablation study. Analysis of the components of our DiCoMoGAN model.

Model     IS ($\uparrow$)     FID ($\downarrow$)     FVD ($\downarrow$)     $\text{MP}_{\text{CLIP}}$ ($\uparrow$)

[t]3mm3*[origin=c]903D Shapes     DiCoMoGAN     2.76     9.08     69.30     0.26

$w/o$ Latent ODE     2.91     12.50     99.79     0.26

$w/o$ RepNet     2.82     12.32     113.89     0.25

Model     IS ($\uparrow$)     FID ($\downarrow$)     FVD ($\downarrow$)     $\text{MP}_{\text{CLIP}}$ ($\uparrow$)

[t]3mm3*[origin=c]90Fashion Videos    DiCoMoGAN     2.96     15.34     53.75     0.25

$w/o$ Latent ODE     3.10     5.35     225.58     0.24

$w/o$ RepNet     3.06     13.97     498.21     0.24

§ CONCLUSION AND FUTURE WORK

We presented DiCoMoGAN to tackle the challenging task of manipulation of videos using textual descriptions. As a first step towards solving this problem, we developed a new neural model that incorporates multiple observations to disentangle motion dynamics and visual content to better perform semantically relevant and temporally coherent edits. Our approach gives significantly better results than existing frame-based methods. As such, there are also a number of ways this work could be extended. It is possible to explore more complicated feature aggregation schemes like self-attention [Vaswani et al., 2017]) to learn permutation invariant representations. Our current model assumes that input video clips include a single object of interest. An exciting future research direction is to incorporate an object-centric approach [Greff et al., 2019, Locatello et al., 2020, Emami et al., 2021] so that it can support manipulation of multiple objects.

[Akan et al., 2021]
Adil Kaan Akan, Erkut Erdem, Aykut Erdem, and Fatma Güney.
SLAMP: Stochastic latent appearance and motion prediction.
In ICCV, 2021.

[Alaluf et al., 2023]
Yuval Alaluf, Or Patashnik, Zongze Wu, Asif Zamir, Eli Shechtman, Dani
Lischinski, and Daniel Cohen-Or.
Third time’s the charm? image and video editing with stylegan3.
arXiv preprint arXiv:2201.13433, 2023.

[Balaji et al., 2019]
Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter
Conditional GAN with discriminative filter generation for
text-to-video synthesis.
In Proceedings of the International Joint Conference on
Artificial Intelligence (IJCAI), 2019.

[Bengio et al., 2013]
Yoshua Bengio, Aaron Courville, and Pascal Vincent.
Representation learning: A review and new perspectives.
PAMI, 350 (8):0 1798–1828, 2013.

[Brock et al., 2018]
Andrew Brock, Jeff Donahue, and Karen Simonyan.
Large scale GAN training for high fidelity natural image synthesis.
In ICLR, 2018.

[Burgess and Kim, 2018]
Chris Burgess and Hyunjik Kim.
3d shapes dataset.
<https://github.com/deepmind/3dshapes-dataset/>, 2018.

[Burgess et al., 2018]
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters,
Guillaume Desjardins, and Alexander Lerchner.
Understanding disentangling in $\beta$-VAE.
arXiv preprint arXiv:1804.03599, 2018.

[Carreira and Zisserman, 2017]
Joao Carreira and Andrew Zisserman.
Quo vadis, action recognition?
In CVPR, 2017.

[Chen et al., 2021]
Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu.
Hierarchical cross-modal talking face generation with dynamic
pixel-wise loss.
In CVPR, 2021a.

[Chen et al., 2018]
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud.
Neural ordinary differential equations.
Advances in Neural Information Processing Systems,

[Chen et al., 2021]
Ricky T. Q. Chen, Brandon Amos, and Maximilian Nickel.
Learning neural event functions for ordinary differential equations.
International Conference on Learning Representations,

[Chen et al., 2018]
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud.
Isolating sources of disentanglement in variational autoencoders.

[Chen et al., 2018]
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.
Neural ordinary differential equations.
In NeurIPS, 2018c.

[Chen et al., 2016]
X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P Abbeel.
InfoGAN: Interpretable representation learning by information
maximizing generative adversarial nets.
In NeurIPS, 2016.

[Coddington and Levinson, 1955]
Earl A Coddington and Norman Levinson.
Theory of ordinary differential equations.
Tata McGraw-Hill Education, 1955.

[Das et al., 2020]
Dipanjan Das, Sandika Biswas, Sanjana Sinha, and Brojeshwar Bhowmick.
Speech-driven facial animation using cascaded GANs for learning of
motion and texture.
In ECCV, 2020.

[Deng et al., 2009]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In CVPR, pages 248–255, 2009.

[Denton and Birodkar, 2017]
E. L. Denton and V. Birodkar.
Unsupervised learning of disentangled representations from video.
In NeurIPS, 2017.

[Denton and Fergus, 2018]
Emily Denton and Rob Fergus.
Stochastic video generation with a learned prior.
In ICML, 2018.

[Donahue et al., 2016]
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell.
Adversarial feature learning.
In ICLR, 2016.

[Dong et al., 2017]
Hao Dong, Simiao Yu, Chao Wu, and Yike Guo.
Semantic image synthesis via adversarial learning.
In ICCV, 2017.

[Dubois et al., 2018]
Yann Dubois, Alexandros Kastanos, Dave Lines, and Bart Melman.
Axis alignment metric.
<https://github.com/YannDubs/disentangling-vae>, 2018.

[Dumoulin et al., 2017]
Vincent Dumoulin, Mohamed Ishmael Diwan Belghazi, Ben Poole, Alex Lamb, Martin
Arjovsky, Olivier Mastropietro, and Aaron Courville.
Adversarially learned inference.
In ICLR, 2017.

[Dupont et al., 2019]
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh.
Augmented neural odes.
In NeurIPS, volume 32, pages 3140–3150, 2019.

[Emami et al., 2021]
Patrick Emami, Pan He, Sanjay Ranka, and Anand Rangarajan.
Efficient iterative amortized inference for learning symmetric and
disentangled multi-object representations.
In ICML, 2021.

[Finn et al., 2016]
C. Finn, I. Goodfellow, and S. Levine.
Unsupervised learning for physical interaction through video
In NeurIPS, 2016.

[Fu et al., 2023]
Tsu-Jui Fu, Xin Eric Wang, Scott T. Grafton, Miguel P. Eckstein, and
William Yang Wang.
M3L: Language-based video editing via multi-modal multi-level
In CVPR, 2023.

[Goodfellow et al., 2014]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,
Sherjil Ozair, Aaron Courville, and Yoshua Bengio.
Generative adversarial nets.
In NeurIPS, pages 2672–2680, 2014.

[Greff et al., 2019]
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters,
Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and
Alexander Lerchner.
Multi-object representation learning with iterative variational
In ICML, 2019.

[He et al., 2018]
Jiawei He, Andreas Lehrmann, Joseph Marino, Greg Mori, and Leonid Sigal.
Probabilistic video generation using holistic attribute control.
In Proceedings of the European Conference on Computer Vision
(ECCV), pages 452–467, 2018.

[Heusel et al., 2017]
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp
GANs trained by a two time-scale update rule converge to a local
nash equilibrium.
In In Proceedings of the Advances in Neural Information
Processing Systems (NeurIPS), 2017.

[Higgins et al., 2017]
Irina Higgins, Loïc Matthey, Arka Pal, Christopher Burgess, Xavier Glorot,
Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner.
$\beta$-VAE: Learning basic visual concepts with a constrained
variational framework.
In ICLR, 2017.

[Huang et al., 2018]
Huaibo Huang, zhihang li, Ran He, Zhenan Sun, and Tieniu Tan.
Introvae: Introspective variational autoencoders for photographic
image synthesis.
In NeurIPS, pages 52–63, 2018.

[Huang and Belongie, 2017]
Xun Huang and Serge Belongie.
Arbitrary style transfer in real-time with adaptive instance
In ICCV, 2017.

[Isola et al., 2017]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.
Image-to-image translation with conditional adversarial networks.
In CVPR, pages 1125–1134, 2017.

[Jeon et al., 2021]
Insu Jeon, Wonkwang Lee, Myeongjang Pyeon, and Gunhee Kim.
Ib-gan: Disentangled representation learning with information
bottleneck generative adversarial networks.
In AAAI, 2021.

[Jiang et al., 2021]
Wentao Jiang, Ning Xu, Jiayun Wang, Chen Gao, Jing Shi, Zhe Lin, and Si Liu.
Language-guided global image editing via cross-modal cyclic
In ICCV, 2021.

[Johnson et al., 2016]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei.
Perceptual losses for real-time style transfer and super-resolution.
In ECCV, pages 694–711, 2016.

[Karras et al., 2019]
Tero Karras, Samuli Laine, and Timo Aila.
A style-based generator architecture for generative adversarial
In CVPR, pages 4401–4410, 2019.

[Karras et al., 2020]
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and
Timo Aila.
Analyzing and improving the image quality of StyleGAN.
In CVPR, 2020.

[Kim and Mnih, 2018]
Hyunjik Kim and Andriy Mnih.
Disentangling by factorising.
In ICML, 2018.

[Kingma and Welling, 2014]
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
In ICLR, 2014.

[Li et al., 2020]
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Torr.
ManiGAN: Text-guided image manipulation.
In CVPR, 2020a.

[Li et al., 2020]
Bowen Li, Xiaojuan Qi, Philip H. S. Torr, and Thomas Lukasiewicz.
Lightweight generative adversarial networks for text-guided image
In NeurIPS, 2020b.

[Li et al., 2018]
Yitong Li, Martin Renqiang Min, Dinghan Shen, David Carlson, and Lawrence
Video generation from text.
In AAAI, 2018.

[Liang et al., 2017]
X. Liang, L. Lee, W. Dai, and E. P. Xing.
Dual motion GAN for future-flow embedded video prediction.
In NeurIPS, 2017.

[Locatello et al., 2019]
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly,
Bernhard Schölkopf, and Olivier Bachem.
Challenging common assumptions in the unsupervised learning of
disentangled representations.
In ICML, 2019.

[Locatello et al., 2020]
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran,
Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf.
Object-centric learning with slot attention.
In NeurIPS, 2020.

[Marwah et al., 2017]
Tanya Marwah, Gaurav Mittal, and Vineeth N. Balasubramanian.
Attentive semantic video generation using captions.
In CVPR, 2017.

[Mathieu et al., 2019]
Emile Mathieu, Tom Rainforth, N Siddharth, and Yee Whye Teh.
Disentangling disentanglement in variational autoencoders.
In ICML, 2019.

[Mathieu et al., 2016]
M. Mathieu, C. Couprie, and Y. LeCun.
Deep multi-scale video prediction beyond mean square error.
In ICLR, 2016.

[Mirza and Osindero, 2014]
Mehdi Mirza and Simon Osindero.
Conditional generative adversarial nets.
arXiv preprint arXiv:1411.1784, 2014.

[Nam et al., 2018]
Seonghyeon Nam, Yunji Kim, and Seon Joo Kim.
Text-adaptive generative adversarial networks: Manipulating images
with natural language.
In NeurIPS, pages 42–51, 2018.

[Nguyen et al., 2017]
Anh Nguyen, Jeff Clune, Yoshua Bengio, Alexey Dosovitskiy, and Jason Yosinski.
Plug & play generative networks: Conditional iterative generation of
images in latent space.
In CVPR, 2017.

[Nie et al., 2020]
Weili Nie, Tero Karras, Animesh Garg, Shoubhik Debnath, Anjul Patney, Ankit
Patel, and Animashree Anandkumar.
Semi-supervised StyleGAN for disentanglement learning.
In International Conference on Machine Learning, pages
7360–7369, 2020.

[Odena et al., 2017]
Augustus Odena, Christopher Olah, and Jonathon Shlens.
Conditional image synthesis with auxiliary classifier GANs.
In ICML, 2017.

[Pan et al., 2017]
Yingwei Pan, Zhaofan Qiu, Ting Yao Houqiang Li, and Tao Mei.
To create what you tell: Generating videos from captions.
In ACMMM, 2017.

[Park et al., 2019]
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.
Semantic image synthesis with spatially-adaptive normalization.
In CVPR, 2019.

[Patashnik et al., 2021]
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.
StyleCLIP: text-driven manipulation of stylegan imagery.
In ICCV, 2021.

[Pontryagin, 2018]
Lev Semenovich Pontryagin.
Mathematical theory of optimal processes.
Routledge, 2018.

[Qi et al., 2017]
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.
PointNet++: Deep hierarchical feature learning on point sets in a
metric space.
In NeurIPS, 2017.

[Radford et al., 2021]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
arXiv preprint arXiv:2103.00020, 2021.

[Ravanbakhsh et al., 2016]
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos.
Deep learning with sets and point clouds.
arXiv preprint arXiv:1611.04500, 2016.

[Reed et al., 2016]
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and
Honglak Lee.
Generative adversarial text to image synthesis.
In ICML, pages 1060–1069, 2016.

[Rempe et al., 2020]
Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojcic, Srinath Sridhar, and
Leonidas J. Guibas.
Caspr: Learning canonical spatiotemporal point cloud representations.
In Advances in Neural Information Processing Systems
(NeurIPS), 2020.

[Rezende et al., 2014]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.
Stochastic backpropagation and approximate inference in deep
generative models.
In ICML, 2014.

[Rubanova et al., 2019]
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud.
Latent ordinary differential equations for irregularly-sampled time
NeurIPS, 32:0 5320–5330, 2019.

[Saito et al., 2017]
M. Saito, E. Matsumoto, and S. Saito.
Temporal generative adversarial nets with singular value clipping.
In ICCV, 2017.

[Salimans et al., 2016]
Tim Salimans, Ian J. Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford,
and Xi Chen.
Improved techniques for training GANs.
In In Proceedings of the Advances in Neural Information
Processing Systems (NeurIPS), 2016.

[Simonyan and Zisserman, 2015]
K. Simonyan and A. Zisserman.
Very deep convolutional networks for large-scale image recognition.
In ICLR, 2015.

[Szegedy et al., 2016]
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna.
Rethinking the inception architecture for computer vision.
In CVPR, 2016.

[Thies et al., 2020]
Justus Thies, Mohamed Elgharib, Ayush Tewari, Christian Theobalt, and Matthias
Neural voice puppetry: Audio-driven facial reenactment.
In ECCV, 2020.

[Träuble et al., 2021]
Frederik Träuble, Elliot Creager, Niki Kilbertus, Francesco Locatello,
Andrea Dittadi, Anirudh Goyal, Bernhard Schölkopf, and Stefan Bauer.
On disentangled representations learned from correlated data.
In ICML, 2021.

[Tulyakov et al., 2018]
S. Tulyakov, M.-Y. Liu, X. Yang, and J. Kautz.
MoCoGAN: Decomposing motion and content for video generation.
In CVPR, 2018.

[Unterthiner et al., 2018]
Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier,
Marcin Michalski, and Sylvain Gelly.
Towards accurate generative models of video: A new metric &
arXiv preprint arXiv:1812.01717, 2018.

[Vaswani et al., 2017]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
In NeurIPS, 2017.

[Vondrick et al., 2016]
C. Vondrick, H. Pirsiavash, and A. Torralba.
Generating videos with scene dynamics.
In NeurIPS, 2016.

[Wang et al., 2018]
Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan
High-resolution image synthesis and semantic manipulation with
conditional GANs.
In CVPR, pages 1–13, 2018.

[Weissenborn et al., 2020]
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkorei.
Scaling autoregressive video models.
In ICLR, 2020.

[Xia et al., 2021]
Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.
TediGAN: Text-guided diverse face image generation and
In CVPR, 2021.

[Xu et al., 2018]
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and
Xiaodong He.
AttnGAN: Fine-grained text to image generation with attentional
generative adversarial networks.
In CVPR, pages 1316–1324, 2018.

[Xue et al., 2016]
T. Xue, J. Wu, K. Bouman, and B. Freeman.
Visual dynamics: Probabilistic future frame synthesis via cross
convolutional networks.
In NeurIPS, 2016.

[Yang et al., 2018]
Dingdong Yang, Seunghoon Hong, Yunseok Jang, Tianchen Zhao, and Honglak Lee.
Diversity-sensitive conditional generative adversarial networks.
In ICLR, 2018.

[Yao et al., 2021]
Xu Yao, Alasdair Newson, Yann Gousseau, and Pierre Hellier.
A latent transformer for disentangled face editing in images and
In ICCV, 2021.

[Zaheer et al., 2017]
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos,
Ruslan R. Salakhutdinov, and Alexander J. Smola.
Deep sets.
In NeurIPS, 2017.

[Zhang et al., 2017]
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang,
and Dimitris Metaxas.
StackGAN: Text to photo-realistic image synthesis with stacked
generative adversarial networks.
In ICCV, pages 5908–5916, 2017.

[Zhou et al., 2019]
Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, and Ziwei
Pose-controllable talking face generation by implicitly modularized
audio-visual representation.
In CVPR, 2019.

[Zhou et al., 2020]
Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis,
and Dingzeyu Li.
MakeltTalk: Speaker-aware talking-head animation.
ACM Trans. Graph., 390 (6), nov 2020.

[Zhu et al., 2017]
Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.
Unpaired image-to-image translation using cycle-consistent
adversarial networks.
In ICCV, pages 1558–1566, 2017.

[Zhu et al., 2019]
Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang.
DM-GAN: Dynamic memory generative adversarial networks for
text-to-image synthesis.
In CVPR, 2019.


# SRG-Net: Unsupervised Segmentation for Terracotta Warrior Point Cloud with
3D Pointwise CNN methods

Yao Hu<EMAIL_ADDRESS>Guohua Geng<EMAIL_ADDRESS>Kang Li
<EMAIL_ADDRESS>Wei Zhou<EMAIL_ADDRESS>Xingxing Hao
<EMAIL_ADDRESS>Xin Cao<EMAIL_ADDRESS>School of Information Science and
Technology, Northwest University, Xi’an 710127, Shaanxi, China

###### Abstract

At present, the repairing works of terracotta warriors in Emperor
Qinshihuang’s Mausoleum Site Museum are mainly done by handwork. However, with
the increasing amount of unearthed pieces of terracotta warriors, the
restoration efficiency of archaeologists is greatly reduced. We hope to
segment the 3D point cloud data of the terracotta warriors automatically, and
store the fragment data in the database to assist the archaeologists to match
the actual fragments with the fragments in the database, thus could result in
the higher repairing efficiency of terracotta warriors for archaeologists.
Besides, previous neural network researches in 3D are mainly about supervised
classification, clustering, unsupervised representation and reconstruction.
There are few researches focusing on unsupervised point cloud part
segmentation. To address these problems, we present a seed-region-growing
CNN(SRG-Net) for unsupervised part segmentation with 3D point clouds of
terracotta warriors. Firstly, we propose our customized seed region growing
algorithm to coarsely segment the point cloud. Then we present our supervised
segmentation and unsupervised reconstruction networks to better understand the
characteristics of 3D point clouds. Finally, we combine the SRG algorithm with
our improved CNN using a refinement method called SRG-Net to conduct the
segmentation tasks on the terracotta warriors. Our proposed SRG-Net are
evaluated on the terracotta warriors data and the benchmark dataset of
ShapeNet with measuring mean intersection over union(mIoU) and latency. The
experimental results show that our SRG-Net outperforms the state-of-the-art
methods. Our code is available at https://github.com/hyoau/SRG-Net.

###### keywords:

Point Cloud , Unsupervised Segmentation , Terracotta Warrior , CNN , Seed
Region Growing

††journal: Computer-Aided Design

##  1 Introduction

Nowadays, the repairing works of terracotta warriors are mainly accomplished
by the handwork of archaeologists at the Emperor Qinshihuang’s Mausoleum Site
Museum. However, with more and more fragments of terracotta warriors unearthed
from the archaeological site, the restoration efficiency of terracotta
warriors is extremely low. Moreover, the lack of archaeological technicians to
repair the terracotta warriors has exacerbated this situation. As a result,
there are large amounts of fragments excavated from the archaeological
siteremaining to be repaired. Fortunately, with the development of 3D sensors
(Apple Li-DAR, Kinect and RealSense) and the improvement of hardware (GPU and
CPU) performance, it is possible to digitize the terracotta warriors. We can
segment the 3D object of the terracotta warriors and store the fragments in
the database of the museum. The researchers in the museum can match fragments
excavated from the excavation site with the pre-segmented fragments in the
database to restore the terracotta warriors. In order to improve the
efficiency of the repairing work, we propose an unsupervised 3D point cloud
segmentation method for the terracotta warrior data. With the segmentation
method proposed in this paper, the complete terracotta warriors can be
segmented into the fragments corresponding to various parts automatically,
e.g. head, hand and foot, etc. Compared with the traditional manual method,
the method proposed in this paper can greatly improve the efficiency and
accuracy of the repair process.

3D data can usually be expressed in different formats, such as point clouds,
meshes, volumetrics and multi-view images. Regarding which data format is used
to process the model, we compare the commonly used data formats. For the mesh,
it is irregular and time-consuming, which will cause high space and time
complexity. Therefore, mesh is not a good way to learn the features. As for
multi-view images, one image can only show the feature of one surface, but not
the overall structure of an object. Recently, there have been many researches
focusing on how to voxelize the point clouds to make them evenly distributed
in regular 3D space, and then implement 3D-CNN on them. However, voxelization
owns the high space and time complexity, and there may be quantization errors
in the process of voxelization, which would result in low accuracy. Compared
with other data formats, point cloud is a data structure suitable for 3D scene
calculation of terracotta warriors. We choose to segment the terracotta
warriors in the form of point clouds (see samples in Figure 1).

(a) 005413

(b) 005420

(c) 005422

(d) 005423

(e) 005424

(f) 005427

(g) 005429

(h) 005433

(i) 005442

(j) 005445

(k) 005452

(l) 005453

(m) 005455

(n) 005456

(o) 005457

(p) 005458

(q) 005459

(r) 005462

Figure 1: Simplified Terracotta Warrior Point Clouds

Our terracotta warriors are represented with $\\{x_{n}\in R^{p}\\}_{n=1}^{N}$,
where $R^{p}$ is the feature space, $x_{n}$ means the features of one point,
such as $x$, $y$, $z$, $N_{x}$, $N_{y}$, $N_{z}$(xyz coordinates and normal
value), $N$ means the number of points in one terracotta warrior 3D object.
Our goal is to design a function $f:R^{p}\rightarrow L$, where $L$ means the
segmentation mapping labels. $\\{c_{n}\in L\\}_{n=1}^{N}$, where $c_{n}$ is
each point’s label after the segmentation. In our problem, $\\{c_{n}\\}$ is
fixed while the mapping function $f$ and $\\{x_{n}\\}$ are trainable, so this
problem can also be regarded as a supervised problem. To solve the
unsupervised segmentation problem, we can split the problem into two sub-
problems. Firstly, we need to design an algorithm to make prediction of
optimal $L$. Secondly, we need to design a network to better learn the labels
$\\{c_{n}\\}$ and make the most of the terracotta warrior point cloud
features. To get the optimal cluster labels $\\{c_{n}\\}$, we think that a
good point cloud segmentation acts like what a human does. Similar to image
segmentation [1], when a person is asked to segment a point cloud, he is most
likely to cluster the points according to the similar features or semantic
meaning of different parts. In 2D images, spatially continuous pixels tend to
have similar colors or texture. While in 3D point cloud, grouping spatially
continuous points that have similar normal value or colors are reasonable to
be given the same label in 3D point cloud segmentation. Besides, the euclidean
distance between the points will not be very long. In conclusion, we design
two criteria to predict the $\\{c_{n}\\}$:

  1. 1.

Points with similar features are desired to be given the same label.

  2. 2.

The euclidean distance between spatially continuous points should not be quite
long.

We design a SRG algorithm to learn the normal feature of point clouds as much
as possible. The experiment [2] shows that the encode-decoder structure of
FoldingNet[3] does help neural network learn from point cloud. And DG-CNN is
good at learning local features. So we combine FoldingNet [3] and DG-CNN [4]
to better learn the part feature of terracotta warriors. In Section3, we
describe the details of the process of assigning labels $\\{c_{n}\\}$.
Section4 will show that our method achieve better methods compared with other
methods. Compared with existing methods, the key contribution of our work can
be summarized as follows:

  * 1.

We design a novel SRG algorithm for point cloud segmentation to make the most
of point cloud xyz coordinates by estimating normal feature.

  * 2.

We design our CNN by combining DG-CNN and FoldingNet to better learn the local
and global features of 3D point clouds.

  * 3.

We combine the SRG algorithm and CNN neural network with our refinement method
and achieve state-of-the-art segmentation results.

  * 4.

Our end-to-end model not only can be used in terracotta warrior point clouds,
but also can obtain quite good results on other point clouds and we evaluate
our SRG-Net on ShapeNet dataset.

##  2 Related Work

Segmentation is common both in 2D images and 3D point clouds. In 2D image,
segmentation accomplish a task that assigns labels to all the pixels in one
image to cluster the pixels with feature. Similarly, point cloud segmentation
is a process assigning labels to all the points in point cloud. The expected
result is the points with similar feature are assigned with the same label.

In terms of image segmentation, k-means is a classical segmentation method in
both 2D and 3D, which leads to partition n observations into k clusters with
the nearest mean. This method is quite popular in data mining. Graph-based
method is another popular method like Prim and Kruskal [5] which implements
simple greedy decisions in segmentation. The above methods focus on global
features instead of the local differentiate features, so when it comes to the
complex context, usually they cannot get promising results. Among unsupervised
deep learning methods, there are many methods learning feature using the
generative methods, such as [6, 7, 8]. They follow the model of neuroscience,
where each neuron represents specific semantic meaning. Meanwhile, CNN is
widely used in image unsupervised segmentation. For example, in [1], Kanezaki
combines the superpixel [9] method and CNN, and employs superpixel for back
propagation to tune the unsupervised segmentation results. Besides, [10] uses
a spatial continuity loss as an alternative to settle the limitation of the
former work [9], whose method is also quite useful in 3D point cloud feature
learning.

In the field of 3D point cloud segmentation, the state-of-the-art methods for
2D images is not quite suitable for being directly used on point cloud because
3D point cloud segmentation requires the understanding both the global feature
and geometric details of each point. 3D point cloud segmentation methods can
be classified into three types: semantic segmentation, instance segmentation
and object segmentation. Semantic segmentation focuses on scene level
segmentation, instance segmentation lay particular emphasis on object level
segmentation, and object segmentation is centered on part level segmentation.

As to semantic segmentation, semantic segmentation aims at separating a point
cloud into several parts with the semantic meaning of each point. There are
four main paradigms in semantic segmentation including projection-based
methods, discretization-based methods, point-based methods and hybrid methods.
Projection-based methods always project a 3D point cloud to 2D images, such as
multi-view [11, 12], spherical [13, 14]. Discretization-based method usually
project a point cloud into a discrete representation, such as volumetric [15]
and sparse permutohedral lattices [16, 17]. Instead of learning a single
feature on 3D scans, several methods try to learn different features from 3D
scans, such as [16, 18, 17].

Point-based network can directly learn features on a point cloud and separate
it into several parts. Point clouds are irregular, unordered and unstructured.
PointNet [19] can directly learn features from point cloud and retain the
point cloud permutation invariance with a symmetric function like maximum
function and sum function. PointNet can learn point wise feature with the
combination of several MLP layers and a max-pooling layer. PointNet is a
pioneer that directly learn on point cloud. A series of point-based network
has been proposed based on PointNet. However, PointNet can only learn features
on each point instead on the local structure. So PointNet++ is presented to
get local structure from neighborhood with a hierarchy network [20]. PointSIFT
[21] is proposed to encode orientation and reach scale awareness. Instead of
using K-means to cluster and KNN to generate neighborhoods like the grouping
method PointNet++, PointWeb [22] is proposed to get the relations between all
the points constructed in a local fully-connected web. As to convolution based
method. RS-CNN takes a local point cloud subset as its input and maps the low-
level relation to high level relation as the convolution to better learn the
feature. PointConv [23] uses the existing algorithm, using a Monte Carlo
estimation to define the convolution. PointCNN [24] uses $\chi-conv$
transformation to convert point cloud into a latent and cananical order. As to
point convolution methods, Parametric Continuous Convolutional Neural
Network(PCCN) [25] is proposed based on a parametric continuous convolution
layers, whose kernel function is parameterized by MLPs and spans the
continuous vector space. As to Graph based method, compared with other
methods, graph-based methods can better learn the features like shapes and
geometric structures in point clouds. Graph Attention Convolution(GAC) [26]
can learn several relevant features from local neighborhoods by dynamically
assigning attention weights to points in different neighborhoods and feature
channels. Dynamic Graph CNN(DG-CNN) [4] constructs several dynamic graphs in
neighborhood, and concatenate the local and global features to extract better
features and update each graph after each layer of the network dynamically.
FoldingNet adopts the auto-encoder structure to encode the point cloud
$N\times 3$ to $1\times 512$ and decode it to $M\times 3$ with the aid of
chamfer loss to construct the auto-encoder network.

In terms of part segmentation, compared with semantic and instance
segmentation, part segmentation is more difficult because there are large
geometric differences between the points with the same labels and the number
of parts with the same semantic meanings may differ. Z. Wang et al. [27]
propose VoxSegNet to achieve promising part segmentation results on 3D
voxelized data, which proposes a Spatial Dense Extraction(SDE) module to
extract multi-scale features from volumetric data. Synchronized Spectral CNN
(SyncSpecCNN) [28] is proposed to achieve fine-grained part segmentation on
irregularity and non-isomorphic shape graphs with convolution. [29] is
proposed to segment unorganized noisy point clouds automatically by extracting
clusters of points on the Gaussian sphere. [30] uses three shape indexes: the
smoothness indicator, shape index and flatness index, which is based on a
fuzzy parameterization. [31] present a segmentation method for conventional
engineering objects based on local estimates of various geometric features.
Branched AutoEncoder network (BAE-NET) [32] is proposed to perform
unsupervised and weakly supervised 3D shape co-segmentation. Each branch of
the network can learn features from a specific part shape for a specific part
shape with representation, based on the auto-encoder structure.

##  3 Proposed Method

In this paper, our input data for terracotta warrior is the form of 3D point
clouds (see also samples in Figure 1). A point cloud is represented as a set
of 3D points $\\{P_{i}|i=1,2,3...n\\}$, where each point is a vector $R^{n}$
containing coordinates x, y, z and other features like normal, color. Our
method contains three steps:

  1. 1.

We estimate normal value with the xyz coordinates.

  2. 2.

We use our seed region growing method to pre-segment the point cloud.

  3. 3.

We use our pointwise CNN called SRG-Net self-trained to segment the point
cloud with the refinement of pre-segmentation results.

Given a point cloud only with 3D coordinates, there are several effective
normal estimation methods like [33] [34] [35]. In our method, we follow the
simplest method in [36] because of its low average-case complexity and quite
good accuracy. We propose an unsupervised segmentation method for the
terracotta warrior point cloud by combining our seed-region-growing clustering
method with our 3D pointwise CNN called SRG-Net. This method contains two
phases, SRG segmentation and CNN deep learning. The problem we want to solve
can be described as follows. First, SRG algorithm is implemented on the point
cloud to do pre-segmentation. Then we use SRG-Net to self-train for
unsupervised with pre-segmentation labels of SRG.

###  3.1 Seed Region Growing

Unlike 2D images, not all point cloud data has features such as color and
normal. For example, our terracotta warriors 3D object do not have any color
feature. However, normal vectors can be calculated and predicted by point
coordinates. It is worth noting that there are many similarities and
differences between color in 2D and normal features in 3D. For color feature
in 2D image, if the pixels are semantically related, the color of the pixel in
the neighborhood generally do not change. For 3D point cloud normal features,
compared with the color features in 2D images, the normal values of points in
the neighborhood of the point cloud often differs. In terms of similarities,
even though each point of point cloud neighborhood has different normal
values, they usually do not change much unless they are not semantically
continuous. We design a seed-region-growing method to pre-segment the point
cloud based on the above characteristics of the normal feature of point cloud.

Input: $P=\\{p_{n}\in R^{3}\\}$

// x, y, z coordinates

$N=\\{n_{n}\in R^{3}\\}$

// normal value

$E=\\{e_{n}\in R^{n}\\}$

// euclidean distance

$\Omega(\cdot)$

// nearest neighbour function

Output: $L=\\{l_{n}\in R^{1}\\}$

// segmentation label of each point

Initialize: $S:=\diameter$

// set seed list to empty

$A:=rand\\{1,2...|P|\\}$

// select random point to available point list as seed

for  _t = 1 to T_ do

while  _$A\neq\diameter$_ do

$a\leftarrow first\;item\;in\;A$

$neighbours\leftarrow\Omega(a)$ for  _$neighbours\neq\diameter$_ do

$neighbour\leftarrow rand(neighbours)$

if  _$E(a,neighbour)\leq e_{t}h\land N(a,neighbour)\leq e_{t}h\land
neighbour\not\in S$ _ then

$append\;neighbour\;to\;S$ $remove\;neighbour\;from\;neighbours$

else

$remove\;neighbour\;from\;neighbours$

$add\;a\;to\;S$

Algorithm 1 SRG unsupervised segmentation Figure 2: Network Structure

First, we implement KNN to the point cloud to the point cloud to get the
nearest neighbors of each point. Then we pick up a random point as the start
seed and add to the available points to start the algorithm. Then we choose
the first seed from the available list to judge the points in its
neighborhood. If the normal value and euclidean distance are within the
threshold we set, then it can be judged that the two points are semantically
continuous and we can group two points into one cluster. An outline about the
SRG-clustering is given in Algorithm 1. The description of the algorithm is as
follows:

  1. 1.

Input:

    1. (a)

xyz coordinates of each point ${p_{n}\in R^{3}}$

    2. (b)

normal value of each point ${n_{n}\in R^{3}}$

    3. (c)

euclidean distance of every two points ${e_{n}\in R^{n}}$

    4. (d)

neighborhood of each point ${b_{n}\in R^{k}}$

  2. 2.

The random chosen point is added to the set $A$, we can call it seed $a$.

  3. 3.

For other points which are not transformed to seeds in $A$:

    1. (a)

Each of point in seed’s neighbor $N_{a}$, check the normal distance and
euclidean distance between the point in $N_{a}$ and the seed $a$.

    2. (b)

We set the normal distance and the euclidean distance threshold. If both of
the two value is less than the two threshold, we can add it to $A$.

    3. (c)

The current seed is added to checked list $S$.

###  3.2 SRG-Net

We propose our SRG-Net, which is inspired by dynamic graph in DG-CNN and auto-
encoder in FoldingNet. Unlike classical graph CNN, our graph is dynamic and
updated in each layer of the network. Compared with the methods that purely
focus on the relation-ship between points, we also propose an encoder
structure to better express the features of the entire point cloud, aiming at
better learn the structure of the point cloud and optimize the pre-
segmentation results of SRG. Our network structure can be shown in Figure2,
which consists of two sub-network, the first part is encoder that generate
features from dynamic graph and the whole point cloud while the second is the
segmentation network.

We denote the point cloud as $S$. We use lower-case letters to represent
vectors, such as $x$, and use upper-case letter to represent matrix, such as
A. We call a matrix m-by-n or $m\times n$ if it has m rows and n columns. In
addition, the terracotta warrior point cloud data is N points with 6 features
$x$, $y$, $z$, $N_{x}$, $N_{y}$, $N_{z}$(xyz coordinates and normal value),
denoted as $X=\\{x_{1},x_{2},x_{3},...,x_{n}\\}\subseteq R^{6}$.

####  3.2.1 Encoder Architecture

The SRG-Net encoder follows a similar design of [3], the structure of SRG-Net
is shown Figure 2. Compared with [3], our encoder concatenate several multi-
layer perceptrons(MLP) and several dynamic graph-based max-pooling layers. The
dynamic graphs are constructed by applying KNN on point clouds.

First, for the entire point cloud. We compute a spatial transformer network
and get a transformer matrix of 3-by-3 to maintain invariance under
transformations. Then for the transformed point cloud, we compute three
dynamic graphs and get graph features respectively. In graph feature
extracting process, we adopt the Edge Convolution in [7] to compute the graph
feature of each layer, which uses an asymmetric edge function in Eq. (1):

${f_{i}}_{j}=h(x_{i},x_{i}-x_{j})$ (1)

where it combines the coordinates of neighborhood center $x_{i}$ with the
subtraction of neighborhood point and the center point coordinates
$x_{i}-x_{j}$ to get local and global information of neighborhood. Then we
define our operation in Eq. (2):

$g_{ij}=\Theta(\mu\cdot(x_{i}-x_{j})+\omega\cdot x_{i})$ (2)

where $\mu$ and $\omega$ are parameters and $\Theta$ is a ReLU function. Eq.
(2) is implemented as a shared MLP with Leaky ReLU. Then we define our max-
pooling operation in Eq. (3):

$x_{i}=\max_{j\in N(i)}g_{ij}$ (3)

where $N(i)$ means neighborhood of point $i$.

Bottleneck is computed by graph feature extraction layer, whose structure is
show in Figure2. First, we compute covariance $3\times 3$ matrix for every
point and vectorize it to $1\times 9$. Then the $n\times 3$ matrix of point
coordinates is concatenated with the $n\times 9$ covariance matrix into a
$n\times 12$ matrix and enter the matrix into a 3-layer perceptron. Then we
feed the output of the perceptron to two subsequent graph layers. In each
layer, max-pooling is added to the neighbor of each node. At last, we apply a
3-layer perceptron to the former output and get the final output. The whole
process of the graph feature extraction layer is summarized in Eq. (4):

$Y=I_{max}(X)K$ (4)

In Eq. (4), $X$ is the input matrix to the graph layer and $K$ is a feature
mapping matrix. $I_{max}(X)$ can be represented in Eq. 5:

$(I_{max}(X))_{ij}=\Theta(\max_{k\in N(i)}x_{kj})$ (5)

where $\Theta$ is a ReLU function and $N(i)$ is the neighborhood of point $i$.
The max-pooling operation in Eq. (5) can get local feature based on the graph
structure. So the graph feature extraction layer can not only get local
neighborhood features, but also global features.

####  3.2.2 Segmenter Architecture

Segmenter gets dynamic graph features and bottleneck as input and assign
labels to each point to segment the whole point cloud. The structure of
segmenter is shown in Figure 2. First, bottleneck is replicated $N$ times in
Eq. (6):

$B^{\prime}=\underbrace{B\cup B\cup...\cup B}_{N}$ (6)

where $N$ is the number of points in point cloud and $B$ is the bottleneck.
The output of replication is concatenated with dynamic features in Eq. (7):

$C=B^{\prime}\cup D_{1}\cup D_{2}\cup D_{3}$ (7)

where $D_{1}$, $D_{2}$, $D_{3}$ represent dynamic graph features. At last, we
feed the output of concatenation to a multi-layer-perceptron to segment the
point cloud in Eq. (8).

$D=\Theta(\Psi\cdot C+\Omega)$ (8)

where $\Psi$ and $\Omega$ represent parameters in linear function, and
$\Theta$ represents a ReLU function.

###  3.3 Refinement

Instead of working on the SRG pre-segmentation results straightly, point
refinement like [37] is implemented in our SRG-Net method. There is no need to
set cluster number very big in SRG, because DG-CNN and FoldingNet have the
ability of learning the local and global features and refinement can achieve
quite better results compared with straight learning in Figure 1.

##  4 Experiments

We train our data on our terracotta warrior dataset and ShapeNet v2 dataset.
The pipeline is implemented by using PyTorch and Python3.7. All the result is
based on experiment under GeForce RTX 2080 Ti and Intel(R) Core(TM) i9-9900K
CPU @ 3.60GHz. We also evaluate our pipeline on the above datasets. The
performance is evaluated by mean intersection-over-union(mIoU).

In our work, the 3D models of terracotta warriors are obtained by Artec Eva
[38]. We got 500 intact terracotta warriors 3D models. Each model consists of
about 2 million points, verture color, verture normal, face color, wedge
coordinates and texture images. We convert the 3D obj files to point clouds
with $x$, $y$, $z$ and estimate normal value $N_{x}$, $N_{y}$, $N_{z}$ to aid
to our segmentation work. We also down sample our point clouds to 10,000
points to limit the memory under 11GB. In order to evaluate our method, we
also split the models into three parts, 80% models for training, 20% for
testing. Some of our simplified 3D models are shown in Figure 1. In order to
further evaluate our method, we also test our data on ShapeNet [39].

###  4.1 Experiments on Terracotta Warrior

In our experiments on terracotta warrior. We got 500 intact terracotta
warriors 3D models. Each model consists of about 2 million points, verture
color, verture normal, face color, wedge coordinates and texture images.
Firstly, We convert the 3D obj files to point clouds with $x$, $y$, $z$ and
estimate normal value $N_{x}$, $N_{y}$, $N_{z}$. Secondly, We down sample our
point clouds to 10,000 points. Thirdly, we implement our SRG-Net to our
terracotta warrior dataset. We set number of iterations as 2000 to achieve a
balance between time-consuming and accuracy. The segmentation results of
different iterations in one terracotta warrior model is shown in Figure 3.

(a) iter 250

(b) iter 500

(c) iter 750

(d) iter 1000

(e) iter 1250

(f) iter 1500

(g) iter 1750

(h) iter 2000

Figure 3: Results of Different Iterations in SRG-Net Method |  005413 |  005420 |  005422 |  005423 |  005427 |  005455
---|---|---|---|---|---|---
SRG-Net |  |  |  |  |  |
SRG-DGCNN |  |  |  |  |  |
Kmeans-DGCNN |  |  |  |  |  |
SRG-Pointnet2 |  |  |  |  |  |
Kmeans-Pointnet2 |  |  |  |  |  |
SRG-Pointnet |  |  |  |  |  |
Kmeans-Pointnet |  |  |  |  |  |
Table 1: Results of Different Methods Method |  mIoU |  Latency
---|---|---
SRG-Net |  0.8263 |  32.11
SRG-DGCNN |  0.7832 |  37.26
Kmeans-DGCNN |  0.7064 |  36.30
SRG-Pointnet2 |  0.7055 |  24.10
Kmeans-Pointnet2 |  0.5233 |  25.38
SRG-Pointnet |  0.7603 |  13.35
Kmeans-Pointnet |  0.5394 |  13.47
Table 2: Comparison of Different Methods

###  4.2 Comparison of Different Methods

In this section, we compare our SRG-Net with the state-of-the-art methods
sharing the similar structure of our SRG-Net. we compare our SRG-Net with the
state-of-the-art methods. We choose K-means as a comparison with our seed
region growing method and DG-CNN, PointNet++, PointNet as the comparison with
our convolution neural network. So we compare our SRG-Net with SRG-DGCNN, SRG-
PointNet++, SRG-PointNet, K-means-DGCNN, K-means-PointNet++, K-means-PointNet.

The comparison experiments are implemented on our terracotta warriors
datasets. We trained these methods on 400 terracotta warriors 3D point cloud,
and we test the trained model on the other 120 point clouds. According to the
suggestions of experts in the school of archaeology, it is reasonable to split
the intact terracotta warriors into five to seven parts. So we split the point
cloud into six pieces both in SRG, K-Means. We also set segmentation number
$S=6$ and number of iterations $T=2000$ in CNN methods(PointNet, PointNet2,
DG-CNN, DGCNN-refine) for each point cloud. The results of different
iterations in SRG-Net are show in Figure 3. Comparison of different methods
are shown as in Table 2. The results of different methods can be shown in
Table. 1.

###  4.3 Experiments on ShapeNet

In this section we perform experiments on ShapeNet to evaluate robustness of
our SRG-Net method. The dataset contains 16,881 object from 16 categories. We
set number of part according to different categories separately. Some of the
unsupervised segmentation results are shown in Figure 4. As Figure 4 shows
SRG-Net can segment well-structured objects such as knife, motorbike, etc. But
when faced with objects that are not very well-structured such as mug, the
segmentation results are not very good. We will conduct research on the
robustness of SRG-Net in the future work.

(a) airplane 01

(b) airplane 02

(c) airplane 03

(d) bag 01

(e) bag 02

(f) bag 03

(g) chair 01

(h) chair 02

(i) chair 03

(j) earphone 01

(k) earphone 02)

(l) earphone 03

(m) knife 01

(n) knife 02

(o) knife 03

(p) laptop 01

(q) laptop 02

(r) laptop 03

(s) motorbike 01

(t) motorbike 02

(u) motorbike 03

(v) mug 01

(w) mug 02

(x) mug 03

Figure 4: SRG-Net Segmentation Results of ShapeNet

##  5 Conclusion

In this paper, we provide an end-to-end model called SRG-Net on unsupervised
segmentation for terracotta warrior point cloud, which assist the
archaeologists to speed up the repairing process of terracotta warriors.
Firstly, we design a novel seed region growing methods to pre-segment the
point clouds by making full use of the point cloud coordinates and normal
information of the 3D terracotta warriors point clouds. Based on this pre-
segmentation process, we proposed a CNN which is inspired by dynamic graph in
DG-CNN and auto-encoder in FoldingNet to better learn the local and global
characteristics of 3D point clouds. And then we presented a refinement method,
and embed it after the above two processes. Finally, we evaluate our method on
the terracotta warriors data, and we outperform the state-of-the-art methods
with better segmentation results and higher efficiency. Besides, we also
conduct experiments on the ShapeNet dataset, which demonstrates our method is
also useful in human body and normal object part segmentation.

Our work still has some limitations. Since the number of points from the hand
of terracotta is quite small, it is difficult for our SRG-Net to learn the
characteristics of hands. We will try to work on this problem in the future.
We hope our work can be a useful tool for the research of terracotta warrior
in archaeology and for other researchers’ point cloud work.

##  6 Acknowledgments

This work is equally and mainly supported by the National Key Research and
Development Program of China (No. 2019YFC1521102) and the Key Research and
Development Program of Shaanxi Province (No. 2020KW-068). Besides, this work
is also partly supported by the Key Research and Development Program of
Shaanxi Province (No. 2019GY-215), the Major research and development project
of Qinghai(No. 2020-SF-143), the National Natural Science Foundation of China
under Grant (No.61701403) and the Young Talent Support Program of the Shaanxi
Association for Science and Technology under Grant (No.20190107).

## References

  * [1] A. Kanezaki, Unsupervised image segmentation by backpropagation, in: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, 2018, pp. 1543–1547.
  * [2] A. Tao, Unsupervised point cloud reconstruction for classific feature learning, https://github.com/AnTao97/UnsupervisedPointCloudReconstruction.
  * [3] Y. Yang, C. Feng, Y. Shen, D. Tian, Foldingnet: Point cloud auto-encoder via deep grid deformation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 206–215.
  * [4] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, J. M. Solomon, Dynamic graph cnn for learning on point clouds, Acm Transactions On Graphics (tog) 38 (5) (2019) 1–12.
  * [5] J. B. Kruskal, On the shortest spanning subtree of a graph and the traveling salesman problem, Proceedings of the American Mathematical society 7 (1) (1956) 48–50.
  * [6] H. Lee, P. Pham, Y. Largman, A. Ng, Unsupervised feature learning for audio classification using convolutional deep belief networks, Advances in neural information processing systems 22 (2009) 1096–1104.
  * [7] Q. V. Le, Building high-level features using large scale unsupervised learning, in: 2013 IEEE international conference on acoustics, speech and signal processing, IEEE, 2013, pp. 8595–8598.
  * [8] H. Lee, R. Grosse, R. Ranganath, A. Y. Ng, Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations, in: Proceedings of the 26th annual international conference on machine learning, 2009, pp. 609–616.
  * [9] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, S. Süsstrunk, Slic superpixels compared to state-of-the-art superpixel methods, IEEE transactions on pattern analysis and machine intelligence 34 (11) (2012) 2274–2282.
  * [10] W. Kim, A. Kanezaki, M. Tanaka, Unsupervised learning of image segmentation based on differentiable feature clustering, IEEE Transactions on Image Processing 29 (2020) 8055–8068.
  * [11] F. J. Lawin, M. Danelljan, P. Tosteberg, G. Bhat, F. S. Khan, M. Felsberg, Deep projective 3d semantic segmentation, in: International Conference on Computer Analysis of Images and Patterns, Springer, 2017, pp. 95–107.
  * [12] A. Boulch, B. Le Saux, N. Audebert, Unstructured point cloud semantic labeling using deep segmentation networks., 3DOR 2 (2017) 7.
  * [13] B. Wu, A. Wan, X. Yue, K. Keutzer, Squeezeseg: Convolutional neural nets with recurrent crf for real-time road-object segmentation from 3d lidar point cloud, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 1887–1893.
  * [14] A. Milioto, I. Vizzo, J. Behley, C. Stachniss, Rangenet++: Fast and accurate lidar semantic segmentation, in: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2019, pp. 4213–4220.
  * [15] B. Graham, M. Engelcke, L. Van Der Maaten, 3d semantic segmentation with submanifold sparse convolutional networks, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9224–9232.
  * [16] A. Dai, M. Nießner, 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 452–468.
  * [17] M. Jaritz, J. Gu, H. Su, Multi-view pointnet for 3d scene understanding, in: Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019, pp. 0–0.
  * [18] S. Wang, S. Suo, W.-C. Ma, A. Pokrovsky, R. Urtasun, Deep parametric continuous convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2589–2597.
  * [19] C. R. Qi, H. Su, K. Mo, L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652–660.
  * [20] C. R. Qi, L. Yi, H. Su, L. J. Guibas, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, in: Advances in neural information processing systems, 2017, pp. 5099–5108.
  * [21] M. Jiang, Y. Wu, T. Zhao, Z. Zhao, C. Lu, Pointsift: A sift-like network module for 3d point cloud semantic segmentation, arXiv preprint arXiv:1807.00652.
  * [22] H. Zhao, L. Jiang, C.-W. Fu, J. Jia, Pointweb: Enhancing local neighborhood features for point cloud processing, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5565–5573.
  * [23] W. Wu, Z. Qi, L. Fuxin, Pointconv: Deep convolutional networks on 3d point clouds, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 9621–9630.
  * [24] Y. Li, R. Bu, M. Sun, W. Wu, X. Di, B. Chen, Pointcnn: Convolution on $\mathcal{X}$-transformed points (2018).  arXiv:1801.07791.
  * [25] S. Wang, S. Suo, W.-C. Ma, A. Pokrovsky, R. Urtasun, Deep parametric continuous convolutional neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2589–2597.
  * [26] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, Y. Bengio, Graph attention networks, arXiv preprint arXiv:1710.10903.
  * [27] Z. Wang, F. Lu, Voxsegnet: Volumetric cnns for semantic part segmentation of 3d shapes, IEEE transactions on visualization and computer graphics.
  * [28] L. Yi, H. Su, X. Guo, L. J. Guibas, Syncspeccnn: Synchronized spectral cnn for 3d shape segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2282–2290.
  * [29] Y. Liu, Y. Xiong, Automatic segmentation of unorganized noisy point clouds based on the gaussian map, Computer-Aided Design 40 (5) (2008) 576–594.
  * [30] L. Di Angelo, P. Di Stefano, Geometric segmentation of 3d scanned surfaces, Computer-Aided Design 62 (2015) 44–56.
  * [31] P. Benkő, T. Várady, Segmentation methods for smooth point regions of conventional engineering objects, Computer-Aided Design 36 (6) (2004) 511–523.
  * [32] Z. Chen, K. Yin, M. Fisher, S. Chaudhuri, H. Zhang, Bae-net: Branched autoencoder for shape co-segmentation, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 8490–8499.
  * [33] D. OuYang, H.-Y. Feng, On the normal vector estimation for point cloud data from smooth surfaces, Computer-Aided Design 37 (10) (2005) 1071–1079.
  * [34] J. Zhou, H. Huang, B. Liu, X. Liu, Normal estimation for 3d point clouds via local plane constraint and multi-scale selection, Computer-Aided Design 129 (2020) 102916.
  * [35] Y. Wang, H.-Y. Feng, F.-É. Delorme, S. Engin, An adaptive normal estimation method for scanned point clouds with sharp features, Computer-Aided Design 45 (11) (2013) 1333–1348.
  * [36] R. B. Rusu, S. Cousins, 3d is here: Point cloud library (pcl), in: 2011 IEEE international conference on robotics and automation, IEEE, 2011, pp. 1–4.
  * [37] A. Kanezaki, Unsupervised image segmentation by backpropagation, in: 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, 2018, pp. 1543–1547.
  * [38] A. Europe, 3d object scanner artec eva — best structured-light 3d scanning device.
URL https://www.artec3d.com/portable-3d-scanners/artec-eva

  * [39] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al., Shapenet: An information-rich 3d model repository, arXiv preprint arXiv:1512.03012.