[table]capposition=top

# OccluMix: Towards De-Occlusion Virtual Try-on by Semantically-Guided Mixup

Zhijing Yang†, Junyang Chen†, Yukai Shi1, Hao Li, Tianshui Chen, Liang Lin  †
The first two authors share equal contribution. Code is available at:
https://github.com/JyChen9811/DOC-VTON

###### Abstract

Image Virtual try-on aims at replacing the cloth on a personal image with a
garment image (in-shop clothes), which has attracted increasing attention from
the multimedia and computer vision communities. Prior methods successfully
preserve the character of clothing images, however, occlusion remains a
pernicious effect for realistic virtual try-on. In this work, we first present
a comprehensive analysis of the occlusions and categorize them into two
aspects: i) Inherent-Occlusion: the ghost of the former cloth still exists in
the try-on image; ii) Acquired-Occlusion: the target cloth warps to the
unreasonable body part. Based on the in-depth analysis, we find that the
occlusions can be simulated by a novel semantically-guided mixup module, which
can generate semantic-specific occluded images that work together with the
try-on images to facilitate training a de-occlusion try-on (DOC-VTON)
framework. Specifically, DOC-VTON first conducts a sharpened semantic parsing
on the try-on person. Aided by semantics guidance and pose prior, various
complexities of texture are selectively blending with human parts in a copy-
and-paste manner. Then, the Generative Module (GM) is utilized to take charge
of synthesizing the final try-on image and learning to de-occlusion jointly.
In comparison to the state-of-the-art methods, DOC-VTON achieves better
perceptual quality by reducing occlusion effects.

###### Index Terms:

Deep Learning, Virtual Try-on, Occlusion Handing, Data Augmentation.

##  I Introduction

Virtual try-on is a popular application by transferring a desired in-shop
clothing onto a reference person. With the demand of e-business, virtual try-
on has been attracted rising attention. Although recent developments in the
try-on network have helped to perform realistic cloth warping by generating
fitness shape and realistic visual quality try-on images [1, 2, 3, 4, 5, 6,
7], it remains a big challenge to locate and resolve the occlusion effect in
the distorted try-on image.

Figure 1: On Viton [1] dataset, the try-on results of the state-of-the-art
models [2, 7] appear undesired occlusion from former and new clothes. To
address the occlusion phenomenon, we attempt to categorize the majority of
occlusion into two types (i.e., Inherent-Occlusion and Acquired-Occlusion). In
the supplementary file, we provide massive occlusion samples to verify
Inherent- and Acquired- types.

(a)

(b)

Figure 2: (a) To investigate the proportion of occlusion samples, we count
occlusion examples of PF-AFN and ACGPN results on CP-VTON+ [8] dataset aided
by human annotation. (b) The Frechet Inception Distance (FID) scores of
different body parts generated by PF-AFN [7] and ACGPN [2].

To investigate the occlusion effect in existing visual try-on methods, we
follow the pipeline of CP-VTON+ [8] to explore the results of two
representative methods [2, 7]. In our experiments, ACGPN [2] stands for the
parser-based methods and PF-AFN [7] stands for the parser-free methods. As
shown in Fig. 1, in the first and the second rows, the occlusion in the try-on
image is represented as the ghost of the previous garment. We denote this
occlusion as Inherent-Occlusion. Typically, inherent-occlusion is caused by
poor generalization and wrong human parsing. As shown in the third and the
fourth rows in Fig. 1, the occlusion effect is caused by the wrong shape
warping of new clothes. We denote it as Acquired-Occlusion. Hence, we mainly
demonstrate the occlusion problems into two forms (i.e., Inherent- and
Acquired- Occlusion). The cloth warping module in the try-on workflow is
easily misled by spatial transformation, and the occlusion effect becomes an
obvious degradation when the human pose exhibits large variance. To this end,
some pioneering synthesized-based models [1, 3, 9, 10] lock into the above
drawbacks and demonstrate limited image quality.

In Fig. 2, we conducted a human annotation to investigate the occlusion effect
in virtual try-on methods. Specifically, 4064 images [8] are labeled with the
existence as well as location of occlusion by manual effort. The statistical
results show that occlusion is still the main challenge in the try-on task
with 20% percent occurrence. We further calculate the mean Frechet Inception
Distance (FID) value of each body part in the results. The arm and clothing
parts account for a large FID score and spatial variance, which motivates us
to improve the image quality by focusing on these challenging human parts.

To tackle the aforementioned issues, we present a robust De-OCclusion
framework for Virtual Try-on (DOC-VTON), which fully applies the semantic
layout of the try-on image. DOC-VTON adaptively performs a crop-and-paste
operation [11, 12] for the generative module (GM) to implement de-occlusion.
Specifically, the DOC-VTON consists of three modules: i) Cloth Warping Module
(CWM), which warps in-shop clothing and its corresponding mask into the
fitting shape by using appearance flow; ii) Occlusion Mixup Module (OccluMix),
which simulates different occlusion cases as OccluMix samples based on the
semantic layout of Sharpened Parsing Network (SPN); iii) Generative Module
(GM) is applied to transfer the clothes on the OccluMix sample to the real
person image, enabling final try-on image generation and de-occlusion jointly.

With the above modules, DOC-VTON learns de-occlusion by adopting a
semantically-guided mixup strategy in virtual try-on. In summary, the main
contributions of our paper are as follows:

  * •

We present a comprehensive analysis of the occlusion effect of current visual
try-on algorithms. This is the first attempt to analyze this point, and it can
facilitate further research on de-occlusion virtual try-on.

  * •

We propose a simple yet non-trivial occlusion mixup strategy for virtual try-
on (OccluMix), which obtains the challenging occluded try-on person by
blending various complexity of texture with semantic and posture guidance.

  * •

Compared with the general parsing pipeline, we investigate a Sharpened Parsing
Network (SPN) to parse try-on images iteratively. SPN not only handles the
irrational warping parts surgically, but also affords regions for OccluMix.

  * •

Extensive experiments and evaluations demonstrate that our method can achieve
the best state-of-the-art results in the VITON task qualitatively and
quantitatively.

In the remaining parts of this paper, Section II briefly surveys the occlusion
phenomenon in existing virtual try-on approaches, and the derivatives of data
augmentation. Section III presents a comprehensive occlusion analysis of
state-of-the-art methods. Section IV presents DOC-VTON pipeline and detailed
explanations of OccluMix. In Section V, we discuss the comparison between
OccluMix and other data augmentation methods. In Section VI, we perform
experiments to verify the effectiveness and efficiency of DOC-VTON by
comparing it with existing state-of-the-arts. Finally, we conclude our work
with future research directions in Section VII.

Figure 3: Analysis of Inherent-Occlusion. For parser-free try-on results, the
ghost of previous garment will remain no matter try on arbitrary garments.
Compared to parser-free method, the parser-based method will generate twisted
images with failed parsing results.

##  II Related Work

Virtual Try-on. Existing deep learning-based methods on virtual try-on can be
mainly categorized as 3D model based approaches [13, 14, 15, 16] and 2D image-
based ones [1, 2, 7, 9, 17, 18, 19]. As the former methods require 3D
measurements, which bring extra computation resources. Instead, 2D image-based
approaches are more feasible to real-world scenarios. For example, VITON [1],
CP-VTON [9], ACGPN [2], ClothFlow [3], DCTON [20] and RT-VTON [21] use human
representation as the input to generate a clothed person. Besides, WUTON [10]
and PF-AFN [7] employ a parser-free approach. In garment deformation methods,
the VITON [1], CP-VTON [9], ACGPN [2], DCTON [20] and VITON-HD [22] use thin-
plate-spline (TPS) [23] transformation to warp target cloth into fitness
shape. However, TPS transformation exhibits limited deformation ability. To
this end, RT-VTON [21] proposed a semi-rigid deformation to align the warped
cloth with the predicted semantics. ClothFlow [3] and PF-AFN [7] use
appearance flow [24], which easily warps the target clothes smoothly onto the
target person.

As typical try-on pipelines, VITON and CP-VTON use rough shapes and pose maps
to ensure the generalization of arbitrary clothes. However, parser-based
methods [1, 2, 9, 25] generate poor quality try-on images when parsing results
become inaccurate. Recently, PFAFN [7] proposes a pioneering parser-free
knowledge distilling approach that gets rid of the interference from
inaccurate segmentation. Nevertheless, the generated images of PFAFN still
encounter the Inherent-Occlusion and Acquired-Occlusion. To this end, we
propose a novel De-occlusion method for virtual try-on (DOC-VTON) . DOC-VTON
handles the misalignment between target clothes and the reference person, and
reduces the ghost effect caused by previous clothes.

De-Occlusion. Removing the partial occlusion from the target object is a
crucial computer vision task [26, 27, 28, 29, 30, 31]. Sail [32] proposes a
novel self-supervised framework that tackles scene de-occlusion on real-world
data without manual annotations. GAN-based [29] methods are used to inpaint
the occluded region on the face. In the virtual try-on task, the occlusion
problem also exists by covering the clothes and human body up. To avoid the
occlusion phenomenon, the parser-based methods [1, 2, 9, 25] use a human
parser to understand the spatial layout of the body part. Nevertheless,
parser-based methods tend to generate poor quality try-on images with
noticeable occlusion. Recently, PFAFN [7] proposes a pioneering parser-free
approach, however, the image quality still suffers from unsuitable shape-
warping clothes and general model generation capabilities. To address the
above problems, we introduce the OccluMix strategy into the Virtual Try-On
task.

Data augmentation. Recently, several studies employ data augmentation (DA)
[33, 34, 35, 36, 37, 38, 39] to enhance generalization ability. Mixup [12, 40]
uses convex linear interpolation on the image level for data augmentation.
CutMix [11] proposes to cut and paste a cropped area from an input image to
other images for data augmentation. Although MixUp and CutMix demonstrate
practical improvements, they do not utilize image prior knowledge, such as
saliency, semantics and optical flow, for guidance. Random Crop [39] crops the
images into a particular dimension and creates synthetic data. Compared with
Mixup and Cutmix, it flexibly preserves the prior knowledge and solves the
distortion problem caused by different scale try-on images [38]. However, it
still remains a big challenge to solve the occlusion phenomenon (i.e.,
Inherent- and Acquired- Occlusion) on the simple scale image. Inspired by
Supermix [36], we investigate a goal-oriented data augmentation method by
using human parsing priors [41] for the try-on generation model.

Figure 4: Try-on images inherit the twisted pattern of the twisted warped
cloth. Figure 5: An illustration of our proposed method. OccluMix generates an
augmented image by crop-and-pasting different clothing textures onto the
challenging region of the try-on image. To simulate a challenging occlusion
emergence on the try-on image, we need to identify the human component of the
try-on image. We first obtain parser-based input image $X_{A}$, then use the
Cloth Warping Module (CWM) to predict the warping cloth $T_{C}^{W}$ and
warping clothing mask $M_{C}^{W}$; In Step I, the Sharpened Parsing Generator
($G$) first multiplies the Clothing Mask $M_{C}$ and $\tilde{M}_{C}^{W}$ to
get the strange area $M_{e}$, then combine it with the body parts $M_{w}$ to
produce the potential location of body parts $M_{w}^{p}$ (including head,
arms, and pants). Then, $G$ generates the rough mask of body parts $M_{w}^{g}$
by using Pose Map $M_{p}$ and the potential location of body parts
$M_{w}^{p}$; In step II, the Sharpened Parsing Restorer ($R$) refines the
rough mask of torso $M_{t}^{o}$ to get the complete torso mask $M_{t}^{c}$. In
step III, we use DensePose [42] to select a challenging segment $Y_{A}^{r}$ to
multiply with an auxiliary cloth $X_{B}$ to obtain the texture occlusion
$C_{B}$. And we mix $C_{B}$ with the try-on image $X_{A}^{s}$ to get the
OccluMix sample $\tilde{X}$. Finally, we exploit a generative module to
generate the try-on images $\hat{X}_{A}$ by utilizing the warped cloth
information $\hat{I}_{c}$,$\hat{M}_{c}$ and $\tilde{X}$.

##  III Occlusion Analysis

Existing state-of-the-art methods on 2D virtual try-on can be classified as a
parser-based approach and a parser-free approach. However, both of them still
remain the occlusion phenomenon in the try-on images. To investigate the
occlusion effects in both approaches, we follow the pipeline of CP-VTON+ to
explore the occlusion phenomenon on ACGPN and PF-AFN. As the former represents
the parser-based approach, and PF-AFN represents the parser-free approach. We
first explore the occlusion effect caused by twisted warped cloth. As shown in
Fig. 4, if the warped results remain twisted pattern, the generated results
will remain in this pattern, and express it in the form of occlusion. We
denoted this occlusion as acquired-occlusion, which was caused by the twisted
warped results. It seems that the generator will generate clean try-on images
with proper warping results, however, the occlusion effect also exists in the
generative stage. As shown in Fig. 3, try-on results will remain the ghost of
the previous garment. For a parser-free generator, we find that the ghost will
remain no matter the try-on arbitrary garments. Since the parser-free
generator needs to classify the try-on region of the reference images, we
assume that the ghost is caused by poor generalization ability. In comparison
with the parser-free generator, the parser-based generator will generate
twisted images with failed parsing results. We denote this occlusion effect
caused by previous garments as inherent-occlusion. Hence, we mainly
demonstrate the occlusion problems in two forms. (_i.e._ , Inherent- and
Acquired- Occlusion).

Figure 6: (a) Generator is fed with Pose Map $M_{p}$ and the potential
location $M_{w}^{p}$ to generate the coarse mask of body. (b) Restorer is
trained to refine the body mask. At this stage, we divide two training cases.
Case 1 is encouraged to partially complete the mask of body parts. Case 2
prevents Restorer from over-completing.

##  IV Methodology

The proposed DOC-VTON is composed of three modules, as shown in Fig. 5. First,
the Clothes Warping Module is designed to warp the target clothing image onto
a real image. Second, the OccluMix module progressively generates the mask of
body parts of the try-on image via semantic information, yielding occlusion
effect by utilizing crop-and-paste strategy on try-on image to generate
OccluMix samples. Finally, the generative module synthesizes the try-on image.

###  IV-A Cloth Warping Module (CWM)

Following the training pipeline of PF-AFN [7], we use the second-order
constraint to better preserve the cloth characteristics, and the constraint is
defined as follows:

$L_{sec}=\sum^{N}_{i=1}\sum_{p}\sum_{\pi\in
N_{p}}Char(f^{p-\pi}_{i}+f^{p+\pi}_{i}-2f^{p}_{i}),\vspace{-3mm}$ (1)

where $f^{p}_{i}$ denotes the p-th point on the flow maps of i-th scale.
$Char$ is the generalized charbonnier loss function [43]. $N_{p}$ consists of
the set of vertical, horizontal, and both diagonal neighborhoods around the
p-th point.

CWM warps the target cloth into a fit shape, which also maintains the details
of the cloth. It performs well when the reference person stands simply. When a
reference person stands in a complex posture, such as both torso twisting and
two hands blocking in front of the body, the warping cloth may cover part of
the human body.

###  IV-B OccluMix

####  IV-B1 Sharpened Parsing Network (SPN)

The sharpened parsing network (SPN) is proposed to refine the warping clothes
as well as to generate the body parts (e.g., arms) of the person. Many
previous works neglect the fact that accurate parsing results can correct the
unreasonable warping process. To address this issue, a sharpened parsing
mechanism is adopted to refine the detail distortion of human parsing during
try-on transformation.

Specifically, suppose a person tries on new clothes, we define the masks of
clothing and body parts as $M_{c}$ and $M_{w}$ (including head, arms and
pants) in the original image, $M_{c}^{s}$ and $M_{w}^{s}$ are the masks of
clothing and body parts in the try-on image, and $M_{p}$ is the skeleton
information of the reference person. Since $M_{w}^{s}$ is absent, we use cloth
and torso prior with warping cloth to obtain it. The formulation of
$M_{w}^{s}$ is defined as follows:

$M_{w}^{s}=M_{w}\odot(1-M_{c}^{s}).$ (2)

where $\odot$ indicates element-wise multiplication. And $M_{e}$ is the mask
of strange fabric between $M_{c}^{s}$ and $M_{c}$, which indicates the
potential location for generating torso region. The formulation of $M_{e}$ is:

$M_{e}=M_{c}^{s}\odot(1-M_{c}).$ (3)

Then, we get the potential location of body parts $M_{w}^{p}$ in try-on image
by combining $M_{e}$ and $M_{w}^{s}$ as follows:

$M_{w}^{p}=M_{w}^{s}+M_{e}.$ (4)

As shown in Fig. 6 (a), since $M_{w}^{p}$ and $M_{w}$ are obtained, we can use
the process of $(M_{p},M_{w}^{p})$ ${\xrightarrow{G}}M_{w}$ to realize $G$ to
generate the rest masks of torso from the potential location $M_{w}^{p}$ under
the supervision of the label $M_{w}$.

However, when the reference people stand in a twisted posture, the pernicious
distortion of the target cloth may destroy the body details in $M_{w}$. To
address this problem, we introduce $M_{d}$ from the Irregular Mask Dataset
[44] and merge it with $M_{c}$ to simulate the failure case $M_{t}^{o}$, where
some details of the body are lost. The formulation is defined as follows:

${M_{t}^{o}}=M_{w}\odot(1-M_{d})+M_{c}\cup M_{d}.$ (5)

We then feed it into restorer $R$, and $(M_{t}^{o},M_{p})$
${\xrightarrow{R}}M_{w}$ is the process to refine the details of body parts.
Note that training details of the restorer are presented in Fig. 6 (b).

As shown in Fig. 7, the detail distortion of the body parts in $M_{o}$ is
enhanced by Restorer. After refining the parsing mask of the try-on image, we
can use it to mine the regions to crop-and-paste the texture occlusion, and
model the OccluMix data.

Figure 7: Effect of semantic restoration component. When the reference people
stands in a twisted posture, Sharpened Parsing Generator provides wrong
semantic information on try-on images, the restoration step can fixed this
question.

####  IV-B2 Occlusion Mixup

In this section, we describe OccluMix, a data augmentation (DA) strategy that
is designed for the try-on task. A practical DA method for try-on needs to
simulate the challenging occlusion and serve as a good regularizer for the
try-on model. Literally, OccluMix overlays different clothes on the try-on
images, enforcing the network to restore realistic details during the try-on
transformation.

Figure 8: The left clothes are complicated by involving dots, stripes, and
various other textures. And the right clothes are more simple with less
texture.

In the top right of Fig. 1, the personal image will retain the ghost of the
complex texture when a person wearing complex clothes wants to put on new
cloth. To this end, we ensure a certain percentage of complex textures
existing in OccluMix. In our experiment, we use Gray-Level Co-occurrence
Matrix [45] to estimate the entropy of the clothing complexity. Besides, we
divide the clothes into two categories (i.e., simple and complex) based on
their texture complexities:

$ENT=-\sum^{k}_{i=1}\sum^{k}_{j=1}G(i,j)logG(i,j),$ (6)

where $G(i,j)$ represents the normalized occurrence of different gray scale
values.

As shown in Fig. 8, the left clothes are categorized into complex textures,
and the right are simple textures. Then we use texture categorization to
divide the training pairs of the reference person.

$T=\left\\{\begin{array}[]{l}1,~{}ENT>=2.5,\\\
0,~{}ENT<2.5,\end{array}\right.$ (7)

where $T=1$ indicates that the clothes belong to the complex texture category
and the other to the simple texture category.

Let $x\in\mathbb{R}^{W\times H\times C}$ and $y$ denote a random sampled image
and corresponding label, respectively. Example $A$ (i.e., $(x_{A},y_{A})$) is
the training sample. And example $B$ (i.e., $(x_{B},y_{B})$) is chosen from
the complex texture set or the simple texture set by complex coefficient
$\lambda$.

As shown in Fig. 9, the goal of try-on mixup is to generate a new training
sample $\tilde{x}$ by combining two training samples $(x_{A},y_{A})$ and
$(x_{B},y_{B})$. The formulations are as follows:

$\begin{split}&C=y_{A}^{r}\odot y^{c}_{B},\\\ \vspace{3mm}\tilde{x}=C&\odot
x_{B}+(1-C)\odot x_{A},\end{split}$ (8)

where $y_{A}^{r}$ is the selected region for occlusion mixing. We first count
the occlusion distribution of each human part in try-on images. DensePose [42]
is then used to forward the results of Sharpened Parsing Network, and select
$y_{A}^{r}$ out w.r.t the occlusion distribution. Besides, $y_{B}^{c}$ is the
clothing mask of the person $x_{B}$ and $C$ is the mask of the texture
occlusion. As shown in Fig. 5, we mix the cropped clothes into the input data
to generate the augmented images. Finally, we feed the augmented images into
the generator to obtain the try-on images with clothes on the real images.

Figure 9: Process of Occlusion Mixup. We attempt to model occlusion in try-on
images, however, it is hard to enforce texture occlusion to be gathered in a
typical region to model real occlusion. To tackle this challenge, we utilize
Sharpened Parsing Network (SPN) in OccluMix.

###  IV-C Generative Module

In this module, we adopt Res-UNet [3] as the backbone architecture of the
generative module (GM). It can not only retain the characteristics of warped
clothes, but also keep the details of human body parts.

In the training phase, the parameters of GM are optimized by minimizing
$L_{g}$, as follows:

$L_{g}=\alpha_{l}L_{euc}+\alpha_{p}L_{per},$ (9)

where $L_{euc}$ is the pixel-wise L1 loss and $L_{per}$ is the perceptual loss
[46] to encourage the improvement of the try-on image visual quality. The
formulations are as below:

$L_{euc}=\lVert I^{G}-I\rVert_{1},$ (10)
$L_{per}=\sum_{m}\lVert\phi_{m}(I^{G})-\phi_{m}(I)\rVert_{1},$ (11)

where $I^{G}$ and $I$ are the generated and real image, respectively. And
$\phi_{m}$ indicates the $m$-th feature map in a VGG-19 [47] network pre-
trained on ImageNet [48].

##  V Discussion

###  V-A Differences from Mixup and its derivatives.

Mixup and its derivatives mix image contents within a random image patch,
which does not consider both the spatial as well as semantic information of
the image patch carefully. From Table I, we find that these random patch can
degrade the performance on FID.

In comparison with the traditional Mixup pipelines, OccluMix performs cut-and-
paste between fake images and other reference images under semantic guidance,
which breaks the defect of random patch mix.

Figure 10: Qualitative effects of OccluMix during virtual try-on inference.
(a) OccluMix successfully generates the clean try-on images while the baseline
generates the try-on images with the occlusion from previous clothes. (b)
$\triangle$ is the absolute residual intensity map between the try-on image
and the ground truth. From the comparison on $\triangle$, it can be
demonstrated that the OccluMix resolves the ghost of clothing texture. In both
(a) and (b), OccluMix generates the try-on images with better visual
coherence. Zooming up for better view.

###  V-B What does the model learn with Occlusion Mixup?

TABLE I: FID comparison with mixup and its derivatives. We report the baseline model (PF-AFN [7]) performance that is trained on the VITON[1] dataset. $\delta$ denotes the performance gap between with and without augmentation. Method | FID $\downarrow$ ($\delta$)
---|---
PF-AFN | 10.09 (+0.00)
Cutout [49] | 10.36 (-0.27)
CutMix [11] | 10.31 (-0.22)
Mixup [12] | 10.17 (-0.08)
CutBlur [35] | 9.94 (+0.15)
OccluMix | 9.66 (+0.43)

Similar to the other DA methods that prevent the models from making a
prediction over-confidently, OccluMix prevents the model from having no
distinction between simple images or distorted images and helps it to
alleviate the occlusion effect. This can be demonstrated in Fig. 10. Note that
the superiority of OccluMix is not only reflected in the visual coherence of
the try-on images, but also in the decreasing of the residual intensity map.
We hypothesize that this enhancement is due to the balanced distribution of
fake images. Now the model has learned to distinguish between simple and
distorted data, and this leads the model to learn ”where” and ”how” it should
address the occlusion effects.

##  VI Experiments

In this section, we first compare the DOC-VTON with state-of-the-art methods.
Then, a detailed ablation study is made to analyze each component. Finally, we
provide massive occlusion samples to verify Inherent- and Acquired-Occlusion
and further demonstrate the negative effects of occlusion on the human visual
system. Besides, to further validate the performance of the proposed DOC-VTON,
we perform the visual comparison of the occlusion images (Generated by PF-AFN)
and clean images (Generated by our DOC-VTON).

###  VI-A Dataset

Experiments are conducted on the VITON dataset [1] that used in CP-VTON [9],
CP-VTON+ [8], ACGPN [2] and PF-AFN [7]. VITON contains a training set of 14221
image pairs and a testing set of 2032 image pairs, each of which has a target
clothing image and woman photo with the resolution of 256 $\times$ 192\.
VITON-HD [22] consists of 11,647 groups for training set and 2032 groups for
testing set, each image with the resolution of 1024 $\times$ 768\. All of our
evaluations and visualizations are performed on the testing set.

###  VI-B Implementation Details

####  VI-B1 Architecture

DOC-VTON contains CWM, OccluMix and GM. The structure of CWM consists of a
dual pyramid feature extraction network (PFEN) [50] and a progressive
appearance flow estimation network (AFEN) [7]. The generators of Sharpened
Parsing Network in OccluMix have the same structure of U-Net [51]. And the
structure of GM is Res-UNet [3]. In our experiments, the resolution of images
is 256 $\times$ 192 for VITON, and 1024 $\times$ 768 for VITON-HD dataset.

Figure 11: Visual comparison of VITON dataset. Compared with four state-of-
the-art try-on methods [1, 2, 7, 9], our model generates more realistic try-on
images. With the proposed De-occlusion strategy, our approach not only
processes the irrational parts of the warping clothes, but also clearly avoids
the ghost of former clothes. Figure 12: Extensive visual comparisons on VITON-
HD dataset. From diverse perspectives, our approach generates high-quality
images.

####  VI-B2 Training

We train Sharpened Parsing Generator ($G$) and Restorer ($R$) with 200 epochs.
Due to the large spatial transformation between arms and clothes, we use a
hard-example mining strategy. In the first 100 training epochs, we selectively
train our model with challenging body parts (e.g., arms). In the last 100
epochs, we train our model on the whole body parts (including head, arms and
pants) together. Then, we use OccluMix to generate samples for GM. All of them
use the initial learning rate as 0.0005 and the network is optimized by Adam
optimizer with the hyper-parameter $\beta_{1}$ = 0.5, and $\beta_{2}$ = 0.999.

####  VI-B3 Testing

The testing process follows the same procedure as training. The reference
person images, target clothes, human parsing results, and human pose
estimations are given as input of DOC-VTON to generate the try-on image.

###  VI-C Qualitative Results

To further validate the performance of the proposed DOC-VTON framework, we
perform the visual comparison of our proposed method with CP-VTON [9],
ClothFlow [3], ACGPN [2] and PF-AFN [7].

####  VI-C1 Results on VITON

As shown in the first and the second rows of Fig. 11, when a person strikes a
complex posture, such as standing with one arm raised around the face, the
occlusion of target cloth occurs on the arms. In such cases, baseline models
all fail to handle the warping process, leading to distorted arm images or
broken sleeves. The warping methods of the baseline model fail to restore the
severe non-rigid deformation, due to the limited degrees of freedom in TPS
[23] or misalignment in the Appearance Flow [3].

In the last row, images generated by baseline methods contain the obvious
artifacts, as CP-VTON [9], ClothFlow [3] and ACGPN [2] own cluttered texture,
boundary-blurring and color mixture problems. To this end, they are vulnerable
to segmentation errors as they heavily rely on parsing results to drive image
generation. Although PF-AFN [7] drives image generation without using parsing
results, it has not performed better on generating the fake body parts.
Furthermore, when a huge displacement exists between the target clothes and
the original clothes (e.g., the person wears long sleeve clothes while the
target clothes are short sleeves), PF-AFN [7] fails to generate arms at the
cuffs of long sleeves since the model does not understand the shape of the
clothes thoroughly. To this end, state-of-the-art models are less robust to
handle some special cases (e.g., cuffs of long sleeves, complex clothes
texture).

In comparison, the proposed DOC-VTON performs a realistic virtual try-on,
which simultaneously handles the warping process to avoid the pernicious
occlusion, and preserves the details of both target clothes and human body
parts. Benefiting from the OccluMix, our model is robust to generate non-label
body parts (e.g., arms, hands, and fingers) from the original cloth. All
qualitative results of DOC-VTON clearly verify the superiority against CP-
VTON, ClothFlow, ACGPN and PF-AFN.

####  VI-C2 Results on VITON-HD

To verify the generalization of our method, we visualize the results on VITON-
HD [22] dataset. Since the artifacts in higher resolution are more obvious to
be observed, it is more challenging to generate highly-realistic try-on
results. As shown in Fig. 12, the results demonstrate that our DOC-VTON is
effective to generate high-quality images on VITON-HD dataset.

###  VI-D Quantitative Evaluation

For virtual try-on, the try-on image is generated by a target cloth and a
reference person image. Since the ground truth of the try-on image is absent,
we can not use point-to-point indicators (e.g., SSIM, PSNR and LPIPS) that
require computation with labels. In this paper, we adopt the Frechet Inception
Distance (FID) [52] to measure the diversity of the try-on images. The lower
score of FID indicates a higher quality of the results. Besides, the Inception
Score (IS) [53] will not be used, owing to Rosca et.al [54] have pointed out
that applying the IS to the models trained on datasets other than ImageNet
will give misleading results.

TABLE II: The FID score among different methods on the VITON dataset. Method | Region of Arm | Warped Clothes | Try-on Results
---|---|---|---
FID $\downarrow$
CP-VTON | 24.42 | 33.17 | 24.43
CP-VTON+ | 22.36 | 30.21 | 21.08
ACGPN | 16.62 | 24.95 | 15.67
ClothFlow | 18.42 | 22.50 | 14.43
DCTON | 13.27 | 30.18 | 14.82
RT-VTON | 12.70 | 20.59 | 11.66
PF-AFN | 12.86 | 18.67 | 10.09
Ours | 11.14 | 18.18 | 9.54

Table II lists the FID score of the try-on results for CP-VTON [9], CP-VTON+
[8], ClothFlow [3], ACGPN [2], DCTON [20], RT-VTON [21], PF-AFN [7] and our
DOC-VTON on the VITON dataset. Our proposed DOC-VTON outperforms other
methods, which indicates that DOC-VTON can improve the perceptual quality of
try-on images, handle large misalignment between clothes and person, and
synthesize realistic try-on results.

Figure 13: We make an in-depth quality assessment on user study by recruiting
25 volunteers to complete two tasks. The left image is the quality ranking of
each method. The right image is the comparison between DOC-VTON and other
methods by a A/B test. In the right image, the label A denotes the percentage
where our DOC-VTON is considered better over the compared method, and the
label b denotes the percentage where the compared method is considered better
over our DOC-VTON.

###  VI-E Human Perception Study

The subtle changes (e.g. removing partial occlusion or ghost) on try-on images
play an important role in human visual coherence. Because FID is insensitive
to subtle changes on synthetic images, which can not demonstrate the
effectiveness of our method. We further conduct two user study by recruiting
25 volunteers. For the first task, our goal is to verify the superiority of
the individual methods. Specifically, CP-VTON [9], ClothFlow [3], ACGPN [2],
PF-AFN [7], and DOC-VTON generate the try-on images from 300 specified
reference images respectively. Each volunteer is asked to rank the methods in
each group images. The left image of Fig. 13 demonstrate a clear advantage.
Our method ranks the first 61.83 $\%$. The second task is to compare our
method with other methods one by one. We divide the previous synthetic images
into five a / b (i.e., a is our methods and b is other methods) groups. Each
volunteer is asked to choose the one with better visual quality. As shown in
the right image of Fig. 13, our DOC-VTON is always rated better than the other
methods.

As shown in Fig. 13, our DOC-VTON achieves the highest voter turnout in both
tasks. It verifies the great superiority of DOC-VTON over the other methods.
The human perception study demonstrates the effectiveness of the proposed
method in removing pernicious occlusions and enjoying the human visual system
in the try-on task.

###  VI-F Ablations

In this section, we analyze each component in OccluMix which brings to the
robustness of the model.

Figure 14: Ablation study of complex coefficient $\lambda$.

Analysis of Complex Coefficient $\lambda$.

As discussed above, we introduce the OccluMix scheme in the training stage. To
validate the effectiveness of OccluMix, we design a baseline method (i.e.,
‘w/o OccluMix’) that is finetuned on the original model. As shown in Fig. 14,
‘w/o OccluMix’ meets a 0.08 FID drop when removing OccluMix scheme. To analyze
the OccluMix scheme continuously, we conduct experiments by setting $\lambda$
from 0 to 1, with intervals of 0.1. The continuous sampling of $\lambda$
indicates the mixup paradigm from simple to complex. As shown in Fig. 14, we
find that the model shows the trivial result when the simplest mixup pattern
is used ($\lambda=0.0$ indicates the DOC-VTON only use simple textures for
OccluMix). We also conduct a balance distribution with $\lambda=0.5$, which
indicates two well-proportioned texture complexity are used for OccluMix. As
shown in Fig. 14, the balance distribution shows the best performance than
others. Since the unbalanced mixup distribution lead to the performance
suffering from an obvious drop, we adopt a balanced complex coefficient in
OccluMix.

TABLE III: Ablation studies of the DOC-VTON. Lower FID indicates better results. Method |  FID $\downarrow$
---|---
w/o SPN | 9.80
w/ SPN | 9.68
w/o OccluMix | 9.80
w/ OccluMix | 9.66
Ours | 9.54

Ablations of OccluMix.

We show the ablation studies on the effects of using the OccluMix scheme in
the training stage. Meanwhile, the ‘w/o OccluMix’ owns the same network
architecture as ‘w/ OccluMix’, other than the training process abandons the
OccluMix strategy and SPN. As shown in Fig. 10, we notice that the try-on
image generated by the model trained with the OccluMix scheme outperforms the
plain model. With the OccluMix, the model enables to learn where to generate
realistic body parts. As depicted in Table III, ‘w/ OccluMix’ achieves a 0.14
FID gains compared with ‘w/o OccluMix’.

Sharpened Parsing Network.

We show the ablation studies on the effects of the ‘Sharpened Parsing Network
(SPN)’. Since the result of SPN includes semantic information about the body
parts of a try-on image, we can apply them to guide the warping process when
the target clothes produce unreasonable distortion on the human body (e.g.,
arms, hands). As shown in Fig. 15, with the guidance of the SPN, the warping
clothes overcome the complex spatial interactions, and the distortion between
the rough warping clothes and the body part. Otherwise, the rough shape of
warping clothes will cause body distortion. As depicted in Table III, ‘w/ SPN’
achieves a 0.12 FID gains compared with ‘w/o SPN’.

Figure 15: Ablation studies on the effects of the Sharpened Parsing Network.

##  VII Conclusion and limitation

In this paper, we have introduced Occlusion Mixup (OccluMix), a new DA method
and strategy for training a stronger try-on model. We proposed a novel De-
Occlusion approach by a data augmentation manner, which enables our model to
remove the partial occlusion to produce realistic try-on images. Extensive
evaluations clearly verify the obvious superiority of DOC-VTON over the state-
of-the-art methods with less occlusion effect.

Though DOC-VTON addresses the occlusion problem in specific try-on datasets,
it still shows limited performance on out-of-distribution (OOD) images. Thus,
DOC-VTON may suffer restrictions of light, posture, and background conditions.
In the following work, we will continue to apply unsupervised 2D to 3D
transformation into DOC-VTON to develop an ODD virtual try-on framework. In
addition, since our method can be used not only to remove occlusions, but also
to expose body parts, we have declared BSD license in the open source code to
avoid this potential social implications.

## References

  * [1] X. Han, Z. Wu, Z. Wu, R. Yu, and L. S. Davis, “Viton: An image-based virtual try-on network,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 7543–7552.
  * [2] H. Yang, R. Zhang, X. Guo, W. Liu, W. Zuo, and P. Luo, “Towards photo-realistic virtual try-on by adaptively generating-preserving image content,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 7850–7859.
  * [3] X. Han, X. Hu, W. Huang, and M. R. Scott, “Clothflow: A flow-based model for clothed person generation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 10 471–10 480.
  * [4] S. Choi, S. Park, M. Lee, and J. Choo, “Viton-hd: High-resolution virtual try-on via misalignment-aware normalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 14 131–14 140.
  * [5] F. Yang and G. Lin, “Ct-net: Complementary transfering network for garment transfer with arbitrary geometric changes,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 9899–9908.
  * [6] C. Ge, Y. Song, Y. Ge, H. Yang, W. Liu, and P. Luo, “Disentangled cycle consistency for highly-realistic virtual try-on,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 16 928–16 937.
  * [7] Y. Ge, Y. Song, R. Zhang, C. Ge, W. Liu, and P. Luo, “Parser-free virtual try-on via distilling appearance flows,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 8485–8493.
  * [8] M. R. Minar, T. T. Tuan, H. Ahn, P. Rosin, and Y.-K. Lai, “Cp-vton+: Clothing shape and texture preserving image-based virtual try-on,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_ , 2020.
  * [9] B. Wang, H. Zheng, X. Liang, Y. Chen, and L. Lin, “Toward characteristic-preserving image-based virtual try-on network,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 589–604.
  * [10] T. Issenhuth, J. Mary, and C. Calauzenes, “Do not mask what you do not need to mask: a parser-free virtual try-on,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020, pp. 619–635.
  * [11] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 6023–6032.
  * [12] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” _International Conference on Learning Representations_ , 2017.
  * [13] I. Santesteban, M. A. Otaduy, and D. Casas, “Learning-based animation of clothing for virtual try-on,” _Computer Graphics Forum_ , vol. 38, no. 2, pp. 355–366, 2019.
  * [14] D. Rohmer, T. Popa, M.-P. Cani, S. Hahmann, and A. Sheffer, “Animation wrinkling: augmenting coarse cloth simulations with realistic-looking wrinkles,” _ACM Transactions on Graphics (TOG)_ , vol. 29, no. 6, pp. 1–8, 2010.
  * [15] I. Santesteban, M. A. Otaduy, and D. Casas, “Learning-based animation of clothing for virtual try-on,” _Computer Graphics Forum_ , vol. 38, no. 2, pp. 355–366, 2019.
  * [16] R. Brouet, A. Sheffer, L. Boissieux, and M.-P. Cani, “Design preserving garment transfer,” _ACM Transactions on Graphics (TOG)_ , vol. 31, no. 4, pp. 1–11, 2012.
  * [17] H. Dong, X. Liang, X. Shen, B. Wang, H. Lai, J. Zhu, Z. Hu, and J. Yin, “Towards multi-pose guided virtual try-on network,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 9026–9035.
  * [18] B. Hu, P. Liu, Z. Zheng, and M. Ren, “Spg-vton: Semantic prediction guidance for multi-pose virtual try-on,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 1233–1246, 2022.
  * [19] C. Du, F. Yu, M. Jiang, A. Hua, X. Wei, T. Peng, and X. Hu, “Vton-scfa: A virtual try-on network based on the semantic constraints and flow alignment,” _IEEE Transactions on Multimedia_ , 2022.
  * [20] C. Ge, Y. Song, Y. Ge, H. Yang, W. Liu, and P. Luo, “Disentangled cycle consistency for highly-realistic virtual try-on,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 16 928–16 937.
  * [21] H. Yang, X. Yu, and Z. Liu, “Full-range virtual try-on with recurrent tri-level transform,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 3460–3469.
  * [22] S. Choi, S. Park, M. Lee, and J. Choo, “Viton-hd: High-resolution virtual try-on via misalignment-aware normalization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 14 131–14 140.
  * [23] J. Duchon, “Splines minimizing rotation-invariant semi-norms in sobolev spaces,” in _Constructive theory of functions of several variables_. Springer, 1977, pp. 85–100.
  * [24] T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros, “View synthesis by appearance flow,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2016, pp. 286–301.
  * [25] R. Yu, X. Wang, and X. Xie, “Vtnfp: An image-based virtual try-on network with body and clothing feature preservation,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2019, pp. 10 511–10 520.
  * [26] X. Zhan, X. Pan, B. Dai, Z. Liu, D. Lin, and C. C. Loy, “Self-supervised scene de-occlusion,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 3784–3792.
  * [27] J. Hur and S. Roth, “Iterative residual refinement for joint optical flow and occlusion estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 5754–5763.
  * [28] L. Qi, L. Jiang, S. Liu, X. Shen, and J. Jia, “Amodal instance segmentation with kins dataset,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3014–3023.
  * [29] N. Zhang, N. Liu, J. Han, K. Wan, and L. Shao, “Face de-occlusion with deep cascade guidance learning,” _IEEE Transactions on Multimedia_ , 2022.
  * [30] X. Dong, J. Shen, D. Yu, W. Wang, J. Liu, and H. Huang, “Occlusion-aware real-time object tracking,” _IEEE Transactions on Multimedia_ , vol. 19, no. 4, pp. 763–771, 2016.
  * [31] F. Angelini, Z. Fu, Y. Long, L. Shao, and S. M. Naqvi, “2d pose-based real-time human action recognition with occlusion-handling,” _IEEE Transactions on Multimedia_ , vol. 22, no. 6, pp. 1433–1446, 2020.
  * [32] Y.-T. Hu, H.-S. Chen, K. Hui, J.-B. Huang, and A. G. Schwing, “Sail-vos: Semantic amodal instance level video object segmentation-a synthetic dataset and baselines,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 3105–3115.
  * [33] H. Guo, Y. Mao, and R. Zhang, “Mixup as locally linear out-of-manifold regularization,” in _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI)_ , 2019, pp. 3714–3722.
  * [34] M. Hong, J. Choi, and G. Kim, “Stylemix: Separating content and style for enhanced data augmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 14 862–14 870.
  * [35] J. Yoo, N. Ahn, and K.-A. Sohn, “Rethinking data augmentation for image super-resolution: A comprehensive analysis and a new strategy,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020, pp. 8375–8384.
  * [36] A. Dabouei, S. Soleymani, F. Taherkhani, and N. M. Nasrabadi, “Supermix: Supervising the mixing data augmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 13 794–13 803.
  * [37] Q. Wang, W. Min, Q. Han, Q. Liu, C. Zha, H. Zhao, and Z. Wei, “Inter-domain adaptation label for data augmentation in vehicle re-identification,” _IEEE Transactions on Multimedia_ , vol. 24, pp. 1031–1041, 2022.
  * [38] T. Kang, S. Park, S. Choi, and J. Choo, “Data augmentation using random image cropping for high-resolution virtual try-on (viton-crop),” _arXiv preprint arXiv:2111.08270_ , 2021.
  * [39] R. Takahashi, T. Matsubara, and K. Uehara, “Data augmentation using random image cropping and patching for deep cnns,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 30, no. 9, pp. 2917–2931, 2019\.
  * [40] Y. Tokozume, Y. Ushiku, and T. Harada, “Between-class learning for image classification,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 5486–5494.
  * [41] P. Li, Y. Xu, Y. Wei, and Y. Yang, “Self-correction for human parsing,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [42] R. A. Güler, N. Neverova, and I. Kokkinos, “Densepose: Dense human pose estimation in the wild,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 7297–7306.
  * [43] D. Sun, S. Roth, and M. J. Black, “A quantitative analysis of current practices in optical flow estimation and the principles behind them,” _International Journal of Computer Vision_ , vol. 106, no. 2, pp. 115–137, 2014.
  * [44] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 85–100.
  * [45] M. Partio, B. Cramariuc, M. Gabbouj, and A. Visa, “Rock texture retrieval using gray level co-occurrence matrix,” in _Proc. of 5th Nordic Signal Processing Symposium_ , vol. 75, 2002.
  * [46] J. Johnson, A. Alahi, and L. Fei-Fei, “Perceptual losses for real-time style transfer and super-resolution,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2016, pp. 694–711.
  * [47] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [48] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , vol. 25, 2012.
  * [49] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [50] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, “Feature pyramid networks for object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2117–2125.
  * [51] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention (MICCAI)_ , 2015, pp. 234–241.
  * [52] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “Gans trained by a two time-scale update rule converge to a local nash equilibrium,” in _Advances in neural information processing systems_ , vol. 30, 2017.
  * [53] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, “Improved techniques for training gans,” in _Advances in neural information processing systems_ , vol. 29, 2016.
  * [54] M. Rosca, B. Lakshminarayanan, D. Warde-Farley, and S. Mohamed, “Variational approaches for auto-encoding generative adversarial networks,” _arXiv preprint arXiv:1706.04987_ , 2017.


# QuadricsNet: Learning Concise Representation for Geometric Primitives in
Point Clouds

Ji Wu∗ Huai Yu∗ Wen Yang Gui-Song Xia ∗ These authors contribute equally to
this work.Ji Wu and Gui-Song Xia are with the School of Computer Science, Huai
Yu and Wen Yang are with the School of Electronic Information. All authors are
with Wuhan University, Wuhan, China 430072. {ji.wu,yuhuai,
<EMAIL_ADDRESS>
###### Abstract

This paper presents a novel framework to learn a concise geometric primitive
representation for 3D point clouds. Different from representing each type of
primitive individually, we focus on the challenging problem of how to achieve
a concise and uniform representation robustly. We employ quadrics to represent
diverse primitives with only 10 parameters and propose the first end-to-end
learning-based framework, namely QuadricsNet, to parse quadrics in point
clouds. The relationships between quadrics mathematical formulation and
geometric attributes, including the type, scale and pose, are insightfully
integrated for effective supervision of QuaidricsNet. Besides, a novel
pattern-comprehensive dataset with quadrics segments and objects is collected
for training and evaluation. Experiments demonstrate the effectiveness of our
concise representation and the robustness of QuadricsNet. Our code is
available at https://github.com/MichaelWu99-lab/QuadricsNet.

##  I Introduction

Geometric primitive representation and parsing are fundamental problems for
compact 3D object representation and scene modeling, providing crucial
features for structured mapping [1, 2, 3], CAD reverse engineering [4, 5], and
SLAM optimization [6, 7, 8]. Unlike non-vectorized, sparse and large point
clouds, geometric primitives can model the geometric information in a compact
manner with vectorized representation. Several geometric primitives have
proven to be successful in various fields during the last decades, such as
planes [9], ellipsoids [10], B-splines [11] and cuboids [8]. However, unified
representation and modeling of these primitives remain a challenging problem.
Moreover, although deep learning has achieved great success in 2D and 3D
object detection and classification, end-to-end unified mathematical modeling
and geometric perception are pressing issues that need to be explored when
parsing geometric primitives.

Real-world objects and scenes are generally composed of multiple geometric
elements. For example, a cup typically consists of a cylinder and two planes,
while streets usually include planes, cylinders, and cones. As shown in Fig.
1, most methods employ one or several geometric primitives for simplicity [12,
11, 13, 14]. Eiffient-RANSAC [15] integrated in CGAL is a standard analytical
algorithm. However, the configurations must be carefully tuned to accommodate
different types of primitives. Learning-based techniques can obtain more
robust configurations from massive training data [12]. Nevertheless, each
primitive is still represented individually, which requires the design of
different learning models for different primitives. Therefore, if we can unify
the representation of different primitives, it will significantly simplify the
design complexity and parameter settings of parsing methods, thereby improving
the robustness and generalization of primitive parsing.

For the unified representation and modeling, we find that quadrics in 3D space
can concisely represent 17 types of geometric primitives with only 10
parameters, which has been introduced into 3D vision for abstract mapping [16]
and SLAM [7]. However, existing methods often use non-learning geometric
clustering and fitting strategies, which require high-fidelity point clouds
and careful parameter configurations. Point cloud noises and parameter errors
will lead to fragmented segmentations and inaccurate primitive fitting. Deep
learning models have stronger adaptability than traditional methods in solving
data noise and parameter fitting. With a large amount of training data, an
intuition is that using deep models to parse these primitives may have better
performance than traditional methods. However, the learning of the quadrics
representation for parsing geometric primitives remains unexplored.

Figure 1: For point clouds with various geometric primitives, representing
them individually poses a challenge for the algorithm design and downstream
tasks. We propose QuadricsNet to learn a concise representation with 10
parameters for diverse geometric primitives, thus yielding robust primitive
parsing and structure mapping results.

In this work, we propose a robust learning-based quadrics detection and
fitting framework for the parsing of geometric primitives in point clouds,
namely QuadircsNet. It consists of a quadrics detection module and a quadrics
fitting module. The detection module segments and classifies point clouds into
several quadrics primitives, while the fitting module generates quadrics
parameters from the detected primitives and vectorizes the primitive models.
Our key insight is utilizing the characteristics obtained from quadrics
mathematical formulation, such as type, scale, and pose, to supervise the
quadrics detection and fitting modules. The key contributions can be
summarized as:

  * $\bullet$

We propose QuadricsNet, the first end-to-end quadrics detection and fitting
framework for learning the concise representation of geometric primitives in
point clouds.

  * $\bullet$

We leverage the geometric attributes from quadrics mathematical decomposition,
such as the type, scale, and pose, to supervise the detection and fitting
modules.

  * $\bullet$

We build QuadricsDataset, which consists of quadrics segments and objects.
Experiments demonstrate the effectiveness of our quadrics representation and
robustness of QuadricsNet.

##  II Related Work

In this section, we review the representations of geometric primitives and the
detection-fitting techniques.

###  II-A Geometric Primitive Representation

The representation of geometric primitives has been widely investigated in
computer vision and robotics [17]. Planes receive the earliest attention as
the simplest primitive, which are represented as a normal and an offset in
most methods [18, 19, 9]. Further, curved surfaces such as cylinders, spheres,
and cones are represented by the parameters like apex, axis, or radius,
depending on their geometric structures [12, 15, 20]. There are also more
particular surfaces, e.g., B-spline [11] and extrusion cylinders [5], which
are also modeled according to their properties. The above individual
representations lack uniformity, resulting in the necessity of multiple
designs for detecting diverse primitives, as well as inconveniences for
downstream tasks.

A more general representation could address the above deficiencies. For
example, cuboids [8, 21] are used to represent objects as bounding boxes.
Ellipsoids [10] and superquadircs [22] have a better expression in terms of
scale, orientation, and position. But they are limited by their basic models,
resulting in only being able to express a coarse structure. Surfaces based on
control points [23, 11] can express arbitrary shapes. However, the parameters
of the control points are redundant and have insufficient geometric
attributes. In contrast, quadrics can concisely cover the 17 most common
geometric primitives, and can also figure out their attributes, such as type,
scale, and pose [16, 7]. However, the relationships between quadrics
mathematical formulation and geometric attributes are not fully considered,
and non-learning methods suffer from weak robustness and generalization.

###  II-B Detection and Fitting of Primitives

Detection and fitting of geometric primitives in point clouds is a time-
honored problem [24]. Hough transforms based on voting in parameter space [18,
25, 26], region growing based on similarity matching [27], and iterative
heuristic RANSAC [28, 15, 20] have achieved great success and continuous
extension. Particularly, Efficient-RANSAC [15] implemented in CGAL is
considered the standard. However, the non-learning methods suffer from
laborious tuning when confronted with different primitives. In addition, noise
and challenging cases like fragmentation and occlusions are also intractable.

Learning-based methods are highly anticipated in these issues. SPFN [12] first
detects four primitives and then fits the corresponding parameters with
separate differentiable estimators. ParseNet [11] and Point2Cyl [5] extend the
primitive types to B-splines and extrusion cylinders. HPNet [13] proposes
hybrid features to combine multiple primitive detection cues, thereby
improving detection performances. CPFN [14] assembles the detection results of
global and local networks by an adaptive patch sampling network to improve the
detection of fine-scale primitives. However, these networks are designed with
individual estimators for specific primitives. There are only a few networks
for general primitives [29, 30, 22], but they are limited by the
expressiveness of the representations used and hence can only represent
structures at a coarse level.

Different from prior works, we are the first to adopt a learning approach to
represent point clouds with quadrics concisely, and we also exploit the
relationships between quadrics mathematics and geometry to improve robustness.

##  III Quadrics

Before delving into the methodological details, we first define the quadrics,
a parametric representation of surfaces that is more exhaustive in terms of
primitive types and more concise with respect to the mathematical form than
existing representations [12, 11, 29, 22]. We further decompose the
formulation of quadrics to reveal the relationships between their mathematics
and geometric attributes (type, scale and pose). Finally, we explain the
degeneracy of quadrics.

###  III-A Quadrics Representation

In algebraic geometry, quadrics are a class of surfaces defined implicitly by
a second-degree polynomial equation:

$\displaystyle f(\mathbf{x,q})$ $\displaystyle=Ax^{2}+By^{2}+Cz^{2}+2Dxy+2Exz$
(1) $\displaystyle+2Fyz+2Gx+2Hy+2Iz+J=0,$

where $\mathbf{x}$ is a point with homogeneous coordinate
$[x,y,z,1]^{\mathrm{T}}$, $\mathbf{q}=[A,B,C,D,E,F,G,H,I,J]$ and the quadratic
term is not all zero. The compact matrix form is
$\mathbf{x}^{\mathrm{T}}\mathbf{Q}\mathbf{x}=0$, where

$\mathbf{Q}=\left[\begin{array}[]{@{}cccc@{}}A&D&E&G\\\ D&B&F&H\\\ E&F&C&I\\\
G&H&I&J\end{array}\right],\nabla\mathbf{Q}=2\left[\begin{array}[]{@{}cccc@{}}A&D&E&G\\\
D&B&F&H\\\ E&F&C&I\end{array}\right].$ (2)

The gradient at $\mathbf{x}$ is $\nabla
f(\mathbf{x,q})=\nabla\mathbf{Q}\mathbf{x}$, and also the direction of its
normal $\mathbf{n}\in\mathbb{R}^{3}$.

Despite having only 10 parameters, quadrics can uniformly represent 17
geometric primitives, including points, lines, planes, spheres, cylinders, and
cones, which encompass most cases in artificial structures.

###  III-B Mathematical Decomposition of Quadrics

For any quadric $\mathbf{Q}$ in space, all its axes can be aligned to the
coordinate axes by precisely applying rotation and translation. After that,
$\mathbf{Q}$ is reduced to a diagonal matrix $\mathbf{C}$, namely the
canonical matrix of a quadric. The canonical matrices of typical quadrics are
summarized in Table I. It is worth noting that the form of $\mathbf{C}$ and
value of $\lambda$ in $\mathbf{C}$ determine respectively the type and scale
of a quadric, and $\mathbf{I}_{\mathbf{s,R,t}}\in\\{0,1\\}^{3}$ indicates the
degeneracy of the scale, rotation, and translation.

TABLE I: Characteristics of typical quadrics Type | $\mathrm{Diag}(\mathbf{C})$ | $\mathbf{I_{s}}$ | $\mathbf{I_{R}}$ | $\mathbf{I_{t}}$
---|---|---|---|---
Line | $[\lambda_{a},\lambda_{b},0,0]$ | $[0,0,0]$ | $[0,0,1]$ | $[1,1,0]$
Plane | $[\lambda_{a},0,0,0]$ | $[0,0,0]$ | $[1,0,0]$ | $[1,0,0]$
Sphere | $[\lambda_{a},\lambda_{b},\lambda_{c},-1]$ | $[1,1,1]$ | $[0,0,0]$ | $[1,1,1]$
Cylinder | $[\lambda_{a},\lambda_{b},0,-1]$ | $[1,1,0]$ | $[0,0,1]$ | $[1,1,0]$
Cone | $[\lambda_{a},\lambda_{b},-\lambda_{c},0]$ | $[1,1,0]$ | $[0,0,1]$ | $[1,1,1]$

Inversely, as illustrated in Fig. 2, a given quadric $\mathbf{Q}$ can be
regarded as being transformed from $\mathbf{C}$ by

$\mathbf{Q}=\mathbf{P}^{-\mathrm{T}}\mathbf{C}\mathbf{P}^{-1}=\left[\begin{array}[]{@{}cc@{}}\mathbf{R}&\mathbf{t}\\\
\mathbf{0}^{\mathrm{T}}&1\end{array}\right]^{-\mathrm{T}}\left[\begin{array}[]{@{}cc@{}}\mathbf{\Lambda}&\mathbf{0}\\\
\mathbf{0}^{\mathrm{T}}&c_{44}\end{array}\right]\left[\begin{array}[]{@{}cc@{}}\mathbf{R}&\mathbf{t}\\\
\mathbf{0}^{\mathrm{T}}&1\end{array}\right]^{-1}.$ (3)

where $\mathbf{P}(\mathbf{R},\mathbf{t})\in SE(3)$ denotes the transform
matrix from $\mathbf{C}$ to $\mathbf{Q}$, namely the pose of a quadric.
$\mathbf{R}\in SO(3)$ and $\mathbf{t}\in\mathbb{R}^{3}$ are the rotation and
translation blocks of $\mathbf{P}$, $\mathbf{\Lambda}\in\mathbb{R}^{3\times
3}$ and $c_{44}$ are the diagonal blocks of $\mathbf{C}$. Furthermore,
$\mathbf{Q}$ can be decomposed as

$\mathbf{Q}=\left[\begin{array}[]{@{}cc@{}}\mathbf{R}\mathbf{\Lambda}\mathbf{R}^{\mathrm{T}}&-\mathbf{R}\mathbf{\Lambda}\mathbf{R}^{\mathrm{T}}\mathbf{t}\\\
&k\end{array}\right]=\left[\begin{array}[]{@{}cc@{}}\mathbf{Q}_{33}&\mathbf{l}\\\
&k\end{array}\right].$ (4)

Here, it is obvious that the scale and pose of a quadric can be inferred by
mathematical analysis of $\mathbf{Q}_{33}\in\mathbb{R}^{3\times 3}$ and
$\mathbf{l}\in\mathbb{R}^{3}$, which is explained in Sect. IV-C.

Figure 2: Illustration of quadrics derivation and degeneracy. Ellipsoid
$\mathbf{Q}_{1}$ and cylinder $\mathbf{Q}_{2}$ are derived from the canonical
quadric $\mathbf{C}_{1}$ and $\mathbf{C}_{2}$ through transformations
$\mathbf{P}_{1}$ and $\mathbf{P}_{2}$. For $\mathbf{Q}_{1}$, the rotation
around axis $a$ is degenerate because the ellipsoid is symmetric on the axes
$b$ and $c$ $(\mathbf{I_{R}}=[1,0,0]$). For $\mathbf{Q}_{2}$, the translation
along axis $c$ and the rotation around it are degenerate because the cylinder
is open on axis $c$ $(\mathbf{I_{t}}=[1,1,0])$ and symmetric on the axes $a$
and $b$ ($\mathbf{I_{R}}=[0,0,1]$).

###  III-C Geometric Degeneracy of Quadrics

The degeneracy means that the scale, rotation, and translation of a quadric
won’t affect its shape, which can be judged from $\mathbf{Q}$. If $\mathbf{Q}$
is rank-deficient, this will lead to the degeneracy of scale at certain axes.
Since $\mathbf{Q}_{33}$ is a real symmetric matrix, it can be similarly
diagonalized as
$\mathbf{Q}_{33}=\hat{\mathbf{R}}\hat{\mathbf{\Lambda}}\hat{\mathbf{R}}^{\mathrm{T}}$,
where $\hat{\mathbf{\Lambda}}=\mathbf{\Lambda}$ can be denoted by the
eigenvalues of $\mathbf{Q}_{33}$ as
$\mathrm{diag}\left(\lambda_{a},\lambda_{b},\lambda_{c}\right)$ and
$\hat{\mathbf{R}}$ consists of the eigenvectors. The zero values in the
eigenvalues make the translation along the corresponding axis degenerate. If
there are duplicates in the eigenvalues, i.e., the quadric is symmetric, and
the rotation around the non-symmetric axis will degenerate. Fig. 2 illustrates
this more intuitively.

##  IV Methodology

We presume that the inputs are point clouds $\mathcal{X}\in\mathbb{R}^{N}$
with 3D positions and optional normals. We seek to represent it with a set of
quadrics
$\\{\hat{\mathbf{Q}}_{1},\hat{\mathbf{Q}}_{2},...,\hat{\mathbf{Q}}_{\hat{K}}\\}$
that closely approximate its underlying surfaces. For this purpose, an end-to-
end framework QuadricsNet is proposed, as illustrated in Fig. 3. We first
introduce the two modules of QuadricsNet: quadrics detection module and
quadrics fitting module, and then we specify the losses customized for
quadrics.

Figure 3: Overview of QuadricsNet. The detection module segments point clouds
into several quadrics segments and identifies the quadrics types, then the
fitting module fits the quadrics parameters of each segment, and finally the
structure map is reconstructed using the fitted quadrics.

###  IV-A Quadrics Detection Module

The purpose of this module is to decompose the input $\mathcal{X}$ into
several quadrics segments
$\\{\hat{\mathbf{X}}_{1},\hat{\mathbf{X}}_{2},...,\hat{\mathbf{X}}_{\hat{K}}\\}$
while simultaneously identifying their quadrics types
$\\{\hat{l}_{1},\hat{l}_{2},...,\hat{l}_{\hat{K}}\\}$. To this end, we
implement it in three steps: embedding, clustering, and classification.

Embedding. We primarily employ an embedding network to learn the point-wise
features $\mathbf{Z}\in\mathbb{R}^{N\times 128}$ that can distinguish
different quadrics segments in $\mathcal{X}$. Its backbone is derived from
EdgeConv [31], which can learn local features singly and global features when
stacking in multiple.

Clustering. Based on the features $\mathbf{Z}$ with implicit
distinguishability, we proceed to modify them by the mean shift [32] in a
differentiable way to explicitly cluster the points according to the quadrics
segments:

$\mathbf{Z}_{t+1}=\mathbf{Z}_{t}+\eta(\mathbf{Z}_{t}\mathbf{KD}^{-1}-\mathbf{Z}_{t}),$
(5)

where $\mathbf{K}$ is a Gaussian kernel, $\mathbf{D}=\sum\mathbf{K}$ and the
step size $\eta=1$. After the convergence of Eq. 5, the cluster centers are
determined by a non-maximal suppression. Eventually, each point is assigned to
a segment according to its nearest center. We employ a membership matrix
$\hat{\mathbf{W}}\in\\{0,1\\}^{N\times\hat{K}}$ to indicate the affiliations
between points $\mathcal{X}$ and clustered segments
$\\{\hat{\mathbf{X}}_{1},\hat{\mathbf{X}}_{2},...,\hat{\mathbf{X}}_{\hat{K}}\\}$
, where we ensure that $\hat{K}\geq K$ even if $K$ varies for different
objects.

Classification. Simultaneously with clustering, we combine an MLP and a
softmax to construct primitive classifiers that predict the point-wise
quadrics type (e.g., plane, sphere, cylinder, cone) based on $\mathbf{Z}$. We
use another membership matrix $\hat{\mathbf{L}}\in[0,1]^{N\times L}$ to
indicate the probability of each point in $\mathcal{X}$ being predicted as
which quadrics type. Eventually, the types
$\\{\hat{l}_{1},\hat{l}_{2},...,\hat{l}_{\hat{K}}\\}$ of clustered segments
are determined by a majority voting overall types of points in each segment.

###  IV-B Quadrics Fitting Module

This module fits quadrics parameters
$\\{\hat{\mathbf{Q}}_{1},\hat{\mathbf{Q}}_{2},...,\hat{\mathbf{Q}}_{\hat{K}}\\}$
for all detected quadrics segments
$\\{\hat{\mathbf{X}}_{1},\hat{\mathbf{X}}_{2},...,\hat{\mathbf{X}}_{\hat{K}}\\}$.

Prior to fitting, the points $\hat{\mathbf{X}}_{k}$ of each segment are
derived by filtering $\mathcal{X}$ according to $\hat{\mathbf{W}}$:

$\hat{\mathbf{X}}_{k}=\hat{\mathbf{W}}_{:,k}\odot\mathcal{X},$ (6)

where $\odot$ denotes the element-wise multiplication. We construct a quadrics
fitting network to robustly estimate the quadrics parameters
$\hat{\mathbf{Q}}_{k}$ for the underlying surface of the segment
$\hat{\mathbf{X}}_{k}$. The backbone of the fitting network is similar to the
embedding network. Furthermore, we leverage two MLPs to respectively estimate
a canonical matrix $\hat{\mathbf{C}}_{k}$ and a inverse pose matrix
$\hat{\mathbf{P}}_{k}^{-1}(\hat{\mathbf{R}}_{k},\hat{\mathbf{t}}_{k})$, which
correspond to the scale and pose respectively. According to Eq. 3, the quadric
is finally determined as

$\hat{\mathbf{Q}}_{k}=\hat{\mathbf{P}}_{k}^{-\mathrm{T}}\hat{\mathbf{C}}_{k}\hat{\mathbf{P}}_{k}^{-1}.$
(7)

It is worth noting that depending on the symmetry and sparsity of
$\hat{\mathbf{C}}_{k}$ and $\hat{\mathbf{P}}_{k}^{-1}$, the number of
parameters in the output of the fitting network is correspondingly no more
than 10 and 12. According to Table I, the form of $\hat{\mathbf{C}}_{k}$
determines the type of a quadric. To guarantee that the fitted
$\hat{\mathbf{Q}}_{k}$ is consistent with the type $\hat{l}_{k}$, we leverage
$\hat{l}_{k}$ as a prior to constrain the form of output
$\hat{\mathbf{C}}_{k}$, e.g., the form is constrained to be
$\mathrm{diag}([\hat{\lambda}_{a},\hat{\lambda}_{b},0,-1])$ if a cylinder is
to be fitted. In such a concise representation, even a small parameter error
can lead to the wrong fitted type. Our network avoids this issue and
significantly improves the fitting accuracy.

###  IV-C Losses

Six losses customized for quadrics are defined from mathematic and geometric
perspectives to train these modules.

Losses for Detection. We adopt the GT segment membership $\mathbf{W}$ and type
membership $\mathbf{L}$ to supervise this module.

We leverage the triplet loss [33] to achieve the point-wise features
$\mathbf{Z}$ with segment distinguishabilities:

$\mathcal{L}_{\mathrm{emb}}=\frac{1}{M}\sum_{i=1}^{M}\max\left(\left\|\mathbf{z}^{\mathrm{A}}_{i}-\mathbf{z}^{\mathrm{P}}_{i}\right\|^{2}-\left\|\mathbf{z}^{\mathrm{A}}_{i}-\mathbf{z}^{\mathrm{N}}_{i}\right\|^{2}+\alpha,0\right),$
(8)

where $\mathbf{z}\in\mathbb{R}^{128}$ is the point-wise feature, $\alpha$ is
the margin, $M$ is the total number of triplet sets
$\\{\mathbf{z}^{\mathrm{A}},\mathbf{z}^{\mathrm{P}},\mathbf{z}^{\mathrm{N}}\\}$
sampled from features of different segments according to $\mathbf{W}$. The
cross entropy loss $H$ is employed for quadrics classification:

$\mathcal{L}_{\mathrm{type}}=\frac{1}{N}\sum_{i=1}^{N}H\left(\hat{\mathbf{L}}_{i,:},\mathbf{L}_{i,:}\right).$
(9)

Losses for Fitting. To supervise the fitting module, we employ the GT
$\mathbf{Q}$ and normals $\mathbf{N}$.

Previously, to match the GT signals to the fitted signals of each clustered
segment, we first compute the Relaxed Intersection over Union of
$\hat{\mathbf{W}}_{:,k}$ and $\mathbf{W}_{:,k}$, then the best one-to-one
correspondence between GT segments $\mathbf{X}_{k}$ and clustered segments
$\mathbf{\hat{X}}_{k}$ is matched by the Hungarian algorithm [34]. According
to Eq. 1, we define the primal loss for $\mathbf{\hat{Q}}$, which roughly
mirrors the quality of fitting:

$\mathcal{L}_{\mathrm{primal}}=\frac{1}{\hat{K}}\sum_{k=1}^{\hat{K}}\frac{1}{|\mathbf{\hat{X}}_{k}|}\sum_{\mathbf{\hat{x}}\in\mathbf{\hat{X}}_{k}}\left\|\mathbf{\hat{x}}^{\mathrm{T}}\mathbf{Q}_{k}\mathbf{\hat{x}}\right\|^{2}.$
(10)

Further, based on the correlation between the normals and parameters of
quadrics (Eq. 2), we define the normal loss:

$\mathcal{L}_{\mathrm{normal}}=\frac{1}{\hat{K}}\sum_{k=1}^{\hat{K}}\frac{1}{|\mathbf{\hat{X}}_{k}|}\sum_{\mathbf{\hat{x}}\in\mathbf{\hat{X}}_{k}}\left\|\nabla\hat{\mathbf{Q}}_{k}\hat{\mathbf{x}}\otimes\mathbf{n}\right\|^{2},$
(11)

where $\otimes$ denotes column-wise cross product, $\mathbf{n}$ is GT normal
on $\mathbf{\hat{x}}$. Additionally, we explicitly define the regression loss:

$\mathcal{L}_{\mathrm{reg}}=\frac{1}{\hat{K}}\sum_{k=1}^{\hat{K}}\left\|\hat{\mathbf{Q}}_{k}-\mathbf{Q}_{k}\right\|^{2}.$
(12)

The scale and pose of fitted quadrics can be learned directly in Eq. 7, but
those of GT need to be inferred by mathematical analysis of its $\mathbf{Q}$.
Prior to inference, $\mathbf{Q}$ has to be normalized to eliminate proportion
ambiguity in Eq. 1:

${\mathbf{Q}}=\left\\{\begin{array}[]{cc}\left|{\dfrac{{\prod{\lambda_{i}^{\mathbf{Q}_{33}}}}}{{\prod{\lambda_{i}^{\mathbf{Q}}}}}}\right|{\mathbf{Q}},&{c_{44}}\neq
0\\\
\dfrac{1}{\left\|\mathbf{Q}\right\|}{\mathbf{Q}},&{c_{44}}=0\end{array},\right.$
(13)

where $\lambda^{\mathbf{Q}_{33}}$ and $\lambda^{\mathbf{Q}}$ are the non-zero
eigenvalues of $\mathbf{Q}_{33}$ and $\mathbf{Q}$. According to Sect. III-C,
$\mathbf{Q}_{33}=\tilde{\mathbf{R}}\mathbf{\Lambda}\tilde{\mathbf{R}}^{\mathrm{T}}$,
where
$\mathbf{\Lambda}=\mathrm{diag}\left(\lambda_{a},\lambda_{b},\lambda_{c}\right)$.
Without loss of generality, we assume that
$\lambda_{a}>\lambda_{b}>\lambda_{c}$, and the scale
$\mathbf{s}\in\mathbb{R}^{3}$, rotation $\mathbf{R}\in SO(3)$ and translation
$\mathbf{t}\in\mathbb{R}^{3}$ are

$\left\\{\begin{array}[]{lll}\left[s_{a},s_{b},s_{c}\right]^{\mathrm{T}}&=&\mathrm{diag}(\mathbf{I_{s}})\sqrt{\left|\left[\frac{1}{\lambda_{a}},\frac{1}{\lambda_{b}},\frac{1}{\lambda_{c}}\right]\right|},\\\
\left[\mathbf{r}_{a},\mathbf{r}_{b},\mathbf{r}_{c}\right]&=&\pm(\mathrm{diag}(\mathbf{I}_{\mathbf{R}})\tilde{\mathbf{R}}^{\mathrm{T}})^{\mathrm{T}},\\\
\left[t_{a},t_{b},t_{c}\right]^{\mathrm{T}}&=&\mathrm{diag}(\mathbf{I_{t}})\tilde{\mathbf{t}},\end{array}\right.$
(14)

where the direction of $\mathbf{r}$ can either be identical or opposite to the
column of $\tilde{\mathbf{R}}$ and
$\tilde{\mathbf{t}}\in\left\\{\tilde{\mathbf{t}}\mid\mathbf{Q}_{33}\tilde{\mathbf{t}}+\mathbf{l}=0\right\\}$.
Based on these theoretical guides, we define the geometric loss:

$\mathcal{L}_{\mathrm{geo}}=\frac{1}{\hat{K}}\sum_{k=1}^{\hat{K}}\left[||\hat{\mathbf{s}}_{k}-\mathbf{s}_{k}||^{2}+||\hat{\mathbf{R}}_{k}\otimes\mathbf{R}_{k}||^{2}+||\hat{\mathbf{t}}_{k}-\mathbf{t}_{k}||^{2}\right],$
(15)

where we approximate them for gradient stabilization:

$\left\\{\begin{array}[]{lll}\hat{\mathbf{s}}-\mathbf{s}&\approx&\hat{\mathbf{C}}-\mathbf{\Lambda},\\\
\hat{\mathbf{R}}\otimes\mathbf{R}&=&\sum_{i=1}^{3}\hat{\mathbf{R}}_{:,i}\times\mathbf{R}_{:,i},\\\
\hat{\mathbf{t}}-\mathbf{t}&\approx&\mathbf{\Lambda}\mathbf{R}^{\mathrm{T}}\hat{\mathbf{t}}+\mathbf{R}^{\mathrm{T}}\mathbf{l}.\end{array}\right.$
(16)

##  V Experiments

In this section, we detail the experimental dataset, evaluation metrics, and
result comparisons to the state-of-the-art.

###  V-A Quadrics Dataset and Training Strategy

Since there is no publicly available point cloud dataset with labeled quadrics
parameters $\mathbf{Q}$, we build a Quadrics Dataset for QuadricsNet training
and evaluation using simulated quadric segments and CAD objects selected from
ABC dataset [35]. To improve the network robustness to incomplete point
clouds, we randomly trim them to simulate fragmentation and incompletion.
Limited by the ABC dataset, we mainly consider four types of quadrics, that
is, planes, spheres, cylinders, and cones. We compute $\mathbf{Q}$ for all
segments in this dataset and also randomly add noise along the normal
direction in a uniform range $[-0.01,0.01]$. It contains two subsets: a)
Segment Dataset is mainly designed for quadrics fitting network, which has 20k
quadrics segments for training and 3k segments for testing; b) Object Dataset
is built for both quadrics detection and fitting networks, which has 20k CAD
objects for training and 3k objects for testing.

We adopt a two-step pre-training and fine-tuning strategy to train the
QuadricsNet. We first pre-train the detection network using
$\mathcal{L}_{\mathrm{emb}}+\mathcal{L}_{\mathrm{type}}$ loss on the Object
Dataset. At the same time, we pre-train the fitting network with
$\mathcal{L}_{\mathrm{primal}}+\mathcal{L}_{\mathrm{normal}}+\mathcal{L}_{\mathrm{reg}}+\mathcal{L}_{\mathrm{geo}}$
loss on the Segment Dataset. Then, the whole QuadricsNet is fine-tuned with
the six losses on the Object Dataset in an end-to-end manner.

###  V-B Evaluation Metrics

To quantitatively evaluate the performance of QuadricsNet, we use Seg-IoU and
Type-IoU metrics to measure the performance of quadrics detection, while
Residual and P-coverage metrics measure the accuracy of quadrics fitting.

  * $\bullet$

Seg-IoU (S-IoU):
$\frac{1}{K}{\textstyle\sum_{k=1}^{K}}IoU(\mathbf{\hat{W}}_{:,k},\mathbf{W}_{:,k})$
measures the accuracy of quadrics segment clustering.

  * $\bullet$

Type-IoU (T-IoU):
$\frac{1}{K}{\textstyle\sum_{k=1}^{K}}\mathbb{I}(\hat{l}_{k}=l_{k})$ measures
the accuracy of quadrics classification.

  * $\bullet$

Residual (Res):
$\frac{1}{K}{\textstyle\sum_{k=1}^{K}}\frac{1}{|\mathbf{X}_{k}|}{\textstyle\sum_{\mathbf{x}\in\mathbf{X}_{k}}}D(\mathbf{\hat{Q}}_{k},\mathbf{x})$
is the average Euclidean distance of raw points $\mathbf{x}$ to the predicted
quadric surfaces.

  * $\bullet$

P-coverage (P-cov):
$\frac{1}{|\mathcal{X}|}{\textstyle\sum_{\mathbf{x}\in\mathcal{X}}}\mathbb{I}[\mathrm{min}_{k=1}^{K}D(\mathbf{\hat{Q}}_{k},\mathbf{x})<\epsilon]$
measures the percentage of input points covered by the predicted quadric
surfaces.

###  V-C Result Comparisons

To evaluate the performance, we compare QuadricsNet with five state-of-the-art
geometric primitive parsing methods, including the traditional nearest
neighbor (NN) [36] and Efficient RANSAC [15], the learning-based SPFN [35],
ParseNet [11] and HPNet [13]. Unlike our unified quadrics representation for
different geometric primitives, these competitors employ individual
representations for them. All these methods are evaluated on the Object
Dataset test set for a fair comparison.

Quantitative Results. As reported in Table II, we test these methods with two
input cases: points (p) and points with normals (p+n). Under the same input
information, QuadricsNet generally outperforms other methods for all metrics.
Especially for the Residual and P-coverage metrics, our unified quadrics
representation yields better performance than other non-unified methods with
different representations for each primitive. These results demonstrate the
effectiveness of our quadrics-based framework on geometric primitive detection
and fitting.

Figure 4: Qualitative comparisons. Structure mapping of the raw point clouds using different primitive parsing methods. TABLE II: Quantitative comparisons Method | Input | S-IoU$(\%)$ $\uparrow$ | T-IoU$(\%)$ $\uparrow$ | Res $\downarrow$ |  P-cov $(\%)$ $\uparrow$
---|---|---|---|---|---
$\epsilon=0.01$ | $\epsilon=0.02$
NN [36] | p | 54.10 | 61.10 | - | - | -
E-RANSAC [15] | p+n | 67.21 | - | 0.022 | 83.40 | 87.73
SPFN [35] | p | 61.42 | 74.56 | 0.023 | 82.55 | 90.67
p+n | 73.19 | 85.91 | 0.019 | 86.79 | 92.14
ParseNet [11] | p | 74.12 | 79.90 | 0.018 | 83.20 | 92.32
p+n | 85.70 | 90.21 | 0.013 | 89.76 | 93.98
HPNet [13] | p | 80.32 | 87.19 | 0.014 | 86.07 | 94.12
p+n | 88.17 | 92.25 | 0.009 | 93.12 | 96.27
Ours | p | 84.12 | 88.00 | 0.013 | 88.64 | 95.88
p+n | 92.16 | 95.87 | 0.009 | 93.76 | 97.12

Qualitative Results. Fig. 4 qualitatively shows the structure mapping of point
clouds using different primitive parsing methods. The mapping results using
QuadricsNet are more reasonable with clean boundaries and better structure
integrity because QuadricsNet can detect and fit primitives more precisely.
Furthermore, to evaluate the generalizability of QuadricsNet in real-world
data, we extend the experiment to the large-scale indoor S3DIS dataset [37].
As shown in Fig. 5, our method effectively represents the real scene with
quadrics on the object-level scale and yields a robust structure mapping
result.

Figure 5: Stucture mapping of a scene in S3DIS using quadrics. TABLE III: Impacts of loss design on quadrics fitting Loss settings | Res $\downarrow$ | P-cov $(\%)$ $\uparrow$
---|---|---
$\mathcal{L}_{\mathrm{primal}}$ | $\mathcal{L}_{\mathrm{normal}}$ | $\mathcal{L}_{\mathrm{reg}}$ | $\mathcal{L}_{\mathrm{geo}}$ | $\epsilon=0.01$ | $\epsilon=0.02$
✓ |  |  |  | 0.032 | 68.10 | 71.87
✓ | ✓ |  |  | 0.029 | 73.03 | 78.62
✓ | ✓ | ✓ |  | 0.012 | 89.45 | 90.34
✓ | ✓ | ✓ | ✓ | 0.008 | 94.12 | 96.10

###  V-D Ablation Study

We mainly discuss the effectiveness of the designed loss functions on the
quadrics fitting module. By gradually adding the four losses, we obtain their
impacts in Table III. On the basis of the $\mathcal{L}_{\mathrm{primal}}$
loss, adding other supervision terms generally improves the fitting accuracy.
Especially for adding the $\mathcal{L}_{\mathrm{reg}}$ loss, we obtain a
noticeable performance improvement with a large margin. Eventually, adding the
$\mathcal{L}_{\mathrm{geo}}$ loss produces the best result, which demonstrates
the effectiveness of combining mathematical factors of quadrics and geometric
attributes on geometric primitive parsing.

##  VI Conclusions and Future Work

In this paper, we propose an end-to-end QuadricsNet to learn a concise
representation of geometric primitives in point clouds. Quadrics
representation successfully unifies different geometric primitives, while the
geometric attributes from quadrics mathematical formulation effectively
supervise the quadrics detection and fitting networks. Experiments of
primitive parsing on the collected dataset and structure mapping on real-world
scenes demonstrate that our quadrics representation is effective and the
QuadricsNet framework is robust. In the future, we will explore the fusion of
geometry and semantics for primitive parsing and structure mapping.

##  VII Acknowledgement

This work was supported by the NSFC grants under contracts Nos. 62301370 and
62325111, and Wuhan University-Huawei Geoinformatics Innovation Laboratory.

## References

  * [1] L. Nan and P. Wonka, “Polyfit: Polygonal surface reconstruction from point clouds,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2353–2361.
  * [2] A. Monszpart, N. Mellado, G. J. Brostow, and N. J. Mitra, “Rapter: rebuilding man-made scenes with regular arrangements of planes.” _ACM Transactions on Graphics_ , vol. 34, no. 4, pp. 103–1, 2015.
  * [3] S. Wang, G. Cai, M. Cheng, J. M. Junior, S. Huang, Z. Wang, S. Su, and J. Li, “Robust 3d reconstruction of building surfaces from point clouds based on structural and closed constraints,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 170, pp. 29–44, 2020.
  * [4] P. Benko, G. Kós, T. Várady, L. Andor, and R. Martin, “Constrained fitting in reverse engineering,” _Computer Aided Geometric Design_ , vol. 19, no. 3, pp. 173–205, 2002.
  * [5] M. A. Uy, Y.-Y. Chang, M. Sung, P. Goel, J. G. Lambourne, T. Birdal, and L. J. Guibas, “Point2cyl: Reverse engineering 3d objects from point clouds to extrusion cylinders,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022, pp. 11 850–11 860.
  * [6] M. Kaess, “Simultaneous localization and mapping with infinite planes,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2015, pp. 4605–4611.
  * [7] W. Zhen, H. Yu, Y. Hu, and S. Scherer, “Unified representation of geometric primitives for graph-slam optimization using decomposed quadrics,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 5636–5642.
  * [8] Y. Taguchi, Y.-D. Jian, S. Ramalingam, and C. Feng, “Point-plane slam for hand-held 3d sensors,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2013, pp. 5182–5189.
  * [9] C. Sommer, Y. Sun, L. Guibas, D. Cremers, and T. Birdal, “From planes to corners: Multi-purpose primitive detection in unorganized 3d point clouds,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1764–1771, 2020\.
  * [10] L. Nicholson, M. Milford, and N. Sünderhauf, “Quadricslam: Dual quadrics from object detections as landmarks in object-oriented slam,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 1, pp. 1–8, 2018.
  * [11] G. Sharma, D. Liu, S. Maji, E. Kalogerakis, S. Chaudhuri, and R. Mech, “Parsenet: A parametric surface fitting network for 3d point clouds,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , vol. 12352, 2020, pp. 261–276.
  * [12] L. Li, M. Sung, A. Dubrovina, L. Yi, and L. J. Guibas, “Supervised fitting of geometric primitives to 3d point clouds,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 2652–2660.
  * [13] S. Yan, Z. Yang, C. Ma, H. Huang, E. Vouga, and Q.-X. Huang, “Hpnet: Deep primitive segmentation using hybrid representations. 2021 ieee,” in _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_ , 2021, pp. 2733–2742.
  * [14] E.-T. Lê, M. Sung, D. Ceylan, R. Mech, T. Boubekeur, and N. J. Mitra, “Cpfn: Cascaded primitive fitting networks for high-resolution point clouds,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 7457–7466.
  * [15] R. Schnabel, R. Wahl, and R. Klein, “Efficient ransac for point-cloud shape detection,” in _Computer graphics forum_ , vol. 26, no. 2, 2007, pp. 214–226.
  * [16] T. Birdal, B. Busam, N. Navab, S. Ilic, and P. Sturm, “Generic primitive detection in point clouds using novel minimal quadric fits,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 42, no. 6, pp. 1333–1347, 2019.
  * [17] G. Sharma, R. Goyal, D. Liu, E. Kalogerakis, and S. Maji, “Neural shape parsers for constructive solid geometry,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 5, pp. 2628–2640, 2022.
  * [18] D. Borrmann, J. Elseberg, K. Lingemann, and A. Nüchter, “The 3d hough transform for plane detection in point clouds: A review and a new accumulator design,” _3D Research_ , vol. 2, no. 2, pp. 1–13, 2011.
  * [19] H. Fang, F. Lafarge, and M. Desbrun, “Planar shape detection at structural scales,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 2965–2973.
  * [20] Y. Li, X. Wu, Y. Chrysathou, A. Sharf, D. Cohen-Or, and N. J. Mitra, “Globfit: Consistently fitting primitives by discovering global relations,” in _ACM SIGGRAPH 2011 papers_ , 2011, pp. 1–12.
  * [21] S. Yang and S. Scherer, “Cubeslam: Monocular 3-d object slam,” _IEEE Transactions on Robotics_ , vol. 35, no. 4, pp. 925–938, 2019.
  * [22] D. Paschalidou, A. O. Ulusoy, and A. Geiger, “Superquadrics revisited: Learning 3d shape parsing beyond cuboids,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 10 344–10 353.
  * [23] G. Farin, _Curves and surfaces for CAGD: a practical guide_ , 5th ed. Morgan Kaufmann Publishers Inc., 2001.
  * [24] A. Kaiser, J. A. Ybanez Zepeda, and T. Boubekeur, “A survey of simple geometric primitives detection methods for captured 3d data,” in _Computer Graphics Forum_ , vol. 38, no. 1, 2019, pp. 167–196.
  * [25] B. Drost and S. Ilic, “Local hough transform for 3d primitive detection,” in _Proceedings of the International Conference on 3D Vision (3DV)_ , 2015, pp. 398–406.
  * [26] C. Sommer, Y. Sun, E. Bylow, and D. Cremers, “Primitect: Fast continuous hough voting for primitive detection,” in _Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 8404–8410.
  * [27] S. Oesau, F. Lafarge, and P. Alliez, “Planar shape detection and regularization in tandem,” in _Computer Graphics Forum_ , vol. 35, no. 1, 2016, pp. 203–215.
  * [28] M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” _Communications of the ACM_ , vol. 24, no. 6, pp. 381–395, 1981.
  * [29] S. Tulsiani, H. Su, L. J. Guibas, A. A. Efros, and J. Malik, “Learning shape abstractions by assembling volumetric primitives,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 2635–2643.
  * [30] C. Zou, E. Yumer, J. Yang, D. Ceylan, and D. Hoiem, “3d-prnn: Generating shape primitives with recurrent neural networks,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR)_ , 2017, pp. 900–909.
  * [31] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic graph CNN for learning on point clouds,” _ACM Transactions on Graphics_ , vol. 38, no. 5, pp. 146:1–146:12, 2019.
  * [32] K. Fukunaga and L. D. Hostetler, “The estimation of the gradient of a density function, with applications in pattern recognition,” _IEEE Transactions on Information Theory_ , vol. 21, no. 1, pp. 32–40, 1975.
  * [33] A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” _arXiv preprint arXiv:1703.07737_ , 2017.
  * [34] H. W. Kuhn, “The hungarian method for the assignment problem,” _Naval Research Logistics Quarterly_ , vol. 2, no. 1-2, pp. 83–97, 1955.
  * [35] S. Koch, A. Matveev, Z. Jiang, F. Williams, A. Artemov, E. Burnaev, M. Alexa, D. Zorin, and D. Panozzo, “ABC: A big CAD model dataset for geometric deep learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 9601–9611.
  * [36] G. Sharma, R. Goyal, D. Liu, E. Kalogerakis, and S. Maji, “Csgnet: Neural shape parser for constructive solid geometry,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018, pp. 5515–5523.
  * [37] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese, “3d semantic parsing of large-scale indoor spaces,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.