# CloSe: A 3D Clothing Segmentation Dataset and Model

Dimitrije Antić 1 Garvita Tiwari 2,3,4 Batuhan Ozcomlekci 2 Riccardo Marin 2,3
Gerard Pons-Moll 2,3,4
1University of Amsterdam, Netherlands 2University of Tübingen, Germany
3Tübingen AI Center, Germany
4Max Planck Institute for Informatics, Saarland Informatics Campus, Germany
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>tuebingen.de,
{riccardo.marin<EMAIL_ADDRESS>
###### Abstract

3D Clothing modeling and datasets play crucial role in the entertainment,
animation, and digital fashion industries. Existing work often lacks detailed
semantic understanding or uses synthetic datasets, lacking realism and
personalization. To address this, we first introduce CloSe-D: a novel large-
scale dataset containing 3D clothing segmentation of $3167$ scans, covering a
range of $18$ distinct clothing classes. Additionally, we propose CloSe-Net,
the first learning-based 3D clothing segmentation model for fine-grained
segmentation from colored point clouds. CloSe-Net uses local point features,
body-clothing correlation, and a garment-class and point features-based
attention module, improving performance over baselines and prior work. The
proposed attention module enables our model to learn appearance and geometry-
dependent clothing prior from data. We further validate the efficacy of our
approach by successfully segmenting publicly available datasets of people in
clothing. We also introduce CloSe-T, a 3D interactive tool for refining
segmentation labels. Combining the tool with CloSe-Net in a continual learning
setup demonstrates improved generalization on real-world data. Dataset, model,
and tool can be found at https://virtualhumans.mpi-inf.mpg.de/close3dv24/.

##  1 Introduction

Clothing plays a vital role in shaping our identity. Our dressing choices
contribute to our representation, conveying cultural traits, religious
beliefs, geographical origin, or mood. In light of the recent attention on the
expressiveness of avatarization processes, Computer Vision scholars dedicate
tremendous efforts in acquiring, modeling, and comprehending digital clothing,
enabling uncountable applications: from digital fashion to AR/VR; from home
entertainment to industrial-scale content creation.

Semantic understanding of humans in clothing from 2D images has seen
significant progress [16, 40, 17, 57], but the lack of geometrical information
is an obstacle for AR/VR applications, where actions occur in a 3D world.
Despite the development of 3D data capture techniques [1, 3, 7] and the
consequent abundant geometrical data of 3D people in clothing [12, 59, 29,
20], garment analysis and its semantic understanding remains an open problem.

The primary challenge lies in representing 3D digital garments and their
semantics. Certain existing approaches rely on synthetic clothing meticulously
designed by experts [9, 35] or involve expensive acquisitions [66, 23], where
accurate information comes at the expense of scalability. Some recent methods
address human and clothing models within a unified representation [48, 42, 14]
but lack a semantic understanding of the distinct parts. We posit that the
absence of large-scale and high-quality segmented datasets serves as a
fundamental barrier.

Consequently, the development of robust methods for understanding the 3D cloth
semantics has been hampered. Prior work like MGN-Seg [10] requires expensive
pipelines involving rendering, SMPL+D registration [24], 2D segmentation [16],
and hand-crafted clothing priors, making it time-consuming (15-20 minutes per
scan) and prohibitive for complex clothing items. In contrast, GIM3D [36] uses
a SotA part segmentation method trained on synthetic data, accounting only for
geometric information, namely location and normals. Both are limited by
3-class prediction: upper and lower garments and the human body, which
trivialize the problem. Considering the clothes variations in style and
textures, getting semantic information from 3D scans requires fast, accurate,
generalizable, and scalable method.

This work aims to fill this gap with a three-fold contribution. First, we
introduce CloSe-D, a large-scale dataset of people in clothing with fine-
grained segmentation labels, for a total of $3167$ scans and 18 garments
categories. To our knowledge, this is the first real-world dataset with such
fine-grained segmentation labels. Secondly, we use CloSe-D to train CloSe-Net,
a 3D clothing segmentation model to predict 18 distinct types of clothing. Our
model is based on two key intuitions: we correlate body parts with clothing
classes, leveraging the SMPL [28] body model(Sec. 4.1.2); and we address the
relationship between local geometric-appearance cues and clothing class in
segmentation using an attention module (Sec. 4.1.3), learning associations
between point features and clothing classes. We demonstrate a significant
improvement over baselines and prior works (Sec. 5.2 and Sec. 5.2.1). Finally,
to achieve the most comprehensive generalization possible, we develop CloSe-T,
an interactive 3D tool to provide quick human feedback/annotation. This tool
allows users to refine segmentation predictions and rectify segmentation
labels. Our tool can also be integrated with CloSe-Net, where the feedback is
backpropagated, and the network is fine-tuned in a continual learning setup(
Sec. 5.3). We leverage this tool to prepare high-quality segmentation training
data CloSe-D. We also use the tool in conjunction with CloSe-Net to prepare
high-quality segmentation labels on public datasets, which we release as
CloSe-D++.

In summary, our contributions are:

  * •

CloSe-D: A high-quality fine-grained clothing segmentation dataset containing
segmentation labels for $3167$ scans, covering $18$ clothing classes.

  * •

CloSe-Net: A human prior and clothing classes attention-based 3D clothing
segmentation method that outperforms baselines and prior work.

  * •

CloSe-T: A 3D interactive tool to refine the model in a continual learning
framework, improving generalization to new datasets.

  * •

CloSe-D++: Fine-grained semantic segmentation for a subset of publicly
available real-world datasets.

We release our data, model, and tool for further research.

##  2 Related Work

Our work includes a new 3D clothing dataset, a 3D segmentation model, and an
interactive refinement method, thus the related work covers these three areas.

###  2.1 3D Clothing Datasets

The rise of learning-based digital fashion and virtual try-on led to the
creation of image-based clothing datasets with semantic labels [17, 27].
However, these datasets lack pose variation, are 2D and mostly frontal, and
are unsuitable for learning overall human/clothing shape, deformations, and
3D/4D models.

3D/4D Clothing datasets can be grouped into two categories: Synthetic and
Captured ones. Synthetic datasets [35, 51, 9, 30] are obtained using physics-
based simulation software [4], their generation requires expert intervention
and mostly contain geometric information without texture. Moreover, this may
not scale for complex clothing and multiple layers, often resulting in non-
realistic deformations.

On the other hand, the accessibility of capturing datasets has increased
recently, courtesy of advancements in 3D/4D capture systems [1, 7]. THuman1-4
[59, 45, 64, 43] propose medium to high-quality static scans of individuals in
limited clothing styles/variations with limited pose variations. BUFF/CAPE
[60, 29, 37], and HuMMan [12] provide dynamic scans of subjects in different
clothing items. These datasets contain point clouds, occasionally texture, and
SMPL parameters, but lack clothing semantic segmentation. Prior work such as
MGN [10], SIZER [47], GIM3D [36] provide coarse 3D clothing segmentation
labels, containing only three categories, namely _upper garment, lower
garment, and body_. Deepfashion3D [66] is a dataset of high-quality and
diverse 3D Clothing items, scanned on mannequins consisting of 10 clothing
classes with keyline annotations and SMPL pose parameters. However, the
clothing items are scanned separately and cannot be used to create a fully
clothed person. In contrast, our dataset is more realistic, featuring a
broader range of clothing classes.

None of the existing real-world clothing datasets contains fine-grained
clothing segmentation labels of clothed humans. In CloSe-D, we provide clean
and fine-grained clothing segmentation labels of scans along with colored
point clouds, and SMPL [28] parameters. We compare CloSe-D with existing real-
world datasets in Table 1.

Dataset |  $\\#$ scans | Segmentation | Garment Class | Texture |
---|---|---|---|---|---
MGN [10] |  $\sim$300 |  1F5F8 | 3 |  1F5F8 |
SIZER [47] |  $\sim$2000 |  1F5F8 | 3 |  1F5F8 |
DeepFashion3D [66] | 2078(563) |  1F5F8 | 10 |  1F5F8 |
THuman [59, 45, 63] |  $\sim$1000 |  2718 | - |  1F5F8 |
CloSe-D |  $\sim$3000 |  1F5F8 | 18 |  1F5F8 |
CloSe-D++ |  $\sim$( +1000 ) |  1F5F8 | 18 |  1F5F8 |

Table 1: Compared to current 3D clothing(real and static) datasets, CloSe-D is
the first extensive dataset featuring fine-grained clothing segmentation
labels and a variety of clothing items.

###  2.2 3D Clothing Segmentation

2D clothing segmentation and human parsing have been extensively studied.
Several prior works [56, 62, 25, 16, 55] propose learning-based human parsing
in images, trained using 2D clothing datasets such as [15, 17]. These methods
exploit human body parts and pose information to improve accuracy and
generalization. However, they cannot be used for 3D segmentation, as they are
not trained to produce multi-view consistent results, and lifting labels from
2D to 3D requires a slow optimization process.

MGN-Seg [10] is an optimization-based 3D clothing segmentation method that
involves registering scans to SMPL+D [10, 11], applying PGN [16] for 2D
segmentation of multi-view renderings of a mesh. These 2D segmentation images
are then lifted back to 3D by solving GrabCut [41] in SMPL-UV space with a
handcrafted clothing class based prior. This takes roughly $20$ minutes per
scan, is limited to 3 clothing classes, and requires expensive SMPL+D
registration and hand-crafted clothing prior. On the other hand, CloSe-Net
only needs colored point cloud, SMPL($\boldsymbol{\theta},\boldsymbol{\beta}$)
parameters, and clothing classes present in the scan and learns clothing prior
from data.

3D clothing segmentation from a point cloud resembles a part segmentation
setup. There are various existing works in learning-based 3D part
segmentation, such as the seminal PointNets [13, 38] and self-attention-based
PointTransformer [61]. More recent methods like KPConv [46] use kernel points
defined in Euclidean space to apply convolution. SGPN [52] uses a single
network to predict point grouping proposals and a corresponding semantic class
for each proposal. Recent SotA methods [18] like DGCNN [53] introduces
EdgeConv, a differentiable layer representing data on graphs dynamically
computed in each network layer. DeltaConv [54] uses anisotropic convolution
layers and introduces a convolution layer that combines geometric operators
from vector calculus to construct anisotropic filters on point clouds.
Existing works lack evaluations on clothing segmentation and do not integrate
human-clothing-specific knowledge. In contrast, CloSe-Net employs a DGCNN-
based point-feature module, leveraging human body and clothing priors for
improved performance in clothing segmentation. We also show in our experiments
that SotA DGCNN and DeltaConv, have limitations when applied to this task.

Most relevant to our work is GIM3D [36], which uses SotA part segmentation
methods to segment directly from a point cloud, but similar to [10, 47] it is
limited to three classes. To our knowledge, no existing method directly
operates on a colored 3D point cloud/mesh to generate fine-grained 3D clothing
segmentation. GIM3D+ [33] extends GIM3D, by including more diverse fabrics,
sizes, and poses in the dataset, but is still limited to three classes and
doesn’t consider texture information.

###  2.3 3D Interactive Segmentation and Refinement

Tools for interactive segmentation refinement are crucial for developing
large-scale datasets and incorporating human feedback for improving
segmentation. In 2D, classical methods like GrabCut [41] and a more recent
interactive segmentation refinement [44] use human feedback input to improve
segmentation. In 3D, 3D-GrabCut [31] and mesh cutting tools [19] are used for
foreground/background segmentation. However, these methods do not leverage
learned neural features and initial predictions. Recent works like [39, 22]
are related to 3D interactive part segmentation annotation tools. iSeg3D [39]
utilizes primitive-aware embedding and doesn’t consider color information of
data. InterObject3D [22] proposes a generalizable pipeline for interactive 3D
segmentation, where the network is refined based on user clicks for a given
target domain. Yet, none of these methods is designed for clothing
segmentation and doesn’t consider catastrophic forgetting. We introduce
CloSe-T, a novel, fast, and easy-to-use interactive tool, tailored for the
clothing domain. We propose the network refinement in a continual learning
setup [50], such that the network not only performs well on the target domain
but also learns from it, improving generalization. Prior works like [26, 21]
introduce weighted loss term and EWC(elastic weight consolidation) to avoid
catastrophic forgetting. We use [26] based weighted loss in our setup.

##  3 CloSe-D: 3D Clothing Segmentation Dataset

We introduce CloSe-D, a large-scale 3D clothing segmentation dataset that
contains labels for $3167$ scans comprising of $18$ garment classes. CloSe-D
consists of scans from two kinds of sources: 1) CloSe-Di: Scans captured using
Treddy [7] scanner, and 2) CloSe-Dc: Scans from the commercial datasets, such
as twindom, renderpeople [6, 8, 7, 2]. For CloSe-Di we will release scans,
SMPL parameters, and segmentation labels, while for CloSe-Dc we will provide
SMPL parameters and segmentation labels only, due to license concerns. We show
examples from our dataset in Fig. LABEL:fig:teaser(left), and the details in
Table 2.

Ground Truth Segmentation Labels. To obtain the ground truth segmentation
labels, we adopt the pipeline utilized in [10, 47], as mentioned in Sec. 2.2.
However, unlike MGN-Seg [10], our pipeline does not require SMPL+D
registration, as we directly apply all the steps on scan and use [5] for
lifting labels to 3D. Due to inconsistent predictions across views and limited
generalization of 2D segmentation methods, the 3D segmentation labels may
contain noise. To address this, we manually refine the segmentation using
CloSe-T, which is explained in Sec. 4.2.

Data Class | T-shirt | Shirt | Vest | Coat | Jacket | Hoodies | Short-Pants | Pants | Skirts
---|---|---|---|---|---|---|---|---|---
CloSe-Di | 415 | 556 | 85 | 191 | - | 209 | 500 | 897 | 59
CloSe-Dc | 775 | 739 | 107 | 306 | 42 | 42 | 252 | 1404 | 34
Data Class | Dress | JumpS. | SwimS. |  UnderG. | Scarf | Hat | Shoes | Body | Hair
CloSe-Di | - | - | - | - | - | 114 | 1437 | 1455 | 1382
CloSe-Dc | 50 | 6 | 23 | 10 | 25 | 64 | 1686 | 1732 | 1717

Table 2: Number of scans per clothing class in CloSe-D (_i.e_., the union of
CloSe-Di and CloSe-Dc).

##  4 Method

\begin{overpic}[width=486.91699pt,unit=1bp,tics=20]{images/overview_blur.pdf}
\par\put(5.0,10.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathbf{p}_{i}=\\{\mathbf{x}_{i}|\mathbf{c}_{i}|\mathbf{n}_{i}\\}$} }}
\par\put(5.0,0.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt
$\mathrm{SMPL:(\boldsymbol{\theta},\boldsymbol{\beta})},\&\;\mathrm{{\mathrm{g}}}$}
}}
\par\put(127.0,102.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathrm{PointEncoder{}}$} }}
\par\put(283.0,116.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathrm{F^{\mathrm{p}}_{i}}$} }}
\put(225.0,90.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathrm{\mathbf{p^{\prime}}^{2}_{i}}\in\mathbb{R}^{l}$} }}
\put(230.0,50.5){\makebox(0.0,0.0)[]{\rotatebox{90.0}{Attention}}}
\par\par\put(132.0,50.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathbf{G}\in\mathbb{R}^{18\times l}$} }}
\put(283.0,60.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathrm{F^{\mathrm{c}}_{i}}$} }}
\par\put(283.0,20.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{74.52539pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathrm{F^{\mathrm{b}}_{i}}$} }}
\par\put(134.0,10.0){\pgfsys@fill@opacity{0.0}\hbox{\pagecolor{white}\parbox{124.21405pt}{\pgfsys@fill@opacity{1}
\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\hskip
1.0pt $\mathrm{BodyEncoder}$} }}
\par\put(389.0,60.0){\makebox(0.0,0.0)[]{\rotatebox{90.0}{Segmentation
Decoder}}} \put(455.0,10.0){\makebox(0.0,0.0)[]{$y_{i}$}} \par\end{overpic}
Figure 1: CloSe-Net: Given a colored point cloud
$\mathbf{P}=\\{\mathbf{p}_{i}\ldots\mathbf{p}_{n}\\}$ with SMPL parameters
($\boldsymbol{\theta},\boldsymbol{\beta}$), and clothing classes
(${\mathrm{g}}$) detected in the scan, where
$\mathbf{p}_{i}=\\{\mathbf{x}_{i}|\mathbf{c}_{i}|\mathbf{n}_{i}\\}$ represent
point location, color and normal of a point, CloSe-Net predicts fine-grained
per-point segmentation labels. (a) Point Encoder(Sec. 4.1.1) takes
$\mathbf{P}$, as input and predicts per-point features $F^{\mathrm{p}}$. (b)
Clothing Encoder(Sec. 4.1.3) consists of a learnable codebook $G$ and an
attention module, which predicts $F^{\mathrm{c}}$, based on per-point feature
$\mathrm{\mathbf{p^{\prime}}^{2}_{i}}$ and $G$. This
$\mathrm{\mathbf{p^{\prime}}^{2}_{i}}$ is intermediate feature of Point
Encoder. (c) Body Encoder(Sec. 4.1.2), finds per-point canonical vertex in
SMPL template, given SMPL $\boldsymbol{\theta}$, $\boldsymbol{\beta}$
parameters. (d) Finally, the Segmentation Decoder(Sec. 4.1.4) takes
$F^{\mathrm{p}},F^{\mathrm{c}},F^{\mathrm{b}}$ and predicts segmentation
labels, $y_{i}$ for $i^{\mathrm{{th}}}$ point. Solid boxes in model are
learnable, while others are fixed.

In this section, we introduce CloSe-Net (Sec. 4.1), a 3D clothing segmentation
model that predicts fine-grained clothing labels from a colored point cloud.
Additionally, we present CloSe-T (Sec. 4.2), a 3D interactive tool used for
creating high-quality segmentation labels of CloSe-D. We demonstrate the
utility of CloSe-T in enhancing the generalization of our model on real-world
datasets.

###  4.1  CloSe-Net: 3D Clothing Segmentation Network

Overview. CloSe-Net predicts fine-grained clothing segmentation labels
directly from colored point clouds, given SMPL parameters and clothing classes
as input. As shown in Fig. 1, CloSe-Net consists of four modules: Point
Encoder, Body Encoder, Clothing Encoder, and Segmentation Decoder. Previous
methods (e.g., MGN-Seg [10]) manually define clothing priors, leading to poor
generalization across garment styles. In contrast, our model learns clothing
priors by establishing the correlation between body parts and local clothing,
utilizing the Body Encoder, and understanding the connection between point
features and clothing class through the Clothing Encoder. As a result, our
approach learns a prior that incorporates the garment’s style, body
information, and the combined local geometric and appearance features.

Input/Output. CloSe-Net takes a point cloud $\mathbf{P}\in\mathbb{R}^{n\times
9}$ as input, consisting of $n$ points, denoted as
$\mathbf{p}_{i}=\\{\mathbf{x}_{i}|\mathbf{c}_{i}|\mathbf{n}_{i}\\}$, where
$\mathbf{x}_{i}\in\mathbb{R}^{3}$ represent Euclidean coordinates,
$\mathbf{c}_{i}\in\mathbb{R}^{3}$ represent per-point colors and
$\mathbf{n}_{i}\in\mathbb{R}^{3}$ represent normals. CloSe-Net predicts per-
point segmentation labels, $y_{i}\in\\{1\ldots K\\},\;$ where $K$ is the
number of classes.

As CloSe-Net incorporates human body prior and clothing class-based attention
module, our method also needs SMPL ($\boldsymbol{\theta},\boldsymbol{\beta}$)
parameters and clothing classes (${\mathrm{g}}$) of the scan. We use SMPL
registration library [11, 24] to obtain SMPL parameters. For clothing class
labels, we render a single viewpoint of the scan and infer the clothing
classes using a SotA human parsing network [40].

####  4.1.1 Point Encoder

Semantic/part segmentation of a point cloud needs meaningful local and global
geometric features [18]. Following this, we implement our Point Encoder,
$f_{\mathrm{point}}$ using SotA _EdgeConv_ based architecture, called DGCNN
[53]. DGCNN operates on a point cloud by constructing a directed graph
$\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}\in\\{1,\ldots
n\\}$ and $\mathcal{E}\subseteq\mathcal{V}\times\mathcal{V}$. The edges
$\mathcal{E}$ are obtained using $k$-nearest neighbors of
$\mathbf{p}_{i}\in\mathbb{R}^{F}$, where $F=9$ for first layer, and $64$ for
subsequent layers. We then calculate per-point features using Edge Convolution
given in Eq. (1), where $h_{\theta}$ is the learnable edge feature layer.

$\mathbf{p^{\prime}}_{i}=\max_{j:(i,j)\in\mathcal{E}}\;h_{\theta}(\mathbf{p}_{i},\mathbf{p}_{j})\;$
(1)

Similar to [53], we use 3 EdgeConv layers followed by an MLP
($f_{\mathrm{MLP}}$) to learn a global encoding. This results in multi-scale
per-point features given by
$F^{\mathrm{p}}_{i}=\\{\mathbf{p^{\prime}}^{s}_{i}|\mathbf{p^{\prime}}_{\mathrm{g}}\\}$,
where $s=\\{0,1,2\\}$, $\mathbf{p^{\prime}}^{s}_{i}\in\mathbb{R}^{l}$ is per-
point feature learned by $s^{\mathrm{th}}$ EdgeConv layer and
$\mathbf{p^{\prime}}_{\mathrm{g}}\in\mathbb{R}^{1024}$ is a global encoding of
the point cloud. $\mathbf{p^{\prime}}_{\mathrm{g}}$ is obtained using an MLP,
$\mathbf{p^{\prime}}_{\mathrm{g}}=f_{\mathrm{MLP}}(\mathbf{P^{\prime}})$,
where
$\mathbf{P^{\prime}}=(\mathbf{p^{\prime}}_{0},\dots,\mathbf{p^{\prime}}_{n})\in\mathbb{R}^{n\times(l+l+l)},\mathbf{p^{\prime}}_{i}=\\{\mathbf{p^{\prime}}^{0}_{i}|\mathbf{p^{\prime}}^{1}_{i}|\mathbf{p^{\prime}}^{2}_{i}\\}$
is the concatenation of intermediate per-point features.

####  4.1.2 Body Encoder

We incorporate the correlation between body parts and clothing class using
SMPL mesh, given by $M(\boldsymbol{\beta},\boldsymbol{\theta})$ and SMPL
template mesh $\mathbf{T}$. In particular, for every point $\mathbf{x}_{i}$ in
the input point cloud, we find the index ($j$) of the nearest vertex on SMPL
mesh, given by $M_{j}(\boldsymbol{\beta},\boldsymbol{\theta})$. We then find
the corresponding vertex location in SMPL template $\mathbf{T}$. In this way,
we associate each point in the point cloud with fine-grained semantic
information about the human body. This module is not learnable and only
requires a nearest-neighbor search in $\mathbb{R}^{n\times 3}$. We represent
the encoded body feature as $F^{\mathrm{b}}_{i}$ = $\mathbf{T}_{j}$ and call
this a Canonical Body Encoder.

####  4.1.3 Clothing Encoding and Class-based Attention

Clothing classification is ambiguous due to its subjective nature (_e.g_.,
think about the difference between jackets and coats). To address this, we use
a learnable codebook $\mathbf{G}\in\mathbb{R}^{18\times l}$. Our model learns
distinct latent vectors [49] for each clothing class in an auto-decoding
manner [34]. This enables the model to acquire per-clothing attributes and
relevant characteristics for segmentation. The learned codebook is fixed
during inference. Compared to a non-learnable binary/one-hot encoding-based
representation, the learnable codebook yields better performance in
challenging regions, like clothing boundaries and uncommon clothing items like
jumpsuits(see Sec. 5.2.3). The key idea of our model is learning an explicit
association between per-point features and clothing class. We implement this
using an attention module, where we compute how much each point feature
attends to a clothing feature. The attention between a point feature and a
specific clothing latent code increases when that particular clothing is
present at the query point.
We define query vector using per-point EdgeConv features
$\mathbf{p^{\prime}}^{s}_{i},$ $s$ = $2$, and key-value pair using the
learnable codebook $\mathbf{G}$ and define the attention mechanism in Eq. (2),
where $\circ$ is masking operator. Simply using
$\mathbf{p^{\prime}}^{s}_{i}\times\mathbf{G}$, would also yield features for
clothing items not present in the scan. This might result in learning spurious
correlations between features and labels. To avoid this, we mask the key
matrix $\mathbf{G}$, using a binary encoding $\mathbf{g}$ of length $K=18$,
where $g_{j}=1$, if $j^{\mathrm{th}}$ class is present in the scan, and
$g_{j}=0$ otherwise. Our model is steered towards learning the clothing-
specific prior, leveraging fine-grained local point features and clothing
latent codes.

$F^{\mathrm{c}}_{i}=\mathrm{softmax}(\mathbf{p^{\prime}}^{s}_{i}\times(\mathbf{g}\circ\mathbf{G})^{T})\mathbf{G}.$
(2)

####  4.1.4 Segmentation Decoder

Finally, we concatenate all the features
$F^{\mathrm{all}}_{i}=\\{F^{\mathrm{p}}_{i}|F^{\mathrm{b}}_{i}|F^{\mathrm{c}}_{i}\\}$,
and pass it through a segmentation decoder $f_{\mathrm{dec}}$, which predicts
per-point segmentation labels. The network $f_{\mathrm{dec}}$ is implemented
as an MLP.

Loss. We train CloSe-Net with the cross-entropy loss in Eq. (3), where
$\hat{y}^{k}_{i}$ is the true label, $y^{k}_{i}$ is the predicted probability
of the $k^{\mathrm{th}}$ class, and $K$ is the number of classes.

$\mathcal{L}_{\mathrm{CE}}=-\sum_{k=1}^{K}\hat{y}^{k}_{i}\log({y}_{i}^{k})$
(3)

###  4.2 CloSe-T

In the previous section, we presented our method which learns clothing prior
from data for improved generalization over garment styles, appearance and
categories. However, we foresee that a variety of clothes often shows unique
characteristics that are difficult to catch statistically, especially with the
datasets available at the present date. For this reason, we introduce CloSe-T,
a fast-interactive tool to streamline the label refinement process. It
provides a graphical interface built explicitly for clothing segmentation and
relies on Open3D library [65], offering a broad set of functionalities
(_e.g_., points selection, labels updating, segmentation prediction, model
refinement). We use CloSe-T to refine the training dataset, and to improve our
network generalization on publicly available dataset by backpropagating user
feedback in a continual learning setup [50].

Continual Learning CloSe-Net Refinement. Let $y^{k}_{i}$ be the predicted
probability of the $k^{\mathrm{th}}$ class label of $i^{\mathrm{th}}$ point in
given pointcloud, and $\hat{y}^{k}_{i}$ be the correct segmentation label
provided by user using CloSe-T. We define the loss as:

$\mathcal{L}_{\mathrm{refine}}=\lambda_{\mathrm{c}}\underset{i\in\mathcal{C}}{\mathcal{L}_{\mathrm{CE}}}(y_{i}^{k},\hat{y}_{i}^{k})+\lambda_{\mathrm{f}}\underset{{i\in\mathcal{F}}}{\mathcal{L}_{\mathrm{CE}}}(y_{i}^{k},\hat{y}_{i}^{k})+\lambda_{\mathrm{w}}{\mathcal{L}_{\mathrm{W}}}(\theta,\theta^{\prime}),$
(4)

where $\mathcal{C}$ is the set of indices of point cloud corrected by the user
and $\mathcal{F}$ is the set of remaining points. $\mathcal{L}_{\mathrm{CE}}$
is cross-entropy loss defined in Eq. (3), $\mathcal{L}_{\mathrm{W}}$ is weight
regularization, penalizing weights difference between refined model
($\theta^{\prime}$) and original pre-trained model ($\theta$).
$\lambda_{\mathrm{c}},\lambda_{\mathrm{f}},\lambda_{\mathrm{w}}$ are the
weights associated with each loss term. We only fine-tune the last layer of
the segmentation decoder and MLP of the Point Encoder. Following [26], we use
$\lambda_{\mathrm{c}}\ll\lambda_{\mathrm{f}}$, in order to avoid catastrophic
forgetting.

##  5 Experiments and Results

We describe the experiment setup in Sec. 5.1, evaluate our proposed CloSe-Net
and compare it with SotA part segmentation in Sec. 5.2, and with prior methods
in Sec. 5.2.1. We analyze each module of CloSe-Net in Sec. 5.2.3 and discuss
the attention-based clothing prior in Sec. 5.2.2. Additionally, we present
results on publicly available clothing datasets in Sec. 5.3 and showcase
improvements in generalization through CloSe-T-based refinement.

###  5.1 Experimental Setup

Implementation Details: We use the official DGCNN implementation [53] with 3
EdgeConv layers (feature-length $l=64$ and
$|\mathbf{p^{\prime}}_{\mathrm{global}}|=1024$). For the clothing
codebook($\mathbf{G}$), we use $l=64$. The clothing-class-based attention
module is based on multi-head attention. The train-val-test splits of CloSe-D
are 2652/265/270.

Error Metric. Intersection over Union (IoU) is a popular metric for
segmentation, quantifying the overlap between predicted and ground truth
labels. We consider both per-class IoU and mean IoU ($\mathrm{IoU_{mean}}$)
over all classes.

###  5.2 3D Clothing Segmentation

We evaluate CloSe-Net on test split of CloSe-D, both qualitatively (Fig. 13)
and quantitatively (Table 3). Given the similarity between clothing
segmentation and part segmentation tasks [36], we compare our method with SotA
3D part segmentation models trained on CloSe-D. For this, we employ methods
[18] like DGCNN [53] and DeltaConv [54]. As observed in Fig. 13-middle, DGCNN
and DeltaConv struggle with multi-layer clothing. This arises from the absence
of clothing-related information in the model and its inclination to overfit to
single-layer scans, which constitute the majority of the dataset. On the other
hand, our model incorporates clothing information via the learned distinct
codebook and hence predicts correct labels with crisp boundaries. Moreover, as
seen in Fig. 13-top both baseline methods inadequately segment less common
classes, such as hats. We also observe texture bias in both baseline methods
(Fig. 13-bottom); both networks fail to predict the same class for nearby
points with different texture colors. We attribute this to the codebook and
masked attention module in our model, which enables the network to learn
distinct features for each clothing and avoid learning spurious correlations
between point features and absent classes.

Method | Mean | T-shirt | Shirt | Vest | Coat | Jacket | Hoodies | Short-Pants | Pants | Skirts | Dress | JumpS. | SwimS. | UnderG. | Scarf | Hat | Shoes | Body | Hair
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
DGCNN [53] | 87.11 | 87.88 | 81.02 | 90.58 | 81.60 | 96.16 | 97.46 | 94.60 | 82.50 | 96.44 | 73.61 | 76.67 | 98.89 | 99.26 | 73.33 | 95.19 | 79.47 | 80.45 | 82.83
DeltaConv [54] | 84.78 | 87.22 | 73.56 | 84.68 | 80.06 | 98.52 | 96.99 | 89.58 | 78.37 | 94.11 | 67.08 | 77.04 | 99.26 | 99.26 | 73.33 | 95.19 | 72.98 | 76.69 | 82.13
Ours | 91.23 | 95.47 | 92.94 | 98.86 | 90.12 | 99.23 | 99.43 | 98.32 | 85.96 | 98.12 | 79.11 | 77.73 | 99.78 | 99.96 | 73.23 | 97.72 | 82.96 | 85.76 | 87.49

Table 3: We quantitatively compare the results of our method SotA part
segmentation methods, DGCNN [53] and DeltaConv [54]. We report IoU for every
class and mean over all the classes($\mathrm{IoU_{mean}}$).

\begin{overpic}[width=233.52304pt,unit=1bp,tics=10]{images/baseline/camready/axyz_WMan0306-HD2-O01P01-S_comparison.pdf}
\put(15.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Input}}}}
\put(50.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{DGCNN~{}\cite[cite]{[\@@bibref{Number}{dgcnn}{}{}]}}}}}
\put(100.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{DeltaConv~{}\cite[cite]{[\@@bibref{Number}{Wiersma2022DeltaConv}{}{}]}}}}}
\put(155.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{{Ours}}}}}
\put(205.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{GT}}}}
\end{overpic}

Figure 2: Comparison with SotA part segmentaiton models: DGCNN [53] and
DeltaConv [54]. Our model predicts accurate clothing classes and finer
boundaries in complex scans. This can be attributed to our model’s utilization
of local point features, body priors, and clothing class-based attention
features.

####  5.2.1 Comparison with Prior Work

We compare CloSe-Net with prior 3D clothing segmentation methods like MGN-Seg
[10] and GIM3D [36]. Since both GIM3D and MGN-Seg [10] predict 3 classes, we
merge the predictions of our model into the respective classes. For GIM3D, we
only compare with PointNet++ [38], as it is the only one authors provided to
us. We observe from Table 4 that CloSe-Net largely improves over prior work.
Moreover, CloSe-Net takes approximately $5-6\;\mathrm{seconds}$ to infer the
segmentation labels for the whole scan($270k$ vertices) on 12 GiB GPU
(3080Ti). Whereas for MGN-Seg [10] it takes roughly $\sim 20$ minutes to get
the segmentation labels for SMPL+D mesh ($27k$ vertices).

We further provide qualitative results in Fig. 3. We observe that MGN-Seg [10]
is not able to generate precise labels at boundaries, _e.g_. at the leg. This
is because the prior designed for lower garments comes from a fixed template
of “long pants”, whereas the pant in this scan is smaller than the pre-defined
template. We also observe that sometimes the handcrafted features are not able
to correct segmentation errors due to texture bias and inconsistent multiview
prediction of 2D human parsing [16, 40]. This is visible in the right hand of
the scan. We notice that GIM3D fails near boundaries. This stems from the fact
that normal information is not sufficient to distinguish between different
clothing, especially in the case of real-world scans, where normals can be
noisy. On the other hand, our model results in accurate boundaries as it takes
texture information into account as well and doesn’t rely on any pre-defined
prior.

\begin{overpic}[width=233.52304pt,unit=1bp,tics=20]{images/baseline/134911533743174_prior_comparison.pdf}
\put(10.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Input}}}}
\put(50.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{MGN-
Seg~{}\cite[cite]{[\@@bibref{Number}{bhatnagar2019mgn}{}{}]}}}}}
\put(100.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{GIM3D~{}\cite[cite]{[\@@bibref{Number}{gim3d}{}{}]}}}}}
\put(155.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{{Ours}}}}}
\put(210.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{GT}}}}
\end{overpic} Figure 3: Comparison with MGN-Seg [10] and GIM3D [36].

Dataset | MGN [10] | GIM3D [36] | Ours
---|---|---|---
CloSe-D-Test | 88.88 | 72.04 | 92.47

Table 4:  Comparison with MGN [10] and GIM3D [36] on three class(upper, lower
and body) segmentation.

####  5.2.2 Learned Clothing Prior

A good clothing prior is crucial for segmentation. In MGN-Seg [10], a geodesic
distance-based prior was manually crafted. This approach lacks scalability for
new clothing and struggles with varying shapes within the same class (e.g.,
different jacket lengths or shirt sleeve styles _etc_.). Interestingly, our
model’s attention-based clothing encoder learns clothing prior from point
features ($F^{\mathrm{p}}$) and a garment codebook ($F^{\mathrm{c}}$). This is
visualized in Fig. 4, where attention for the $j^{\mathrm{th}}$ garment class
at point $i$ is calculated as
$\mathbf{p^{\prime}}^{k}_{i}\times\mathbf{G}_{j}$. As compared to MGN-Seg [10]
prior, this is not manually defined and also incorporates the local properties
of the clothing.

\begin{overpic}[width=198.73946pt,unit=1bp,tics=10]{images/results/10014_2425_attention.pdf}
\put(13.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Coat}}}}
\put(63.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Pants}}}}
\put(110.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Shirt}}}}
\par\put(160.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Hoodies}}}}
\end{overpic}

\begin{overpic}[width=198.73946pt,unit=1bp,tics=10]{images/results/10074_7292_v2_attention.pdf}
\scriptsize\put(15.0,76.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Hat}}}}
\put(60.0,76.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{TShirt}}}}
\put(109.0,76.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Short
Pants}}}}
\par\put(157.0,76.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Hoodies}}}}
\par\par\put(195.0,12.0){0} \put(195.0,40.0){1} \end{overpic}

Figure 4: Clothing prior learned using attention module: Attention module in
Clothing Encoder, learns a robust clothing prior based on point features. Here
we visualise the attention of point feature on different clothing class.

####  5.2.3 Ablation

We experimentally validate and discuss the design choices of each module of
CloSe-Net in Table 5 and Fig. 5.

Point Encoder. We experiment with two part segmentation models: DGCNN [53] and
DeltaConv [54] as Point Encoder. Table 5 shows the DGCNN-based model
significantly outperforms the DeltaConv-based one. This behavior is similar to
the standalone DGCNN and DeltaConv, as shown in Table 3. DGCNN surpasses
DeltaConv by learning feature spaces where semantically similar features are
closely clustered. Hence, we use DGCNN as Point Encoder in our model.

Body Encoder. We explore different body encodings: _Canonical Body Encoder_ ,
$F^{\mathrm{b}}$ (Sec. 4.1.2), and a fusion of $F^{\mathrm{b}}$ with a coarse
feature encoder based on COAP [32], resulting in the _Hybrid Body Encoder_.
The lack of Body Encoder in the model leads to mislabeled regions in
improbable locations, such as a t-shirt label appearing in the skirt region.
These mislabels are evident as patches in Fig. 5-top. The _Hybrid Body
Encoder_ learns vertex and body part associations with the clothing, but
results in smudged boundaries(see skirt and legs in Fig. 5-top ). This is
because the hybrid model contains body-part features, so it tends to associate
the same labels to all the points in a body part if there is no significant
difference in geometry or appearance. Our proposed Body Encoder uses fine-
grained correspondence and establishes accurate correlations between clothing
labels and body locations.

Clothing Encoder. We compare our attention-based clothing encoder, with a
binary encoding-based one. In binary encoding, we use $\mathbf{g}$, instead of
$F^{\mathrm{c}}$. The attention-based model boosts performance and also learns
garments prior from data. Binary encoding is not consistently effective in
predicting the correct garment, particularly when confronted with uncommon
clothing styles, as exemplified by the jumpsuit case in Fig. 5-bottom, similar
to models without Clothing Encoder. Moreover, without any Clothing Encoder,
models exhibit texture bias. Leveraging the learned clothing prior(Sec. 5.2.2)
significantly improves performance, and alleviates the mentioned problems.

Points Encoder |  Body Encoder |  Clothing Encoder |  $\mathrm{IoU_{mean}}$ $\uparrow$
---|---|---|---
DGCNN | Canonical | Attention | 91.23
Canonical | Binary | 90.41
Hybrid | Attention | 89.70
Hybrid | Binary | 89.68
$\times$ | Attention | 89.90
Canonical | $\times$ | 87.18
$\times$ | $\times$ | 87.10
DeltaConv | Canonical | Attention | 86.84
Table 5: Quantitative evaluation of the ablation study on different modules of
the proposed CloSe-Net model. The table shows the performance of the model
with different combinations of the Point Encoder, Body Encoder, and Clothing
Encoder.

\begin{overpic}[width=233.52304pt,unit=1bp,tics=10]{images/ablation/camready/top/renderpeople_rp_eve_posed_002_30k_comparison.pdf}
\put(15.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Input}}}}
\put(65.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{$\times$}}}}
\put(100.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{ +
Hybrid}}}}
\put(150.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{ +
Canonical}}}}
\put(205.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{GT}}}}
\end{overpic}

\begin{overpic}[width=233.52304pt,unit=1bp,tics=20]{images/ablation/camready/bottom/twindom_10k_133411517842814_comparison.pdf}
\put(15.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Input}}}}
\put(65.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{$\times$}}}}
\put(100.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{ +
Binary}}}}
\put(150.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{ +
Attention}}}}
\put(205.0,75.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{GT}}}}
\par\vspace{-1cm} \end{overpic}

Figure 5: Body Encoder(Top): As opposed to others, the proposed Body
Encoder(Canonical) is simple, generalizes to difficult poses, and produces
fine boundaries. Clothing Encoder(Bottom): Attention-based encoder and
codebook learn distinct garment features and prior, achieving accurate
segmentation prediction.

###  5.3 CloSe-D++

\begin{overpic}[width=118.25505pt,unit=1bp,tics=20]{images/public/public_3dhumans.pdf}
\put(30.0,65.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{3DHumans-
IIITH~{}\cite[cite]{[\@@bibref{Number}{Jinka2022}{}{}]}}}}}
\par\par\end{overpic}\begin{overpic}[width=118.25505pt,unit=1bp,tics=20]{images/public/public_thuman2.pdf}
\put(30.0,65.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{THuman2.0~{}\cite[cite]{[\@@bibref{Number}{thuman2}{}{}]}}}}}
\end{overpic}

Figure 6: Results of CloSe-Net on publicly available datasets [20, 59],
showing generalization capability of the model.

Our model generalizes well on real-world public datasets, showing good results
in Fig. 6. However, it exhibits blurry boundaries and texture bias for some
scans, as seen in Fig. 7 (middle). Generalizing to out-of-distribution real-
world datasets is challenging due to the vast variability of clothing styles
and few differences between classes.

To address this, we use the proposed CloSe-T in a continual learning
approach(Sec. 4.2). The goal is to improve performance over new datasets,
without catastrophic forgetting. We show the results of the original model and
fine-tuned model on a scan from THuman2 [59] in Fig. 7. After fine-tuning,
results on CloSe-D-test is a mean IoU of $90.35$, which is a small decrease
from the original model($91.23$) and still better than prior work and
baselines. We will provide the labels of publicly available datasets such as
THuman2,3 [59, 45], CAPE [29](textured-scans), 3DHumans-IIITH [20], a subset
of HuMMan [12] and we call this dataset CloSe-D++.

\begin{overpic}[width=243.46228pt,unit=1bp,tics=20]{images/public/THuman2_org_0089.pdf}
\put(30.0,95.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Input}}}}
\par\put(80.0,95.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{
Original Network}}}}
\par\put(160.0,95.0){\hbox{\pagecolor{white}\parbox{74.52539pt}{\scriptsize{Refined
Network}}}} \end{overpic} Figure 7: Improving CloSe-Net performance on
THuman2.0 [59] by fine-tuning the model on few samples from THuman2.0 using
CloSe-T.

##  6 Conclusions

We present a novel fine-grained 3D clothing segmentation model that works
directly on point clouds, and to train it we introduce a large-scale dataset
of people in diverse clothing items, poses, and with high-quality segmentation
labels. We incorporate human body information to improve the pose-
generalization of our model and introduce a novel garment class attention
module, which learns clothing prior from data, as opposed to hand-crafted
priors [10]. Our model outperforms prior work and baselines and generalizes to
public out-of-distribution datasets. We further introduce a continual
learning-based refinement strategy to improve the generalization of the model,
without catastrophic forgetting.

Limitations and Future Work To the best of our knowledge, this is the first
dataset and model for 3D clothing segmentation from colored point clouds,
which contains diverse and fine-grained segmentation labels. Future work may
broaden our work by adding more clothing items through CloSe-T, _e.g_., to
include a variety of cultural styles. Our approach necessitates the garment
class worn by the subject as network input, requiring a preprocessing step.
Potentially, this can be obviated by incorporating clothing prediction
directly within the network. Lastly, to enhance network generalization, we
have integrated the continual learning [26] framework, paving the way for
future exploration of recent strategies such as EWC [21].

Acknowledgments: Thanks to RVH, CVLab team, and reviewers for valuable
feedback. The project was made possible by funding from the Carl Zeiss
Foundation. This work is supported by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) - 409792180 (Emmy Noether Programme,
project: Real Virtual Humans), German Federal Ministry of Education and
Research (BMBF): Tübingen AI Center, FKZ: 01IS18039A. Gerard Pons-Moll is a
member of the Machine Learning Cluster of Excellence, EXC number 2064/1 -
Project number 390727645. Riccardo Marin has been supported by the European
Union’s Horizon 2020 research and innovation program under the Marie
Skłodowska-Curie grant agreement No 101109330.

## References

  * [1] 3dMD.
  * [2] AXYZ design.
  * [3] LumaAI Labs.
  * [4] Marvelous Designer.
  * [5] Agisoft Metashape: Reconstruction from Images.
  * [6] 3D People from Renderpeople.
  * [7] Treedy static scanner.
  * [8] Twindom 3D Scans.
  * Bertiche et al. [2019] Hugo Bertiche, Meysam Madadi, and Sergio Escalera.  CLOTH3D: clothed 3d humans.  _CoRR_ , abs/1912.02792, 2019.
  * Bhatnagar et al. [2019] Bharat Lal Bhatnagar, Garvita Tiwari, Christian Theobalt, and Gerard Pons-Moll.  Multi-garment net: Learning to dress 3D people from images.  In _ICCV_ , 2019.
  * Bhatnagar et al. [2020] Bharat Lal Bhatnagar, Cristian Sminchisescu, Christian Theobalt, and Gerard Pons-Moll.  Combining implicit function learning and parametric models for 3d human reconstruction.  In _European Conference on Computer Vision (ECCV)_. Springer, 2020.
  * Cai et al. [2022] Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, Fangzhou Hong, Mingyuan Zhang, Chen Change Loy, Lei Yang, and Ziwei Liu.  HuMMan: Multi-modal 4d human dataset for versatile sensing and modeling.  In _17th European Conference on Computer Vision, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part VII_ , pages 557–577. Springer, 2022.
  * Charles et al. [2017] R. Qi Charles, Hao Su, Mo Kaichun, and Leonidas J. Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 77–85, 2017.
  * Chen et al. [2022] Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J Black, Andreas Geiger, and Otmar Hilliges.  gdna: Towards generative detailed neural avatars.  _arXiv_ , 2022.
  * Gong et al. [2017] Ke Gong, Xiaodan Liang, Xiaohui Shen, and Liang Lin.  Look into person: Self-supervised structure-sensitive learning and A new benchmark for human parsing.  _CoRR_ , abs/1703.05446, 2017.
  * Gong et al. [2018] Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin.  Instance-level human parsing via part grouping network.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018.
  * Guo et al. [2019] Sheng Guo, Weilin Huang, Xiao Zhang, Prasanna Srikhanta, Yin Cui, Yuan Li, Matthew R.Scott, Hartwig Adam, and Serge Belongie.  The imaterialist fashion attribute dataset.  _arXiv preprint arXiv:1906.05750_ , 2019.
  * He et al. [2021] Yong He, Hongshan Yu, Xiaoyan Liu, Zhengeng Yang, Wei Sun, Yaonan Wang, Qiang Fu, Yanmei Zou, and Ajmal Mian.  Deep learning based 3d segmentation: A survey.  _CoRR_ , abs/2103.05423, 2021.
  * Ji et al. [2006] Zhongping Ji, Ligang Liu, Zhonggui Chen, and Guojin Wang.  Easy mesh cutting.  _Computer Graphics Forum_ , 25(3):283–291, 2006\.
  * Jinka et al. [2022] Sai Sagar Jinka, Astitva Srivastava, Chandradeep Pokhariya, Avinash Sharma, and P. J. Narayanan.  Sharp: Shape-aware reconstruction of people in loose clothing.  _International Journal of Computer Vision_ , 2022.
  * Kirkpatrick et al. [2017] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the National Academy of Sciences_ , 114(13):3521–3526, 2017.
  * Kontogianni et al. [2022] Theodora Kontogianni, Ekin Celikkan, Siyu Tang, and Konrad Schindler.  Interactive object segmentation in 3d point clouds, 2022.
  * Lähner et al. [2018] Zorah Lähner, Daniel Cremers, and Tony Tung.  Deepwrinkles: Accurate and realistic clothing modeling.  In _Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV_ , page 698–715, Berlin, Heidelberg, 2018. Springer-Verlag.
  * Lazova et al. [2019] Verica Lazova, Eldar Insafutdinov, and Gerard Pons-Moll.  360-degree textures of people in clothing from a single image.  In _2019 International Conference on 3D Vision (3DV)_ , pages 643–653. IEEE, 2019.
  * Li et al. [2017] Jianshu Li, Jian Zhao, Yunchao Wei, Congyan Lang, Yidong Li, and Jiashi Feng.  Towards real world human parsing: Multiple-human parsing in the wild.  _CoRR_ , abs/1705.07206, 2017.
  * Li and Hoiem [2018] Zhizhong Li and Derek Hoiem.  Learning without forgetting.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(12):2935–2947, 2018.
  * Liu et al. [2016] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In _Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  _ACM Trans. Graphics (Proc. SIGGRAPH Asia)_ , 34(6):248:1–248:16, 2015.
  * Ma et al. [2020] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J. Black.  Learning to Dress 3D People in Generative Clothing.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Ma et al. [2021] Qianli Ma, Jinlong Yang, Siyu Tang, and Michael J. Black.  The power of points for modeling humans in clothing.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Meyer and Do [2015] Gregory P. Meyer and Minh N. Do.  3d grabcut: Interactive foreground extraction for reconstructed 3d scenes.  In _3DOR@Eurographics_ , 2015.
  * Mihajlovic et al. [2022] Marko Mihajlovic, Shunsuke Saito, Aayush Bansal, Michael Zollhoefer, and Siyu Tang.  COAP: Compositional articulated occupancy of people.  In _Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Musoni et al. [2023] Pietro Musoni, Simone Melzi, and Umberto Castellani.  Gim3d plus: A labeled 3d dataset to design data-driven solutions for dressed humans.  _Graphical Models_ , 129:101187, 2023.
  * Park et al. [2019] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 165–174, 2019.
  * Patel et al. [2020] Chaitanya Patel, Zhouyincheng Liao, and Gerard Pons-Moll.  The virtual tailor: Predicting clothing in 3D as a function of human pose, shape and garment style.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2020.
  * Pietro Musoni [2022] Umberto Castellani Pietro Musoni, Simone Melzi.  GIM3D: A 3d dataset for garment segmentation.  _placeholder_ , 2022.
  * Pons-Moll et al. [2017] Gerard Pons-Moll, Sergi Pujades, Sonny Hu, and Michael Black.  Clothcap: Seamless 4d clothing capture and retargeting.  _ACM Transactions on Graphics, (Proc. SIGGRAPH)_ , 36(4), 2017.  Two first authors contributed equally.
  * Qi et al. [2017] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , page 5105–5114, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * Qian et al. [2021] Sucheng Qian, Liu Liu, Wenqiang Xu, and Cewu Lu.  iseg3d: An interactive 3d shape segmentation tool.  2021\.
  * Qin et al. [2020] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar Zaiane, and Martin Jagersand.  U2-net: Going deeper with nested u-structure for salient object detection.  page 107404, 2020.
  * Rother et al. [2004] C. Rother, V. Kolmogorov, and A. Blake.  Grabcut: Interactive foreground extraction using iterated graph cuts.  2004\.
  * Saito et al. [2021] Shunsuke Saito, Jinlong Yang, Qianli Ma, and Michael J. Black.  SCANimate: Weakly supervised learning of skinned clothed avatar networks.  In _Proceedings IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Shao et al. [2022] Ruizhi Shao, Zerong Zheng, Hongwen Zhang, Jingxiang Sun, and Yebin Liu.  Diffustereo: High quality human reconstruction via diffusion-based stereo using sparse cameras.  In _ECCV_ , 2022.
  * Sofiiuk et al. [2020] Konstantin Sofiiuk, Ilya Petrov, Olga Barinova, and Anton Konushin.  F-brs: Rethinking backpropagating refinement for interactive segmentation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, 2020.
  * Su et al. [2023] Zhaoqi Su, Tao Yu, Yangang Wang, and Yebin Liu.  Deepcloth: Neural garment representation for shape and style editing.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(2):1581–1593, 2023.
  * Thomas et al. [2019] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  _Proceedings of the IEEE International Conference on Computer Vision_ , 2019.
  * Tiwari et al. [2020] Garvita Tiwari, Bharat Lal Bhatnagar, Tony Tung, and Gerard Pons-Moll.  SIZER: A dataset and model for parsing 3D clothing and learning size sensitive 3D clothing.  In _ECCV_ , 2020.
  * Tiwari et al. [2021] Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll.  Neural-GIF: Neural generalized implicit functions for animating people in clothing.  In _ICCV_ , 2021.
  * van den Oord et al. [2017] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , page 6309–6318, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * Wang et al. [2023] Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu.  A comprehensive survey of continual learning: Theory, method and application, 2023.
  * Wang et al. [2018a] Tuanfeng Y. Wang, Duygu Ceylan, Jovan Popovic, and Niloy J. Mitra.  Learning a shared shape space for multimodal garment design.  _ACM Trans. Graph._ , 37(6):1:1–1:14, 2018a.
  * Wang et al. [2018b] Weiyue Wang, Ronald Yu, Qiangui Huang, and Ulrich Neumann.  Sgpn: Similarity group proposal network for 3d point cloud instance segmentation.  In _CVPR_ , 2018b.
  * Wang et al. [2019] Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon.  Dynamic graph cnn for learning on point clouds.  _ACM Transactions on Graphics (TOG)_ , 2019.
  * Wiersma et al. [2022] R Wiersma, A. Nasikun, E Eisemann, and K Hildebrandt.  Deltaconv: Anisotropic operators for geometric deep learning on point clouds.  _Transactions on Graphics_ , 41(4), 2022.
  * Yamaguchi et al. [2012] Kota Yamaguchi, Mohammad Hadi Kiapour, Luis E. Ortiz, and Tamara L. Berg.  Parsing clothing in fashion photographs.  _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3570–3577, 2012.
  * Yamaguchi et al. [2013] Kota Yamaguchi, M. Hadi Kiapour, and Tamara L. Berg.  Paper doll parsing: Retrieving similar styles to parse clothing items.  In _IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013_ , pages 3519–3526. IEEE Computer Society, 2013.
  * Yang and Luo [2014] W. Yang and L. Luo, P.and Lin.  Clothing co-parsing by joint image segmentation and labeling.  2014\.
  * Yenamandra et al. [2021] T Yenamandra, A Tewari, F Bernard, HP Seidel, M Elgharib, D Cremers, and C Theobalt.  i3dmm: Deep implicit 3d morphable model of human heads.  In _Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * Yu et al. [2021] Tao Yu, Zerong Zheng, Kaiwen Guo, Pengpeng Liu, Qionghai Dai, and Yebin Liu.  Function4d: Real-time human volumetric capture from very sparse consumer rgbd sensors.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR2021)_ , 2021.
  * Zhang et al. [2017] Chao Zhang, Sergi Pujades, Michael J. Black, and Gerard Pons-Moll.  Detailed, accurate, human shape estimation from clothed 3d scan sequences.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Zhao et al. [2020] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun.  Point transformer, 2020.
  * Zhao et al. [2018] Jian Zhao, Jianshu Li, Yu Cheng, Li Zhou, Terence Sim, Shuicheng Yan, and Jiashi Feng.  Understanding humans in crowded scenes: Deep nested adversarial learning and A new benchmark for multi-human parsing.  _CoRR_ , abs/1804.03287, 2018.
  * Zheng et al. [2021] Yang Zheng, Ruizhi Shao, Yuxiang Zhang, Tao Yu, Zerong Zheng, Qionghai Dai, and Yebin Liu.  Deepmulticap: Performance capture of multiple characters using sparse multiview cameras.  In _IEEE Conference on Computer Vision (ICCV 2021)_ , 2021.
  * Zheng et al. [2022] Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, and Yebin Liu.  Structured local radiance fields for human avatar modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Zhou et al. [2018] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Open3d: A modern library for 3d data processing, 2018.  cite arxiv:1801.09847Comment: http://www.open3d.org.
  * Zhu et al. [2020] Heming Zhu, Yu Cao, Hang Jin, Weikai Chen, Dong Du, Zhangye Wang, Shuguang Cui, and Xiaoguang Han.  Deep fashion3d: A dataset and benchmark for 3d garment reconstruction from single images, 2020.

SUPPLEMENTARY MATERIALS
CloSe: A 3D Clothing Segmentation Dataset and Model

##  7 Dataset

Our dataset CloSe-D comes from two sources, 1) CloSe-Di, which is dataset
captured in our lab and 2) CloSe-Dc, dataset from commercial data sources. We
explain the dataset capturing details in the following section, followed by
the process for obtaining segmentation labels.

##### CloSe-Dc Data.

We collect scans from different commercial dataset such as AXYZ [2], Twindom
[8], Treedy [7], Renderpeople [6]. Due to licensing issues, we will not
provide the scans from these datasets, but we will release the segmentation
labels and detailed instructions to purchase these datasets from respective
sources.

##### CloSe-Di Data Capture.

Following data capture setup in [47, 58], we create a dataset of approximately
100 subjects in 7 diverse poses, wearing 12 garment classes. We use Treedy’s
scanner [7], which consists of $\sim 130$ high-resolution camera at a fixed
position. We use Metashape [5] for 3D reconstruction, which is photogrammetry-
based reconstruction. Reconstructed scans are highly detailed and have high-
resolution texture maps associated with them. We also register SMPL [28] to
each scan, with the registration method used in [10, 47, 24].

Figure 8: _Top_ : Multiview rendered images of a scan._Bottom_ : Clothing
segmentation obtained using 2D Parsing method [16]. 2D Parsing method
generates inconsistent labels across views. Consequently, when these labels
are elevated from 2D to 3D using the 2D-to-3D lifting technique, the resulting
segmentation is noisy.

##### Ground Truth Segmentation Labels of CloSe-Dc Scans.

We follow the pipeline similar to the one in MGN-Seg [10]. We first register
the scans to SMPL and SMPL+D [10]. We then render the registered meshes from
72 different views and apply SotA 2D Human Parsing method, PGN [16]. One of
the major limitations of such a pipeline is inconsistent multiview prediction
of the 2D Human Parsing method, as shown in Fig. 8. This is expected behavior
from such methods as 1) they are not trained with any explicit loss to produce
multi-view consistent results, and 2) they are not trained on multi-view
images of the same scene. As a result, we observe many patches of undesired
clothing classes in the 2D segmentation and hence in the lifted 3D
segmentation as well. MGN-Seg [10] tried to solve this problem by using a pre-
defined prior, but these priors are limited to 3 classes. We propose to clean
such inconsistency using our hand-crafted heuristics and CloSe-T(see Fig.
12(left)). Moreover, PGN labels are inconsistent with our CloSe-Net labels, so
we apply some merging and splitting in labels. We first explain heuristics for
merging and segregation of labels in the following points:

  * •

Merging body parts: In PGN there are separate labels for left-leg, right-leg,
left-arm and right-arm. We instead use a single label for all these parts, so
we merge them into a single category.

  * •

Separate labels for Upper and Lower Garments : PGN generates only two kinds of
upper garment labels, namely ‘Shirt’ and ‘Coat’. On the other hand, our model
uses more fine-grained labels, _e.g_. ‘Shirt’ is further divided into
‘TShirt’, ‘Vest’, ‘Hoodies’ _etc_. We use the _change all_ option provided in
CloSe-T to correct such labels, as shown in Fig. 10. Similarly, there is only
one label for lower garments: ‘Pants’, which we split into ‘Pants’ and ‘Short-
Pants’.

We show some examples of heuristics-based segmentation and manually refined
segmentation in Fig. 9 and Fig. 10. We explain more details about our
interactive tool in Sec. 9.

##### Ground Truth Segmentation Labels of CloSe-Di Scans.

For CloSe-Di, we follow a similar idea, but instead of using SMPL+D
registration and SMPL UV space, we use Metashape [5] to perform 2D-to-3D
lifting of segmentation labels. The recovered 3D segmentation might be
inaccurate because of 1) inaccurate 2D segmentation prediction, and 2)
inconsistent 2D segmentation labels across different views. Similar to our
processing of CloSe-Dc, we clean noise using heuristics. We define heuristics-
based priors on SMPL mesh and clean the labels in for scan points directly.
This alleviates the problem of obtaining SMPL+D [10] registrations. We
deployed two different classes of heuristics:

  * •

Body Parts Heuristics: We rely on the prior knowledge that some garments
should not belong to unusual body parts (e.g., t-shirts on feet, trousers on
arms _etc_.).

  * •

Garments Class Heuristics: In some cases, we observed artifacts related to
specific combinations of garments. In these cases, we deploy an additional set
of rules to address these issues specifically.

\begin{overpic}[width=223.58379pt,unit=1bp,tics=10]{images/suppl_im/CWom0337-HD2-O03P01-S_input_comparison.pdf}
\par\put(35.0,120.0){\makebox(0.0,0.0)[]{\scriptsize{Textured Scan}}}
\put(110.0,120.0){\makebox(0.0,0.0)[]{\scriptsize{Initial Segmentation}}}
\put(190.0,120.0){\makebox(0.0,0.0)[]{\scriptsize{Clean Segmentation}}}
\put(190.0,112.0){\makebox(0.0,0.0)[]{\scriptsize{(using CloSe-T{})}}}
\par\end{overpic}

Figure 9: Segmentation labels obtained using our heuristics might result in
unclear boundaries(top, middle) and undesired noisy patches (bottom, middle).
We clean such noise using CloSe-T and obtain high-quality labels, as shown on
the right.

\begin{overpic}[width=223.58379pt,unit=1bp,tics=10]{images/suppl_im/CGirl0302-HD2-O07P01-S_input_comparison.pdf}
\par\put(40.0,125.0){\makebox(0.0,0.0)[]{\scriptsize{Textured Scan}}}
\put(110.0,125.0){\makebox(0.0,0.0)[]{\scriptsize{Initial Labels}}}
\put(180.0,125.0){\makebox(0.0,0.0)[]{\scriptsize{Corrected Labels}}}
\put(180.0,118.0){\makebox(0.0,0.0)[]{\scriptsize{(using CloSe-T{})}}}
\par\end{overpic}

Figure 10: Due inherent uncertainty in clothing classification, the
segmentation labels acquired through [16] might be incoherent. However, such
labeling discrepancies can be easily corrected using CloSe-T.

##  8 Method

We explain the details of our model CloSe-Net in this section.

##### Point Encoder.

We use the official implementation of DGCNN [53] and use 3 layers of
EdgeConvoluation operation, followed by a single-layer MLP.

##### Clothing Encoder.

We use a multi-head attention module in the encoder, where
$\mathrm{n_{head}}=4$ in our case. We also apply positional encoding to the
query vector($\mathrm{\mathbf{p^{\prime}}^{2}_{i}}$), before calculating the
attention score.

##### Body Encoder.

$F^{\mathrm{b}}$ requires the computation of nearest neighbors for each point
within the batch, potentially leading to computational overhead during the
training process. To mitigate this, we opt to precompute $F^{\mathrm{b}}$.
This is done by finding the nearest point for each scan point from the posed
SMPL mesh ($M(\boldsymbol{\beta},\boldsymbol{\beta})$). Subsequently, during
inference, a preprocessing step is employed to calculate $F^{\mathrm{b}}$
beforehand, which is then used during inference.

##  9 Interactive Tool

In this section, we explain common functionalities provided by our tool and
its usage in data annotation and network refinement.

##### Interactive Tool Interface.

We implement CloSe-T using Open3D [65] in C++ and introduce an easy-to-use,
light-weight interactive 3D tool, which provides following functionalities:

  * •

I/O operations: Loading/Saving meshes and labels, Loading/evaluating pre-
trained model, Saving/Evaluating refined network.

  * •

Scene: Move in the scene with mouse control, change lights, background, _etc_.

  * •

User selection: Easy polygon-based region selection by selecting the polygon
edges by clicking.

  * •

Labeling: Relabel region based on user selection/majority vote.

Figure 11: Functionalities provided in CloSe-T interface includes based I/O
operations, mouse-controlled camera movement in the scene, region selection,
relabelling, evaluation, and fine-tuning CloSe-Net.

##### Label Correction.

There are multiple options to label the selected regions

  * •

User selected class : Manually set the class assigned to the selected areas.
The predefined list of classes is shown in the dropdown menu; see Fig.
11(right).

  * •

Majority Vote: If a partial/inaccurate initial segmentation of the scan
already exists, the selected region can be labeled more efficiently using the
”majority vote” procedure. More precisely, for example, if there is a patch of
mislabeled points, the user can select the wider region around it, and label
the whole region by the class that is the most commonly present in the region.
This makes the labeling procedure much faster.

##### CloSe-T for Data Annotation

.

We use CloSe-T to manually clean segmentation and generate high quality
segmentation data, as shown in Fig. 9 and Fig. 10. We provide a demo of
labeling process in the supplementary video and visualize key-stage of
pipeline in Fig. 12.

\begin{overpic}[width=223.58379pt,unit=1bp,tics=20]{images/suppl_im/tool1_1.pdf}
\par\put(40.0,140.0){\makebox(0.0,0.0)[]{\scriptsize{Load scan with texture}}}
\put(160.0,140.0){\makebox(0.0,0.0)[]{\scriptsize{Visualize current
segmentation}}}
\par\par\end{overpic}\begin{overpic}[width=223.58379pt,unit=1bp,tics=20]{images/suppl_im/tool1_2.pdf}
\par\par\put(40.0,140.0){\makebox(0.0,0.0)[]{\scriptsize{Select region and
relabel }}} \put(160.0,140.0){\makebox(0.0,0.0)[]{\scriptsize{Visualize new
segmentation}}} \par\end{overpic}

Figure 12: Annotation using CloSe-T.Using CloSe-T, we first _load the scan
with texture_ to understand the scan. We then _visualize current segmentation_
as an overlay on the textured scan. After inspection, we identify and _select
mislabeled regions_ and assign them the correct label from a predefined set.
Finally, we _visualize the new segmentation_ and inspect by moving the camera
around the scene.

Due to the inherently error-prone nature of the segmentation label generation
pipeline, numerous scans displayed noisy boundaries and improperly labeled
clothing classes. To address this issue, approximately 1000 scans were
annotated using CloSe-T within the CloSe-D dataset, while the remaining were
carefully verified. Consequently, CloSe-D comprises a curated segmentation
label dataset that has been meticulously verified.

##### CloSe-T for Network Refinement.

We also use CloSe-T to improve the generalization of our model for real-world
datasets. We first predict the segmentation label for a given scan using the
pre-trained CloSe-Net. Since the given scan is out-of-distribution, network
results might be incorrect and noisy. We then refine the network prediction
for the given scan using the steps mentioned in _data annotation_. We explain
training details and experiments in Sec. 10. The new network is used to infer
the given scan again and also evaluated on the test-set of CloSe-D. All these
functions are implemented as a simple button click in the tool, see Fig. 11.
The newly trained model can be saved and used of this new out-of-distribution
dataset for better generalization.

##  10 Results

In this section, we provide more results of our model. In Sec. 10.1, we
provide more comparison with baseline methods, followed by comparison on BUFF
[60] dataset in Sec. 10.2. Finally, we provide ablation studies for continual
learning setup of our model and show more results on real-world datasets in
Sec. 10.3.

###  10.1 Comparison with baseline

In this section, we analyze more comparisons with part segmentation methods to
understand the cause of superior performance of CloSe-Net. We broadly classify
them into 5 factors, as discussed below. These factors act mutually in many
cases, widening the disparity between the performance levels of baseline
techniques and our proposed approach. In the table Table 6 we provide a
quantitative comparison on the test split of CloSe-Di.

Method | Mean | T-shirt | Shirt | Vest | Coat | Hoodies | Short-Pants | Pants | Skirts | Hat | Shoes | Body | Hair
---|---|---|---|---|---|---|---|---|---|---|---|---|---
DGCNN [53] | 92.65 | 97.50 | 93.23 | 95.78 | 86.89 | 99.54 | 96.89 | 87.27 | 98.90 | 97.26 | 86.17 | 84.19 | 88.14
DeltaConv [54] | 91.30 | 97.19 | 88.12 | 96.57 | 86.98 | 98.55 | 94.39 | 86.87 | 98.69 | 97.26 | 83.42 | 80.33 | 87.29
Ours | 95.19 | 99.12 | 96.18 | 99.48 | 87.93 | 99.69 | 97.98 | 89.39 | 99.05 | 99.06 | 89.97 | 89.78 | 94.66

Table 6: We quantitatively compare the results of our method SotA part-
segmentation methods, DGCNN [53] and DeltaConv [54]. We report IoU for every
class and mean over all the classes($\mathrm{IoU_{mean}}$).

##### Clothing Information.

Baseline methods DGCNN [53] and DeltaConv [54] have no prior about clothing
present in the scan. As a result, these methods rely on local/global geometric
and appearance features. Given the diversity and complexity of clothing items,
it is challenging to learn about robust semantics from limited information. As
a result, baseline methods seem to generate multiple clothing classes in a
vicinity, mislabel clothing classes, and are not able to learn the
shape/structure of clothing items. This is evident from all the examples shown
in Fig. 13. CloSe-Net not only takes advantage of clothing information but
also learns a more distinctive feature for each clothing class and
consequently learns clothing prior based on local features and these clothing
features(via attention module).

##### Texture Bias.

As observed in Fig. 13(first and second row), baseline methods are highly
sensitive to changes in texture. As a result any steep change in texture
results in a new clothing class. However CloSe-Net produces accurate results.
For baseline methods color, normal and location are the only guiding signal
without any prior. Given limited training data, they tend to overfit to
textures scene during training.

##### Multi-layer Clothing.

We also observe that baseline methods are not able to recover multi-layer
clothing labels see Fig. 13(third row). As there is no prior knowledge about
clothing present in the scan, baselines rely on texture and geometry
information. In such cases, baselines seem to predict the most commonly seen
example with texture during training such as hoodies or shirts. On the other
hand, the clothing information used in CloSe-Net helps with better
comprehension, even if local features are very similar.

##### Shape/geometry Bias.

Similar to texture bias, the baseline method also has geometry bias to some
extent. As shown in Fig. 13(fourth row) loose upper clothing with larger
shapes are classified as hoodies, although the labels are not noisy.

##### Sparse Clothing Classes.

We also observe that CloSe-Net performs well for rare clothing classes such as
dresses, hats, etc. On the other hand baseline methods fail to generate
consistent labels.

\begin{overpic}[width=496.85625pt,unit=1bp,tics=20]{images/suppl_im/full_fig_6_supmat.pdf}
\par\put(20.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{Input}}}
\put(78.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{DGCNN~{}\cite[cite]{[\@@bibref{Number}{dgcnn}{}{}]}}}}
\put(125.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{DeltaConv~{}\cite[cite]{[\@@bibref{Number}{Wiersma2022DeltaConv}{}{}]}}}}
\put(170.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{Ours}}}
\put(220.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{Ground Truth}}}
\par\put(270.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{Input}}}
\put(320.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{DGCNN~{}\cite[cite]{[\@@bibref{Number}{dgcnn}{}{}]}}}}
\put(370.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{DeltaConv~{}\cite[cite]{[\@@bibref{Number}{Wiersma2022DeltaConv}{}{}]}}}}
\put(420.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{Ours}}}
\put(470.0,480.0){\makebox(0.0,0.0)[]{\scriptsize{Ground Truth}}}
\par\put(250.0,440.0){\makebox(0.0,0.0)[]{a)}}
\put(250.0,360.0){\makebox(0.0,0.0)[]{b)}}
\put(250.0,280.0){\makebox(0.0,0.0)[]{c)}}
\put(250.0,200.0){\makebox(0.0,0.0)[]{d)}}
\put(250.0,120.0){\makebox(0.0,0.0)[]{e)}}
\put(250.0,40.0){\makebox(0.0,0.0)[]{f)}} \par\end{overpic} Figure 13:
Baseline method like DGCNN [53] and DeltaConv [54] have Texture bias (a, b),
are unable to distinguish between multi-layer clothing (c), produces incorrect
labels if geometry deviates significantly from average body and clothing
shapes (d) and underperform for unbalanced classes such as dress and hats(e,
f).

###  10.2 Comparison with Prior Work

We compare or model with prior work GIM3D [36] on Buff dataset [60]. We use 15
scans from BUFF, as in [36] for evaluation on the 3-class segmentation
problem. We use PointNet++ [38] based model from GIM3D and report the number
in Table 7. We observe that for both CloSe-D-test and BUFF dataset, our model
significantly outperforms GIM3D [36].

Dataset | MGN [10] | GIM3D [36] | Ours
---|---|---|---
CloSe-D-Test | 88.88 | 72.04 | 92.47
Buff [60] | - | 75.41 | 90.13

Table 7:  Comparison with MGN [10] and GIM3D [36] on CloSe-D and BUFF dataset.

###  10.3 CloSe-Net on Real-world Datasets

We qualitatively evaluate CloSe-Net on publicly avaialble real-world datasets
such as THuman2.0 [59], THuman3.0 [45], HuMMan [12], 3DHumans [20]. We have
added more results in Fig. 15. We observe that for all datasets, CloSe-Net
generates good results and generalizes well. However, in some cases, it
results in blurry boundaries and noisy patches of labels, as shown in Fig. 14.

We propose to improve the performance of our model for such out-of-
distribution scans, by fine-tuning the model in a continual learning
framework. We follow [50] and experiment with various loss combinations and
training configurations to find an optimal setup, such that network
performance improves on new out-of-distribution scans without catastrophic
forgetting. We show the ablation in Table 8. We compare the mean IoU on test
split of CloSe-D, after iteratively fine-tuning on 2 sets of scans from this
new distribution. Based on experiments, we pick the full loss(eq. 5, main
paper) as training loss and only train the last layer of the segmentation
decoder and MLP of the Point Encoder. We fine-tune the model for 2 epochs
only.

Figure 14: CloSe-Net predicts blurry boundaries for out-of-distributions
scans.

\begin{overpic}[width=119.24821pt,unit=1bp,tics=20]{images/suppl_im/public/00001_1_00001_0070_outp_mesh_inp.png}
\put(25.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Input}}}
\put(80.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Prediction}}} \end{overpic}

\begin{overpic}[width=119.24821pt,unit=1bp,tics=20]{images/suppl_im/public/00003_2_00003_0024_outp_mesh_inp.png}
\put(25.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Input}}}
\put(80.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Prediction}}}
\put(130.0,0.0){\makebox(0.0,0.0)[]{\scriptsize{THuman3.0
Scans~{}\cite[cite]{[\@@bibref{Number}{thuman3}{}{}]}}}} \end{overpic}

\begin{overpic}[width=119.24821pt,unit=1bp,tics=20]{images/suppl_im/public/00001_3_00001_0046_outp_mesh_inp.png}
\put(25.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Input}}}
\put(80.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Prediction}}} \end{overpic}

\begin{overpic}[width=119.24821pt,unit=1bp,tics=20]{images/suppl_im/public/00004_1_00004_0003_outp_mesh_inp.png}
\put(25.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Input}}}
\put(80.0,100.0){\makebox(0.0,0.0)[]{\scriptsize{Prediction}}} \end{overpic}

\begin{overpic}[width=119.24821pt,unit=1bp,tics=20]{images/suppl_im/public/3DHumans_IIITH_org_20_outp_mesh_inp.png}
\par\put(130.0,0.0){\makebox(0.0,0.0)[]{\scriptsize{3DHumans
Scans~{}\cite[cite]{[\@@bibref{Number}{Jinka2022}{}{}]}}}} \end{overpic}

\begin{overpic}[width=119.24821pt,unit=1bp,tics=20]{images/suppl_im/public/p000443_a000218_000030_outp_mesh_inp.png}
\put(130.0,0.0){\makebox(0.0,0.0)[]{\scriptsize{HuMMan
Scans~{}\cite[cite]{[\@@bibref{Number}{cai2022humman}{}{}]}}}} \end{overpic}

\begin{overpic}[width=119.24821pt,unit=1bp,tics=20]{images/suppl_im/public/THuman2_org_0069_outp_mesh_inp.png}
\put(130.0,0.0){\makebox(0.0,0.0)[]{\scriptsize{THuman2.0
Scans~{}\cite[cite]{[\@@bibref{Number}{thuman2}{}{}]}}}} \end{overpic}

Figure 15: CloSe-Net results on real-world public datasets [59, 45, 20, 12].
Table 8: Performance($\mathrm{IoU_{mean}}$) on CloSe-D-test after network
refinement.

Layers trained | Naive loss | Weighted cross-entropy | Full
---|---|---|---
$f_{\mathrm{dec}}$-last | 90.33 | 90.37 | 90.25
$f_{\mathrm{dec}}$-full | 89.14 | 88.53 | 88.50
$f_{\mathrm{dec}}$-last+ $f_{\mathrm{MLP}}$ | 90.62 | 90.53 | 90.33
$f_{\mathrm{dec}}$-full+ $f_{\mathrm{MLP}}$ | 89.00 | 88.53 | 88.95
$f_{\mathrm{dec}}$-last+ $f_{\mathrm{MLP}}$ \+ $f^{3}$ | 90.53 | 90.18 | 90.35
$f_{\mathrm{dec}}$-full+ $f_{\mathrm{MLP}}$ \+ $f^{3}$ | 89.00 | 88.53 | 88.62

##### Segmenting 4D Scans using CloSe-Net and CloSe-T.

We use the aforementioned setup to improve segmentation accuracy for a given
4d sequence. We randomly pick one frame of a 4D sequence and refine the model
as per this scan. This is similar to one-shot fine-tuning. Then we generate
the segmentation labels for the whole sequence. Since the model has now
learned appearance and geometry features of one frame, this results in
improved accuracy for remaining frames. We show results on a set of poses from
THuman3.0 and HuMMan inFig. 16.

Figure 16: CloSe-Net is fine-tuned using CloSe-T on a single frame of a
sequence to improve generalization on the remaining frames. We show results of
fine-tuned CloSe-Net on THuman3.0 [45](top) and HuMMan [12](bottom). Fine-
tuned networks result in consistent predictions.

Finally, we have generated high quality segmentation labels of approximately
1000 scans(from diverse sources [59, 45, 12, 20]) using CloSe-Net and CloSe-T.
We will release this as CloSe-D++.