(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Georgia Institute of Technology 22institutetext: Meta AI,
FAIR33institutetext: University of Illinois Urbana-Champaign

#  $3\times 2$: 3D Object Part Segmentation by 2D Semantic Correspondences

Anh Thai 1122 Weiyao Wang 22 Hao Tang 22 Stefan Stojanov 11 James M. Rehg 33
Matt Feiszli 22

###### Abstract

3D object part segmentation is essential in computer vision applications.
While substantial progress has been made in 2D object part segmentation, the
3D counterpart has received less attention, in part due to the scarcity of
annotated 3D datasets, which are expensive to collect. In this work, we
propose to leverage a few annotated 3D shapes or richly annotated 2D datasets
to perform 3D object part segmentation. We present our novel approach, termed
3-By-2 that achieves SOTA performance on different benchmarks with various
granularity levels. By using features from pretrained foundation models and
exploiting semantic and geometric correspondences, we are able to overcome the
challenges of limited 3D annotations. Our approach leverages available 2D
labels, enabling effective 3D object part segmentation. Our method 3-By-2 can
accommodate various part taxonomies and granularities, demonstrating
interesting part label transfer ability across different object categories.
Project website: https://ngailapdi.github.io/projects/3by2/.

††Work done as an intern at Meta AI (FAIR). Figure 1: We propose 3-By-2, a
novel training-free method for low-shot 3D object part segmentation that
achieves SOTA performance on both zero-shot and few-shot settings.

##  1 Introduction

3D object part understanding is essential in various research fields and
applications, such as robotics [40, 28, 23] and graphics [17]. Through our
understanding of the world, objects can be decomposed into parts based on
diverse properties (e.g., geometry or affordance [24, 9]). However, these
different decompositions do not always align with one another—the same object
can be segmented into parts differently depending on the specific use case.
For instance, a driver might perceive a car in terms of its functional
components like the steering wheel, accelerator pedal, and brake pedal.
Conversely, a manufacturing worker may view the car as an assembly of
structural parts, such as the frame, bumper, and windshield. Further, various
parts with similar functionalities or structures can be shared among different
object classes (e.g., the term “leg" can apply to multiple furniture items).
How can we design a 3D part segmentation system that has high performance
across such different requirements and scenarios?

Recent works in 3D part segmentation have integrated language as an additional
input [22, 2, 55] by leveraging vision-language models to prompt the
segmentation. However, grounding visual parts using language is inherently
ambiguous. This is because parts can be described using diverse phrases that
may include synonyms, various levels of detail, and differences in terminology
(structural vs functional), which presents challenges for these models [22].
In contrast, images capture rich information about object shapes, textures and
spatial part relationships. These properties can directly be parsed and
compared using visual similarities between objects despite differences in
linguistic expression. Therefore, it is important to study the limits and
potentials of reasoning about visual similarity for generalization across
different objects and categories.

In this work, we investigate the 3D part segmentation task from this different
perspective and propose a novel method called 3-By-2. Since labeling 3D data
is expensive, we design 3-by-2 to leverage existing extensively annotated 2D
part segmentation datasets [34, 13] or a few-labeled 3D shapes to perform
object part segmentation _without additional training or finetuning_. Our
method does not need any language input and can flexibly handle segmentation
tasks at various levels of granularity.

We build our method based on the observation that because objects are
constructed from parts, and because various objects often share a common set
of parts with similar visual structures, this should allow part label transfer
from one object to another without any language description. Recent studies
[39, 50] have demonstrated the strong 2D semantic correspondences encoded by
features of image diffusion models that generalize across different domains
(e.g. sketch vs real images). To label a query 3D object point cloud, we
leverage these strong representations to perform 2D pixel correspondence-based
label transfer from in-the-wild 2D datasets or 2D renders of a few labeled 3D
objects. To the best of our knowledge, we are the first to use diffusion model
features for semantic label transfer in the context of 3D part segmentation.

Figure 2: Overview of our proposed method 3-By-2. (1) Render the input object
in multiple camera viewpoints, (2) Perform 2D part segmentation on each view
individually by leveraging 2D semantic correspondences and 2D class-agnostic
segmentation model, (3) Aggregate the 2D predictions from multiple views using
our proposed mask-consistency module, (4) Back-project the predictions to 3D
using depth information.

While it might seem that obtaining 2D part labels for multi-view renders of an
object through label transfer and back-projection into 3D is intuitively
straightforward, a high performance and efficient implementation requires
careful consideration of the challenges of 3D part segmentation: 1) Precise
determination of 3D object part boundaries, which is particularly challenging
for unstructured data like point clouds, and 2) Flexible adaptation to
different levels of part granularity. To this end, we introduce three novel
elements of our method: non-overlapping generation, mask-level label transfer
and mask-consistency modules (see Fig. 2). These components work efficiently
together to ensure precise 3D part segmentation masks and boundaries across a
range of object categories and part levels (Fig. $3\times 2$: 3D Object Part
Segmentation by 2D Semantic Correspondences and Tables 1, 2, 3).

Overall, 3-By-2 is a training-free method _independent_ of language inputs,
instead relying solely on the 2D labels provided by a 2D database. Unlike
previous methods that require 3D segmentation priors like point-cloud clusters
[22] or mesh surface information [2, 35], our approach has only a single
requirement: calibrated cameras for back-projection. This can be known during
the rendering process or predicted using SfM approaches.

We validate the performance of our approach with PartNet-Ensembled [22], a
dataset tailored for language-input models, and PartNet [27], which is not
tailored for language. These datasets exhibit multiple levels of granularity.
Notably, unlike previous approaches that require category-specific fine-tuning
for few-shot scenarios [22, 35], 3-By-2 achieves SOTA performance without any
training or fine-tuning requirements in either a zero-shot or few-shot
setting. Additionally, we identify that models with language inputs exhibit
suboptimal performance with highly fine-grained part terminologies. This
highlights the advantages of our approach, which effectively handles these
fine-grained object parts. Furthermore, we conduct comprehensive ablation
studies and demonstrate the transferability of parts across different object
categories, which benefits the understanding of object part compositionality.

In summary, our contributions are 4-fold:

  * •

A novel, training-free method, 3-By-2, that achieves SOTA performance on
benchmarks with different levels of granularity for zero-shot and few-shot 3D
object part segmentation.

  * •

The first to provide an effective approach for leveraging image diffusion
model’s features [39] to establish 2D semantic correspondences in the context
of 3D part segmentation.

  * •

Novel non-overlapping mask generation, mask-level label transfer, and mask-
consistency modules that effectively transfer part labels from 2D database and
extrapolate them to 3D.

  * •

Demonstrating the flexibility of 3-By-2 in accommodating various database
settings and in generalizing between different object categories.

##  2 Related Work

###  2.1 3D Part Segmentation

In contrast to its 2D counterpart, the progress in this field has been
relatively limited, primarily due to the high cost associated with collecting
and annotating 3D datasets. Currently, all of the available large-scale
annotated 3D object part datasets are synthetic [27, 48, 43, 21]. The most
widely used benchmarks [27, 48] are predominantly derived from objects within
the ShapeNetCore [6] dataset. This problem has been tackled using
architectures that take 3D representations [27, 32] as inputs. These methods
were trained in a supervised manner, requiring large-scale annotated data.
More recent approaches have attempted to investigate data-efficient training
scenarios where only a few 3D shapes are annotated [22, 42, 35, 53].

###  2.2 Multi-view 2D-3D Segmentation Using Foundation Models

Although multi-view approaches have been widely utilized in the past for 3D
segmentation [8, 16, 51], the rapid advancement of 2D foundation models [20,
19] has encouraged more SOTA research aimed at leveraging these models to
perform 3D segmentation in a multi-view fashion. CLIP [33] and GLIP [20] have
been employed to integrate language information from multiple 2D views into 3D
for open-vocabulary segmentation [30, 38, 22, 53, 2]. SAM [19], due to its
ability to output per-pixel masks, has been used as an effective tool for
multi-view 2D-3D segmentation, both on 3D structures like point clouds [53,
47, 38, 49, 45] or in NeRF-style [5].

Scene Segmentation. Various combinations of foundation models have been
explored for this task. While [38] leverages CLIP and SAM to support open-
vocabulary 3D part segmentation, others use SAM with carefully designed
prompts [5] or post-processing techniques [47]. Building upon these successes,
concurrent works [45, 29, 15] seek to improve SAM utilization strategies. Our
work differs by focusing on part segmentation, which requires finer
granularity. This distinction in objectives directly influences the processing
of SAM predictions, tailored to suit their specific characteristics. For
example, while scene segmentation methods may disregard or merge masks
covering parts of objects, part segmentation approaches might encourage
splitting, depending on the desired level of detail.

Part Segmentation. PartSLIP [22] and SATR [2] were among the first to employ
foundation models for this task, pioneering the use of GLIP for open-
vocabulary segmentation. Concurrent works have seen the integration of SAM
into their pipelines [53, 46, 18]. Zhou et al. [53] and Kim et al. [18] use
SAM with GLIP-predicted bounding boxes, while Xue et al. [46] employ SAM with
furthest point sampling for each view, extending predictions to 3D with GLIP
labels. Our approach shares with these works the use of SAM for 2D
segmentation before 3D aggregation. In contrast, our method focuses solely on
visual cues without language inputs, employing image diffusion model’s
features [39]. To improve SAM’s accuracy, we introduce a novel non-overlapping
mask generation module, eliminating the need for GLIP-generated bounding
boxes.

###  2.3 Part Label Transfer using Correspondences

Transferring labels from annotated datasets to non-annotated datasets has been
considered recently in [37] for open-vocabulary 2D part segmentation and
previously in [54, 7] for 3D part segmentation. While [37] used DINOv1 [4]
feature representations for dense label transfer between related objects in
the base classes and novel object classes, Zhu et al. [54] relied on classical
SIFT [25] features for establishing correspondences in 2D images. Chen et al.
[7], in contrast, train a network to regress the correspondences directly on
the input point cloud.

We share with these approaches the use of semantic correspondences to identify
optimal candidates for label transfer. However, our primary objective sets us
apart significantly from [37], as we focus on segmenting 3D objects. Compared
to [54], we leverage class-agnostic segmentation models to avoid dense
pixel/patch sampling. Furthermore, unlike [7], we do not require direct
operations on 3D point clouds or any specific 3D representations.
Additionally, we introduce a mask-consistency module for per mask label
voting, rather than relying solely on small local patches.

Semantic Correspondences from Foundation Models. Many vision foundation models
have demonstrated an inherent capability to implicitly capture semantic
correspondences across different instances within the same category (e.g.,
matching chair backs) and across diverse categories (e.g., aligning dog’s legs
with cat’s legs) [39, 50, 14, 3]. In this work, we leverage semantic
correspondences established by [39] to transfer part labels from annotated 2D
datasets to query 3D objects.

##  3 Method

Given a database $\mathcal{D}$ consisting of 2D part annotations, our goal is
to segment each query object $q$ into parts using the visual part vocabulary
provided by $\mathcal{D}$. Note that $\mathcal{D}$ can either be gathered from
2D (image) part datasets or from renders of a few 3D objects captured at
different view-points. Our method consists of three main steps (Fig. 2): (1)
render a set of 2D RGB images $\mathcal{I}q$ of 3D object $q$ from $K$
distinct camera viewpoints; (2) perform 2D part segmentation on the rendered
images; (3) aggregate image-level predictions through a mask-consistency
aggregation module to obtain 3D predictions.

Figure 3: The process of pixel-level part label transferring. For each pixel
$p$ in the query image $I_{k}$, we perform the following: (1) Extract the
feature $f(p)$, along with the feature grid for each image $I_{\mathcal{D}}$
in the database $\mathcal{D}$, (2) Measure cosine similarity between $f(p)$
and the feature of each pixel within each feature grid, (3) Obtain the best
match of $p$ over $\mathcal{D}$ by determining the most similar pixel
$p_{\mathcal{D}}$ over all images $I_{\mathcal{D}}$, (4) Assign the label of
$p$ is to be the label of $p_{\mathcal{D}}$.

(a)

(b)

Figure 4: (4(a)) Non-overlapping 2D Mask Proposal. We address the issue of
overlapping masks produced by SAM. The masks are first sorted by their areas.
Subsequently, the smaller masks are stacked on top of the larger ones. Non-
overlapping masks are obtained by taking the visible segment of each mask.
(4(b)) Different mask sampling strategies for label transfer. Our strategy
provides accurate, dense prediction with clear part boundaries.

###  3.1 2D Part Segmentation

There are two primary approaches to tackle this task: (1) Top-down, using
segmentation mechanisms such as SAM [19], or (2) Bottom-up, which involves
labeling each pixel individually. While SAM produces high-quality 2D masks
with sharp boundaries, it operates in a class-agnostic manner, often leading
to high overlap between sub-parts, parts, and instances. Simply selecting the
mask with the highest score may result in incorrect granularity and lacks the
flexibility required for part segmentation.

Conversely, doing label transfer for each pixel individually in the image is
computationally impractical, particularly for part segmentation tasks where
high resolution is preferred. Sparsely sampling and labeling pixels can result
in under-segmented masks, particularly for smaller parts that are less likely
to be sampled compared to larger parts (see Fig. 4(b)). Moreover, accurately
determining part boundaries for individual pixels can be challenging, which
may result in increased errors when extrapolating to 3D, particularly with
unstructured 3D representations like point clouds. These issues raise the
important question: how do we transfer part labels and preserve part
boundaries without sacrificing computational resources?

To address this question, we propose a 2D segmentation method that combines
the strengths of both approaches which consists of 3 novel components: (1)
Single-pixel 2D label transfer using semantic correspondences derived from
DIFT [39], (2) Non-overlapping 2D mask proposal module, which refines SAM’s
multi-granularity predicted masks into non-overlapping part masks, and (3)
Mask-level label transfer by integrating (1) and (2).

Single-pixel 2D Label Transfer. At the core of our method is the 2D label
transfer process. The goal is to transfer pixel labels from the annotated 2D
database $\mathcal{D}$ to the query RGB image $I_{k}\in\mathcal{I}q$: for a
pixel $p$ in the foreground object in $I_{k}$, we aim to identify the best-
matched pixel $p^{\prime}$ in each image $I_{\mathcal{D}}$ in the database
$\mathcal{D}$ and assign initial label to $p$ by $p^{\prime}$. To this end, we
leverage the established semantic correspondence of DIFT [39]. While recent
works have demonstrated the effectiveness of image diffusion models in
extracting semantic correspondences, as evidenced by evaluations on datasets
like SPair-71K [26], we are the first to leverage these features for
transferring semantic labels in the context of 3D part segmentation.
Specifically, $p^{\prime}=\arg\max_{p^{\prime}\in
I_{\mathcal{D}}}\cos(f(p),f(p^{\prime}))$ where $\cos$ and $f(x)$ denotes the
cosine similarity score and the feature representing pixel $x$. The best pixel
correspondence $p_{\mathcal{D}}$ of $p$ over the entire database is obtained
by taking the most similar match within all the images in the database.
Formally, $p_{\mathcal{D}}=\arg\max_{\mathcal{D}}p^{\prime}$. The label of $p$
is then assigned to be the label of $p_{\mathcal{D}}$ (see Fig. 3).

Coarse-to-fine correspondence search. Finding the nearest neighbor for a query
pixel across the entire database can be prohitively costly, especially for
part segmentation which operates in high resolutions. We propose a coarse-to-
fine strategy: using the coarse feature maps generated by DIFT [39], we first
conduct the search at the coarse level to localize the region of the best
match. We then extract the $3\times 3$ window centered at this region (in
feature space) for a fine search (see Fig. 7). This approach ensures that we
compute per-pixel similarity scores only within the region of interest, rather
than across the entire image, improving computational efficiency. For
instance, when processing a pair of images with a resolution of $800\times
800$, coarse-to-fine correspondence search achieves a speed improvement of
approximately 2000 times in terms of wall time.

Figure 5: Two approaches to aggregate 3D part labels from multiple 2D views.
Aggregating 3D part labels from multiple 2D views through geometric
correspondence can be achieved by either point or mask label consistency.

Figure 6: Effectiveness of mask label consistency. Enforcing consistency at
the mask level can mitigate discrepancies at each individual point and
contributes to smoother segmentation.

Non-overlapping 2D Mask Proposal. We propose the use of class-agnostic 2D part
mask proposal, specifically from SAM [19]. By assuming that each mask proposal
corresponds to a subset of a part, we can then selectively sample pixels
within each mask proposal for label transferring. The labels are subsequently
propagated to each pixel of the 2D masks through a majority voting process
based on the sampled pixels within the mask. To address the issue posed by the
highly overlapping predictions from SAM’s multi-granularity model, we
introduce a non-overlapping 2D mask generation module. This module takes SAM
masks as inputs and outputs a set of mutually exclusive 2D masks.

We arrange the SAM output masks in descending order of mask area and stack
smaller masks on top of larger ones. This ensures that if mask $A$ is a subset
of mask $B$, stacking $A$ on top of $B$ results in non-overlapping masks,
namely $A$ and $B\setminus A$. Non-overlapping masks are finally obtained by
taking the visible segments of each mask (see Fig. 4(a)).

2D Mask Label Assignment. After obtaining the non-overlapping masks, we
sparsely sample pixels in each mask to transfer label. We then perform
majority voting to assign the dominant label for each 2D mask, weighted by the
confidence score (cosine similarity) of the best pixel correspondence matches.

Figure 7: Coase-to-fine correspondence search. We first conduct searching on a
coarse level to identify the region of best match. We then extract the
$3\times 3$ window centered at this region in feature space for a fine search.
This approach is approximately 2000 times faster in terms of wall time for
large $N$ ($800\times 800$).

###  3.2 Mask-consistency Aggregation Module

Given a set of 2D RGB images with part segmentation predictions, we aim to
extrapolate these segmentation labels to 3D using geometric correspondences.
Prior works [22, 35] aggregate multi-view information for each 3D point or
mesh triangle face through a weighted sum of multi-view 2D predictions. To
fully maintain the high-quality part boundaries predicted by SAM in 2D, we
choose to aggregate multi-view predictions for each 2D mask instead. This
observation is based on the fact that part identities remain constant across
multiple views (e.g., the seat in view 1 should be segmented as the seat in
view 2, see Fig. 6). Intuitively, mask consistency can be seen as an
additional constraint on point consistency, encouraging points within the same
2D mask to remain associated with the same masks in the 3D space

We present a novel mask-consistency aggregation module that takes a set of 2D
part segmentation predictions for multiple views as input. Our approach
involves constructing an undirected unweighted graph, denoted as
$G:V\rightarrow E$, where each vertex corresponds to a 2D mask in a given
view. The edges of the graph connect masks from different views that capture
the projection of the same 3D points. We construct a set of mask
correspondences for each vertex $v\in V$,
$\mathcal{M}_{v}=\\{v,u_{1},u_{2},\dots u_{N}\\}$ where an edge $e_{i}$
connects $v$ and $u_{i}$. A mask $v$ is defined as undersegmented when there
exists at least 2 masks in $\mathcal{M}_{v}$ that belong to the same image but
are assigned with different labels. For instance, in Fig. 8, vertex $v_{1}$
corresponds to an undersegmented mask. Formally,

$\mathcal{S}_{v}=\\{u_{i},u_{j}\in I_{k}\text{ and }l(u_{i})\neq
l(u_{j})|u_{i},u_{j}\in\mathcal{M}_{v}\\}$ (1)

where $l(x)$ denotes the label of $x$. We discard $v$ if
$|\mathcal{S}_{v}|>\epsilon$. That is, if $v$ is consistently determined as
undersegmented across multiple views, we discard the contribution of $v$ in
the final label assignment. We then traverse the graph simultaneously from
each vertex using breadth-first-search to accumulate the labels for each
$\mathcal{M}_{v}$. Subsequently, we perform majority voting to assign labels
to each $\mathcal{M}_{v}$. Finally, for each mask, we identify the most
frequently assigned label as the final label.

Figure 8: Mask-consistency process. (1) Each vertex of $G$ corresponds to a
mask in a given image. The edge connecting each pair of vertices denotes that
the pair contains the projection of the same 3D points. Mask consistency set
$\mathcal{M}_{v}$ for each $v$ is obtained via the first-order neighborhood of
$v$. (2) $v_{1}$ is detected as under-segmented since $\mathcal{M}_{v_{1}}$
consists of masks from the same view with different labels ($v_{2}$, $v_{3}$)
and hence, is discarded. (3) Traverse $\mathcal{M}_{v_{i}}$ to obtain labels
for $\mathcal{M}_{v_{i}}$. (4) Obtain label for each mask by majority voting.
Here we show a simple example for visualization purpose.

The simple intuition behind this approach is: if a part occasionally receives
incorrect labels in some challenging views, employing majority voting within
the mask correspondence set can calibrate these errors. Further, performing
this on the mask level ensures that if two 2D points share the same mask label
in the majority of the views, they will ultimately be assigned with the same
final label. This approach calibrates potential discrepancies in individual
point-wise aggregations (see Fig. 6).

##  4 Experiments

In this section, we first report the performance of 3-By-2 against baselines
on PartNet-Ensembled (PartNetE) [22] in Sec. 4.1 and on PartNet [27] with
“level-3" annotation in Sec. 4.2. Note the distinction between these datasets
since PartNetE consists of a distinct set of articulated objects from [44].
These datasets also exhibit different granularity of part annotations. While
PartNetE consists of both basic parts like chair back and fine-grained parts
like scissors screw, PartNet with “level-3" annotation contains all fine-
grained parts such as “back_frame_vertical_bar". In Sec. 4.3, we conduct
comprehensive ablation studies to verify the necessity of each components in
3-By-2. Our few-shot experiments refer to the setting where a few labeled 3D
objects are available for each object category while there is no annotated 3D
part labels in the zero-shot setting. In this setting, we leverage labels from
the 2D domain instead.

Table 1: Few-shot performance on PartNetE [22] dataset. The left columns show
performance on the 17 categories that supervised methods [32, 31, 41] (first 3
rows) were trained on with additional 28K objects. The right columns show
performance on the 28 categories with only 8 objects/category in the training
set. [36, 52, 22, 53] and ours (last 5 rows) only have access to 8
objects/category during training for all 45 categories. Please refer to the
Supplement for the full table on all 45 categories.

Methods | Chair |  | Sci-
---
ssors
| Lap-
---
top
Door |  | Micro-
---
wave
| Key-
---
board
| Avg.
---
(17)
| Cam-
---
era
USB |  | Stap-
---
ler
| Disp-
---
enser
| Ket-
---
tle
| Eye-
---
gl.
| Avg.
---
(28)
| Avg.
---
(45)
PointNext [32] | 0.918 | 0.573 | 0.325 | 0.438 | 0.405 | 0.450 | 0.591 | 0.332 | 0.679 | 0.886 | 0.260 | 0.451 | 0.881 | 0.457 | 0.502
PointNet++ [31] | 0.847 | 0.500 | 0.554 | 0.457 | 0.436 | 0.745 | 0.533 | 0.065 | 0.524 | 0.516 | 0.121 | 0.209 | 0.762 | 0.250 | 0.365
SoftGroup [41] | 0.883 | 0.760 | 0.184 | 0.531 | 0.383 | 0.589 | 0.505 | 0.236 | 0.441 | 0.801 | 0.189 | 0.574 | 0.724 | 0.313 | 0.384
ACD [36] | 0.390 | 0.391 | 0.111 | 0.189 | 0.066 | 0.261 | 0.196 | 0.101 | 0.252 | 0.500 | 0.194 | 0.402 | 0.782 | 0.259 | 0.235
Prototype [52] | 0.708 | 0.430 | 0.279 | 0.334 | 0.270 | 0.449 | 0.419 | 0.320 | 0.654 | 0.807 | 0.534 | 0.607 | 0.779 | 0.470 | 0.451
PartSLIP [22] | 0.854 | 0.603 | 0.297 | 0.408 | 0.427 | 0.536 | 0.567 | 0.583 | 0.561 | 0.848 | 0.738 | 0.770 | 0.883 | 0.625 | 0.603
PartSLIP++ [53] | 0.853 | 0.605 | 0.297 | 0.451 | 0.495 | 0.724 | 0.574 | 0.632 | 0.575 | 0.630 | 0.720 | 0.856 | 0.883 | 0.642 | 0.615
3-By-2 (ours) | 0.844 | 0.657 | 0.453 | 0.544 | 0.402 | 0.896 | 0.604 | 0.626 | 0.790 | 0.901 | 0.782 | 0.815 | 0.928 | 0.665 | 0.642

###  4.1 Performance on PartNet-Ensembled

Data & Metric. We use the dataset provided by Liu et al. [22] for both the
few-shot and zero-shot settings. For each object in both few-shot and test
sets, we render 20 RGB images from different views with resolution $800\times
800$. We report mean IoU (mIoU) performance of all baselines using the
evaluation protocol provided by [22] on the input point clouds. Specifically,
the performance of a part is not considered if it does not exist in the
queried object.

Few-shot Baselines. We compare 3-By-2 against fully-supervised semantic
segmentation [32, 31, 41], few-shot semantic segmentation [36, 52] and
language-based [22, 53] methods. The fully supervised methods [32, 31, 41]
were trained on 28K objects of 17 overlapping categories between PartNetE
[22], in addition to the few-shot set consisting of 8 objects/category. The
second group of baselines [36, 22, 53, 52] were only trained on the few-shot
set. PartSLIP and PartSLIP++, a concurrent work, rely on large vision-language
model (GLIP [20]) to guide the 2D part detection before extending to the 3D
point cloud segmentation. We provide more detailed descriptions in the
Supplement. We omit the evaluation of MvDeCor [35] on this benchmark since it
requires ground-truth 3D meshes, whereas PartNetE only provides dense point
clouds as inputs.

Few-shot Setting. In this setting, 8 objects/category serve as the few-shot
set. We evaluate on the entire test set of PartNetE [22]. For a fair
comparison, we remove part labels in the test set that do not exist in the
few-shot set. We present our few-shot results in Table 1. Compared to fully-
supervised 3D methods, we outperform by 1-10% mIoU on these categories.
Additionally, we demonstrate a significant performance boost on the remaining
28 categories (21-41% mIoU). We further outperform PartSLIP and PartSLIP++ on
both subsets, achieving $\sim$3% mIoU improvements overall.

Performance on Real-world Scans. Please note that there is currently no
publicly available real-world 3D part segmentation dataset for direct
comparison. However, we demonstrate the robustness of our method using real-
world objects, as shown in Fig. $3\times 2$: 3D Object Part Segmentation by 2D
Semantic Correspondences. These objects were originally introduced by Liu et
al. [22] and captured using an iPhone12 camera.

Zero-shot Baselines. We compare 3-By-2 with PartSLIP [22], VLPart [37]-MC and
SAMPro3D [45]+OpenMask3D [38]. For PartSLIP, we prompt the pre-trained GLIP
model with the language inputs without finetuning, following Liu et al. [22].
VLPart [37] is a SOTA 2D part segmentation method that was trained on a
combination of various large-scale 2D part datasets. We replace our 2D part
segmentation module with a pre-trained VLPart model, retaining the 3D mask-
consistency aggregation module as 3-By-2, and term this baseline VLPart-MC.
During inference, to guide VLPart effectively, we prompt the model with
language inputs as in PartSLIP. SAMPro3D [45] is a SOTA zero-shot instance
segmentation method for 3D scenes using SAM at its core. For semantic
segmentation evaluation, we integrate SAMPro3D with OpenMask3D [38], an open-
vocabulary 3D scene segmentation method.

Zero-shot Setting. Since we do not have access to any labeled 3D objects in
this setting, to effectively transfer part labels, we leverage PACO [34]. This
dataset is a fine-grained and richly annotated 2D datasets consisting of
objects from COCO-LVIS [12]. We crop and mask each annotated object using the
provided object bounding box and segmentation mask to form the database.
Further, we filter out small objects or objects with limited visibility, using
the area of the object segmentation mask as a criterion.

Table 2: Zero-shot performance on the subset of PartNetE [22] that overlaps
with PACO [34]. Our method effectively leverages 2D in-the-wild part
segmentation dataset to perform 3D part segmentation.

Methods |  | Ket-
---
tle
| Micro-
---
wave
| Sci-
---
ssors
| F.-
---
Chair
Mouse |  | Bot-
---
tle
Laptop | Clock | Remote | Lamp |  | Avg.-
---
(18)
| SAMPro3D [45]+
---
OpenMask3D [38]
0.026 | 0.001 | 0.118 | 0.437 | 0.019 | 0.103 | 0.017 | 0.007 | 0.084 | 0.074 | 0.146
PartSLIP [22] | 0.208 | 0.166 | 0.218 | 0.917 | 0.270 | 0.763 | 0.270 | 0.267 | 0.115 | 0.371 | 0.341
VLPart [37]-MC | 0.211 | 0.192 | 0.193 | 0.813 | 0.000 | 0.216 | 0.060 | 0.205 | 0.132 | 0.166 | 0.222
3-By-2 (ours) | 0.765 | 0.348 | 0.594 | 0.712 | 0.307 | 0.807 | 0.394 | 0.253 | 0.239 | 0.500 | 0.430

In Table 2 we show the the performance of all baselines and 3-By-2 on the
subset of PartNetE that overlaps with PACO dataset [34]. By leveraging the
abundance and fine-grained of 2D in-the-wild part segmentation datasets, we
achieve superior performance compared to all baselines (9-29$\%$ mIoU). We
significantly outperform PartSLIP on challenging categories with small or thin
parts (e.g. scissors and lamp by $28\%$ and $13\%$ mIoU respectively). These
results highlight the effectiveness of 3-By-2 even when the database includes
challenging real-world images with partial occlusion and truncation.

Table 3: Performance on PartNet dataset with “level-3" annotations in the few-
shot setting. Bold and underline denote best and second best performance
respectively.

Methods |  | Bot-
---
tle
| Micro-
---
wave
| Dis-
---
play
| Dish-
---
washer
| Fau-
---
cet
Knife |  | Ear-
---
phone
Clock | Bed |  | Trash-
---
can
Avg.
MvDeCor [35] | 0.421 | 0.377 | 0.600 | 0.327 | 0.212 | 0.187 | 0.205 | 0.143 | 0.099 | 0.199 | 0.277
PartSLIP [22] | 0.344 | 0.143 | 0.386 | 0.228 | 0.009 | 0.023 | 0.064 | 0.017 | 0.003 | 0.031 | 0.125
3-By-2 (ours) | 0.454 | 0.389 | 0.567 | 0.429 | 0.203 | 0.196 | 0.225 | 0.116 | 0.096 | 0.134 | 0.281

Effectiveness of Our 2D Segmentation Module. We demonstrate the effectiveness
of our 2D segmenter, leveraging SAM and DIFT, by showcasing its strong
performance against VLPart [37], a SOTA 2D part segmentation method (see Table
2, last 2 rows). Note that VLPart was trained on PACO [34] among other 2D part
datasets. Therefore, it is reasonable to anticipate that this method can
effectively use knowledge from PACO to accurately segment the 18 overlapping
categories between PartNetE and PACO. For both VLPart-MC and 3-By-2, we
maintain the same 3D aggregation module. Our method significantly outperforms
VLPart-MC, demonstrating the advantage of our proposed 2D segmentation module.

Comparison to SOTA Scene Segmentation Approach. SAMPro3D [45] is a concurrent
work with SOTA performance on zero-shot instance segmentation in 3D scene.
This is a training-free model that effectively prompts SAM within the 2D
domain using 3D point projections. As in Table 2, we outperform this baseline
by a significant margin, highlighting the non-trivial nature of adapting scene
segmentation methods for 3D part segmentation tasks, particularly those
involving post-processing of 2D foundation models.

###  4.2 Performance on Level-3 PartNet

In this experiment, we select 10 categories from PartNet [27] that come with
fine-grained (“level-3") annotations. We randomly select 10 objects per
category from the training set (following [35]) to form our few-shot set, and
up to 50 objects per category from the test set for evaluation, ensuring
overlap with ShapeNetCore.v2 [6]. Given that PartSLIP [22] employs point cloud
RGB for superpoint generation, which serves as 3D priors, our decision to
choose overlapping objects with ShapeNetCore.v2 is to preserve object texture
information. We use the same few-shot and test set for all baselines.

Data. As inputs to our approach, we render 15 overlapping views for each
textured mesh using Blender cycle renderer with realistic lighting from HDRI
environment maps.

Baselines. The baselines are reproduced following the papers’ recommended
training procedure. Specifically, we pre-train MvDeCor [35] on the entire
training set of the selected categories consisting of 86 views per non-
textured object, with rendered RGB, depth and normal maps as inputs. We then
fine-tune the segmentation heads for each individual object category in the
few-shot set with 15 views per object. Note that the input for this stage also
includes RGB, depth and normal maps.

For PartSLIP [22], we derive the language prompt by traversing the part
hierarchy and concatenating labels from each level along the path, spanning
from root to leaf. For example, the path "bottle/jug/handle" is transformed
into "bottle jug handle". This adaptation is due to the potential for
different leaf nodes to share identical labels (e.g.,
bottle/normal_bottle/handle and bottle/jug/handle), as relying solely on the
leaf node label could introduce confusion in predictions. We adopt PartSLIP’s
point cloud, image rendering and data processing pipeline with default
parameters.

Evaluation & Metric. We uniformly sample 300K points on the surface of each
labeled ground truth mesh and employ nearest neighbor assignment to associate
a ground-truth label with each point. This point set is used for evaluating
all methods for a fair comparison and eliminating any randomness introduced by
the point cloud sampling step. We use part mIoU on the sampled point set as
the evaluation metric. We employ the standard mIoU calculation, which
considers the performance of all parts in the vocabulary, even in cases where
they may not exist in certain objects. Additionally, different from MvDeCor,
we do not exclude the “others" label during evaluation based on ground-truth
labels. For a fair comparison, we applied the same evaluation approach across
all methods.

Results. We show results in Table 3. Compared to PartSLIP [22], we
outperformed on all categories by a significant margin (16% mIoU on average),
demonstrating the challenges posed by fine-grained settings for GLIP [20].
While our performance is on par with MvDeCor [35], it is important to note
that MvDeCor is both pretrained and finetuned on PartNet [27], using ground
truth depth and normal maps as additional inputs. In contrast, our method
requires no training on the target data distribution.

###  4.3 Ablation Study

Non-overlapping Mask Generation. In Table 5, we illustrate the effectiveness
of our proposed non-overlapping mask generation module. The comparison
involves evaluating the performance of our method with and without this
module. In the case of the model without the non-overlapping mask generation
module, we directly utilize the predicted SAM outputs for label transferring.
The results indicate that our non-overlapping mask generation module is
necessary for achieving an optimal performance.

Table 4: Ablation of the non-overlapping mask generation module.

2D Mask Proposal | Scissors | Mouse | Suitcase | Bottle | Chair
---|---|---|---|---|---
SAM | 0.457 | 0.440 | 0.285 | 0.004 | 0.638
Non-overlap | 0.675 | 0.684 | 0.813 | 0.810 | 0.844

Table 5: Ablation of our proposed mask-consistency component.

3D Label Aggregation | Scissors | Suitcase | Printer | Clock
---|---|---|---|---
Point-Consistency | 0.619 | 0.579 | 0.009 | 0.363
Mask-Consistency | 0.675 | 0.684 | 0.085 | 0.458

Mask-consistency Module. In Table 5, we demonstrate the effectiveness of our
proposed mask-consistency component, which improves the final performance
especially on objects with small parts.

Properties of Database. In this section, we investigate two key questions: 1)
Can 3-By-2 accurately segment the query object within a database containing
multiple object categories? and 2) Is it possible to transfer parts with the
same semantic meaning between different object categories?

Multi-category database. To address question 1, we perform experiments using
databases containing 1, 2, and 8 categories respectively (see Table 7).
Specifically, taking the query category as “Kettle", for the 2-category
setting we construct a database consisting of "Kettle, Kitchen Pot". We
selected these categories due to their shared semantic parts with "Kettle",
which could potentially lead to confusion (e.g., kettle lid vs. kitchen pot
lid). With 8-category setting, we add in categories that are completely
different and do not share any parts with “Kettle" (e.g. “Eyeglasses"). In
general, with more categories in the database, there is a slight decrease in
the average performance. Notably, there are marginal differences between
2-category and 8-category (second and third rows), highlighting the the
ability of 3-By-2 in handling both diverse object taxonomy and part
segmentation. This finding is particularly interesting since many prior works
[22, 35] require finetuning each category separately for few-shot evaluation.

Cross-category database. Considering question 2, we note that the few-shot set
of “Table" in PartNetE lacks objects with wheels as a part, whereas such
objects are present in the test set. To address this, we incorporate the
“Chair" category where the wheel part exists in the database. We evaluate on
18 tables in PartNetE test set with the “wheel" part annotated (see Table 7).
Compared to the table only few-shot set, combining the database with “Chair"
improves the performance on “leg" by $\sim 6\%$ mIoU. The improvement in the
“leg" part can be attributed to the inclusion of “Chair" in the database,
which reduces the likelihood of the model incorrectly associating “wheel" with
“leg" due to the absence of “wheel" in the few-shot set. Interestingly, the
performance for “wheel" increases significantly, $+60\%$ mIoU through the
label transfer from chair wheels.

Table 6: Multi-category database experiment. Performance of Kettle in various
database settings is reported with mIoU. Our method shows robustness in
performance even when more categories are added in the database.

Database | Lid | Handle | Spout | Avg.
---|---|---|---|---
1-category | 0.759 | 0.904 | 0.783 | 0.815
2-category | 0.703 | 0.820 | 0.748 | 0.757
8-category | 0.727 | 0.773 | 0.756 | 0.752

Table 7: Cross-category database experiment. We report the performance of 18
tables with wheels in PartNetE. Results show that our method can transfer
wheel annotations from Chair to correct the prediction on Table wheels.

Database | Leg | Tabletop | Wheel | Avg.
---|---|---|---|---
Table only | 0.586 | 0.647 | 0.000 | 0.411
Chair & Table | 0.641 | 0.633 | 0.600 | 0.625

While the concept may seem intuitive, our findings shed new light on object
part compositionality. Despite the diversity in appearances and shapes across
various object categories, there exists a finite set of object parts that are
shared among them. Recognizing the transferability of these parts is important
for facilitating rapid learning of novel objects across a range of tasks.
Further, our results show the ability to correct wrong predictions of our
approach by transferring labels from another category. Please refer to the
Sup. for additional studies.

##  5 Conclusion

In this work, we propose 3-By-2, a novel, training-free method that achieves
SOTA performance on benchmarks with diverse levels of part granularity without
the need for language inputs, on both zero-shot and few-shot settings. We
demonstrate the flexibility of 3-By-2 in transferring part labels between
different object categories. We hope the development of 3-By-2 can encourage
further exploration of visual similarities for this task.

## Acknowledgement

This work was partly supported by NIH R01HD104624-01A1. This Supplementary
Material is structured as follows: in Section 6 we provide more details about
the datasets, the data rendering pipelines and ground-truth processing; in
Section 7 we present full results for the PartNetE few-shot and zero-shot
experiments in the main text; in Section 8 we perform additional ablation
studies for our method; in Section 9, we provide implementation details of our
approach; in Section 10, we demonstrate the effectiveness of the semantic
correspondence matching process; in Section 11 we discuss about the limitation
of our work and potential future directions; in Section 12, we present
additional qualitative results.

##  6 Data

###  6.1 Datasets

PartNet-Ensembled [22]. This dataset combines objects from the PartNet-
Mobility dataset [44] for few-shot and test splits, along with additional
training objects from PartNet [27]. In our study, we exclusively use the few-
shot and test splits. The few-shot split encompasses 8 objects per category
across all 45 categories, while the test split spans from 6 to 338 objects per
category, having 1906 objects in total. PartNetE consists of 106 parts,
ranging from basic components like chair backs to more fine-grained parts such
as scissors screws. Note that compared to the level-3 annotations of PartNet
[27], PartNetE exhibits a coarser granularity and consists of 28 additional
categories from PartNet Mobility dataset [44].

Figure 9: Rendered images for 3-By-2 on PartNetE [22]. Each object is rendered
with resolution $800\times 800$ images on a white background.

PartNet [27]. This dataset consists of annotations at various levels of
granularity, spanning from level-1 to level-3, where level-3 represents the
most fine-grained part annotations, including labels such as
“back_frame_vertical_bar" for chairs. We perform experiments on 10 object
categories with level-3 part annotations. As mentioned in the main text, we
use a subset of PartNet that overlaps with ShapeNet [6].

###  6.2 Data Rendering

PartNetE Experiments. For these experiments, we use the rendering code
provided by Liu et al. [22]. Specifically, we employ PyTorch3D point cloud
rasterization to render the input point clouds. We render 20 RGB views for
each point cloud on a white background. Pixel and 3D point correspondences are
saved for inference. Refer to Figure 9 for examples of rendered images.

PartNet Experiment. For our method, we employ Blender [1] to render textured
meshes from ShapeNet [6], incorporating an HDRI environment map to ensure
photorealistic lighting variations (refer to Figure 10, 2 top rows). For each
object, we generate 15 RGB views with resolution $800\times 800$. At
inference, we use the object’s ground-truth segmentation mask to mask out the
background. We additionally center and square crop the objects based on their
bounding boxes, with a padding of 16 pixels on each side (Figure 10, 2 bottom
rows).

For PartSLIP [22], we first convert each textured mesh into textured point
cloud using BlenderProc [10]. We then use the provided point cloud rendering
code from PartSLIP to render input images with default rendering parameters.

For MvDeCor [35], we follow the rendering code provided by the authors to
render RGB, depth and normal maps for each untextured mesh. For pre-training,
we follow the default rendering parameters from [35] and render 86 views on a
white background with resolution $256\times 256$. During low-shot finetuning
and inference, we use 15 views per object.

###  6.3 Ground-truth Processing

For PartNetE experiments, we use the ground-truth labels provided by Liu et
al. [22]. In the case of PartNet, we follow [35], where we initially convert
the ground-truth labels for 1024 points to labels for triangle faces of the
processed object mesh. Subsequently, we sample a dense point cloud of 300K
points from the surface of each object and transfer the labels from the mesh
to this standardized point cloud via querying the nearest face neighbor for
each point. This point cloud is used to evaluate the performance of all
methods.

Figure 10: (Top) Rendered images for 3-By-2 on PartNet [27]. We use Blender to
render each object with photorealistic lightings from HDRI environment maps.
(Bottom) Square cropped and masked inputs used by 3-By-2 for label
transferring.

##  7 Full Table Results

###  7.1 Metric Details

For each object category, we first record the average IoU for each object
part. The mean IoU reported for each category is the average IoU of all the
parts in that category.

The standard Intersection-over-Union (IoU) metric is computed as the
following:

$IoU_{\text{standard}}=\frac{m_{\text{pred.}}\cap
m_{\text{gt}}}{m_{\text{pred.}}\cup m_{\text{gt}}}$

where $m_{\text{pred.}}$ and $m_{\text{gt}}$ are the binary masks for the
predicted and ground-truth segmentation respectively. We employ this
implementation of IoU in our PartNet experiment.

In the PartNetE experiments, we follow the evaluation protocol outlined in
[22] to ensure a fair comparison with baselines and maintain consistencies.
Specifically, we omit the prediction of a part if that part does not actually
exist in the object. Formally,

$IoU=\mathds{1}\\{m_{\text{gt}}\neq\mathbb{0}\\}\cdot\frac{m_{\text{pred.}}\cap
m_{\text{gt}}}{m_{\text{pred.}}\cup m_{\text{gt}}}$

###  7.2 Few-shot on PartNetE

In this setting, we compare our method against 3D fully-supervised semantic
segmentation methods: PointNext [32], PointNet++ [31] and SoftGroup [41]; few-
shot semantic segmentation methods: ACD [11] and Prototype [52]; language-
based part segmentation methods: PartSLIP [22] and PartSLIP++ [53].

The first group of methods was trained fully-supervised on all objects of 17
categories in PartNet [27], in addition to the few-shot set of PartNetE, which
consists of 8 objects for each of 45 categories. ACD and Prototype exclusively
trained on the few-shot subset of PartNetE. While ACD [11] integrates a proxy
self-supervised task to facilitate few-shot learning, Prototype [52] extracts
features serving as prototypes for each part category. In contrast, PartSLIP
[22] and PartSLIP++ [53] relies on large vision-language model (GLIP [20]) to
guide the 2D part detection before extending to the 3D point cloud
segmentation. All these baselines require either fine-tuning or training on
the few-shot set, whereas our approach is training-free, using the few-shot
set solely for constructing the database to transfer labels. We present the
performance on all 45 categories of PartNetE in the few-shot setting in Table
8 and Table 9.

Table 8: Few-shot performance on PartNetE [22] dataset on the 17 categories
that overlap between PartNet [27] and PartNetE. The first three columns show
the performance of fully-supervised methods. The right columns show
performance of the methods that only have access to the few-shot set.

Categories $\downarrow$ |  | PointNet++
---
[31]
| SoftGroup
---
[41]
| PointNext
---
[32]
| ACD
---
[11]
| Prototype
---
[52]
| PartSLIP
---
[22]
| PartSLIP++
---
[53]
3-By-2
Bottle | 0.488 | 0.414 | 0.684 | 0.224 | 0.601 | 0.834 | 0.855 | 0.809
Chair | 0.847 | 0.883 | 0.918 | 0.390 | 0.708 | 0.854 | 0.853 | 0.844
Clock | 0.192 | 0.025 | 0.284 | 0.000 | 0.105 | 0.376 | 0.541 | 0.458
Dishwasher | 0.495 | 0.530 | 0.692 | 0.253 | 0.483 | 0.625 | 0.609 | 0.536
Display | 0.783 | 0.621 | 0.894 | 0.291 | 0.673 | 0.848 | 0.851 | 0.726
Door | 0.457 | 0.531 | 0.438 | 0.189 | 0.334 | 0.408 | 0.451 | 0.544
Faucet | 0.672 | 0.684 | 0.850 | 0.242 | 0.460 | 0.714 | 0.659 | 0.669
Keyboard | 0.745 | 0.589 | 0.450 | 0.261 | 0.449 | 0.536 | 0.724 | 0.896
Knife | 0.354 | 0.313 | 0.587 | 0.396 | 0.504 | 0.652 | 0.643 | 0.751
Lamp | 0.680 | 0.822 | 0.649 | 0.137 | 0.382 | 0.661 | 0.680 | 0.595
Laptop | 0.554 | 0.184 | 0.325 | 0.111 | 0.279 | 0.297 | 0.297 | 0.453
Microwave | 0.436 | 0.383 | 0.405 | 0.066 | 0.270 | 0.427 | 0.495 | 0.402
Refrigerator | 0.434 | 0.469 | 0.762 | 0.108 | 0.429 | 0.558 | 0.557 | 0.517
Scissors | 0.500 | 0.760 | 0.573 | 0.391 | 0.430 | 0.603 | 0.605 | 0.657
StorageFurniture | 0.469 | 0.602 | 0.685 | 0.076 | 0.302 | 0.536 | 0.573 | 0.517
Table | 0.577 | 0.591 | 0.616 | 0.202 | 0.386 | 0.484 | 0.521 | 0.566
TrashCan | 0.717 | 0.170 | 0.228 | 0.000 | 0.329 | 0.223 | 0.242 | 0.331
Avg. (17) | 0.533 | 0.505 | 0.591 | 0.196 | 0.419 | 0.567 | 0.574 | 0.604

Table 9: Few-shot performance on PartNetE [22] dataset on the remaining 28
categories. The first three columns show the performance of fully-supervised
methods. The right columns show performance of the methods that only have
access to the few-shot set.

Categories $\downarrow$ |  | PointNet++
---
[31]
| SoftGroup
---
[41]
| PointNext
---
[32]
| ACD
---
[11]
| Prototype
---
[52]
| PartSLIP
---
[22]
| PartSLIP++
---
[53]
3-By-2
Box | 0.186 | 0.088 | 0.842 | 0.211 | 0.686 | 0.845 | 0.855 | 0.761
Bucket | 0.000 | 0.250 | 0.041 | 0.000 | 0.313 | 0.365 | 0.855 | 0.784
Camera | 0.065 | 0.236 | 0.332 | 0.101 | 0.320 | 0.583 | 0.632 | 0.626
Cart | 0.064 | 0.239 | 0.363 | 0.315 | 0.368 | 0.881 | 0.849 | 0.812
CoffeeMachine | 0.346 | 0.083 | 0.179 | 0.007 | 0.200 | 0.378 | 0.376 | 0.342
Dispenser | 0.121 | 0.189 | 0.260 | 0.194 | 0.534 | 0.738 | 0.720 | 0.782
Eyeglasses | 0.762 | 0.724 | 0.881 | 0.782 | 0.779 | 0.883 | 0.883 | 0.928
FoldingChair | 0.109 | 0.147 | 0.964 | 0.742 | 0.912 | 0.863 | 0.899 | 0.936
Globe | 0.465 | 0.590 | 0.923 | 0.698 | 0.883 | 0.957 | 0.928 | 0.952
Kettle | 0.209 | 0.574 | 0.451 | 0.402 | 0.607 | 0.770 | 0.856 | 0.815
KitchenPot | 0.158 | 0.455 | 0.570 | 0.519 | 0.633 | 0.696 | 0.729 | 0.650
Lighter | 0.350 | 0.302 | 0.163 | 0.108 | 0.307 | 0.647 | 0.662 | 0.650
Mouse | 0.167 | 0.559 | 0.325 | 0.000 | 0.301 | 0.580 | 0.587 | 0.684
Oven | 0.344 | 0.137 | 0.378 | 0.089 | 0.365 | 0.735 | 0.703 | 0.600
Pen | 0.465 | 0.289 | 0.395 | 0.054 | 0.475 | 0.715 | 0.664 | 0.628
Phone | 0.200 | 0.024 | 0.395 | 0.099 | 0.355 | 0.484 | 0.590 | 0.410
Pliers | 0.577 | 0.742 | 0.996 | 0.835 | 0.910 | 0.332 | 0.297 | 0.994
Printer | 0.000 | 0.012 | 0.000 | 0.000 | 0.002 | 0.043 | 0.062 | 0.085
Remote | 0.036 | 0.371 | 0.578 | 0.000 | 0.296 | 0.383 | 0.364 | 0.541
Safe | 0.319 | 0.053 | 0.256 | 0.093 | 0.201 | 0.322 | 0.372 | 0.347
Stapler | 0.516 | 0.801 | 0.886 | 0.500 | 0.807 | 0.848 | 0.630 | 0.901
Suitcase | 0.407 | 0.183 | 0.136 | 0.132 | 0.355 | 0.704 | 0.700 | 0.652
Switch | 0.018 | 0.210 | 0.397 | 0.103 | 0.409 | 0.594 | 0.561 | 0.546
Toaster | 0.147 | 0.264 | 0.084 | 0.000 | 0.101 | 0.600 | 0.508 | 0.565
Toilet | 0.180 | 0.180 | 0.165 | 0.267 | 0.292 | 0.701 | 0.705 | 0.569
USB | 0.524 | 0.441 | 0.679 | 0.252 | 0.654 | 0.561 | 0.575 | 0.790
WashingMachine | 0.006 | 0.241 | 0.273 | 0.101 | 0.311 | 0.535 | 0.487 | 0.526
Window | 0.263 | 0.392 | 0.833 | 0.668 | 0.766 | 0.754 | 0.728 | 0.739
Avg. (28) | 0.250 | 0.313 | 0.457 | 0.259 | 0.470 | 0.625 | 0.642 | 0.665
Avg. (45) | 0.365 | 0.384 | 0.502 | 0.235 | 0.451 | 0.603 | 0.615 | 0.642

Figure 11: Number of views ablation on PartNetE.

###  7.3 Zero-shot on PartNetE

We present the performance on 18 categories of PartNetE [22] that overlaps
with PACO [34] in Table 10. With each category, we form a database of at most
150 objects from PACO. We first mask and crop each object in PACO using the
ground-truth bounding box and segmentation mask. Objects that are either too
small or not visible in the image are subsequently filtered out (see Figure
12). We manually convert labels from PACO to PartNetE for evaluation. For
parts that exist in PartNetE but do not exist in PACO, we report the
performance as 0 IoU.

We compare 3-By-2 with PartSLIP [22], VLPart [37]-MC and SAMPro3D [45] \+
OpenMask3D [38]. For PartSLIP, we prompt the pre-trained GLIP model with the
language inputs without finetuning, following Liu et al. [22]. VLPart [37] is
a SOTA 2D part segmentation method that was trained on a combination of
various large-scale 2D part datasets. We replace our 2D part segmentation
module with a pre-trained VLPart model, retaining the 3D mask-consistency
aggregation module as 3-By-2, and term this baseline VLPart-MC. During
inference, to guide VLPart effectively, we prompt the model with language
inputs as in PartSLIP. SAMPro3D [45] is a SOTA zero-shot instance segmentation
method for 3D scenes using SAM at its core. For semantic segmentation
evaluation, we integrate SAMPro3D with OpenMask3D [38], an open-vocabulary 3D
scene segmentation method.

Table 10: Zero-shot performance on the subset of PartNetE [22] that overlaps
with PACO [34]. Our method effectively leverages 2D in-the-wild part
segmentation dataset to perform 3D part segmentation.

Categories $\downarrow$ |  | SAMPro [45]+
---
OpenMask3D [38]
| VLPart [37]-
---
MC
| PartSLIP
---
[22]
3-By-2
Bottle | 0.103 | 0.216 | 0.763 | 0.807
Box | 0.325 | 0.313 | 0.575 | 0.369
Bucket | 0.035 | 0.253 | 0.020 | 0.124
Chair | 0.024 | 0.066 | 0.607 | 0.468
Clock | 0.007 | 0.205 | 0.267 | 0.253
FoldingChair | 0.437 | 0.813 | 0.917 | 0.712
Kettle | 0.026 | 0.211 | 0.208 | 0.765
Knife | 0.267 | 0.427 | 0.468 | 0.413
Lamp | 0.074 | 0.166 | 0.371 | 0.500
Laptop | 0.017 | 0.060 | 0.270 | 0.394
Microwave | 0.000 | 0.192 | 0.166 | 0.348
Mouse | 0.019 | 0.000 | 0.270 | 0.307
Pen | 0.088 | 0.016 | 0.146 | 0.010
Pliers | 0.996 | 0.494 | 0.054 | 0.944
Remote | 0.084 | 0.132 | 0.115 | 0.239
Scissors | 0.118 | 0.193 | 0.218 | 0.594
Table | 0.004 | 0.114 | 0.398 | 0.251
TrashCan | 0.000 | 0.119 | 0.301 | 0.239
Avg. (18) | 0.146 | 0.222 | 0.341 | 0.430

Figure 12: Processed images from PACO [34] that are used as the database for
our zero-shot experiment. Figure 13: Non-overlapping masks generated by our
method.

##  8 More Ablation Results

###  8.1 Non-overlapping Mask Generation

In Figure 13, we demonstrate the visualization of non-overlapping mask outputs
from our non-overlapping mask generation module.

Figure 14: (Top) Correspondence retrieval between PartNetE test objects and
PACO. (Bottom) Correspondence retrieval between PartNetE few-shot and test
sets. Our method using Diffusion Features [39] perform well in both cases.

###  8.2 Results on Multiple Runs

Table 11 presents the few-shot setting results across multiple runs. For each
category, we conduct label transferring independently 3 times and report the
average mIoU along with the standard deviation across these runs. Note that
the standard deviation demonstrates a relatively small variation between the
runs, indicating that sparsely sampling points inside the mask for label
transferring does not have a significant impact on the final performance of
our model.

Table 11: Average mIoU and standard deviation for 3 independent runs. Randomly
sampling pixels for label transfer has minimal effect on the final performance
of the method.

Scissors | Kettle | Suitcase | Bottle
---|---|---|---
Avg. | Std. | Avg. | Std. | Avg. | Std. | Avg. | Std.
0.675 | 0.008 | 0.663 | 0.002 | 0.813 | 0.003 | 0.810 | 0.007

Table 12: Ablation for different number of instances in the few-shot database
on Kettle. We demonstrate strong performance even with only 1 labeled
instance.

Database | Lid | Handle | Spout | Avg. | PartSLIP Avg.
---|---|---|---|---|---
1-instance | 0.762 | 0.901 | 0.765 | 0.809 | 0.200
4-instance | 0.784 | 0.893 | 0.774 | 0.817 | 0.690
8-instance | 0.759 | 0.904 | 0.783 | 0.815 | 0.770

###  8.3 Number of Views Ablation

In Fig. 11, we ablate the number of views. Using more than 10 views has
minimal impact on performance, aligning with the observations in PartSLIP
[22]. In all experiments, we follow the optimal number of views for all
baselnes.

###  8.4 More Analyses on Few-shot Database

Table 12 presents the results for Kettle using various numbers of object
instances in the database for few-shot label transfer. The instances are
randomly selected from the few-shot set of 8 objects. Note that even with a
single instance in the database, 3-By-2 achieves remarkable performance,
significantly outperforming PartSLIP [22]. The performance gap between the
1-instance and 8-instance database settings for our method is negligible,
demonstrating that 3-By-2 can effectively operate with a minimal number of
labeled objects.

##  9 Implementation Details

We use the image diffusion feature extractor [39] as our backbone. We extract
features at the first upsampling block of the U-Net at time $t=261$. We use an
empty string as prompt input to the image diffusion model. For SAM [19], we
use point sampling as the prompt with a grid size of $64\times 64$. For all
majority voting operations, we employ a cut-off threshold of $0.6$. That is,
if $x_{i}\geq 0.6~{}\Sigma_{j}x_{j}$ and $x_{i}=\max_{j}x_{j}$ then $x_{i}$ is
the dominant element of $\\{x_{j}\\}_{j=1,\dots,N}$.

##  10 Semantic Correspondence Matching Visualization

We show the visualization of the semantic correspondence matching in Figure
14. For each pixel in the query object, we find the closest match from the
database using cosine distances of the features extracted from DIFT [39]. The
semantic matches are high quality for both PACO (zero-shot) and PartNetE (few-
shot).

##  11 Limitations & Future Work

While our approach offers flexibility across multiple levels of granularity
and diverse object categories, it is important to acknowledge certain
limitations. The performance of our method is directly tied to the performance
of SAM [19] and DIFT [39]. Additionally, for scenarios like animal or human
part segmentation, where part boundaries are usually not determined by edges,
applying SAM in this case may prove suboptimal for 2D mask proposal. However,
we note that one can effortlessly switch out SAM and DIFT for other
segmentation and correspondence matching methods. The choice for leveraging
these models stems from their status as current SOTA approaches. Future
improvements of either one of these tasks can further improve the performance
of our approach. In addition, noise in labels in the database can lead to
incorrect label assignments, even when the correspondences are accurately
identified by the DIFT features.

Due to the strong taxonomy preservation capability of our method (see Sec.
4.3), future work can explore applying similar approach to other 3D tasks such
as object classification or scene semantic segmentation.

##  12 Additional Qualitative Results

In Fig.15, we present a qualitative comparison between PartSLIP[22] and our
approach. Our method, 3-By-2, demonstrates a superior ability to capture small
parts with higher precision. We further demonstrate the qualitative
performance of our method on non-rigid objects, including deformable (teddy
bear) and articulated (door and scissors) objects (Fig. 16).

Figure 15: Few-shot qualitative comparison between PartSLIP [22] and 3-By-2.
Our approach is more successful at capturing small parts and details. Figure
16: Few-shot is the ground-truth segmentation template, Pred1 and Pred2 are
the segmentation predictions on 2 different samples given the few-shot
template masks.

## References

  * [1] Blender, https://blender.org/
  * [2] Abdelreheem, A., Skorokhodov, I., Ovsjanikov, M., Wonka, P.: Satr: Zero-shot semantic segmentation of 3d shapes. arXiv preprint arXiv:2304.04909 (2023)
  * [3] Amir, S., Gandelsman, Y., Bagon, S., Dekel, T.: Deep vit features as dense visual descriptors. arXiv preprint arXiv:2112.05814 2(3), 4 (2021)
  * [4] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9650–9660 (2021)
  * [5] Cen, J., Zhou, Z., Fang, J., Shen, W., Xie, L., Jiang, D., Zhang, X., Tian, Q., et al.: Segment anything in 3d with nerfs. Advances in Neural Information Processing Systems 36 (2024)
  * [6] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)
  * [7] Chen, N., Liu, L., Cui, Z., Chen, R., Ceylan, D., Tu, C., Wang, W.: Unsupervised learning of intrinsic structural representation points. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9121–9130 (2020)
  * [8] Dai, A., Nießner, M.: 3dmv: Joint 3d-multi-view prediction for 3d semantic scene segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 452–468 (2018)
  * [9] Deng, S., Xu, X., Wu, C., Chen, K., Jia, K.: 3d affordancenet: A benchmark for visual object affordance understanding. In: proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1778–1787 (2021)
  * [10] Denninger, M., Winkelbauer, D., Sundermeyer, M., Boerdijk, W., Knauer, M., Strobl, K.H., Humt, M., Triebel, R.: Blenderproc2: A procedural pipeline for photorealistic rendering. Journal of Open Source Software 8(82), 4901 (2023). https://doi.org/10.21105/joss.04901, https://doi.org/10.21105/joss.04901
  * [11] Gadelha, M., RoyChowdhury, A., Sharma, G., Kalogerakis, E., Cao, L., Learned-Miller, E., Wang, R., Maji, S.: Label-efficient learning on point clouds using approximate convex decompositions. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16. pp. 473–491. Springer (2020)
  * [12] Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5356–5364 (2019)
  * [13] He, J., Yang, S., Yang, S., Kortylewski, A., Yuan, X., Chen, J.N., Liu, S., Yang, C., Yu, Q., Yuille, A.: Partimagenet: A large, high-quality dataset of parts. In: European Conference on Computer Vision. pp. 128–145. Springer (2022)
  * [14] Hedlin, E., Sharma, G., Mahajan, S., Isack, H., Kar, A., Tagliasacchi, A., Yi, K.M.: Unsupervised semantic correspondence using stable diffusion. arXiv preprint arXiv:2305.15581 (2023)
  * [15] Huang, R., Peng, S., Takmaz, A., Tombari, F., Pollefeys, M., Song, S., Huang, G., Engelmann, F.: Segment3d: Learning fine-grained class-agnostic 3d segmentation without manual labels. arXiv preprint arXiv:2312.17232 (2023)
  * [16] Jaritz, M., Gu, J., Su, H.: Multi-view pointnet for 3d scene understanding. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) pp. 3995–4003 (2019), https://api.semanticscholar.org/CorpusID:203593088
  * [17] Kalogerakis, E., Hertzmann, A., Singh, K.: Learning 3D Mesh Segmentation and Labeling. ACM Transactions on Graphics 29(3) (2010)
  * [18] Kim, H., Sung, M.: Partstad: 2d-to-3d part segmentation task adaptation (2024)
  * [19] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [20] Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10965–10975 (2022)
  * [21] Li, Y., Upadhyay, U., Habib Slim, A.A., Arpit Prajapati, S.P., Wonka, P., Elhoseiny, M.: 3d compat: Composition of materials on parts of 3d things (eccv 2022). ECCV (2022)
  * [22] Liu, M., Zhu, Y., Cai, H., Han, S., Ling, Z., Porikli, F., Su, H.: Partslip: Low-shot part segmentation for 3d point clouds via pretrained image-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21736–21746 (2023)
  * [23] Liu, W., Mao, J., Hsu, J., Hermans, T., Garg, A., Wu, J.: Composable part-based manipulation. In: 7th Annual Conference on Robot Learning (2023), https://openreview.net/forum?id=o-K3HVUeEw
  * [24] Liu, X., Xu, X., Rao, A., Gan, C., Yi, L.: Autogpart: Intermediate supervision search for generalizable 3d part segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11624–11634 (2022)
  * [25] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International journal of computer vision 60, 91–110 (2004)
  * [26] Min, J., Lee, J., Ponce, J., Cho, M.: Spair-71k: A large-scale benchmark for semantic correspondence. arXiv preprint arXiv:1908.10543 (2019)
  * [27] Mo, K., Zhu, S., Chang, A.X., Yi, L., Tripathi, S., Guibas, L.J., Su, H.: Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 909–918 (2019)
  * [28] Nadeau, P., Giamou, M., Kelly, J.: The sum of its parts: Visual part segmentation for inertial parameter identification of manipulated objects. arXiv preprint arXiv:2302.06685 (2023)
  * [29] Nguyen, P.D.A., Ngo, T.D., Gan, C., Kalogerakis, E., Tran, A., Pham, C., Nguyen, K.: Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance (2023)
  * [30] Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., Funkhouser, T., et al.: Openscene: 3d scene understanding with open vocabularies. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 815–824 (2023)
  * [31] Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems 30 (2017)
  * [32] Qian, G., Li, Y., Peng, H., Mai, J., Hammoud, H., Elhoseiny, M., Ghanem, B.: Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in Neural Information Processing Systems 35, 23192–23204 (2022)
  * [33] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [34] Ramanathan, V., Kalia, A., Petrovic, V., Wen, Y., Zheng, B., Guo, B., Wang, R., Marquez, A., Kovvuri, R., Kadian, A., et al.: Paco: Parts and attributes of common objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7141–7151 (2023)
  * [35] Sharma, G., Yin, K., Maji, S., Kalogerakis, E., Litany, O., Fidler, S.: Mvdecor: Multi-view dense correspondence learning for fine-grained 3d segmentation. In: European Conference on Computer Vision. pp. 550–567. Springer (2022)
  * [36] Singh, C., Murdoch, W.J., Yu, B.: Hierarchical interpretations for neural network predictions. arXiv preprint arXiv:1806.05337 (2018)
  * [37] Sun, P., Chen, S., Zhu, C., Xiao, F., Luo, P., Xie, S., Yan, Z.: Going denser with open-vocabulary part segmentation. arXiv preprint arXiv:2305.11173 (2023)
  * [38] Takmaz, A., Fedele, E., Sumner, R.W., Pollefeys, M., Tombari, F., Engelmann, F.: Openmask3d: Open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306.13631 (2023)
  * [39] Tang, L., Jia, M., Wang, Q., Phoo, C.P., Hariharan, B.: Emergent correspondence from image diffusion. arXiv preprint arXiv:2306.03881 (2023)
  * [40] Varadarajan, K.M., Vincze, M.: Object part segmentation and classification in range images for grasping. In: 2011 15th International Conference on Advanced Robotics (ICAR). pp. 21–27. IEEE (2011)
  * [41] Vu, T., Kim, K., Luu, T.M., Nguyen, T., Yoo, C.D.: Softgroup for 3d instance segmentation on point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2708–2717 (2022)
  * [42] Wang, L., Li, X., Fang, Y.: Few-shot learning of part-specific probability space for 3d shape segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (June 2020)
  * [43] Wang, R., Zhang, Y., Mao, J., Zhang, R., Cheng, C.Y., Wu, J.: Ikea-manual: Seeing shape assembly step by step. Advances in Neural Information Processing Systems 35, 28428–28440 (2022)
  * [44] Xiang, F., Qin, Y., Mo, K., Xia, Y., Zhu, H., Liu, F., Liu, M., Jiang, H., Yuan, Y., Wang, H., et al.: Sapien: A simulated part-based interactive environment. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11097–11107 (2020)
  * [45] Xu, M., Yin, X., Qiu, L., Liu, Y., Tong, X., Han, X.: Sampro3d: Locating sam prompts in 3d for zero-shot scene segmentation. arXiv preprint arXiv:2311.17707 (2023)
  * [46] Xue, Y., Chen, N., Liu, J., Sun, W.: Zerops: High-quality cross-modal knowledge transfer for zero-shot 3d part segmentation (2023)
  * [47] Yang, Y., Wu, X., He, T., Zhao, H., Liu, X.: Sam3d: Segment anything in 3d scenes. arXiv preprint arXiv:2306.03908 (2023)
  * [48] Yi, L., Kim, V.G., Ceylan, D., Shen, I.C., Yan, M., Su, H., Lu, C., Huang, Q., Sheffer, A., Guibas, L.: A scalable active framework for region annotation in 3d shape collections. SIGGRAPH Asia (2016)
  * [49] Yu, Q., Du, H., Liu, C., Yu, X.: When 3d bounding-box meets sam: Point cloud instance segmentation with weak-and-noisy supervision. ArXiv abs/2309.00828 (2023), https://api.semanticscholar.org/CorpusID:261530997
  * [50] Zhang, J., Herrmann, C., Hur, J., Cabrera, L.P., Jampani, V., Sun, D., Yang, M.H.: A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. arXiv preprint arXiv:2305.15347 (2023)
  * [51] Zhao, L., Lu, J., Zhou, J.: Similarity-aware fusion network for 3d semantic segmentation. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) pp. 1585–1592 (2021), https://api.semanticscholar.org/CorpusID:235732071
  * [52] Zhao, N., Chua, T.S., Lee, G.H.: Few-shot 3d point cloud semantic segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8873–8882 (2021)
  * [53] Zhou, Y., Gu, J., Li, X., Liu, M., Fang, Y., Su, H.: Partslip++: Enhancing low-shot 3d part segmentation via multi-view instance segmentation and maximum likelihood estimation. arXiv preprint arXiv:2312.03015 (2023)
  * [54] Zhu, J., Zhang, Y., Guo, J., Liu, H., Liu, M., Liu, Y., Guo, Y.: Label transfer between images and 3d shapes via local correspondence encoding. Comput. Aided Geom. Des. 71(C), 255–266 (may 2019). https://doi.org/10.1016/j.cagd.2019.04.009, https://doi.org/10.1016/j.cagd.2019.04.009
  * [55] Zhu, X., Zhang, R., He, B., Guo, Z., Zeng, Z., Qin, Z., Zhang, S., Gao, P.: Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 2639–2650 (2023)