# Adapting Pre-Trained Vision Models for Novel Instance Detection and
Segmentation

Yangxiao Lu  Department of Computer Science
University of Texas at Dallas
Richardson, TX 75080, USA
<EMAIL_ADDRESS>Jishnu
Jaykumar P  Department of Computer Science
University of Texas at Dallas
Richardson, TX 75080, USA
<EMAIL_ADDRESS>Yunhui
Guo  Department of Computer Science
University of Texas at Dallas
Richardson, TX 75080, USA
<EMAIL_ADDRESS>Nicholas Ruozzi  Department of Computer Science
University of Texas at Dallas
Richardson, TX 75080, USA
<EMAIL_ADDRESS>Yu
Xiang  Department of Computer Science
University of Texas at Dallas
Richardson, TX 75080, USA
<EMAIL_ADDRESS>
###### Abstract

Novel Instance Detection and Segmentation (NIDS) aims at detecting and
segmenting novel object instances given a few examples of each instance. We
propose a unified framework (NIDS-Net) comprising object proposal generation,
embedding creation for both instance templates and proposal regions, and
embedding matching for instance label assignment. Leveraging recent
advancements in large vision methods, we utilize the Grounding DINO and
Segment Anything Model (SAM) to obtain object proposals with accurate bounding
boxes and masks. Central to our approach is the generation of high-quality
instance embeddings. We utilize foreground feature averages of patch
embeddings from the DINOv2 ViT backbone, followed by refinement through a
weight adapter mechanism that we introduce. We show experimentally that our
weight adapter can adjust the embeddings locally within their feature space
and effectively limit overfitting. This methodology enables a straightforward
matching strategy, resulting in significant performance gains. Our framework
surpasses current state-of-the-art methods, demonstrating notable improvements
of 22.3, 46.2, 10.3, and 24.0 in average precision (AP) across four detection
datasets. In instance segmentation tasks on seven core datasets of the BOP
challenge, our method outperforms the top RGB methods by 3.6 AP and remains
competitive with the best RGB-D method111Code is available at:
https://github.com/YoungSean/NIDS-Net.

##  1 Introduction

Novel Instance Detection and Segmentation (NIDS) is a crucial task in computer
vision aimed at identifying and locating unseen instances in images or videos,
given a few examples of each instance. Of particular interest in this work,
are applications of NIDS in robotics, e.g., a robot might need to fetch a
specific, novel object instance from a cluttered desk given only a small
number of multi-view template images of the novel object. Ideally, detectors
for this task should provide precise bounding boxes and masks of the target
instances in a cluttered query image, but standard detection systems, trained
on specific classes, often struggle to recognize new objects.

The current paradigm for NIDS typically encompasses the following steps: (1)
generating proposals from a query image, (2) obtaining embeddings of the
proposals and the target instance templates, and (3) matching the embeddings
of proposals with those of the templates for identification. Recent work [45,
79, 64, 49] has utilized various open-world detectors to obtain object
proposals. However, the use of models such as the Segment Anything Model (SAM)
[43] or FastSAM [101] often results in the generation of region-based
proposals rather than actual object proposals. This misidentification of
regions as objects can to lead to object identification issues. For example, a
single object may be divided into multiple proposals or some background
regions may be mistaken for foreground objects. These false alarms not only
slow down the whole detection progress but also spoil the final results. To
represent proposals and templates, Shen et al. [79], Nguyen et al. [64], and
Lin et al. [49] adopt the $cls$ token of DINOv2 [66, 20] while Li et al. [45]
employ a 3D voxel representation. Ideally, for each specific unseen instance,
the template embeddings from different views should be similar to each other
but markedly different from the embeddings of other instances. Despite these
efforts, the challenge persists that the embeddings of different instances
remain excessively similar. The development of robust feature embeddings could
provide a solution to this issue.

To address the limitations of current methods, we propose a framework for
NIDS. Initially, we utilize Grounding DINO [52] with the text prompt “objects”
on a cluttered query image to obtain bounding boxes for foreground objects.
This method capitalizes on the inherent objectness of Grounding DINO using
specialized prompts rather than standard labels. Subsequently, we employ SAM
to create masks within these bounding boxes, generating precise object
proposals comprising both bounding boxes and masks. For instance embeddings,
we first extract average foreground features from the patch embeddings of the
DINOv2 ViT backbone. We then apply adapters to enhance instance embeddings by
clustering similar instances and distancing different ones. The adapters are
trained with instance template embeddings via InfoNCE loss [65, 15]. To refine
these embeddings, the CLIP-Adapter [28] introduces a residual vector added to
the original embedding, which risks overfitting with a few training examples.
This addition can spoil and destabilize the embeddings of background objects,
disrupting the entire feature space. To mitigate this issue, we introduce a
weight adapter that modifies the original embeddings by applying learned
weights. Finally, we employ a straightforward matching approach [60], such as
stable matching or using $argmax$, to assign instance labels to these
proposals.

Our approach has been validated on four detection datasets and seven
segmentation datasets. The framework significantly surpasses existing state-
of-the-art methods: it demonstrates substantial increases in average precision
(AP), with gains of 22.3, 46.2, 10.3, and 24.0 across these detection
datasets. Moreover, in instance segmentation tasks on the seven core datasets
of the BOP challenge [82], our method outperforms the best RGB and RGB-D
methods.

Our contributions are summarized as follows.

  * •

We propose NIDS-Net, a unified framework for novel instance detection and
segmentation that surpasses existing methods across four detection datasets
and seven segmentation datasets of BOP challenge.

  * •

We leverage the objectness of open-world detector Grounding DINO and region
recognition of Segmentation Anything Model (SAM) to obtain object proposals.

  * •

We introduce the Weight Adapter, a tool designed to refine embeddings within
their feature space while discouraging overfitting. This approach yields
increasingly significant improvements as the adapting of the feature space
becomes more robust and stable.

##  2 Related Work

Pretrained models. Large-scale pretraining has demonstrated utility across
various downstream tasks. Works such as [32, 90, 55, 24, 54, 10, 66] emphasize
large-scale image representation learning, while [72, 46] focus on vision-
language pretraining. Additionally, [3, 4, 95] address depth prediction, and
[70, 71, 6, 84] tackle large-scale language modeling. Object detection is
addressed by [30, 77, 75, 40, 41, 31, 52], while image segmentation is the
focus of [33, 86, 42]. Works such as [93, 48, 14, 74] explore text-to-3D
generation, and [80, 1] address text-to-video synthesis. Finally, 6DoF pose
estimation and tracking are tackled by [87, 21, 47, 85]. These foundational
models offer diverse capabilities, with the challenge lying in effectively
leveraging their wealth of knowledge for specific use cases, such as novel
instance detection. DINOv2 [66, 20] offers robust features to represent unseen
instances.

Open-world object detection (OWOD). OWOD tackles real-world scenarios where
object detection algorithms are tasked not only with identifying known and
labeled objects but also with effectively handling novel and unknown objects
absent during inference. This sets OWOD apart from traditional object
detection, which confines itself to detecting only predetermined object
classes. A key challenge in OWOD lies in detecting and classifying unknown
objects that were not part of the training data. Recent approaches such as
OWL-ViT [63], DITO [40], RO-ViT [41], YOLO-World [18], Sambor [31], and
Grounding DINO [52] offer diverse solutions to address OWOD challenges. OWL-
ViT leverages a standard Vision Transformer [24] architecture with minimal
modifications, incorporating contrastive image-text pre-training and end-to-
end detection fine-tuning. DITO introduces an open-vocabulary detection
approach based on detection-oriented image-text pretraining, replacing the
classification architecture with a detector architecture during pretraining to
better serve region-level recognition needs. RO-ViT proposes cropping and
resizing regions of positional embeddings (PE) at pretraining, aligning better
with region-level detection fine-tuning. YOLO-World enhances YOLO [75] with
open vocabulary detection capabilities through vision-language modeling and
large-scale pretraining. Grounding DINO [52] utilizes features from DINO [10]
to ground text queries, while Sambor extends SAM [42] to detect arbitrary
objects based on human inputs such as category names or reference expressions.
We employ Grounding DINO due to its strong performance.

Few-shot object detection (FSOD). FSOD involves detecting occurrences of a
novel class, unseen during training, given only a few template examples. De-
ViT [100] introduces a region-propagation mechanism for localization,
transforming propagated region masks into bounding boxes using a learnable
spatial integral layer. FS-DETR [7], the first few-shot detection transformer,
extends DETR [9] to process arbitrary numbers of novel objects concurrently
while supporting various examples from each class. Other approaches such as
[27] prioritize salient representations through angle distance and magnitude
dispersion across few-shot samples, [92] leverages variational autoencoders
for data augmentation, and [59] learns geometry-aware features with an offline
simplex equiangular tight frame classifier. Additionally, [56] generates
specific and representative prototypes for each query image, considering both
support and query image features.

Instance detection. Instance detection involves identifying an unseen instance
in a test image using corresponding templates. Some methods, e.g., [62, 2,
53], rely on pure 2D representations and matching techniques. Specifically,
DTOID [62] employs global object attention and a local pose-specific branch
for template-guided heatmap prediction. However, these methods may struggle
with variations in 2D appearance caused by occlusion or pose changes. In
contrast, VoxDet [45] utilizes explicit 3D knowledge from multi-view
templates, providing geometry-invariant representations. We generate 2D robust
instance embeddings from templates with DINOv2 [66, 20].

Unseen object instance segmentation (UOIS). UOIS is a task designed to segment
unseen objects in cluttered scenes. UOIS generates object masks without
providing labels. UOIS-Net [89] utilizes a two-stage framework, initially
generating rough masks using depth information, and then refining them into
precise instance masks with RGB data. Both UCN [88] and MSMFormer [57, 58]
segment objects by clustering pixel embeddings. Although these methods can
generate object proposals, they primarily focus on tabletop settings. In
contrast, our approach is designed to handle RGB query images in various
environments.

Adapter. The application of adapters atop large pre-trained models has emerged
as a prominent strategy, yielding significant improvements across various
tasks. Previous works have leveraged adapters to enhance few-shot image
classification tasks [28, 97, 68, 99, 81]. Other works have demonstrated
effectiveness in language model fine-tuning [36, 37, 98, 29]. Additional works
exhibit the promise of adapters in vision tasks including detection and
segmentation [17, 16, 69, 91, 23]. For novel instance detection and
segmentation, we propose the Weight Adapter to enhance performance. The
adapter assigns weights to the embeddings from frozen backbones.

##  3 Method

Novel instance detection and segmentation (NIDS), also referred to as open-
world detection and segmentation, is a task to locate and label unseen
instances within a query image. We will assume that each of the $N$ target
instances is represented by $K$ template images $I_{T}\in\mathbb{R}^{K\times
3\times W\times H}$ and their corresponding segmentation masks. During
inference, the output of each query image $I_{Q}\in\mathbb{R}^{3\times W\times
H}$ provides these instances’ bounding boxes for detection and instance masks
for segmentation.

###  3.1 Instance Embedding Generation Stage

In our approach, an instance embedding summarizes the pixels of an instance.
The objective of this stage is to generate initial instance embeddings. Each
of the $N$ instances has $K$ multi-view template images and their ground truth
segmentation masks from which we derive template embeddings
$E_{T}\in\mathbb{R}^{N\times K\times C}$, where $C$ is the dimension of the
embeddings. Given an image and its corresponding mask, we initially extract
patch embeddings using the ViT backbone of DINOv2, and subsequently obtain
foreground features as specified by the mask. We then perform average pooling
on these features. This process, termed Foreground Feature Averaging (FFA), is
proposed by [44] to assess object similarity. We employ FFA to generate all
initial instance embeddings.

###  3.2 Object Proposal Stage

The purpose of this stage is to acquire object proposals from a query image.
Previous studies [79, 64, 49] have utilized SAM [43] or FastSAM [101] to
generate regions as object proposals. Such proposals contain a high number of
false alarms. For example, SAM might misclassify background regions or parts
of objects as complete objects. To address this challenge, an off-the-shelf
zero-shot detector, Grounding Dino [52], is employed with the text prompt
“objects” to obtain initial bounding boxes of foreground objects. Then, SAM is
applied to create masks based on these bounding boxes. The integration of
these two models, termed Grounded-SAM (GS) [78], significantly reduces the
number of erroneous object proposals and expedites the subsequent stages.
Moreover, this approach eliminates the need for training of the detector. We
then extract the proposal regions along with their corresponding masks from
the query images. Using the FFA pipeline with proposals, we calculate proposal
embeddings $E_{P}\in\mathbb{R}^{Q\times C}$, which correspond to $Q$ regions
of interest. Fig. 1 illustrates the process of obtaining proposal embeddings.

Figure 1: In our framework NIDS-Net, only adapters are learnable, while other
models are frozen. Instance IDs are the instance labels.

###  3.3 Embedding Refinement via an Adapter

Most instance embeddings are well separated. However, some embeddings from
distinct instances may cluster together. For exxample, background object
embeddings might be similar to targets. Some target instances may resemble
each other. To address this issue, we employ learnable adapters to refine the
embeddings. We train these adapters using InfoNCE loss [65, 15], aiming to
bring the embeddings of the same instance closer together while separating the
embeddings of different instances.

CLIP-Adapter (CA). CLIP-Adapter [28] comprises two trainable linear bottleneck
layers appended to the language and image branches of CLIP. During few-shot
fine-tuning, the original CLIP backbone remains frozen. Nonetheless, adding
extra layers for fine-tuning may result in overfitting to the few examples
available. To mitigate this issue, CLIP-Adapter integrates residual
connections that dynamically merge the newly adapted knowledge with the
foundational knowledge from the original CLIP backbone.

Given an instance embedding $f$ and a $MLP$ (Multi-Layer Perceptron), the
CLIP-Adapter modifies the embedding according to the formula
$f_{c}=\alpha\times MLP(f)+(1-\alpha)\times f$. In this equation, $f_{c}$
represents the adapted embedding, which is a linear combination of the
transformed embedding $MLP(f)$ and the original embedding $f$. The residual
ratio $\alpha$ is set as 0.6.

Our Weight Adapter (WA). The CLIP-Adapter refines instance embeddings by
adding a new feature vector. However, this new vector might not align with the
feature space of original instance embeddings, which can lead to overfitting.
Since only several hundred template embeddings of target instances are trained
and refined in some datasets, the embeddings of background objects can be
spoiled as the process disrupts the distances between target and background
instance embeddings. Assuming the feature space of original embeddings is
robust and effective, it is crucial to fine-tune the instance embeddings
within this space by constraining the scope of the adapted embeddings. This
refinement can be accomplished by simply applying weights to the original
instance embeddings.

Hence, we introduce our Weight Adapter, a compact MLP network depicted in Fig.
2. The operation of this adapter is described through the following equations:

$\displaystyle w$ $\displaystyle=\text{sigmoid}(MLP(\beta f)),$ (1)
$\displaystyle f_{w}$ $\displaystyle=w\odot(\beta f).$ (2)

Here, $\beta$ is a positive scalar. We set $\beta$ as 10. $w$ is the weights
computed by passing the scaled embedding $\beta f$ through the MLP and then
applying a $sigmoid$ function. The output $f_{w}$ is the element-wise product
of $w$ and the scaled embedding $\beta f$, resulting in the weighted
embedding.

We include a ReLU layer before the $sigmoid$ function, ensuring that the
sigmoid’s output is constrained to the range $[0.5,1)$. Subsequently, each
refined embedding is close to its original counterpart. Additionally, $\beta$
allows for broader adjustments within the feature space. Given that cosine
similarity is utilized to compare embeddings in the subsequent phase, our
adapter alters the interrelationships among the values within each embedding.
Further details of adapters are presented in Fig. 2. Qualitative results can
be found in the appendix.

Figure 2: Comparison of two adapters. (left) CLIP-Adapter. (right) Our weight
adapter.

###  3.4 Matching Stage

This stage provides each proposal $i$ with an instance ID $o_{i}$ as its label
and a confidence score $s_{i}$.

Initially, $Q$ proposal embeddings $E_{P}$ are matched with $N\times K$
template embeddings $E_{T}$ using cosine similarity. It yields a matrix of
template scores with dimensions $Q\times N\times K$, as illustrated in Fig. 1.
For each instance, we aggregate all $K$ template scores to derive a matrix of
instance scores with dimensions $Q\times N$. We employ $Max$ as the
aggregation function for optimal results.

Bonus instance score for segmentation. To improve segmentation performance, we
incorporate an additional appearance matching score, $s_{\text{appe}}$, as
proposed by SAM-6D [49], into the instance scores. $s_{\text{appe}}$ is used
to identify objects that are semantically similar yet differ in appearance.
For each proposal, we can identify its most similar template $T_{\text{best}}$
according to template scores. $s_{\text{appe}}$ is derived from the patch
embeddings of a proposal image $I$ and $T_{\text{best}}$. It quantifies the
similarity between the query image and the template in terms of their local
features as follows:

$s_{\text{appe}}=\frac{1}{N_{\text{patch}}^{I}}\sum_{j=1}^{N_{\text{patch}}^{I}}\max_{i=1,\ldots,N_{\text{patch}}^{T_{\text{best}}}}\left(CosineSimilarity(f_{I,j}^{\text{patch}},f_{T_{\text{best}},i}^{\text{patch}})\right)$

Assuming that objects are unique within a query image [79], we employ the
stable matching algorithm [60] on instance scores to assign a unique instance
ID to each proposal. If the assumption of uniqueness is not met, we use the
$Argmax$ function on the instance score matrix, which permits multiple
proposals to share the same instance ID.

After matching, we acquire labeled proposals. Each is defined by
$\\{b_{i},M_{i},o_{i},s_{i}\\}$. Here, $b_{i}$ is the bounding box of an
instance. $M_{i}$ is the modal mask, which covers the visible object surface
[82]. $o_{i}$ is the instance ID. $s_{i}$ is the confidence score. A
confidence score threshold $\delta$ can be used to remove incorrect proposals.

##  4 Experiments

We employ the Grounding DINO model with a Swin-T backbone [52] and a text
prompt “objects”, and the default ViT-H SAM [43] to generate object proposals.
According to the proposals, we obtain the regions of interest (RoIs) and
resize them to $448\times 448$ or $224\times 224$ resolutions. For object
detection, instance embeddings are produced using the DINOv2’s ViT-L model
with registers [20], employing stable matching due to the uniqueness of
instances in these datasets. For image segmentation, following SAM-6D [49], we
utilize the ViT-L model of DINOv2 [66]. In scenarios with identical instances,
such as the cluttered scenes in BOP datasets, we apply the $argmax$ function
for matching.

The experiments are conducted on an RTX A5000 GPU. For each dataset, we train
both adapters on template embeddings using InfoNCE loss [65, 15] with
temperature 0.05. With two linear layers, Both adapters initially decrease the
embedding dimension from $C$ to $C/4$, subsequently restoring it to $C$. The
Weight Adapter is trained with Adam optimizer at a learning rate of 1e-3 and a
batch size of 1024, while the CLIP-Adapter is trained at a learning rate of
1e-4. Dropout is incorporated in the CLIP-Adapter to reduce overfitting. For
additional details, please see the Appendix.

###  4.1 Detection Datasets

We utilize recently developed instance detection datasets and their associated
baselines [79, 45]. The High-resolution and RoboTools datasets employ real
template images, while the LM-O and YCB-V datasets utilize synthetic images.

High-resolution dataset Shen et al. [79] design a high-resolution dataset for
instance detection. This dataset comprises 100 distinct object instances, each
represented by 24 photos taken from multiple viewpoints, with each photo
having a resolution of $3072\times 3072$ pixels. These instances are
integrated into 14 different indoor scenes for testing, captured in even
higher resolution ($6144\times 8192$ pixels). The testing images are further
classified as easy or hard based on the level of scene clutter and how much
the instances are occluded. Easy tags are assigned when objects are sparsely
placed, while hard tags are used for more cluttered setups.

Baselines According to Shen et al. [79], two types of baselines are set for
the high-resolution dataset: Cut-Paste-Learn and a non-learned method. Cut-
Paste-Learn generates synthetic training images with 2D-box annotations,
putting foreground instances in various sizes and aspect ratios onto different
backgrounds. This enables training detectors by viewing each instance as a
class. They train five detectors: FasterRCNN [76], RetinaNet [51], CenterNet
[102], FCOS [83], and the transformer-based DINO [96]. For the non-learned
method, SAM is initially used to generate proposals, followed by employing
DINO [11] and DINOv2 [66] to generate features for both proposal and template
images, ultimately performing proposal matching and selection.

Synthetic-Real test sets Li et al. [45] employ two benchmarks for evaluation.
LineMod-Occlusion (LM-O) [5] includes 8 texture-less objects and 1,514
bounding boxes, primarily challenging due to significant occlusion. The YCB-
Video (YCB-V) [8] features 21 objects and 4,125 bounding boxes, with pose
variation as the main challenge. Since these datasets have real testing images
without reference videos, Li et al. [45] generate synthetic videos using CAD
models. We sample 16 synthetic template images per object from these videos.

RoboTools benchmark Li et al. [45] introduce RoboTools, a challenging real-
world benchmark for robotics. It features 20 distinct instances, 9,109
annotations, and 24 complex scenarios. Compared to previous benchmarks [5, 8],
RoboTools offers greater difficulty due to its cluttered backgrounds and
significant pose variations. Additionally, the benchmark’s reference videos
are made from real images and incorporate realistic lighting conditions,
including shadows. 25 real template images of each instance are sampled from
reference videos.

Baselines For cluttered scenes from the datasets RoboTools, LM-O and YCB-V, Li
et al. [45] have developed several 2D baselines: OLNDINO, OLNCLIP, and
OLNCorr. Initially, they generate open-world 2D proposals using their
detection module [39]. For matching, different methods are employed to select
the proposal with the highest score. In OLNDINO and OLNCLIP, they use robust
features from pre-trained backbones [11, 73] and cosine similarity for
matching. For OLNCorr, a matching head is designed based on correlation [53].
They also employ the class-level one-shot detectors OS2D [67] and BHRL [94].
To address the limitations of traditional 2D methods with pose variations and
occlusions, Li et al. [45] introduced VoxDet which is based on a 3D voxel
representation. To train these methods, Li et al. [45] developed the open-
world instance detection (OWID-10k) dataset as a synthetic instance detection
training set. It features 9,901 objects from ShapeNet [12] and ABO [19], each
rendered into a 40-frame, 360-degree video using Blenderproc [22].

Evaluation metrics For detection, we evaluate our method with Average
Precision (AP). AP is computed by averaging precision scores at various
Intersection over Union (IoU) thresholds, specifically from 0.5 to 0.95, in
increments of 0.05. This metric is the primary evaluation standard in the COCO
Object Detection dataset [50]. Additionally, AP50 and AP75 are variations of
this metric, where precision is averaged across all instances at IoU
thresholds of 0.5 and 0.75, respectively.

###  4.2 Segmentation Datasets

BOP challenge We test our method using seven datasets from the BOP challenge
[82]: LineMod Occlusion (LM-O) [5], T-LESS [34], TUD-L [35], IC-BIN [25],
ITODD [26], HomebrewedDB (HB) [38], and YCB-Video [8]. The objects from these
datasets vary widely. We use 42 template rendering images from CNOS [64],
which are generated via BlenderProc [22].

Baselines We compare our method with ZeroPose [13], CNOS [64], and SAM-6D
[49]. These methods utilize proposals from SAM [43] or FastSAM [101] and use
the $cls$ token from DINOv2 as the instance embedding for matching. In
addition to adjusting the SAM hyperparameters to generate more proposals,
SAM-6D enhances its performance by incorporating an appearance score,
$s_{\text{appe}}$. Moreover, SAM-6D employs a Geometric Matching Score,
$s_{\text{geo}}$, which considers the shapes and sizes of instances during
matching by utilizing depth information.

Evaluation metrics For instance segmentation task, we evaluate our method
using the Average Precision (AP) metrics. The AP is computed by averaging the
precision scores at various IoU thresholds, from 0.50 to 0.95, increasing by
0.05 at each step.

Figure 3: Visual examples of our results using the Weight Adapter on the high-
resolution and RoboTools datasets. Our approach detects specific object
instances in cluttered scenes according to their real template images. Table
1: Detection results on the high-resolution dataset. avg indicates that we are
evaluating all images, including those from the easy and hard scenes. We train
these two adapters with 2400 template embeddings.

Method | AP | AP50 | AP75 | Time (sec)
---|---|---|---|---
avg | hard | easy | small | medium | large |  |
FasterRCNN [76] | 19.5 | 10.3 | 23.8 | 5.0 | 22.2 | 38.0 | 29.2 | 23.3 | 0.00399
RetinaNet [51] | 22.2 | 14.9 | 26.5 | 5.5 | 25.8 | 42.7 | 31.2 | 25.0 | 0.00412
CenterNet [102] | 21.1 | 11.8 | 25.7 | 5.9 | 24.1 | 40.4 | 32.7 | 23.6 | 0.00376
FCOS [83] | 22.4 | 13.2 | 28.7 | 6.2 | 26.5 | 38.1 | 32.8 | 25.5 | 0.00271
DINO [96] | 28.0 | 17.9 | 32.6 | 11.5 | 30.7 | 45.1 | 32.2 | 32.2 | 1.90625
SAM + DINOf [11] | 37.0 | 22.4 | 43.9 | 11.9 | 40.9 | 62.7 | 44.1 | 40.4 | 15.10
SAM + DINOv2f [66] | 41.6 | 28.0 | 47.6 | 14.6 | 45.8 | 69.1 | 49.1 | 46.0 | 14.70
NIDS-Net (Ours) | 59.3 | 41.7 | 67.4 | 12.7 | 58.1 | 78.8 | 71.1 | 65.1 | 6.92
NIDS-Net + CA (Ours) | 61.3 | 42.5 | 69.6 | 17.0 | 59.3 | 81.6 | 73.2 | 67.0 | 6.99
NIDS-Net + WA (Ours) | 63.9 | 43.4 | 72.7 | 18.1 | 62.5 | 84.0 | 76.6 | 70.6 | 6.78

Table 2: Detection performance on the fully real dataset, RoboTools. Proposal
indicates the object proposal model. GS stands for Grounded-SAM. OLN* is
trained with the matching head while OLN employs fixed modules.

Method | Proposal | AP | AP50 | AP75 |
---|---|---|---|---|---
OS2D [67] | N/A | 2.9 | 6.5 | 2.0 |
DTOID [61] | N/A | 3.6 | 9.0 | 2.0 |
OLNCorr. [39, 53] | OLN* | 14.4 | 18.1 | 15.7 |
VoxDet [45] | OLN* | 18.7 | 23.6 | 20.5 |
NIDS-Net (Ours) | GS | 63.3 | 77.2 | 68.7 |
NIDS-Net + CA (Ours) | GS | 64.8 | 79.2 | 70.7 |
NIDS-Net + WA (Ours) | GS | 64.9 | 79.4 | 70.8 |

Table 3: Detection performance on the LM-O and YCB-V datasets. OLN* is trained
alongside the matching head whereas OLN uses fixed modules. †means the model
is trained on OWID and real images. Our running time of YCB-V is based on the
scene 48. $(\cdot)$ represents the number of instances.

| LM-O (8) | YCB-V (21) | Average
---|---|---|---
Method | Proposal | Train | AP | AP50 | AP75 | AP | AP50 | AP75 | AP | AP50 | AP75 | Time (s)
OS2D [67] | N/A | OWID | 0.2 | 0.7 | <0.1 | 5.2 | 18.3 | 1.9 | 2.7 | 9.5 | 1.0 | 0.189
DTOID [61] | N/A | OWID | 9.8 | 28.9 | 3.7 | 16.3 | 48.8 | 42 | 13.1 | 38.9 | 4.0 | 0.357
OLNCLIP [39, 73] | OLN | OWID† | 16.2 | 32.1 | 15.3 | 10.7 | 25.4 | 7.3 | 13.5 | 28.8 | 11.3 | 0.357
Gen6D [53] | N/A | OWID† | 12.0 | 29.8 | 6.6 | 8.1 | 33.0 | 5.5 | 18.4 | 33.5 | 5.9 | 0.769
BHRL [94] | N/A | COCO | 14.1 | 21.0 | 15.7 | 31.8 | 47.0 | 34.8 | 23.0 | 34.0 | 25.3 | N/A
OLNCorr. [39, 53] | OLN* | OWID | 22.3 | 34.4 | 24.7 | 24.8 | 41.1 | 26.1 | 23.6 | 37.8 | 25.4 | 0.182
OLNDino [39, 11] | OLN | OWID† | 23.6 | 41.6 | 24.8 | 25.6 | 53.0 | 21.1 | 24.6 | 47.3 | 23.0 | 0.357
VoxDet [45] | OLN* | OWID | 29.2 | 43.1 | 33.3 | 31.5 | 51.3 | 33.4 | 30.4 | 47.2 | 33.4 | 0.154
NIDS-Net (Ours) | GS | N/A | 38.7 | 66.0 | 41.0 | 53.0 | 72.9 | 61.7 | 45.9 | 69.5 | 51.4 | 3.73
NIDS-Net + CA (Ours) | GS | N/A | 39.2 | 67.0 | 41.4 | 53.9 | 74.1 | 62.7 | 46.6 | 70.6 | 52.1 | 3.71
NIDS-Net + WA (Ours) | GS | N/A | 39.5 | 67.4 | 41.8 | 55.5 | 75.5 | 65.0 | 47.5 | 71.5 | 53.4 | 3.73

Table 4: Novel instance segmentation results on the seven core datasets of the
BOP benchmark. We utilize Average Precision (AP) to compare these methods.
$(\cdot)$ includes the number of instances.

Method | BOP Datasets |
---|---|---
LM-O (8) | T-LESS (30) | TUD-L (3) | IC-BIN (2) | ITODD (28) | HB (33) | YCB-V (21) | Mean
ZeroPose (SAM) [13] | 34.4 | 32.7 | 41.4 | 25.1 | 22.4 | 47.8 | 51.9 | 36.5
CNOS (SAM) [64] | 39.6 | 39.7 | 39.1 | 28.4 | 28.2 | 48.0 | 59.5 | 40.4
CNOS (FastSAM) [64] | 39.7 | 37.4 | 48.0 | 27.0 | 25.4 | 51.1 | 59.9 | 41.2
SAM-6D (FastSAM) [49] | 39.5 | 37.6 | 48.7 | 25.7 | 25.3 | 51.2 | 60.2 | 41.2
SAM-6D (FastSAM) + $s_{\text{appe}}$ [49] | 40.6 | 39.3 | 50.1 | 27.7 | 29.0 | 52.2 | 60.6 | 42.8
SAM-6D (SAM) [49] | 43.4 | 39.1 | 48.2 | 33.3 | 28.8 | 55.1 | 60.3 | 44.0
SAM-6D (SAM) +$s_{\text{appe}}$ [49] | 44.4 | 40.8 | 49.8 | 34.5 | 30.0 | 55.7 | 59.5 | 45.0
SAM-6D (SAM, RGBD) [49] | 46.0 | 45.1 | 56.9 | 35.7 | 33.2 | 59.3 | 60.5 | 48.1
NIDS-Net (Ours) | 42.9 | 43.0 | 52.0 | 30.5 | 28.8 | 56.6 | 59.7 | 44.8
NIDS-Net + CA (Ours) | 42.8 | 47.5 | 50.0 | 29.7 | 27.6 | 62.7 | 63.5 | 46.3
NIDS-Net + WA(Ours) | 43.2 | 49.2 | 53.5 | 30.5 | 29.4 | 62.4 | 64.3 | 47.5
NIDS-Net + WA (Ours) + $s_{\text{appe}}$ | 43.9 | 49.6 | 55.6 | 32.8 | 31.5 | 62.0 | 65.0 | 48.6

###  4.3 Benchmarking Results

Detection The high-resolution dataset results are presented in Table 1. Our
models dramatically outperform existing techniques, achieving the highest AP
scores across all categories. Our basic method surpasses the top baseline by
17.7 AP. Additionally, as this dataset contains 100 instances with some
similarities among them, our Weight Adapter boosts overall performance by 4.6
AP and improves the detection of small objects by 5.4 AP. For the RoboTools
dataset, detailed in Table 2, our method outperforms the state-of-the-art
VoxDet by 46.2 AP. Since RoboTools contains only 20 instances, our adapter has
limited scope to enhance performance. Our approach achieves over 60 AP on
these two fully real datasets.

For the Synthetic-Real datasets, LM-O and YCB-V, we present a comparison of
our approach with other methods in Table 3. Our model outperforms VoxDet by
10.3 AP on LM-O and by 24.0 AP on YCB-V. The improved performance on YCB-V,
which contains more instances, indicates that our adapter functions more
effectively with increased instance variety.

Segmentation CNOS [64] and SAM-6D [49] initially utilize SAM or FastSAM to
obtain proposals and derive instance embeddings through the $cls$ token of
Dinov2. In contrast, we generate proposals using Grounded-SAM and acquire
embeddings through the FFA pipeline. The segmentation results for the seven
principal datasets of the BOP challenge are detailed in Table 4. Overall, our
method surpasses the state-of-the-art SAM-6D and achieves superior results
compared to RGBD results.

Qualitative result We present detection results for both simple and complex
scenes in Fig. 3. Additional detection and segmentation examples are available
in the Appendix.

Runtime The tables of detection results include the runtime of our method.
Despite Grounded-SAM reducing the number of proposals, our approach remains
slower compared to end-to-end detectors. This is primarily due to the process
involved in obtaining proposal embeddings. The inclusion of adapters does not
impede the overall framework. Future research could focus on developing a more
efficient method for robotic applications. Please see the appendix for
segmentation runtime.

###  4.4 Ablation Study

Grounded SAM (GS) vs SAM On the seven BOP datasets containing images of
numerous cluttered scenes, we compare the object proposals from GS and SAM as
detailed in Table 5. The results indicate that GS yields more precise bounding
boxes and masks compared to using SAM alone. Furthermore, GS enhances
efficiency by eliminating false object proposals, thereby reducing runtime.

FFA vs $cls$ token To compare these two types of embedding generation, we
present the results in Table 5. Our weight adapter enhances both two types of
embeddings. Despite close segmentation results, FFA produces embeddings that
possess greater adaptive potential, demonstrated by higher AP scores after
adaption via our weight adapter.

Table 5: Comparison of segmentation performance across various proposal models
and embedding generation options. The first row is from CNOS [64].

Proposal | Embedding | BOP Datasets |
---|---|---|---
LM-O | T-LESS | TUD-L | IC-BIN | ITODD | HB | YCB-V | Mean
SAM | cls token | 39.6 | 39.7 | 39.1 | 28.4 | 28.2 | 48.0 | 59.5 | 40.4
SAM | FFA | 40.0 | 40.0 | 40.8 | 28.0 | 25.8 | 46.0 | 54.4 | 39.3
GS | cls token | 41.7 | 41.7 | 50.8 | 31.5 | 30 | 58 | 63 | 45.2
GS | FFA | 42.9 | 43.0 | 52.0 | 30.5 | 28.8 | 56.6 | 59.7 | 44.8
GS | cls token + WA | 42.5 | 48.2 | 50.7 | 31.3 | 32.0 | 60.5 | 62.8 | 46.9
GS | FFA +WA | 43.2 | 49.2 | 53.5 | 30.5 | 29.4 | 62.4 | 64.3 | 47.5

Table 6: Detection results using different ViT backbones of Dinov2. “reg”
indicates DINOv2 with registers. The results are based on all testing images
of the high-resolution dataset. WA Diff indicates the improvement attributed
to the Weight Adapter.

Dinov2 backbone | AP | AP50 | AP75 | WA Diff
---|---|---|---|---
ViT-S/14 | 47.5 | 56.4 | 51.7 | \+ 0.9 AP
ViT-S/14 + WA | 48.4 | 57.5 | 52.9
ViT-B/14 | 54.7 | 65.2 | 60.1 | \+ 3.2 AP
ViT-B/14 + WA | 57.9 | 69.0 | 63.6
ViT-L/14 | 56.8 | 67.7 | 62.3 | \+ 3.6 AP
ViT-L/14 + WA | 60.4 | 71.8 | 66.4
ViT-L/14 reg | 59.3 | 71.1 | 65.1 | \+ 4.6 AP
ViT-L/14 reg +WA | 63.9 | 76.6 | 70.6

Dinov2 backbones with adapter Our Weight Adapter is compatible with various
backbones of Dinov2. Notably, more powerful backbones, which offer a more
effective feature space, enable our adapter to deliver greater improvements.
Details of this comparison are provided in Table 6. Please refer to the
Appendix for additional ablation studies.

##  5 Discussions

In this study, we introduce NIDS-Net, a framework designed for novel instance
detection and segmentation. We utilize Grounding DINO and SAM to generate
precise foreground object proposals, as opposed to conventional naive region
proposals. We also present the Weight Adapter, which refines features from a
pre-trained vision model effectively and mitigates the risk of overfitting.
With the adapter, template and proposal embeddings of different instances are
separated to facilitate the subsequent matching. Our method surpasses other
approaches significantly in detection performance and also excels in
segmentation compared to existing methods.

However, there are limitations: given that our approach incorporates multiple
pre-trained models, it requires a longer running time and greater
computational resources compared to end-to-end detectors. Additionally,
Grounding DINO may produce large bounding boxes that encompass multiple
objects, leading to under-segmentation and detection failures. Moreover, our
method tends to overlook heavily occluded objects with low confidence scores.

In this work, instances are represented by $K$ template embeddings. For future
research, we propose that each instance should be characterized by a single,
distinctive embedding that acts as its identifier to enable one-shot
detection. This approach will allow the detector to identify and locate a
target instance within a query image using just one template image.
Additionally, developing an efficient method for this process will be crucial
for its application in robotics.

## Acknowledgments

This work was supported in part by the DARPA Perceptually-enabled Task
Guidance (PTG) Program under contract number HR00112220005.

## References

  * [1] Sora: Creating video from text.  https://openai.com/sora.  [Accessed Apr 24 2024].
  * Ammirato et al. [2018] P. Ammirato, C.-Y. Fu, M. Shvets, J. Kosecka, and A. C. Berg.  Target driven instance detection.  _arXiv preprint arXiv:1803.04610_ , 2018.
  * Bhat et al. [2023] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  _arXiv preprint arXiv:2302.12288_ , 2023.
  * Birkl et al. [2023] R. Birkl, D. Wofk, and M. Müller.  Midas v3. 1–a model zoo for robust monocular relative depth estimation.  _arXiv preprint arXiv:2307.14460_ , 2023.
  * Brachmann et al. [2014] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother.  Learning 6d object pose estimation using 3d object coordinates.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part II 13_ , pages 536–551. Springer, 2014.
  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bulat et al. [2023] A. Bulat, R. Guerrero, B. Martinez, and G. Tzimiropoulos.  Fs-detr: Few-shot detection transformer with prompting and without re-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11793–11802, 2023.
  * Calli et al. [2015] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar.  The ycb object and model set: Towards common benchmarks for manipulation research.  In _2015 international conference on advanced robotics (ICAR)_ , pages 510–517. IEEE, 2015.
  * Carion et al. [2020] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , pages 213–229. Springer, 2020.
  * Caron et al. [2021a] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2021a.
  * Caron et al. [2021b] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021b.
  * Chang et al. [2015] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al.  Shapenet: An information-rich 3d model repository.  _arXiv preprint arXiv:1512.03012_ , 2015.
  * Chen et al. [2023a] J. Chen, M. Sun, T. Bao, R. Zhao, L. Wu, and Z. He.  3d model-based zero-shot pose estimation pipeline.  _arXiv preprint arXiv:2305.17934_ , 2023a.
  * Chen et al. [2023b] R. Chen, Y. Chen, N. Jiao, and K. Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22246–22256, 2023b.
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Chen et al. [2023c] T. Chen, L. Zhu, C. Deng, R. Cao, Y. Wang, S. Zhang, Z. Li, L. Sun, Y. Zang, and P. Mao.  Sam-adapter: Adapting segment anything in underperformed scenes.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3367–3375, 2023c.
  * Chen et al. [2022] Z. Chen, Y. Duan, W. Wang, J. He, T. Lu, J. Dai, and Y. Qiao.  Vision transformer adapter for dense predictions.  _arXiv preprint arXiv:2205.08534_ , 2022.
  * Cheng et al. [2024] T. Cheng, L. Song, Y. Ge, W. Liu, X. Wang, and Y. Shan.  Yolo-world: Real-time open-vocabulary object detection.  _arXiv preprint arXiv:2401.17270_ , 2024.
  * Collins et al. [2022] J. Collins, S. Goel, K. Deng, A. Luthra, L. Xu, E. Gundogdu, X. Zhang, T. F. Y. Vicente, T. Dideriksen, H. Arora, et al.  Abo: Dataset and benchmarks for real-world 3d object understanding.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 21126–21136, 2022.
  * Darcet et al. [2023] T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski.  Vision transformers need registers.  _arXiv preprint arXiv:2309.16588_ , 2023.
  * Deng et al. [2021] X. Deng, A. Mousavian, Y. Xiang, F. Xia, T. Bretl, and D. Fox.  Poserbpf: A rao–blackwellized particle filter for 6-d object pose tracking.  _IEEE Transactions on Robotics_ , 37(5):1328–1342, 2021.
  * Denninger et al. [2019] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam.  Blenderproc.  _arXiv preprint arXiv:1911.01911_ , 2019.
  * Dong et al. [2024] B. Dong, G. Yang, W. Zuo, and L. Zhang.  Consept: Continual semantic segmentation via adapter-based vision transformer.  _arXiv preprint arXiv:2402.16674_ , 2024.
  * Dosovitskiy et al. [2020] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Doumanoglou et al. [2016] A. Doumanoglou, R. Kouskouridas, S. Malassiotis, and T.-K. Kim.  Recovering 6d object pose and predicting next-best-view in the crowd.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3583–3592, 2016.
  * Drost et al. [2017] B. Drost, M. Ulrich, P. Bergmann, P. Hartinger, and C. Steger.  Introducing mvtec itodd-a dataset for 3d object recognition in industry.  In _Proceedings of the IEEE international conference on computer vision workshops_ , pages 2200–2208, 2017.
  * Du et al. [2023] J. Du, S. Zhang, Q. Chen, H. Le, Y. Sun, Y. Ni, J. Wang, B. He, and J. Wang.  $\sigma$-adaptive decoupled prototype for few-shot object detection.  In _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 18904–18914. IEEE, 2023.
  * Gao et al. [2021] P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao.  Clip-adapter: Better vision-language models with feature adapters.  _arXiv 2110.04544_ , 2021.
  * Gao et al. [2023] P. Gao, J. Han, R. Zhang, Z. Lin, S. Geng, A. Zhou, W. Zhang, P. Lu, C. He, X. Yue, H. Li, and Y. Qiao.  Llama-adapter v2: Parameter-efficient visual instruction model.  _arXiv preprint arXiv:2304.15010_ , 2023.
  * Girshick [2015] R. Girshick.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1440–1448, 2015.
  * Han et al. [2023] X. Han, L. Wei, X. Yu, Z. Dou, X. He, K. Wang, Z. Han, and Q. Tian.  Boosting segment anything model towards open-vocabulary learning.  _arXiv preprint arXiv:2312.03628_ , 2023.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * He et al. [2017] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2961–2969, 2017.
  * Hodan et al. [2017] T. Hodan, P. Haluza, Š. Obdržálek, J. Matas, M. Lourakis, and X. Zabulis.  T-less: An rgb-d dataset for 6d pose estimation of texture-less objects.  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 880–888. IEEE, 2017.
  * Hodan et al. [2018] T. Hodan, F. Michel, E. Brachmann, W. Kehl, A. GlentBuch, D. Kraft, B. Drost, J. Vidal, S. Ihrke, X. Zabulis, et al.  Bop: Benchmark for 6d object pose estimation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 19–34, 2018.
  * Hu et al. [2021] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ , 2021.
  * Hu et al. [2023] Z. Hu, L. Wang, Y. Lan, W. Xu, E.-P. Lim, L. Bing, X. Xu, S. Poria, and R. K.-W. Lee.  Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models.  _arXiv preprint arXiv:2304.01933_ , 2023.
  * Kaskman et al. [2019] R. Kaskman, S. Zakharov, I. Shugurov, and S. Ilic.  Homebreweddb: Rgb-d dataset for 6d pose estimation of 3d objects.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , pages 0–0, 2019.
  * Kim et al. [2022] D. Kim, T.-Y. Lin, A. Angelova, I. S. Kweon, and W. Kuo.  Learning open-world object proposals without learning to classify.  _IEEE Robotics and Automation Letters_ , 7(2):5453–5460, 2022.
  * Kim et al. [2023a] D. Kim, A. Angelova, and W. Kuo.  Detection-oriented image-text pretraining for open-vocabulary detection.  _arXiv preprint arXiv:2310.00161_ , 2023a.
  * Kim et al. [2023b] D. Kim, A. Angelova, and W. Kuo.  Region-aware pretraining for open-vocabulary object detection with vision transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11144–11154, 2023b.
  * Kirillov et al. [2023a] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al.  Segment anything.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4015–4026, 2023a.
  * Kirillov et al. [2023b] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al.  Segment anything.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4015–4026, 2023b.
  * Kotar et al. [2024] K. Kotar, S. Tian, H.-X. Yu, D. Yamins, and J. Wu.  Are these the same apple? comparing images based on object intrinsics.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Li et al. [2024] B. Li, J. Wang, Y. Hu, C. Wang, and S. Scherer.  Voxdet: Voxel learning for novel instance detection.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Li et al. [2020] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX 16_ , pages 121–137. Springer, 2020.
  * Li et al. [2018] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox.  Deepim: Deep iterative matching for 6d pose estimation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 683–698, 2018.
  * Lin et al. [2023a] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin.  Magic3d: High-resolution text-to-3d content creation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 300–309, 2023a.
  * Lin et al. [2023b] J. Lin, L. Liu, D. Lu, and K. Jia.  Sam-6d: Segment anything model meets zero-shot 6d object pose estimation.  _arXiv preprint arXiv:2311.15707_ , 2023b.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Lin et al. [2017] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017.
  * Liu et al. [2023] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023.
  * Liu et al. [2022a] Y. Liu, Y. Wen, S. Peng, C. Lin, X. Long, T. Komura, and W. Wang.  Gen6d: Generalizable model-free 6-dof object pose estimation from rgb images.  In _European Conference on Computer Vision_ , pages 298–315. Springer, 2022a.
  * Liu et al. [2021] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Liu et al. [2022b] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 11976–11986, 2022b.
  * Lu et al. [2023a] X. Lu, W. Diao, Y. Mao, J. Li, P. Wang, X. Sun, and K. Fu.  Breaking immutable: Information-coupled prototype elaboration for few-shot object detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 1844–1852, 2023a.
  * Lu et al. [2022] Y. Lu, Y. Chen, N. Ruozzi, and Y. Xiang.  Mean shift mask transformer for unseen object instance segmentation.  _arXiv preprint arXiv:2211.11679_ , 2022.
  * Lu et al. [2023b] Y. Lu, N. Khargonkar, Z. Xu, C. Averill, K. Palanisamy, K. Hang, Y. Guo, N. Ruozzi, and Y. Xiang.  Self-supervised unseen object instance segmentation via long-term robot interaction.  _arXiv preprint arXiv:2302.03793_ , 2023b.
  * Ma et al. [2023] J. Ma, Y. Niu, J. Xu, S. Huang, G. Han, and S.-F. Chang.  Digeo: Discriminative geometry-aware learning for generalized few-shot object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3208–3218, 2023.
  * McVitie and Wilson [1971] D. G. McVitie and L. B. Wilson.  The stable marriage problem.  _Communications of the ACM_ , 14(7):486–490, 1971.
  * Mercier et al. [2021a] J.-P. Mercier, M. Garon, P. Giguere, and J.-F. Lalonde.  Deep template-based object instance detection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1507–1516, 2021a.
  * Mercier et al. [2021b] J.-P. Mercier, M. Garon, P. Giguere, and J.-F. Lalonde.  Deep template-based object instance detection.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1507–1516, 2021b.
  * Minderer et al. [2022] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, et al.  Simple open-vocabulary object detection with vision transformers. arxiv 2022.  _arXiv preprint arXiv:2205.06230_ , 2, 2022.
  * Nguyen et al. [2023] V. N. Nguyen, T. Groueix, G. Ponimatkin, V. Lepetit, and T. Hodan.  Cnos: A strong baseline for cad-based novel object segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2134–2140, 2023.
  * Oord et al. [2018] A. v. d. Oord, Y. Li, and O. Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Oquab et al. [2023] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Osokin et al. [2020] A. Osokin, D. Sumin, and V. Lomakin.  Os2d: One-stage one-shot object detection by matching anchor features.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16_ , pages 635–652. Springer, 2020.
  * P et al. [2023] J. J. P, K. Palanisamy, Y.-W. Chao, X. Du, and Y. Xiang.  Proto-clip: Vision-language prototypical network for few-shot learning.  2023\.
  * Pu et al. [2024] X. Pu, H. Jia, L. Zheng, F. Wang, and F. Xu.  Classwise-sam-adapter: Parameter efficient fine-tuning adapts segment anything to sar domain for semantic segmentation.  _arXiv preprint arXiv:2401.02326_ , 2024.
  * Radford et al. [2018] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever.  Improving language understanding with unsupervised learning.  2018\.
  * Radford et al. [2019] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. [2021a] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021a.
  * Radford et al. [2021b] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021b.
  * Raj et al. [2023] A. Raj, S. Kaza, B. Poole, M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein, J. Barron, et al.  Dreambooth3d: Subject-driven text-to-3d generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2349–2359, 2023.
  * Redmon et al. [2016] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * Ren et al. [2015a] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015a.
  * Ren et al. [2015b] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  _Advances in neural information processing systems_ , 28, 2015b.
  * Ren et al. [2024] T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, et al.  Grounded sam: Assembling open-world models for diverse visual tasks.  _arXiv preprint arXiv:2401.14159_ , 2024.
  * Shen et al. [2023] Q. Shen, Y. Zhao, N. Kwon, J. Kim, Y. Li, and S. Kong.  A high-resolution dataset for instance detection with multi-view instance capture.  In _NeurIPS Datasets and Benchmarks Track_ , 2023.
  * Singer et al. [2022] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ , 2022.
  * Song et al. [2024] L. Song, R. Xue, H. Wang, H. Sun, Y. Ge, Y. Shan, et al.  Meta-adapter: An online few-shot learner for vision-language model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Sundermeyer et al. [2023] M. Sundermeyer, T. Hodaň, Y. Labbe, G. Wang, E. Brachmann, B. Drost, C. Rother, and J. Matas.  Bop challenge 2022 on detection, segmentation and pose estimation of specific rigid objects.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2784–2793, 2023.
  * Tian et al. [2019] Z. Tian, C. Shen, H. Chen, and T. He.  Fcos: Fully convolutional one-stage object detection.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 9626–9635, 2019.  doi: 10.1109/ICCV.2019.00972.
  * Touvron et al. [2023] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Wen et al. [2023] B. Wen, W. Yang, J. Kautz, and S. Birchfield.  Foundationpose: Unified 6d pose estimation and tracking of novel objects.  _arXiv preprint arXiv:2312.08344_ , 2023.
  * Wu et al. [2019] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Xiang et al. [2017] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox.  Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes.  _arXiv preprint arXiv:1711.00199_ , 2017.
  * Xiang et al. [2021] Y. Xiang, C. Xie, A. Mousavian, and D. Fox.  Learning rgb-d feature embeddings for unseen object instance segmentation.  In _Conference on Robot Learning_ , pages 461–470. PMLR, 2021.
  * Xie et al. [2021] C. Xie, Y. Xiang, A. Mousavian, and D. Fox.  Unseen object instance segmentation for robotic environments.  _IEEE Transactions on Robotics_ , 37(5):1343–1359, 2021.
  * Xie et al. [2017] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He.  Aggregated residual transformations for deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1492–1500, 2017.
  * Xie et al. [2024] Z. Xie, B. Guan, W. Jiang, M. Yi, Y. Ding, H. Lu, and L. Zhang.  Pa-sam: Prompt adapter sam for high-quality image segmentation.  _2024 IEEE International Conference on Multimedia and Expo (ICME)_ , 2024.
  * Xu et al. [2023a] J. Xu, H. Le, and D. Samaras.  Generating features with increased crop-related diversity for few-shot object detection.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19713–19722, 2023a.
  * Xu et al. [2023b] J. Xu, X. Wang, W. Cheng, Y.-P. Cao, Y. Shan, X. Qie, and S. Gao.  Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20908–20918, 2023b.
  * Yang et al. [2022] H. Yang, S. Cai, H. Sheng, B. Deng, J. Huang, X.-S. Hua, Y. Tang, and Y. Zhang.  Balanced and hierarchical relation learning for one-shot object detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7591–7600, 2022.
  * Yang et al. [2024] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao.  Depth anything: Unleashing the power of large-scale unlabeled data.  _arXiv preprint arXiv:2401.10891_ , 2024.
  * Zhang et al. [2022a] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. M. Ni, and H.-Y. Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection, 2022a.
  * Zhang et al. [2022b] R. Zhang, Z. Wei, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li.  Tip-adapter: Training-free adaption of clip for few-shot classification.  _arXiv preprint arXiv:2207.09519_ , 2022b.
  * Zhang et al. [2023a] R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y. Qiao.  Llama-adapter: Efficient fine-tuning of language models with zero-init attention.  _arXiv preprint arXiv:2303.16199_ , 2023a.
  * Zhang et al. [2023b] R. Zhang, X. Hu, B. Li, S. Huang, H. Deng, Y. Qiao, P. Gao, and H. Li.  Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15211–15222, 2023b.
  * Zhang et al. [2024] X. Zhang, Y. Wang, and A. Boularias.  Detect everything with few examples, 2024.
  * Zhao et al. [2023] X. Zhao, W. Ding, Y. An, Y. Du, T. Yu, M. Li, M. Tang, and J. Wang.  Fast segment anything.  _arXiv preprint arXiv:2306.12156_ , 2023.
  * Zhou et al. [2019] X. Zhou, D. Wang, and P. Krähenbühl.  Objects as points.  _arXiv preprint arXiv:1904.07850_ , 2019.

##  6 Appendix / supplemental material

###  6.1 Training Details

Detection. For detection datasets, the weight adapter is trained with a batch
size of 1024, while the CLIP-Adapter is trained with a batch size of 512 to
enhance performance. Both adapters are trained for the same number of epochs,
as detailed in Table 7. To utilize Grounding DINO, a box threshold of 0.15 is
set for the high-resolution and YCB-V datasets, and 0.10 for other datasets.

Table 7: The training epochs of different detection datasets. | High-resolution | RoboTools | LM-O | YCB-V
---|---|---|---|---
Both Adapters | 40 | 80 | 40 | 40

Segmentation. We combine all instances from seven core datasets of the BOP
benchmark. We train both adapters with a batch size of 1344 (32 instances
$\times$ 42 templates per instance) for 500 epochs. For Grounding DINO, a box
threshold of 0.10 is set for all datasets.

###  6.2 More Ablation Study

FFA vs $cls$ token and Grounded SAM (GS) vs SAM in Detection Tasks. The
detection capabilities of FFA and the $cls$ token are demonstrated using the
high-resolution dataset, as presented in Table 8. For instance embedding
generation, FFA achieves significantly greater enhancements compared to the
$cls$ token. Furthermore, Table 8 illustrates that GS is effective in
generating precise object proposals.

Table 8: The detection results on all images of the high-resolution dataset. “Proposal” refers to the object proposal method. “Embedding” denotes the method of instance embedding generation. Proposal | Embedding | AP | AP50 | AP75
---|---|---|---|---
SAM |  $cls$ token | 41.6 | 49.1 | 46.0
GS |  $cls$ token | 54.9 | 65.4 | 60.1
GS | FFA | 59.3 | 71.1 | 65.1

Aggregation. For cls token embeddings, averaging the top $k$ highest (avg k)
scores yields the best results [64, 49]. For FFA embeddings, the $\max$
aggregation function achieves optimal outcomes. The comparison of these
aggregation functions is detailed in Table 9.

Table 9: Comparison of aggregation functions for segmentation performance. We
report Average Precision (AP). avg k refers to averaging the top $k$ scores.
All results are based on object proposals from Grounded SAM (GS).

Embedding | Aggregation | BOP Datasets |
---|---|---|---
LM-O | T-LESS | TUD-L | IC-BIN | ITODD | HB | YCB-V | Mean
$cls$ token | avg k | 41.7 | 41.7 | 50.8 | 31.5 | 30 | 58 | 63 | 45.2
$cls$ token | $\max$ | 42 | 37.4 | 45.9 | 30.2 | 28.9 | 54.9 | 61.5 | 43.0
FFA | avg k | 42.5 | 42 | 47.4 | 28.2 | 27.3 | 55.1 | 61.5 | 43.4
FFA | $\max$ | 42.9 | 43 | 52 | 30.5 | 28.8 | 56.6 | 59.7 | 44.8

Runtime. We compare the efficiency of existing methods for novel instance
segmentation, as presented in Table 10. Our approach significantly reduces
running time by proposing only high-quality bounding boxes.

Table 10: Runtime comparisons of various methods for novel instance segmentation. Method | Proposal | Server | Time (sec)
---|---|---|---
CNOS | FastSAM | Tesla V100 | 0.22
CNOS | DeForce RTX 3090 | 0.23
SAM-6D (RGB) | DeForce RTX 3090 | 0.25
SAM-6D (RGBD) | DeForce RTX 3090 | 0.45
CNOS | SAM | Tesla V100 | 1.84
CNOS | DeForce RTX 3090 | 2.35
SAM-6D (RGB) | DeForce RTX 3090 | 2.28
SAM-6D (RGBD) | DeForce RTX 3090 | 2.80
NIDS-Net (Ours) | GS | RTX A5000 | 0.49
NIDS-Net + CA (Ours) | 0.48
NIDS-Net + WA (Ours) | 0.48
NIDS-Net + WA + $s_{\text{appe}}$ (Ours) | 0.48

###  6.3 Unseen Detection of BOP Benchmark

We compare our approach with ZeroPose [13], CNOS [64], and SAM-6D [49] for 2D
unseen detection, as illustrated in Table 11. Our method outperforms the best
RGB method by 2.5 AP and competes effectively with the top RGB-D method.

Table 11: Unseen instance detection results across the seven core datasets of
the BOP benchmark, with all results reported as Average Precision (AP).

Method | Proposal | BOP Datasets |
---|---|---|---
LM-O | T-LESS | TUD-L | IC-BIN | ITODD | HB | YCB-V | Mean
ZeroPose [13] | SAM | 36.7 | 30.0 | 43.1 | 22.8 | 25.0 | 39.8 | 41.6 | 34.1
CNOS | SAM | 39.5 | 33.0 | 36.8 | 20.7 | 31.3 | 42.3 | 49.0 | 36.1
CNOS [64] | FastSAM | 43.3 | 39.5 | 53.4 | 22.6 | 32.5 | 51.7 | 56.8 | 42.8
SAM-6D (RGB) [49] | FastSAM | 43.8 | 41.7 | 54.6 | 23.4 | 37.4 | 52.3 | 57.3 | 44.4
SAM-6D (RGBD) [49] | SAM | 46.6 | 43.7 | 53.7 | 26.1 | 39.3 | 53.1 | 51.9 | 44.9
SAM-6D (RGBD) [49] | FastSAM | 46.3 | 45.8 | 57.3 | 24.5 | 41.9 | 55.1 | 58.9 | 47.1
NIDS-Net (Ours) | GS | 44.9 | 42.8 | 43.4 | 24.4 | 34.9 | 54.8 | 56.5 | 43.1
NIDS-Net + WA(Ours) | 44.9 | 48.9 | 46.0 | 24.5 | 36.0 | 59.4 | 62.4 | 46.0
NIDS-Net + WA (Ours) + $s_{\text{appe}}$ | 45.7 | 49.3 | 48.6 | 25.7 | 37.9 | 58.7 | 62.1 | 46.9

###  6.4 More Qualitative Results

####  6.4.1 Adapter

To facilitate a comparison between the CLIP-Adapter and our Weight Adapter, we
present a visual illustration in Figure 4. It is evident that the CLIP-Adapter
alters the feature space and spoils the embeddings of background objects due
to overfitting. In contrast, our Weight Adapter delivers robust embeddings
within the original feature space.

####  6.4.2 Detection

We display the visual outcomes of our methodology employing the weight adapter
on the LMO and YCB-V datasets in Figure 5. The gap between synthetic and real
images results in some instances of detection failure. For example, some LM-O
instances are not found with our method.

####  6.4.3 Segmentation

We present the visual outcomes of our approach using the weight adapter on the
BoP datasets in Figures 6, 7, 8, and 9. These images demonstrate the
effectiveness of our approach in cluttered environments. In some cases of
T-LESS and IC-BIN datasets, Grounding DINO generates large bounding boxes
which include multiple objects, causing under-segmentation. Furthermore, in
IC-BIN and HB datasets, some heavily occluded objects with low confidence
scores are overlooked by our method.

Figure 4: Comparison of different adapters on a hard scene image from the
high-resolution dataset. Red arrows denote background objects that are
erroneously classified as targets. Figure 5: Visual detection results using
the Weight Adapter on the LM-O and YCB-V datasets. Our approach detects object
instances according to their synthetic template images. Figure 6: Qualitative
segmentation results on the LM-O and T-Less datasets. Figure 7: Qualitative
segmentation results on the TUD-L and IC-BIN datasets. Figure 8: Qualitative
segmentation results on the ITODD and HB datasets Figure 9: Qualitative
segmentation results on the YCB-V dataset.