# Supplementary material for
Open-vocabulary object 6D pose estimation

Jaime Corsetti1,2 Davide Boscaini1 Changjae Oh3 Andrea Cavallaro4,5 Fabio
Poiesi1
1Fondazione Bruno Kessler 2University of Trento 3Queen Mary University of
London 4Idiap Research Institute 5EPFL
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
##  1 Introduction

We provide additional material in support of our main paper. This document is
organized as follows:

  * •

In Sec. 2, we compare the assumptions and requirements of Oryon with those of
state-of-the-art methods that address pose estimation in a generalizable
setting.

  * •

In Sec. 3, we provide additional implementation details and display the
architecture of the fusion module ($\phi_{TV}$) in Sec. 3.1 and the decoder
($\phi_{D}$) in Sec. 3.2.

  * •

In Sec. 4, we outline the process we followed to generate our training and
test datasets.

  * •

In Sec. 5, we list the textual prompts we used for evaluation on REAL275 [18]
and Toyota-Light [6] (TOYL for short), including the alternative prompts used
in the ablation studies. We also present some examples of synonym sets from
ShapeNet6D [19] (SN6D for short).

  * •

In Sec. 6, we show additional qualitative results on pose estimation (Sec.
6.1) and segmentation (Sec. 6.2) on TOYL and REAL275. Moreover, we provide
examples of feature distance visualization for $\mathbf{F}^{A}$ and
$\mathbf{F}^{Q}$, demonstrating how the features change when different prompts
are used (Sec. 6.3).

##  2 Setting comparison

In Tab. 1, we present a comparison of the data requirements of state-of-the-
art methods for generalizable pose estimation. For each method, we report the
input data, the reference used to estimate the pose, and whether the method
can estimate the full pose or only the rotation component. We also summarize
the object onboarding process, which has to be carried out at evaluation time
for each novel object. Finally, we report the localization prior used, i.e.,
an external module that is used to crop or segment the object of interest.
Note that some methods include the localization module within their pipeline
([16, 10], and ours), instead of employing an external module.

We identify three main groups in the current literature. The first is model-
based methods, which require a 3D model of the object at test time [2, 8, 16].
When present, the object onboarding phase requires generating a set of
templates (i.e., a set of views of the novel object) from the object 3D model
[2, 16].

The second is model-free methods, which assume a video sequence of the object
to be available [10, 17, 5]. In these methods, the object onboarding phase is
particularly cumbersome. Gen6D [10] requires the user to perform SfM to
reconstruct the object point cloud, which is then manually cropped and
oriented. The resulting point cloud is used as an object model. OnePose [17]
and OnePose++ [5] also perform SfM on the video sequence, but they only use it
to recover the camera poses, so the manual cropping of the point cloud is not
needed. Therefore, the onboarding process of model-free methods requires human
intervention, as opposed to that of model-based methods. OnePose and OnePose++
both adopt a prior detector (YOLOv5 [7]), while in Gen6D the localization is
included in the pipeline.

The third group is composed of methods for relative pose estimation [13, 20].
These methods do not require any type of object model but instead rely on one
or more support views of the novel object. In particular, NOPE [13] adopts a
single support view, while RelPose [20] can work with two or more support
views. Both methods rely on RGB data, and no geometric information is present.
Because of this, they only estimate the relative rotation.

Oryon is fundamentally different from model-based methods as the object 3D
model is not required. With respect to model-free methods instead, Oryon does
not rely on complex object onboarding procedures, and it does not assume that
the novel object is physically available so that a video sequence can be
captured. Instead, to test on a novel object, only a textual description of it
is required. Finally, compared to relative pose estimation methods, Oryon is
able to estimate the complete pose due to the requirement of depth
information.

Table 1:  Comparison of the data requirements of Oryon with examples of state-
of-the-art methods for generalizable pose estimation. We classify the methods
based on: Input: the type of input data, typically RGB or RGBD; Reference:
additional data used to identify the novel object at test time; Full pose:
whether the method is capable of estimating the 6D pose or is limited to the
rotation component; Object onboarding: eventual process required at test time
to acquire data about the object to pose; Localization prior: eventual
external modules used to localize the object, typically a segmentation mask or
a detector.

Method | Input | Reference | Full pose | Object onboarding | Localization prior
---|---|---|---|---|---
OVE6D [2] | D | 3D model | ✓ | Generates and encodes 4K object templates | Segm. mask
MegaPose [8] | RGBD | 3D model | ✓ | - | Detector
OSOP [16] | RGB | 3D model | ✓ | Generates 90 object templates | -
Gen6D [10] | RGB | Video sequence | ✓ | SfM and manual cropping of point cloud | -
OnePose [17] | RGB | Video sequence | ✓ | SfM to retrieve camera viewpoints | Detector
OnePose++ [5] | RGB | Video sequence | ✓ | SfM to retrieve camera viewpoints | Detector
NOPE [13] | RGB | Single supp. view |  | - | -
RelPose [20] | RGB | Two or more supp. views |  | - | -
Oryon (ours) | RGBD | Single supp. view | ✓ | Expression of textual prompt | -

##  3 Implementation details

###  3.1 Text-visual fusion architecture

In Fig. 1, we show the details of a single layer of the fusion module
$\phi_{TV}$. Note that the input visual feature map $\mathbf{E}$ and the
prompt features $\mathbf{e}^{T}$ have different feature sizes, as $\mathbf{E}$
$\in\mathbb{R}^{1024\times 24\times 24}$, while $\mathbf{e}^{T}$
$\in\mathbb{R}^{80\times 768}$, where 80 is the number of prompt templates. To
compute the cost volume, we require the same feature dimension; therefore we
apply a 2D convolution to $\mathbf{E}$ to obtain a matrix
$\in\mathbb{R}^{768\times 24\times 24}$. The guidance backbone $\phi_{G}$ is
used to output a feature map $\mathbf{G}$ $\in\mathbb{R}^{512\times 24\times
24}$, which is projected to a feature map $\in\mathbb{R}^{128\times 24\times
24}$ before concatenating with the keys and queries of the two attention
layers. We also project the prompt features $\mathbf{e}^{T}$ to a lower-
dimensional space to obtain a feature list $\in\mathbb{R}^{80\times 128}$,
which is concatenated to the visual feature map in the text guidance head. The
text guidance head is used to inject prompt information into a lower-
resolution feature map to enhance the semantic content of the features.

Note that the fusion module $\phi_{TV}$ is composed of two of the layers
depicted in Fig. 1. Experimentally, we found beneficial to use the weights of
CATSeg [3] for the window attention and shifted window attention parts of the
module. In CATSeg, these two modules are followed by a Linear Transformer
layer, which is used to model the relationship among the different classes in
the textual prompt. Instead, in our setting, the prompt $T$ only describes a
single class (i.e., object); therefore we replace the Linear Transformer layer
with the text guidance head, which is trained from scratch.

\begin{overpic}[width=496.85625pt]{supp/figures/_fusion.pdf}
\put(6.0,24.0){$\mathbf{G}$} \put(99.5,12.5){$\mathbf{C}$ }
\put(1.5,23.0){$\phi_{G}$ } \put(-1.0,15.6){$\mathbf{E}$ }
\put(-1.0,9.8){$\mathbf{e}^{T}$ } \put(15.3,15.8){[.]} \put(45.7,15.8){[.]}
\put(20.0,-0.5){\footnotesize Window attention} \put(48.0,-0.5){\footnotesize
Shifted window attention} \put(78.0,-0.5){\footnotesize Text guidance head}
\end{overpic}

Figure 1:  Architecture detail of a layer of the fusion module $\phi_{TV}$. W.
Attention: attention layer that performs self-attention within windows, as in
[11]; SW. Attention: attention layer that performs self-attention among
shifted windows, as in [11]; Upsampling: upsampling by interpolation;
$\mathbf{\oplus}$ denotes feature sum; $\mathbf{\otimes}$ denotes cost
computation by cosine similarity; [.] denotes feature concatenation.

###  3.2 Decoder architecture

In Fig. 2, we show the details of the decoder $\phi_{D}$. A single decoder
layer is composed by a transposed 2D convolution that doubles the spatial
resolution of the input feature map. The result is concatenated with a
projection of the feature map derived from the guidance backbone $\phi_{G}$.
We use $\mathbf{G}_{1}$ for the first layer and $\mathbf{G}_{2}$ for the
second, while the third layer does not use any guidance. After the
concatenation, we apply two blocks of operations, each composed of a 2D
convolution followed by group normalization and ReLU activation. For better
visualization, this group of operations is depicted as ConvGroup in Fig. 2.

As for the fusion module $\phi_{TV}$, for the first two upsampling layers and
the guidance projections, we finetune the weights provided by CATSeg [3]. We
find this approach to be beneficial, as the upsampling layers are already
trained for segmentation and therefore provide a good initialization to learn
fine-grained features necessary for matching. The third upsampling layer is
instead trained from scratch.

\begin{overpic}[width=496.85625pt]{supp/figures/_decoder.pdf}
\put(2.0,40.0){$\phi_{G}$ } \put(26.0,42.8){$\mathbf{G}_{1}$}
\put(61.5,42.8){$\mathbf{G}_{2}$} \put(-1.0,17.0){$\mathbf{C}$ }
\put(22.1,17.0){[.]} \put(57.4,17.0){[.]} \put(12.0,-1.0){\footnotesize
1${}^{\circ}$ upsampling} \put(46.0,-1.0){\footnotesize 2${}^{\circ}$
upsampling} \put(73.0,-1.0){\footnotesize 3${}^{\circ}$ upsampling}
\put(99.0,17.0){$\mathbf{F}$ } \end{overpic}

Figure 2:  Architecture detail of the decoder $\phi_{D}$. TransConv:
transposed 2D convolution; ConvGroup: group composed by two blocks, each
contains a 2D convolution, a group normalization and a ReLU; [.] denotes
feature concatenation.

###  3.3 Hardware and implementation

We implement Oryon with PyTorch Lightning [1]. We train each model on four
Nvidia V100 GPUs and set the batch size to 32. Training a single model for 20
epochs takes about 20 hours in our standard setting. We test on a single GPU
of the same model; a complete evaluation on a dataset takes about one hour,
including I/O operations and metric computations.

##  4 Dataset generation

In this section, we describe the process to generate the scene pairs for our
training and test datasets. Given an object $O$ in two scenes $A$ and $Q$, let
PA, PQ be the point clouds resulting from the unprojection of the two depth
maps 0ptA, 0ptQ, and let RGBA, RGBQ be the respective RGB images. The point
clouds PA, PQ are then filtered by using the ground-truth mask of $A$ and $Q$
of the object $O$, thus retaining only the points that belong to the object of
interest in the two scenes. The objective is to find the transformation
$T_{A\rightarrow Q}$ that aligns PA to PQ. Let $T_{A}$, $T_{Q}$ be the ground-
truth pose of $O$ in $A$ and $Q$ respectively.

In order to generate the poses required for the training process, the
following strategy is adopted:

  1. 1.

Sample a random object $O$, and randomly sample a scene $A$ in which $O$ is
present.

  2. 2.

Consider all the images that contain $O$ that belong to a different scene from
$A$. Randomly sample $Q$ from this set.

  3. 3.

Given $T_{A}$ and $T_{Q}$, compute the relative pose $T_{A\rightarrow Q}$ =
$T_{Q}(T_{A})^{-1}$.

  4. 4.

Apply $T_{A\rightarrow Q}$ to $P_{A}$, thus aligning it to $P_{Q}$. Use the
Nearest Neighbor algorithm to compute a set of 3D matches between $P_{A}$ and
$P_{Q}$. To be considered a match, the Euclidean distance between two points
must be $\leq 2$ mm.

  5. 5.

Use the intrinsic camera parameters to project the set of matches in 2D, thus
obtaining the pixel-level correspondences $\textbf{x}^{A}$, $\textbf{x}^{Q}$
between RGBA and RGBQ.

Note that the relative pose $T_{A\rightarrow Q}$ is only used for evaluation,
as the training objectives are the ground-truth matches $\textbf{x}^{A}$,
$\textbf{x}^{Q}$. The above process is repeated until the desired number of
image pairs is obtained (i.e., 2K for REAL275 and TOYL, 20K for SN6D).
Additionally, when generating the training set from SN6D, we reject all scene
pairs with fewer than 100 ground-truth matches.

##  5 Prompt details

In this section, we present the textual prompts used for all our tests with
REAL275 [18] and TOYL [6], along with some examples of textual metadata from
the training set, SN6D [19]. Note that we omit the prompt template used to
augment the object descriptions. We adopt the template list used by CLIP [14],
which includes 80 templates.

###  5.1 REAL275 prompts

In Fig. 3, we provide the exhaustive list of prompts used for each object
model in REAL275. For each object, we show an example crop obtained from our
test set and the textual information used to compose the prompt. We report in
black the object name, in green the description used in our default setting,
and in red the misleading description used in the ablation study.

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/bottle_1.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/bottle_2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/bottle_3.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/bowl_1.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/bowl_2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/bowl_3.png}
\end{overpic}

bottle

red stanford

blue tall

bottle

white shampoo

black water

glass

tall plastic

green plastic

bowl

white and blue

purple and green

bowl

white small

yellow

bowl

white small

big red

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/camera_1.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/camera_2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/camera_3.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/can_1.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/can_2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/can_3.png}
\end{overpic}

camera

black lens

white digital

camera

black lens

red digital

camera

white small

blue lens

can

colorful tall

green small

can

green tall

blue

can

white tall

red

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/laptop_1.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/laptop_2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/laptop_3.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/mug_1.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/mug_2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/nocs/mug_3.png}
\end{overpic}

laptop

brown open

green closed

laptop

open black

closed white

laptop

grey open

white closed

mug

white

purple

mug

brown

white

mug

light blue

red

Figure 3:  Prompts used in the REAL275 [18] dataset. For each object, we show
the object name, the detailed description (i.e., the default one), and the
misleading description we adopt in the ablation study. All the objects are
cropped for better visualization.

###  5.2 Toyota-Light prompts

In Fig. 4, we present the exhaustive list of prompts used for each object
model in the TOYL. For each object, we show an example crop obtained from our
test set and the textual information used to compose the prompt. We report in
black the object name and in green the description used in our default
setting. Note how, with respect to REAL275, the poses of the objects show more
variety (e.g., objects tilted on one side or upside-down), as well as more
object types.

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/1.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/3.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/4.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/5.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/6.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/7.png}
\end{overpic}

remote

white and blue

magazine

green sudoku

mug

red and white small

mug

colorful small

mug

large green

mug

large yellow

mug

black and white

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/8.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/9.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/10.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/11.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/12.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/13.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/14.png}
\end{overpic}

plastic bottle

green

croc

light green

cup

blue and white

cup

brown and red

plate

red and white small

plate

white striped small

cracker box

blue and yellow

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/15.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/16.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/17.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/18.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/19.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/20.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/toyl/21.png}
\end{overpic}

milk carton

white and blue

plastic container

white

can

green and white

can

orange

basket

yellow

mug

black and white

mug

black and white

Figure 4:  Prompts used in the TOYL [6] dataset. For each object, we show the
object name and the detailed description (i.e., the default one). All the
objects are cropped for better visualization.

###  5.3 ShapeNet6D synsets

In Fig. 5, we showcase some examples of objects in the SN6D [19] dataset,
along with the object names and the synsets (i.e., synonym sets) we adopt to
build the prompt. For each object, we show an example crop obtained from the
training set and the list of synonyms obtained from ShapeNetSem [15]. The
first name in the list is the default one, and the subsequent names (if
present) are the synonyms. This dataset is completely synthetic and does not
depict a realistic scenario, as all the objects are rendered in random poses
and on a random background. Nonetheless, SN6D shows a wide variety of poses
and objects, along with rich textual information useful for constructing the
textual prompts.

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/12.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/2.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/3.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/8.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/5.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/6.png}
\end{overpic}

telephone booth;
phone booth;
call box;
telephone box

ashcan;
trash can;
garbage can;
dust bin

bag;
handbag;
pocketbook;
purse

recliner;
reclining chair;
lounger

blender;
liquidizer;
liquidiser

french fries;
french-fried potatoes;
fries;
chips

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/7.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/4.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/9.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/10.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/11.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/prompts/sn6d/1.png}
\end{overpic}

plant;
flora;
plant life

battery;
electric battery

rifle

skateboard

table

alarm clock;
alarm

Figure 5:  Sample objects from the SN6D [19] dataset. For each object, we show
the object default name, followed by its synonyms. Note that not all objects
have an associated synonym set (e.g., the rifle, skateboard, and table
objects). All the objects are cropped for better visualization, and padding is
added when necessary.

##  6 Additional qualitative results

In Sec. 6.1, we report additional qualitative results on pose estimation on
the test datasets. In Sec. 6.2, we show qualitative results of the
segmentation mask obtained by Oryon in our best setting and compare them with
the ground-truth masks and the ones predicted by OVSeg [9].

###  6.1 Pose results

In Fig. 6, we show qualitative results of the object poses on REAL275 [18].
Oryon is generally effective at localizing the object of interest, while the
main source of errors in the pose is related to the rotation components (see
Fig. 6(a,c,f)). In all these cases, the object appears small, and therefore
the retrieved matches are coarse-grained. On the other hand, ObjectMatch [4]
and SIFT [12] show larger errors, often related to the translation components
(Fig. 6(a,b,d) for ObjectMatch, and Fig. 6(e) for SIFT). Note that all methods
in this visualization use our segmentation mask; therefore, translation errors
are due to spurious matches rather than an incorrect mask.

Anchor

Ground truth

ObjectMatch [4]

SIFT [12]

Ours

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/0_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/0_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/0_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/0_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/0_ours.png}
\end{overpic}

(a) Prompt: White and blue bowl

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/1_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/1_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/1_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/1_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/1_ours.png}
\end{overpic}

(b) Prompt: Brown open laptop

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/2_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/2_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/2_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/2_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/2_ours.png}
\end{overpic}

(c) Prompt: Black lens camera

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/3_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/3_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/3_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/3_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/3_ours.png}
\end{overpic}

(d) Prompt: White tall can

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/4_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/4_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/4_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/4_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/4_ours.png}
\end{overpic}

(e) Prompt: Light blue mug

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/5_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/5_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/5_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/5_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_nocs/5_ours.png}
\end{overpic}

(f) Prompt: Red Stanford bottle

Figure 6:  Examples of qualitative pose results from the REAL275 [18] dataset.
All the results use the segmentation mask predicted by Oryon. We show the
object model colored by mapping its 3D coordinates to the RGB space. Query
images are darkened to highlight the object poses.

In Fig. 7, we show qualitative results of the object poses on TOYL [6]. As in
REAL275, Oryon localization on TOYL is quite accurate. Rotation errors are
still present and are related to small objects (see Fig. 7(a, d, f)), which in
TOYL are more common due to higher variation in poses compared to REAL275. We
observe that ObjectMatch’s effectiveness in this dataset is limited: in some
cases, large errors cause the projected object to fall outside the image plane
(Fig. 7(a, b)), while in other cases, significant translation errors are
present (Fig. 7(d, e, f)). As observed in the main results, the high variation
in light conditions of TOYL is not tolerated by ObjectMatch, which fails to
find accurate matches. On the other hand, SIFT performances are closer to
ours. Localization is more effective than ObjectMatch (Fig. 7(a, b, c, d, f)),
but rotation errors are still present (Fig. 7(a, b, f)). The scale invariance
properties of SIFT make it more effective for this scenario.

Anchor

Ground truth

ObjectMatch [4]

SIFT [12]

Ours

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/0_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/0_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/0_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/0_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/0_ours.png}
\end{overpic}

(a) Prompt: White and blue remote

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/1_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/1_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/1_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/1_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/1_ours.png}
\end{overpic}

(b) Prompt: Red and white small plate

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/2_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/2_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/2_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/2_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/2_ours.png}
\end{overpic}

(c) Prompt: Blue and yellow cracker box

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/3_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/3_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/3_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/3_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/3_ours.png}
\end{overpic}

(d) Prompt: Orange can

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/4_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/4_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/4_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/4_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/4_ours.png}
\end{overpic}

(e) Prompt: Green sudoku magazine

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/5_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/5_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/5_om.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/5_sift.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/pose_toyl/5_ours.png}
\end{overpic}

(f) Prompt: Red and white small mug

Figure 7:  Examples of qualitative pose results from the TOYL [6] dataset. All
the results use the segmentation mask predicted by Oryon. We show the object
model colored by mapping its 3D coordinates to the RGB space. Query images are
darkened to highlight the object poses.

###  6.2 Segmentation results

In Fig. 8, we show some qualitative results of the segmentation masks on
REAL275 [18]. On this dataset, Oryon outperforms OVSeg [9] by 10.1 points in
mean Intersection-over-Union (mIoU). In Fig. 8, we can observe that the masks
predicted by Oryon are generally coarser than the ones predicted by OVSeg.
This is caused by the lower resolution we adopt for the segmentation mask
(192$\times$192), while OVSeg generates the masks using the original image
resolution (480$\times$640). There are cases in which this results in a
clearly better performance for OVSeg (Fig. 8(a, d)). However, compared to
Oryon, false positives are much more frequent in OVSeg: the object is
completely missed in Fig. 8(c, e, f), while in Fig. 8(b) the whole table is
selected in the anchor scene, and the wrong object is segmented in the query
one.

On the other hand, Oryon is less prone to false negatives: most errors are
instead due to the inclusion of background in the mask (Fig. 8(b)) or
segmenting only a part of the object (Fig. 8(d)). Both these errors can still
provide a sufficient number of valid matches to perform registration, while a
false positive results in an automatic failure in the pose estimation task.
Note the particular case of Fig. 8(f), in which two mugs are present in the
anchor image. This causes an ambiguity, and results in a msk for each mug.
Similarly, in the query image of Fig. 8(c), the wrong mug is segmented.

Anchor

Query

Anchor

Query

Anchor

Query

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_anchor_gt.png}
\put(-12.0,8.0){\small{\rotatebox{90.0}{Ground truth}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_anchor_ovseg.png}
\put(-12.0,10.0){\small{\rotatebox{90.0}{OVSeg~{}\cite[cite]{[\@@bibref{Number}{liang2023ovseg}{}{}]}}}}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_anchor_ours.png}
\put(-12.0,18.0){\small{\rotatebox{90.0}{Oryon}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/0_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/1_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/2_query_ours.png}
\end{overpic}

(a) Prompt: Gray open laptop

(b) Prompt: White small camera

(c) Prompt: Brown mug

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_anchor_gt.png}
\put(-12.0,8.0){\small{\rotatebox{90.0}{Ground truth}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_anchor_ovseg.png}
\put(-12.0,10.0){\small{\rotatebox{90.0}{OVSeg~{}\cite[cite]{[\@@bibref{Number}{liang2023ovseg}{}{}]}}}}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_anchor_ours.png}
\put(-12.0,18.0){\small{\rotatebox{90.0}{Oryon}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/3_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/4_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_nocs/5_query_ours.png}
\end{overpic}

(d) Prompt: Black lens camera

(e) Prompt: Tall green can

(f) Prompt: White mug

Figure 8:  Examples of qualitative segmentation results from the REAL275 [18]
dataset. Images are darkened to highlight the masks.

In Fig. 9, we show some qualitative results of the segmentation masks on TOYL
[6]. On this dataset, Oryon is outperformed by OVSeg [9] by 7.4 mIoU. Compared
to REAL275, TOYL does not show additional objects other than the object of
interest; therefore it is an easier dataset for the segmentation task. We
observe that both Oryon and OVSeg can correctly locate the objects of
interest, and all the errors in the examples are due to imprecision in the
segmentation mask, instead of false positives or false negatives. Similarly to
REAL275, OVSeg masks are generally more accurate than the ones predicted by
Oryon. The higher resolution of OVSeg is an important advantage in this
dataset, as the object can appear very small due to perspective (see Fig. 9(a,
b, c)). Most of the errors in Oryon are due to partial object segmentation
(Fig. 9(b, d, f)) and background inclusion in the mask (see Fig. 9(a, e)).

Anchor

Query

Anchor

Query

Anchor

Query

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_anchor_gt.png}
\put(-12.0,8.0){\small{\rotatebox{90.0}{Ground truth}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_anchor_ovseg.png}
\put(-12.0,10.0){\small{\rotatebox{90.0}{OVSeg~{}\cite[cite]{[\@@bibref{Number}{liang2023ovseg}{}{}]}}}}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_anchor_ours.png}
\put(-12.0,18.0){\small{\rotatebox{90.0}{Oryon}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/0_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/1_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/2_query_ours.png}
\end{overpic}

(a) Prompt: White and blue remote

(b) Prompt: Red and white small mug

(c) Prompt: Large green mug

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_anchor.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_query.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_anchor_gt.png}
\put(-12.0,8.0){\small{\rotatebox{90.0}{Ground truth}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_anchor_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_query_gt.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_anchor_ovseg.png}
\put(-12.0,10.0){\small{\rotatebox{90.0}{OVSeg~{}\cite[cite]{[\@@bibref{Number}{liang2023ovseg}{}{}]}}}}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_anchor_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_query_ovseg.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_anchor_ours.png}
\put(-12.0,18.0){\small{\rotatebox{90.0}{Oryon}}} \end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/3_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/4_query_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_anchor_ours.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/segm_toyl/5_query_ours.png}
\end{overpic}

(d) Prompt: Black and white mug

(e) Prompt: Green plastic bottle

(f) Prompt: White and blue milk cartoon

Figure 9:  Examples of qualitative segmentation results from the TOYL [6]
dataset. Images are darkened to highlight the masks.

###  6.3 Feature visualization

In Fig. 10, we present some examples of the visualization of the feature
distance in the feature maps $\mathbf{F}^{A}$, $\mathbf{F}^{Q}$ obtained by
Oryon. For each image pair, we first sample a random reference point from the
ground-truth segmentation mask on the anchor image (shown in green). We then
compute the cosine similarity between the feature at the reference point and
all the other features, both on the anchor and on the query image. In this
way, we obtain a distance measure for each pixel, which is normalized across
the pair and mapped to the RGB space. For each example, we show three versions
of the feature maps obtained with the prompts we adopted in the ablation study
in the main paper (i.e., the standard prompt, the prompt with only the object
name, and the prompt with a misleading description).

When the standard prompt is adopted (first row), we can observe a sharp
difference between the features of the background and the ones on the objects.
Note also how in Fig. 10(a), the most similar points to the reference one on
the query image are on the bowl border, as the reference point itself. When
the object description is removed (second row), we can observe noticeable
changes in Fig. 10(a), in which the background appears more noisy, and in Fig.
10(c), where a set of similar features appears on top of the camera in the
query image. Instead, in Fig. 10(b), the difference with the distances of the
original prompt is less relevant. As we reported in the ablation study, this
alternative prompt causes a small drop of 2.2 in AR and of 3.1 in mIoU.
Finally, in the last row, a misleading description is used in the prompt. This
causes a clear drop in feature quality in Fig. 10(a, c): in the first example,
the object outline is still visible due to similarity in the borders of the
bowl, and in the second example only spurious similar features are present.
Instead, the laptop object in Fig. 10(b) is less affected. Note that in the
ablation study, the misleading prompt causes a drop of 6.8 and 10.1 in AR and
mIoU, respectively. These results suggest that the use of a misleading or more
generic prompt can impact Oryon performances differently depending on the
object of interest.

Anchor

Query

Anchor

Query

Anchor

Query

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_anchor_.png}
\put(6.0,28.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_query_.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_anchor_.png}
\put(25.0,72.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_query_.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_anchor_.png}
\put(70.0,75.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_query_.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_anchor_yes.png}
\put(-12.0,9.0){\small{\rotatebox{90.0}{Standard prompt}}}
\put(6.0,28.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\par\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_query_yes.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_anchor_yes.png}
\put(25.0,72.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_query_yes.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_anchor_yes.png}
\put(70.0,75.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_query_yes.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_anchor_no.png}
\put(-12.0,21.0){\small{\rotatebox{90.0}{Name only}}}
\put(6.0,28.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_query_no.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_anchor_no.png}
\put(25.0,72.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_query_no.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_anchor_no.png}
\put(70.0,75.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_query_no.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_anchor_wrong.png}
\put(-12.0,12.0){\small{\rotatebox{90.0}{Misleading desc.}}}
\put(6.0,28.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/0_query_wrong.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_anchor_wrong.png}
\put(25.0,72.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/1_query_wrong.png}
\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_anchor_wrong.png}
\put(70.0,75.0){\tiny{\color[rgb]{0.531,0.531,0.531}\definecolor[named]{pgfstrokecolor}{rgb}{0.531,0.531,0.531}\pgfsys@color@gray@stroke{0.531}\pgfsys@color<EMAIL_ADDRESS>\end{overpic}

\begin{overpic}[width=496.85625pt]{supp/figures/qualitative/feat_nocs/2_query_wrong.png}
\end{overpic}

(a) Prompt: White small bowl

(b) Prompt: Brown open laptop

(c) Prompt: Black lens camera

Figure 10:  Visualization of the feature distance in $\mathbf{F}^{A}$,
$\mathbf{F}^{Q}$ with respect to a reference point on the object (denoted as
$\bullet$ in the anchor image) on REAL275 [18]. For each sample, we test the
three different prompts we adopted in the ablation study: Standard prompt: the
standard prompt we adopt, composed of a description and the object name; Name
only: the standard prompt without the description; Misleading desc.: the
standard description is replaced by a misleading one (see Fig. 3). The RGB
images are modified to highlight the object of interest.

## References

  * [1] Pytorch lightning documentation.  https://lightning.ai/docs/pytorch/stable/.  Last access: 21/11/2023.
  * Cai et al. [2022] D. Cai, J. Heikkila, and E. Rahtu.  OVE6D: Object Viewpoint Encoding for Depth-based 6D Object Pose Estimation.  In _CVPR_ , 2022.
  * Cho et al. [2023] S. Cho, H. Shin, S. Hong, S. An, S. Lee, A. Arnab, P. H. Seo, and Kim.  Cat-seg: Cost aggregation for open-vocabulary semantic segmentation.  In _arXiv:2303.11797_ , 2023.
  * Gümeli et al. [2023] C. Gümeli, A. Dai, and M. Nießner.  ObjectMatch: Robust Registration using Canonical Object Correspondences.  In _CVPR_ , 2023.
  * He et al. [2022] X. He, J. Sun, Y. Wang, D. Huang, H. Bao, and X. Zhou.  OnePose++: Keypoint-Free One-Shot Object Pose Estimation without CAD Models.  In _NeurIPS_ , 2022.
  * Hodan et al. [2018] T. Hodan, F. Michel, E. Brachmann, W. Kehl, A. GlentBuch, D. Kraft, B. Drost, J. Vidal, S. Ihrke, X. Zabulis, et al.  Bop: Benchmark for 6D object pose estimation.  In _ECCV_ , 2018.
  * Jocher [2020] Glenn Jocher.  Yolov5 by ultralytics, 2020.
  * Labbé et al. [2022] Y. Labbé, L. Manuelli, A. Mousavian, S. Tyree, S. Birchfield, J. Tremblay, J. Carpentier, M. Aubry, D. Fox, and J. Sivic.  Megapose: 6D pose estimation of novel objects via render & compare.  In _CoRL_ , 2022.
  * Liang et al. [2023] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  In _CVPR_ , 2023.
  * Liu et al. [2022] Y. Liu, Y. Wen, S. Peng, C. Lin, X. Long, T. Komura, and W. Wang.  Gen6d: Generalizable model-free 6-DoF object pose estimation from rgb images.  In _ECCV_ , 2022.
  * Liu et al. [2021] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _ICCV_ , 2021.
  * Lowe [1999] D. Lowe.  Object recognition from local scale-invariant features.  In _ICCV_ , 1999.
  * Nguyen et al. [2023] V. Nguyen, T. Groueix, Y. Hu, M. Salzmann, and V. Lepetit.  Nope: Novel object pose estimation from a single image.  In _arXiv:2303.13612_ , 2023.
  * Radford et al. [2021] A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Savva et al. [2015] M. Savva, A. Chang, and Hanrahan P.  Semantically-Enriched 3D Models for Common-sense Knowledge.  In _CVPR_ , 2015.
  * Shugurov et al. [2022] I. Shugurov, F. Li, B. Busam, and S. Ilic.  OSOP: a multi-stage one shot object pose estimation framework.  In _CVPR_ , 2022.
  * Sun et al. [2022] J. Sun, Z. Wang, S. Zhang, X. He, H. Zhao, G. Zhang, and X. Zhou.  Onepose: One-shot object pose estimation without cad models.  In _CVPR_ , 2022.
  * Wang et al. [2019] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and Leonidas J. Guibas.  Normalized object coordinate space for category-level 6D object pose and size estimation.  In _CVPR_ , 2019.
  * Yisheng et al. [2022] H. Yisheng, W. Yao, F. Haoqiang, C. Qifeng, and S. Jian.  Fs6d: Few-shot 6D pose estimation of novel objects.  In _CVPR_ , 2022.
  * Zhang et al. [2022] J. Zhang, D. Ramanan, and S. Tulsiani.  Relpose: Predicting probabilistic relative rotation for single objects in the wild.  In _ECCV_ , 2022.