# CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-
Language Knowledge

Xiao Lin, Minghao Zhu, Ronghao Dang, Guangliang Zhou, Shaolong Shu, , Feng
Lin, , Chengju Liu and Qijun Chen† † Corresponding author: Qijun Chen (email:
qjchen@tongji.edu.cn)The authors of this paper is supported by the National
Natural Science Foundation of China under Grants (62073245, 62233013,
62173248).X. Lin, M. Zhu, R. Dang, G. Zhou, S. Shu, C. Liu and Q. Chen are
with the College of Electronics and Information Engineering, Tongji
University, Shanghai 201804, China (email: {2111118, tjdezmh, dangronghao,
tj_zgl, shushaolong, qjchen}@tongji.edu.cn).F. Lin is with the College of
Electronics and Information Engineering, Tongji University, Shanghai 201804,
China, and also with the Department of Electrical and Computer Engineering,
Wayne State University, Detroit, MI 48202 USA (email: flin@wayne.edu).Color
versions of one or more figures in this article are available at
https://doi.org/xx.xxxx/TCSVT.20xx.xxxxxxx.Digital Object Identifier
xx.xxxx/TCSVT.20xx.xxxxxxx

###### Abstract

Most of existing category-level object pose estimation methods devote to
learning the object category information from point cloud modality. However,
the scale of 3D datasets is limited due to the high cost of 3D data collection
and annotation. Consequently, the category features extracted from these
limited point cloud samples may not be comprehensive. This motivates us to
investigate whether we can draw on knowledge of other modalities to obtain
category information. Inspired by this motivation, we propose CLIPose, a novel
6D pose framework that employs the pre-trained vision-language model to
develop better learning of object category information, which can fully
leverage abundant semantic knowledge in image and text modalities. To make the
3D encoder learn category-specific features more efficiently, we align
representations of three modalities in feature space via multi-modal
contrastive learning. In addition to exploiting the pre-trained knowledge of
the CLIP’s model, we also expect it to be more sensitive with pose parameters.
Therefore, we introduce a prompt tuning approach to fine-tune image encoder
while we incorporate rotations and translations information in the text
descriptions. CLIPose achieves state-of-the-art performance on two mainstream
benchmark datasets, REAL275 and CAMERA25, and runs in real-time during
inference (40FPS).

###### Index Terms:

object pose estimation, category-level, CLIP, point cloud, multi-modal
learning.

##  I Introduction

Efficient and accurate estimation of objects’ pose is crucial in numerous
practical applications, including augmented reality [1], autonomous driving
[2] and robotic manipulation [3], _etc._ Recent works [4, 5, 6, 7, 8, 9, 10]
have demonstrated remarkable performance in estimating objects’ pose on
existing benchmarks [11, 12, 13], they generally can only handle a few ”seen”
objects with high-quality CAD models, and sometimes even one instance at a
time, which does not adhere to real dynamic and unknown world.

Figure 1: Illustration of CLIPose. The proposed CLIPose takes as input a
triplet of objects (point cloud, image and text), and aligns the
representations of three modalities in feature space via contrastive learning.
This enables the network to obtain more robust category-specific information
from vision-language pre-trained knowledge.

In recent years, category-level object pose estimation methods [14, 15, 16,
17, 18, 19] have emerged, whose objective is to estimate the pose of previous
”unseen” objects within predefined categories. However, developing such
algorithms is inherently more challenging due to the complicated and diverse
intra-category variation, including shape and texture variation.

To address the above-mentioned issue, a straightforward idea is to extract
category-specific feature from 3D point cloud data, as the objects of the same
category typically exhibit similar geometric characteristics. For instance,
mugs are typically cylindrical with a handle while laptops generally come with
hinged structures. Most of previous category-level methods conducted research
following the idea mentioned above. [15] extracts the point cloud mean shape
of each category, referred to as _shape prior_ , while [17] indirectly
acquires category information by predicting the objects’ bounding boxes.
Although the positive outcomes of existing methods, there is still room for
improvement in two aspects: 1) These methods heavily depend on complicated
manual designs or various intermediate processes to obtain category
information. 2) The category information extracted from point cloud is limited
by the scale of existing category-level pose estimation datasets due to the
prohibitive cost of 3D data collection and annotation [20, 21, 22]. Therefore,
it’s essential to explore more effective methods to extract richer category-
specific feature for better pose estimation performance.

More recently, multi-modal learning [23, 24, 25, 26] has advanced rapidly and
drawn great attention. Numerous proposed models, represented by CLIP [23]
exploits language knowledge to assist vision tasks and demonstrate promising
outcomes [27, 28, 29]. These works indicate that leveraging knowledge from
different modalities can effectively promote semantic understanding in the
original modality. It motivates us to employ rich knowledge from image and
text modalities in category-level pose estimation task. Some recent methods
[30, 31] have explored how multi-modal information can assist 3D data
understanding. However, these works primarily focus on point cloud
classification tasks. As for whether multi-modal learning can further
facilitate other downstream 3D tasks, e.g. object pose estimation, is still
not well studied.

In this paper, we propose a concise yet effective category-level pose
estimation model, named CLIPose, which leverages pre-trained vision-language
knowledge to assist in extracting more robust category-specific feature for
pose estimation. The illustration of CLIPose is shown in fig. 1. Specifically,
we first acquire three modalities features of the target objects: 3D point
clouds, image patches and text descriptions that specify its category.
Subsequently, the features obtained from three modalities are aligned through
the contrastive learning. The alignment operation can reduce the distance
between different modalities in the feature space, which enables more
comprehensive point cloud feature learning. Unlike directly learning with
other data modalities [32, 33] (e.g. RGB), making use of a pre-trained vision-
language model allows us to utilize abundant semantics extracted from image-
text feature space to capture richer category-specific information for
category-level pose estimation task. Furthermore, we expect CLIP to be aware
of the pose information of objects in addition to the category feature. Hence,
we introduce a prompt tuning approach that can update the parameters of image
encoder based on pose estimation results during pre-training. Moreover, we
innovatively create text descriptions with pose parameters, which can further
assist the network to obtain pose-sensitive information.

Our primary contributions are summarized as follows:

  * •

We propose a novel category-level 6D object pose estimation framework,
CLIPose. To the best of our knowledge, our method is the first work that
employs the pre-trained vision-language model of CLIP [23] to the category-
level pose estimation task, which can leverage abundant semantic knowledge in
image and text modalities to extract more robust category-specific feature.

  * •

We align the representations from three modalities in feature space via
contrastive learning. This allows the network to learn object’s category
information in a concise manner, eliminating the need for shape prior or
complicated manual design. Meanwhile, we finely design a prompt tuning
approach to fine-tune the image encoder, which enables the CLIP model to
better adapt to pose estimation task.

  * •

We provide a brand new insight to realize the real open-world object pose
estimation. Our method achieves state-of-the-art or competitive results on two
mainstream benchmarks, REAL275 and CAMERA25, which demonstrates the
effectiveness of CLIPose.

The rest of the paper is organized as follows. Section II reviews several
previous works on category-level object pose estimation and Multi-modal
Representation Learning.

The alignment among different modalities and the prompt fine-tuning approach
for CLIP image encoder are detailed in Section III. Finally, we report and
analyze the experimental results in Section IV, and conclude our paper in
Section V.

##  II Related Works

###  II-A Category-level Object Pose Estimation

Category-level object pose estimation methods [14, 15, 34, 32, 17, 35, 18]
consist of two major lines of research. One line concentrates on the
correspondence between point clouds. NOCS [14] suggests mapping input shape to
a normalized canonical space and recovering the pose via Umeyama algorithm
[36]. SPD [15] trains a shape deformation field and leverages the structural
similarity between shape prior and target object to estimate the pose. ACR-
Pose[32] proposes an generative adversarial training network to acquire more
accurate NOCS models. while SGPA[34] utilizes Transformer[37] for global
similarity modeling of shape priors and target objects. Though effective,
these methods suffer from the high compute complexity of point cloud matching.
The other line of research explores how to achieve end-to-end pose estimation
without explicit point cloud alignment. CASS[38] estimates the object pose via
the learning of a canonical shape space implicitly. GPV-Pose [17] utilizes
point-wise bounding box voting to capture category-specific feature, while HS-
Pose [35] proposes hybrid scope feature extractor to substitute the 3D-GCN
module in GPV-Pose. IST-Net [18] demonstrates the shape prior is not the
credit to achieve precise pose estimation but shape deformation field, then
proposes an implicit space transformation to recover pose parameters. The
above-mentioned works concentrate on acquiring category information from depth
or RGB-D data. However, the category features extracted from limited 3D data
may not be well-suited to address the complicated and diverse intra-category
variations. On the contrary, we propose a novel framework that exploits pre-
train knowledge from image and text modalities to extract more comprehensive
category-specific features.

Figure 2: Overview of CLIPose. The inputs of our framework are a batch of
objects (e.g. _laptop_) represented as triplets (point cloud, image, text),
therein we append rotation and translation information to the text
description. Image and text features are extracted from a pre-trained (Frozen)
vision and language model, and point cloud features are obtained by a 3D
encoder (Tuning). Contrastive losses are applied to align the 3D
representations of an object to its image and text representations during pre-
training. We utilize CLIP’s inherent classification ability to predict the
category of the input object (_m_ is the number of categories) to form the
classification loss. The image encoder would be fine-tuned with additional
prompt tokens $P_{0}$ (Bottom Right).

###  II-B Multi-modal Representation Learning

Most of existing multimodal learning [23, 24, 25, 26] methods are about image
and text modalities. Among these works, CLIP [23] is the pioneering effort
that utilizes image and text encoders to extract feature respectively for each
image-text pair, and then aligns the features from both modalities via
contrastive learning. Thanks to the outstanding success of CLIP, various
image-text related works draw the idea from CLIP, applying contrastive
learning architecture or modal alignment approach in semantic segmentation
[29, 39], vision-language navigation [40, 41] and video retrieval [42], _etc._
. For the 3D vision domain, some recent works [30, 43, 44, 31, 45] explore the
application of multimodal representation learning and present promising
results. The PointCLIP series [30, 43] converts 3D piont cloud to multi-view
2D depth maps and employs CLIP directly for 3D recognition. DepthCLIP [44]
cleverly transforms the depth estimation problem into a the classification
problem via classifying the depth of objects into seven classes from near to
far. The ULIP series [31, 45] learns the unified representation with the goal
of developing a 3D version of CLIP.

However, above methods typically focused on point cloud classification tasks.
The application of multimodal learning and CLIP’s robust pre-trained knowledge
in other 3D downstream tasks, e.g. object pose estimation, is still not
further explored. To this end, our method develops the contrastive learning
framework among three modalities, utilizing the pre-trained knowledge from
image and text modalities to extract more robust category-specific feature for
accurate category-level object pose estimation.

##  III Methodology

###  III-A Preparing Triplets

In datasets of category-level pose estimation, the image patch and point cloud
of a specific object are typically corresponding. The text descriptions can be
obtained according to the category of objects. Naturally, we can treat the
representations from these three modalities as a triplet.

For each object $i$ in dataset, we prepare the triplet containing image patch,
point cloud and text description. Subsequently, we leverage these triplets to
train our network. During each iteration of training, we randomly select a
triplet and take it as input of three modalities encoder to extract features,

$\mathbf{f}_{i}^{M}=\Phi_{M}(M_{i})$ (1)

where $M$ indicates modalities, $\mathbf{f}_{i}^{M}$ denotes the features
extracted from specific modalities and $\Phi_{M}$ represents feature encoder
of modalities.

Image Patch of Object. We first employ the off-the-shelf Mask-RCNN [46] object
detector to segment the object region of interests(ROIs) from RGB images. In
general, the ROIs only contain the target object, so it can also be referred
to as an object image patch. Then, all object images patches after
normalization are fed into the image encoder of CLIP, which is a standard
vision transformer(ViT) [47] in this paper. The output features of image
encoder will be involved in the formation of two loss functions. One is the
alignment with the point cloud features to form the contrastive learning loss,
and the other is the similarity computation with the text features to get the
loss for predicting the category of objects.

Point Cloud Generation. Given a RGBD image, we can obtain the 2D pixel
coordinates $(u,v)$ of the target object. According to the pinhole camera
model, given the determined camera intrinsics $C$, the correspondence between
the 3D point coordinates $[X,Y,Z]$ and the 2D pixel coordinates $[u,v]$ in the
image is as follows,

$\begin{bmatrix}X\\\ Y\\\ Z\end{bmatrix}=C^{-1}Z\begin{bmatrix}u\\\ v\\\
1\end{bmatrix},C=\begin{bmatrix}f_{x}&0&c_{x}\\\ 0&f_{y}&c_{y}\\\
0&0&1\end{bmatrix}$ (2)

where the $Z$ coordinate indicates the depth of the corresponding 2D pixel
point. We can leverage the RGBD images to construct the point cloud of each
target object based on eq. 2. The point cloud features extracted by 3D encoder
will be aligned with the image features and text features respectively via
contrastive learning. In addition, we utilize the point cloud features for
point cloud symmetry reconstruction and final pose estimation.

Text Descriptions. In category-level pose estimation task, the category range
of the target object is already predefined. Hence, we could utilize predefined
category that come with each object as the corresponding text description.
Prior works [30, 31] adopt simple prompts to construct meaningful sentences,
such as ”a point cloud model of [CLS]”, with the aim of enhancing the
adaptability of the CLIP model to 3D data. However, these sentences only
contain objects’ category information, which does not guide the network to
learn pose-sensitive information.

Therefore, to enable the CLIP model more adapted to the pose estimation task,
we leverage the objects’ pose parameters to construct text descriptions. We
extract the rotation (euler angles) and translation parameters directly from
ground truth. Thus the format of text description for pre-training can be
expressed as ”a point cloud model of a [CLS] whose rotation euler angles is
$<$-103.73, 58.98, 71.62$>$ and translation is $<$10.63, 48.78, 41.32$>$.” The
process of generating text descriptions would be done offline and does not
impact the training efficiency of the network. During the inference stage, the
object’s pose parameters are required to be recovered, so the text format is
as follows: ”a point cloud model of a [CLS] whose pose is to be estimated.”

###  III-B Aligning Representations

With the obtained features of three modalities: point cloud, image and text,
CLIPose performs pre-training to align all modal representations into the same
feature space via contrastive learning. We draw the inspiration from the
training strategy of CLIP and achieve alignment by calculating the similarity
between different modalities. Specifically, we calculate the cosine similarity
between point cloud and image modalities, as well as between point cloud and
text modalities. Then, the contrastive learning loss is computed by the cross
entropy between the obtained similarity matrix and the ground truth. The
ground truth is an identity matrix, which represents stronger corresponding
for the same object. Moreover, since the scale of 3D point cloud data is
limited, we find that catastrophic forgetting will emerge if we full-tune and
update the pre-aligned image-text representations of CLIP during training.
Hence, we freeze majority of CLIP’s model parameters and train the 3D point
cloud encoder following the aforementioned contrastive learning approach [31,
44]. A small portion of image and text encoder weights are reserved for fine-
tuning, which is detailed in section III-C.

The alignment operation between different modalities serves two crucial
functions. First, aligning representations of all modalities into the same
feature space enhances the consistency among different modalities. Second, the
wealth of semantics information already captured by CLIP can effectively
assist the feature learning of 3D encoder, which enables better 3D
understanding, thereby improving the performance in category-level pose
estimation tasks.

Multi-Modal Contrastive Learning Loss. The contrastive learning is initially
developed to address multi-categorical Contrastive Estimation (CE) problems.
The ideal contrastive loss is a function whose value will be low when query is
similar to its positive key (correct category) and dissimilar to negative keys
(wrong categories), where the cross entropy function is a commonly contrastive
learning loss function.

Figure 3: Comparison of the loss function in CE and NCE. The darker the color
means the higher similarity value. $m$ denotes the number of predefined
categories. $n$ indicates the number of input samples. The ground truth list
[···] for each input batch could be formed into a similarity ground truth
matrix using one-hot encoding.

Typically, the cross entropy loss is suitable for multi-categorical problems
and it strives to classify each sample accurately. Apart from accurate
classification results, our method is more concerned with increasing the
consistency of the three modal representations via contrastive learning.
Directly using the conventional cross entropy loss does not meet our
requirements effectively. Draw the idea of prior work [48], we treat the
alignment among different modalities as a Noise Contrastive Estimation (NCE)
[49] problem and design the multi-modal noise contrastive loss for our
framework. The comparison between the loss function used in CE and NCE is
shown in fig. 3.

The essence of NCE lies in transforming a multi-categorical problem into a
binary-categorical problem. If the representations of the two modalities are
matched, this sample is considered positive sample and the rest are negative
samples (noise samples). Hence, the ground truth of noise contrastive loss is
a unitary matrix, as shown in bottom right of fig. 3. Through learning the
difference between positive and negative samples, the network is able to learn
the correspondence between two modalities. With the similarity matrices
measured by dot product, we compute the contrastive loss for each pair of
modalities as follows,

$\begin{split}\mathcal{L}_{(M_{1},M_{2})}^{NCE}=\sum_{(i,j)}-\frac{1}{2}log\frac{exp(\mathbf{f}_{i}^{M_{1}}\cdot\mathbf{f}_{j}^{M_{2}}/\tau)}{\sum_{k=1}^{K}exp(\mathbf{f}_{i}^{M_{1}}\cdot\mathbf{f}_{k}^{M_{2}}/\tau)}\\\
-\frac{1}{2}log\frac{exp(\mathbf{f}_{i}^{M_{1}}\cdot\mathbf{f}_{j}^{M_{2}}/\tau)}{\sum_{k=1}^{K}exp(\mathbf{f}_{k}^{M_{1}}\cdot\mathbf{f}_{j}^{M_{2}}/\tau)}\end{split}$
(3)

where $M_{1}$ and $M_{2}$ indicate two different modalities,
$\mathbf{f}_{i}^{M_{1}}$ denotes the $i_{th}$ feature extracted from $M_{1}$
modality, $\tau$ is a temperature hyper-parameter, $(i,j)$ represents a
positive pair in each training batch, $K$ is the number of negative samples
and $k$ indicates each negative sample from 1 to $K$. Eventually, we minimize
all modality pairs of contrastive loss as follows,

$\mathcal{L}_{mm}^{NCE}=\alpha\mathcal{L}_{(PC,img)}^{NCE}+\beta\mathcal{L}_{(PC,text)}^{NCE}+\lambda\mathcal{L}_{(img,text)}^{NCE}$
(4)

where $PC$ indicates point cloud modality. By default, $\alpha$ and $\beta$
are set to be constant 1.0 equally, and $\lambda$ is set to be 0; because the
image and text modalities are pre-aligned in CLIP, these two modalities
already possess a high level of consistency. Since we freeze most of the
parameters of CLIP’s encoder, the similarity values of the NCE loss between
these two modalities consistently remain at low levels and contributes very
little to the updates of the network parameters.

We initially intend to leverage the the first two terms of the eq. 4 directly.
However, the association between image and text contains the most intuitive
information about object categories, which deserves to be explored further. In
zero-shot classification tasks, CLIP computes the text and image
representations by dot product to obtain the similarity and predicts the
category of objects. This suggests that we can leverage CLIP’s inherent
categorization capabilities to further enhance the network’s learning of
category features. Similar to the inference stage of CLIP, we calculate the
similarity matrix by dot product of text and image representations to obtain
preliminary classification results for each object. Subsequently, we construct
the multi-categorical Contrastive Estimation (CE) loss by computing the cross
entropy function as follows,

$\mathcal{L}_{(img,text)}^{CE}=-\sum_{(i,j)}log\frac{exp(\mathbf{f}_{i}^{img}\cdot\mathbf{f}_{j}^{text})}{\sum_{m=1}^{M}exp(\mathbf{f}_{i}^{img}\cdot\mathbf{f}_{m}^{text})}$
(5)

where $CE$ indicates that we utilize the normal cross entropy loss, and $M$
donates the number of categories.

In this way, we enhance the consistency in representations between the point
cloud modality and the image-text modality, while simultaneously introducing
classification results through contrastive estimation. We conduct ablation
experiments section IV-B to validate the effectiveness of proposed loss.

###  III-C Prompt Fine-tuning for CLIP

The original CLIP model is primarily designed to solve image classification
problems. Consequently, CLIP is highly sensitive to category-related features
of objects, while the perceptive of object pose related information is
comparatively less sharp. While fully leveraging the rich semantic knowledge
embedded in the CLIP encoders, we expect that CLIP can be more attuned to the
6D pose of objects. An intuitive idea is to fully update the parameters of
CLIP image and text encoder along with the 3D encoder. Nevertheless, as
mentioned earlier, this way would lead to catastrophic forgetting. If the
parameters are completely frozen, the CLIP model is unable to effectively
obtain pose information. Hence, exploring a appropriate and efficient fine-
tune approach is crucial to serve both purposes.

We first start with the image encoder. The CLIP’s image encoder is a standard
ViT [47], which initially reshape the image $I\in\mathbb{R}^{H\times W\times
C}$ into a sequence of flattened 2D image patches
$I_{0}\in\mathbb{R}^{N\times(P^{2}\cdot C)}$, where $(H,W)$ denotes the
resolution of the original image, $C$ is the number of channels, $(P,P)$ is
the resolution of each image patch, and $N=HW/P^{2}$ is the resulting number
of patches. Then, ViT prepends a learnable embedding [$cls$] token to the
sequence of embedded patches ($I_{0}^{0}=[cls]$), serving as the final input.
The [$cls$] token captures the category information of the image through
multiple Transformer layers, which is then mapped into specific dimensions by
a projection layer and utilized for representations alignment. Recalling the
two main purposes of this paper. We aim to utilize the abundant pre-training
knowledge to assist robust category-specific feature learning and expect the
CLIP model to be more pose-sensitive. The output image features are mainly
related to the input image embedding. Naturally, we can explore fine-tuning in
this aspect. Drawing the idea from [50], we present a prompt fine-tuning
manner to appropriately update the parameters of the CLIP’s image encoder
during training, as shown in the bottom right of the fig. 2. Specifically, we
prepend a set of prompt tokens with the same dimensions as the input image
embedding, then we insert prompt tokens
$P_{0}\in\mathbb{R}^{N\times(P^{2}\cdot C)}$ between the [$cls$] embedding and
sequence of embedded image patches $I_{0}$ as follows,

$\mathcal{I}_{0}=[cls;P_{0};I_{0}^{1};I_{0}^{2};\cdots;I_{N}^{1}]$ (6)

where $\mathcal{I}_{0}$ is the integrated embedding that would be fed into the
image encoder. The [$cls$] embedding outputs the final image features through
the projection layer. Since the default length of an image embedding is 196,
we select 10 as the length of the prompt tokens to strike the balance. This
length won’t affect the original image features while facilitating effective
fine-tuning. To retain pre-trained knowledge of the image modality, we freeze
all parameters of image encoder except for the prompt tokens $P_{0}$ and the
parameters of projection layer.

As for the text encoder, the incorporation of pose parameters and text
description (section III-A) has already provided pose-sensitive information
for text modality, so we do not further fine tune the text encoder. We also
take into account various factors that might affect prompt fine-tuning,
including the length of prompt tokens, insertion location and the number of
involved transformer layers, _etc._ . Extensive ablation experiments section
IV-B are performed to validate the effect of these factors on fine-tuning.

###  III-D Pose Estimation and Overall Loss Function

Given the triplets of target objects, the objective of CLIPose is to estimate
the pose of input object, including rotation $R\in SO(3)$, translation
$t\in\mathbb{R}^{3}$ and $s\in\mathbb{R}^{3}$. We utilize point cloud features
extracted from 3D encoder to recover object 6D pose parameters, as shown in
the top right of the fig. 2. We adopt the pose estimator in GPV-Pose [17],
which decompose the rotation into three mutually perpendicular bounding box
plane normals $\left[r_{x},r_{y},r_{z}\right]$. We predict the first two
normals ($r_{x}$ and $r_{y}$), and the last rotation $r_{z}$ is available by
cross product. Note that we calibrate the plane normals ($r_{x}$, $r_{y}$) to
be perpendicular normals ($r_{x}^{*}$, $r_{y}^{*}$) before calculating the
loss by minimize the following cost function,

$\begin{split}\theta_{1}^{*},\theta_{2}^{*}&=arg\ min\
c_{y}\theta_{1}+c_{x}\theta_{2}\\\ s.t.\
\theta&=\theta_{1}+\theta_{2}+\pi/2,\end{split}$ (7)

where $\theta$ refers to the angle between origin plane normals and $c_{x}$
and $c_{y}$ denote the confidence of $r_{x}$ and $r_{y}$. From eq. 7 we can
then obtain

$\begin{split}\left\\{\begin{matrix}\theta_{1}^{*}=\frac{c_{x}}{c_{x}+c_{y}}(\theta-\frac{\pi}{2})\\\
\theta_{2}^{*}=\frac{c_{y}}{c_{x}+c_{y}}(\theta-\frac{\pi}{2}).\end{matrix}\right.\end{split}$
(8)

The calibrated plane normals ($r_{x}^{*}$, $r_{y}^{*}$) can be calculated from
($\theta_{1}^{*},\theta_{2}^{*}$) via the Rodrigues Rotation Formula [51].
Therefore, the rotation can be obtained as
$\left[r_{x}^{*},r_{y}^{*},r_{z}^{*}\right]$, and the rotation loss as follow,

$\mathcal{L}_{rot}=\left\|r_{x}-r_{x}^{gt}\right\|_{1}+\left\|r_{y}-r_{y}^{gt}\right\|_{1}$
(9)

where $r_{x}^{gt}$ and $r_{y}^{gt}$ indicate the ground truth plane normals,
$\left\|\cdot\right\|_{1}$ refers to the $\mathcal{L}_{1}$-loss.

As for translation $t$, which can be computed with the residual prediction
$t_{*}$ and the mean of input point cloud $M_{P}$ as $t=t_{*}+M_{P}$. Thus we
can calculate the translation loss as,

$\mathcal{L}_{trans}=\left\|t-t^{gt}\right\|_{1}$ (10)

where $t^{gt}$ and $s^{gt}$ donates the ground truth translation. Similarly,
the scale $s$ can be obtained as $s=s_{*}+M_{s}$, where $s_{*}$ is the
residual scale and $M_{s}$ indicates the mean object scale of all instances
within a certain category in the training dataset. The scale loss is defined
as,

$\mathcal{L}_{scale}=\left\|s-s^{gt}\right\|_{1}$ (11)

where $s^{gt}$ donates the ground truth scale. Finally, the total pose loss
term can be expressed as follows,

$\begin{split}\mathcal{L}_{pose}=\mathcal{L}_{rot}+\mathcal{L}_{trans}+\mathcal{L}_{scale}\\\
\end{split}$ (12)

TABLE I: Comparisons with state-of-the-art methods on REAL275 dataset. Overall best results are in bold. Input denotes the format of the input data used by the methods, and Prior refers to whether the method necessitates prior category information. ‘-’ indicates no results reported under this metric in original paper. ($\uparrow$) represents a higher value indicating better performance.  Methods | Input | Prior |  5°2 _cm_ $\uparrow$ |  5°5 _cm_ $\uparrow$ |  10°2 _cm_ $\uparrow$ |  10°5 _cm_ $\uparrow$ |  10°10 _cm_ $\uparrow$ |  Speed(FPS)$\uparrow$
---|---|---|---|---|---|---|---|---
NOCS[14] | RGB-D | $\times$ | - | 9.5 | 13.8 | 26.7 | 26.7 | 9.9
CASS[38] | RGB-D | $\times$ | 19.5 | 23.5 | 50.8 | 58.0 | 58.3 | -
DualPoseNet[52] | RGB-D | $\times$ | 29.3 | 35.9 | 50.0 | 66.8 | - | 2.6
SPD[15] | RGB-D | ✓ | 19.3 | 21.4 | 43.2 | 54.1 | - | 7.9
SGPA[34] | RGB-D | ✓ | 35.9 | 39.6 | 61.3 | 70.7 | - | 6.6
DPDN[33] | RGB-D | ✓ | 46.0 | 50.7 | 70.4 | 78.4 | 80.4 | 31.6
IST-Net[18] | RGB-D | $\times$ | 47.8 | 55.1 | 69.5 | 79.6 | 81.6 | 33.7
FS-Net[16] | D | $\times$ | 19.9 | 33.9 | - | 69.1 | 71.0 | 16.1
GPV-Pose[17] | D | $\times$ | 32.0 | 42.9 | 55.0 | 73.3 | 74.6 | 29.6
SSP-Pose[53] | D | ✓ | 34.7 | 44.6 | - | 77.8 | 79.7 | 37.0
GPT-COPE[19] | D | ✓ | 45.9 | 53.8 | 63.1 | 77.7 | 79.8 | -
DR-Pose[54] | D | ✓ | 41.7 | 46.0 | 67.7 | 76.3 | - | -
HS-Pose[35] | D | $\times$ | 45.3 | 54.9 | 68.6 | 83.6 | 84.6 |  39.8
Ours | D | $\times$ | 48.7 | 58.3 | 70.4 | 85.2 | 86.2 | 40.0

The pose estimation network actually includes a point cloud symmetry
reconstruction component, which is designed to extract more accurate point
clouds for reflection (_mug,laptop_) and rotational (_can,bowl,bottle_)
symmetry objects. We also supervise it by $\mathcal{L}_{1}$-loss as follows,

$\mathcal{L}_{sym}=\left\|P^{{}^{\prime}}-\varepsilon(P,r,t,r_{gt},t_{gt})\right\|_{1}$
(13)

where $P,P^{{}^{\prime}}$ denotes input point cloud and corresponding point
cloud, respectively. $\varepsilon(*)$ depends on the symmetry type. So far,
the overall loss function can be states as follows,

$\mathcal{L}_{all}=\mathcal{L}_{mm}^{NCE}+\mathcal{L}_{(img,text)}^{CE}+\mathcal{L}_{pose}+\mathcal{L}_{sym}$
(14)

Figure 4: Qualitative results of the IST-Net (red line) and our method (green
line). The ground truth results are shown with white lines. The detail
comparison areas are annotated with yellow lines.

##  IV Experiments

Datasets. We evaluate CLIPose on two mainstream benchmarks[14] for category-
level object pose estimation. CAMERA25 is a synthetic dataset that provides
300K synthetic RGB-D images of objects rendered on virtual background, with
25K images are withhold for testing. It contains 6 categories of objects,
_i.e._ _bottle, bowl, camera, can, laptop_ and _mug_. REAL275 is a more
challenge real-world dataset that contains the same categories as CAMERA25.
The training data consists of 4.3K images from 7 scenes, while testing data
includes 2.75K from 6 scenes and 3 objects from each category.

Evaluation Metrics. We report the mean average precision (mAP) in $n$° and $m$
cm to evaluate the accuracy of object pose estimation following NOCS [14]. The
$n$ and $m$ indicate the prediction error of rotation and translation,
respectively. We leverage the combination of rotation and translation metrics
of 5°2 _cm_ , 5°5 _cm_ , 10°2 _cm_ , 10°5 _cm_ and 10°10 _cm_ , which means
the estimation is considered correct when the error is below a threshold.
Similar to NOCS, for symmetry objects (_bottles, bowls, and cans_), we ignore
the rotation error around the symmetry axis.

TABLE II: Comparisons with state-of-the-art methods on CAMERA25 dataset. Overall best results are in bold and the second best results are underlined. Prior refers to whether the method necessitates prior category information. ‘-’ indicates no results reported under this metric in original paper. ($\uparrow$) represents a higher value indicating better performance. Methods | Prior |  5°2 _cm_ $\uparrow$ |  5°5 _cm_ $\uparrow$ |  10°2 _cm_ $\uparrow$ |  10°5 _cm_ $\uparrow$
---|---|---|---|---|---
NOCS[14] | $\times$ | 32.3 | 40.9 | 48.2 | 64.6
DualPoseNet[52] | $\times$ | 29.3 | 35.9 | 50.0 | 66.8
GPV-Pose[35] | $\times$ | 72.1 | 79.1 | - | 89.0
IST-Net[18] | $\times$ | 71.3 | 79.9 | 79.4 |  89.9
HS-Pose[35] | $\times$ | 73.3 |  80.5 | 80.4 | 89.4
SPD[15] | ✓ | 54.3 | 59.0 | 73.3 | 81.5
SAR-Net[55] | ✓ | 66.7 | 70.9 | 75.3 | 80.3
SGPA[34] | ✓ | 70.7 | 74.5 | 82.7 | 88.4
GPT-COPE[19] | ✓ | 70.4 | 76.5 | 81.3 | 88.7
RBP-Pose[56] | ✓ |  73.5 | 79.6 |  82.1 | 89.5
Ours | $\times$ | 74.8 | 82.2 | 82.0 | 91.2

Implementation Details. We follow [15, 17] to generate segmentation masks with
the off-the-shelf object detector, Mask-RCNN[46] and resize them to $224\times
224$. We uniformly sample 1024 points to form the object point clouds and feed
them as input to 3D encoder, whose backbone is optional according to the needs
of the task. We select the modified 3D-GCN in [35] as the encoder. The output
of the 3D encoder is a concatenation of the outputs of 3D-GCN layers, whose
shape is $[1024,1280]$. The point cloud features used for contrastive learning
are the output of final 3D-GCN layer after projection and pooling, whose shape
is $[1,512]$. We fix the temperature parameter $\tau$ and set it as 0.07
similar to [23]. We adopt several commonly used data augmentation in category-
level object pose estimation tasks, including random uniform noise, random
rotational and translational perturbations, and bounding box-based adjustment
similar to [16, 17]. For a fair comparison, the loss terms and their
parameters related to point reconstruction and pose estimation are kept the
same as [17]. We employ the Ranger optimizer and all experiments are conducted
on a single RTX3090 GPU with batch size of 24 and base learning rate of 1e-4,
and the learning rate is annealed at 72% of the training phase using a cosine
schedule.

###  IV-A Comparison with State-of-the-Art Methods

Performance on REAL275 dataset. In table I, we present the results of our
CLIPose with state-of-the-art methods on REAL275 [14] dataset, highlighting a
significant advancement in performance. We deal with all categories with a
single model.

As shown in table I, our method outperforms the state-of-the-art performance
in all metrics. As for the current state-of-the-art method, HS-Pose [35], we
still perform significant improvements in most of the metrics, e.g. , we
achieve 48.7% on 5°2 _cm_ , 58.3% on 5°5 _cm_ and 70.4% on 10°2 _cm_ metrics,
which outperform HS-Pose [35] by 3.4%, 3.4% and 1.8%. Noteworthy, we achieve
results on the REAL275 dataset, surpassing the impressive thresholds of 85%
and 86% in the 10°5 _cm_ and 10°10 _cm_ metrics, respectively. These
exceptional outcomes further support the efficacy of our approach.

Our approach solely relies on depth data as input and does not require a prior
models. Even when compared with the methods that take RGB-D data and category
prior as input, our method is still superior on all metrics. Especially, we
achieve 85.2% on 10°5 _cm_ and 86.2% on 10°10 _cm_ , which surpass the state-
of-the-art method, IST-Net [18], with a large gap by 5.6% and 4.6%. In
addition, we present a qualitative comparison between CLIPose and IST-Net
[18], as shown in fig. 4. Our method achieves better size and pose estimation
(e.g. the laptops in the first three columns), and handles complex and small
shapes better (e.g. the cameras and mugs in the last two columns).

TABLE III: Ablation studies on REAL275 dataset. Overall best results are in bold and the second best results are underlined. ($\uparrow$) represents a higher value indicating better performance. The default length of prompt tokens is 10. $D_{0}$ indicates the full version of the network.  ID | Contrastive Loss (PC) | Fine-Tune CLIP | Classification Learning |  mAP$\uparrow$
---|---|---|---|---
$\mathcal{L}_{(PC,img)}^{NCE}$ | $\mathcal{L}_{(PC,text)}^{NCE}$ |  | Text
---
with Pose
| Prompt
---
Tokens
$\mathcal{L}_{(img,text)}^{CE}$ | $\mathcal{L}_{(img,text)}^{NCE}$ | one-hot |  5°2 _cm_ |  5°5 _cm_ |  10°2 _cm_ |  10°5 _cm_ |  10°10 _cm_
$A_{0}$ |  |  |  |  |  |  |  | 43.8 | 54.5 | 65.4 | 81.8 | 82.6
$B_{0}$ | ✓ |  |  |  |  |  |  | 45.3 | 54.6 | 67.7 | 83.2 | 83.4
$B_{1}$ |  | ✓ |  |  |  |  |  | 42.4 | 53.2 | 65.5 | 82.2 | 84.0
$B_{2}$ | ✓ | ✓ |  |  |  |  |  | 45.9 | 55.5 | 68.3 | 83.5 | 84.4
$C_{0}$ | ✓ | ✓ | ✓ |  |  |  |  | 46.2 | 56.8 | 69.4 | 83.3 | 84.5
$C_{1}$ | ✓ | ✓ |  | ✓ |  |  |  | 47.5 | 57.1 | 69.6 |  84.5 |  85.9
$C_{2}$ | ✓ | ✓ | ✓ | ✓ |  |  |  |  48.5 |  57.5 |  70.3 | 84.2 | 85.6
$D_{0}$ | ✓ | ✓ | ✓ | ✓ | ✓ |  |  | 48.7 | 58.2 | 70.4 | 85.2 | 86.2
$D_{1}$ | ✓ | ✓ | ✓ | ✓ |  | ✓ |  | 44.1 | 52.5 | 66.3 | 80.8 | 84.5
$D_{2}$ | ✓ | ✓ | ✓ | ✓ |  |  | ✓ | 47.3 | 57.1 | 69.8 | 84.0 | 84.8
TABLE IV: Ablation studies on prompt length and the insertion of prompt tokens. Overall best results are in bold and the second best results are underlined. ($\uparrow$) represents a higher value indicating better performance. Other network settings remain default.  ID |  Length of Tokens |  Location of Insertion |  mAP$\uparrow$
---|---|---|---
5°2 _cm_ |  5°5 _cm_ |  10°2 _cm_ |  10°5 _cm_ |  10°10 _cm_
0 | 10 | Prepend | 48.7 |  58.2 | 70.4 | 85.2 | 86.2
1 | Append | 46.2 | 55.8 | 69.2 | 83.5 | 84.9
2 | 5 | Prepend |  48.3 | 58.8 | 68.9 |  84.4 | 85.1
3 | 20 | 47.3 | 56.1 | 69.9 | 83.7 | 84.9
4 | 50 | 47.8 | 57.0 |  70.2 | 84.1 |  85.3
5 | 100 | 44.3 | 55.5 | 64.1 | 83.0 | 84.5
6 | 200 | 40.2 | 53.1 | 61.8 | 82.7 | 84.6
$10^{1}$$10^{2}$4143454749$5$$10$$20$$50$$100$$200$Length of TokensmAP@5°2
_cm_ Performance of different lengths in 5°2cm Figure 5: Visualization of
ablation studies on prompt length. The model exhibits improved performance
when the length is kept at 50 or below. As the length increases, the outcomes
tend to decline.

Performance on CAMERA25 dataset. The results comparison of our method and the
state-of-the-art is presented in table II. From the observation of results, it
can be deduced that CLIPose demonstrates superior performance across all
metrics except for 10°2 _cm_ , where our method also achieves highly
competitive results, closely approaching the best performance. As for the
comparison with prior-free methods, our method achieves a remarkable
improvement over the current state-of-the-art method, HS-Pose [35], exceeding
it by nearly 2% in both the rigorous 5°2 _cm_ and 5°5 _cm_ evaluation metrics.
When compared to methods that take category prior as input, our method still
maintains a notable advantage without relying on any additional information.
This indicates that our method has a strong ability to comprehensively
estimate rotation, translation, and size. It is worth noting that CAMERA25 is
a synthetic dataset that contains many other extraneous objects, our method
can still achieve the state-of-the-art performance. This superior performance
on synthetic dataset further proves the effectiveness of our method.

###  IV-B Ablation Study

To validate the proposed contributions, we meticulously conduct ablation
studies on the REAL275[14] dataset. We divide the contribution terms into
three aspects and study their influences through different combinations. The
full ablation study results are shown in table III. $A_{0}$ indicates the
baseline method that only utilize point reconstruction and pose estimation
losses.

1) Effect of Contrastive Loss. The contrastive learning architecture is the
key ingredient of our method, which includes loss functions between the point
cloud and the image $\mathcal{L}_{(PC,img)}^{NCE}$ as well as the point cloud
and the text $\mathcal{L}_{(PC,text)}^{NCE}$. To demonstrate the effectiveness
of abundant semantics captured in image and text modalities, we train our
network utilizing the single contrastive learning loss function, which in
practice can be thought as aligning the point cloud and image or text features
individually. The results in table III, specifically the [$B_{0}$] and
[$B_{1}$] rows, indicate that aligning the representations of two modalities,
either point cloud and image or point cloud and text, positively influences
the baseline method. By observing the results, we found that aligning point
cloud and image can improve the performance in 5°2 _cm_ to 10°10 _cm_ metrics.
As shown in [$B_{2}$] row, when we incorporate the two losses, _i.e._ ,
perform a complete multi-modal contrastive learning, the performance in all
metrics would be improved.

TABLE V: Ablation studies on the involved transformer layers of image encoder and joint training with text encoder. Overall best results are in bold and the second best results are underlined. ($\uparrow$) represents a higher value indicating better performance. Other network settings remain default. ID |  Transformer Layers |  Joint Training |  mAP$\uparrow$
---|---|---|---
5°2 _cm_ |  5°5 _cm_ |  10°2 _cm_ |  10°5 _cm_ |  10°10 _cm_
0 | 1 $\to$ 1 | - | 48.7 | 58.2 | 70.4 | 85.2 | 86.2
1 | ✓ |  47.0 | 56.6 | 68.6 | 83.4 | 84.5
2 |  1 $\to$ 3 | - | 45.4 | 53.5 | 68.4 | 82.7 | 84.0
3 |  1 $\to$ 6 | - | 46.9 | 56.8 | 69.3 |  84.7 |  85.9
4 |  1 $\to$ 12 | - | 45.9 | 54.7 |  69.9 | 83.6 | 85.0
5 |  6 $\to$ 12 | - |  47.0 |  57.3 | 69.5 | 83.6 | 84.7
6 |  9 $\to$ 12 | - | 45.7 | 55.2 | 67.9 | 82.7 | 84.1
7 |  12 $\to$ 12 | - | 46.2 | 56.7 | 67.8 | 83.0 | 84.1

We can achieve comparable results to HS-Pose[35] without fine-tuning the CLIP
model, which proves the effectiveness of contrastive learning.

2) Effect of Fine-Tune Manner. The fine-tuning of the CLIP model includes both
text and image portions. The category loss of network, the text description
with pose information and the additional prompt tokens are employed to fine-
tune the CLIP model to make it more suitable for the pose estimation task. To
verify the benefit of two contributions, we incorporate each of them on top of
the complete contrastive learning, as shown in [$C_{0}$] and [$C_{1}$] rows.
It is clear that two combinations all boost the performance. Specifically, the
modified text description improves the performance on some metrics, e.g. 5°2
_cm_ , 5°5 _cm_ and 10°2 _cm_. Furthermore, the prompt tokens for image
encoder achieve better results on all metrics. The performance of this version
([$C_{1}$]) in 10°5 _cm_ (84.5%) and 10°10 _cm_ (85.9%), which exhibits that
the additional prompt tokens have a significant improvement in network
performance. Eventually, we combine two manners together for evaluation, as
shown in [$C_{2}$]. This version achieves remarkable results, closely
following the performance of the best version [$E_{0}$].

The length of prompt tokens is also a factor influencing the results. To
achieve better fine-tuning results, we perform further ablation studies on
prompt tokens. We specifically investigate several aspects, including the
length of prompt tokens, the location of token insertion, the number of
transformer layers involved, and the joint fine-tuning training with the text
encoder. All ablation results are presented in table IV and table V. Note that
our default settings are as follows: the length of prompt tokens is 10,
insertion of tokens at the beginning (prepend), and involvement of the first
transformer layer (total is 12).

From table IV, we evaluate the performance _w.r.t._ different prompt lengths,
the length of the tokens is set to 5, 10, 20, 50, 100, and 200 and compare
their performance. As can be observed, the default setting (length is 10)
achieves the best result. As shown in ID-2 row, competitive results are also
obtained when the length is set to 5. ID-3 and ID-4 rows exhibit increasing
the length of tokens within certain limits can also maintain the performance.
The rows ID-5 and ID-6 demonstrate that when the length of the token is close
or equivalent to the default length of the image embedding (196), it results
in a performance decrease. The fig. 5 intuitively illustrates the impact of
prompt length on performance. Additionally, we keep a default length of tokens
and compare the performance between insertion at the front (prepend) and end
(append) of the image embedding, as shown in rows ID-0 and ID-1. The overall
performance shows that the difference between these two ways is not
significant, with token insertion in the prepend position slightly
outperforming the append position.

In table V, we further evaluate the performance when changing the transformer
layers of image encoder, e.g. $1\to 3$ means fine-tuning in the first three
layers, and $9\to 12$ denotes fine-tuning in the last three layers. In the
Transformer encoder, the layers closer to the input primarily focus on
learning the local structures of the input embedding, while the layers towards
the end tend to specialize in capturing global features and high-level
abstractions of the embedding. The default setting, _i.e._ inserting tokens in
the first layer still surpasses other settings. The overall performance
exhibits an interesting result, with the performance being better when the
number of layers added is 6 compared to the cases with 3 and 12 layers, while
adding tokens in the later layers achieves superior results. We argue that
introducing prompt tokens in both the first and later transformer layers does
not significantly impact the extraction of image features while allowing for
effective fine-tuning of the image encoder. Moreover, we validated the joint
training of image and text encoders, assessing their performance when fine-
tuned simultaneously. As can be observed in rows ID-0 and ID-1, the results of
joint training exhibits a slight decline. This further demonstrates the
speculation in section III-C that the incorporation of pose parameters and
text description has already provided the pose-sensitive information, and does
not require fine-tuning of text encoder.

3) Ablate on Classification Learning. In section III-B, we have discussed that
we construct the multi-categorical Contrastive Estimation (CE) loss between
image and text modalities to introduce object classification information and
further enhance the model’s learning of category features. To validate the
assumptions we presented in section III-B, we compare the results of applying
different losses between image and text modalities. As shown in table III
[$D_{0}$] rows, the CE loss can effectively assist the network to achieve the
best performance. This version donates that we are leveraging CLIP’s inherent
categorization ability, and we treat [$D_{0}$] as the final version of our
method. On the contrary, applying NCE loss ([$D_{1}$]) means aligning the
image and text representations that are already aligned in the CLIP model. The
results of [$D_{1}$] has a large gap compared to [$D_{0}$], with some metrics
even inferior to the baseline ([$A_{0}$]). We speculate that the text and
image re-alignment does not directly facilitate the feature learning of the 3D
encoder. Additionally, since we have fine-tuned the CLIP’s image encoder, the
re-alignment between text and image may even impact the performance of the
CLIP model itself.

TABLE VI: Ablation studies on feature fusion. ID-0 denotes baseline method and is colored in gray. [Major] term of fusion manner represents the core concepts of fusion and [Specific] term indicates the manner of detailed. GP denotes global pooling and PC represents Point Cloud. Other network settings remain default. (↓) indicates the difference that has decreased compared to the baseline ID-0. ($\uparrow$) represents a higher value indicating better performance.  ID | Fusion Manner | Feature for Fusion |  mAP$\uparrow$
---|---|---|---
Major | Specific |  | Point Cloud
---
& Image
| Point Cloud
---
& Text
5°2 _cm_ |  5°5 _cm_ |  10°2 _cm_ |  10°5 _cm_ |  10°10 _cm_
0 | - | - | - | - | 43.8 | 54.5 | 65.4 | 81.8 | 82.6
1 | Concat | Before GP | ✓ | - |  41.4↓2.4 |  51.7↓2.8 |  61.7↓3.7 |  79.2↓2.6 |  80.1↓2.5
2 | After GP | - | ✓ |  42.2↓1.6 |  51.9↓2.6 |  63.9↓1.5 |  80.4↓1.4 |  81.2↓1.4
3 |  Cross- Attention | PC as Q | ✓ | - |  0.1↓43.7 |  0.4↓54.1 |  2.3↓63.1 |  5.1↓76.7 |  5.3↓77.3
4 | - | ✓ |  0.8↓43.0 |  1.1↓53.4 |  5.7↓59.7 |  8.1↓73.7 |  8.2↓74.4
5 | PC as KV | ✓ | - |  5.1↓38.7 |  14.6↓39.9 |  18.5↓46.9 |  46.6↓35.2 |  47.6↓35.0
6 | - | ✓ |  10.2↓33.6 |  13.6↓40.9 |  29.0↓36.4 |  44.4↓37.4 |  46.6↓36.0
7 |  Transformer Encoder | Plus(regular) | ✓ | - |  38.3↓5.5 |  47.3↓7.2 |  60.2↓5.2 |  73.4↓8.4 |  75.4↓7.2
8 | Concat | - | ✓ |  24.4↓19.4 |  36.6↓17.9 |  45.1↓20.3 |  67.1↓14.7 |  68.7↓13.9
TABLE VII: Comparisons of using different 3D Encoder. CLIPose adopts the 3D Encoders from GPV-Pose [17] or HS-Pose [35] and performs a comparison with these two methods. ($\uparrow$) represents a higher value indicating better performance. Methods |  5°2 _cm_ $\uparrow$ |  5°5 _cm_ $\uparrow$ |  10°2 _cm_ $\uparrow$ |  10°5 _cm_ $\uparrow$
---|---|---|---|---
GPV-Pose[17] | 32.5 | 43.3 | 58.2 | 76.6
Ours(3D-GCN) | 42.9 | 54.4 | 63.7 | 81.5
HS-Pose[35] | 45.3 | 54.9 | 68.6 | 83.6
Ours(HS-Layer) | 48.7 | 58.3 | 70.4 | 85.2

Actually, there are several intuitive ways to incorporate object
classification information, such as using categorical one-hot vectors [17,
35]. We further perform ablation studies to investigate whether our method
could outperforms explicit one-hot vectors modeling of object categories, as
shown in [$D_{2}$]. It can be clearly observed that the performance of the
model is not significantly different from [$C_{2}$] after the introduction of
categorical one-hot vectors, with a slight decrease in the results for some
metrics. We suppose that during the multimodal learning process, the category
information carried by the one-hot vectors may mix with the spatial
information of the point cloud, preventing the network from effectively
learning valuable classification information.

4) Ablate on 3D Encoder. CLIPose can leverage any 3D encoder to extract point
cloud features. The default setting of CLIPose’s 3D encoder is the HS-Layer
from HS-Pose [35]. We also adopt the 3D-GCN [57] similar to GPV-Pose [17] to
evaluate the performance. The overall results are shown in table VII, as can
be observed that different 3D encoders can give varying performance outcomes.
What’s more, under the condition of identical 3D encoders, our method
demonstrates superior performance compared to the original methods.

Specifically, our method surpasses GPV-Pose [17] by more than 10% in rigorous
5°2 _cm_ and 5°5 _cm_ metrics. Moreover, it exceeds HS-Pose [35] by 3.4% and
3.4% in the same two metrics, which further proves the effectiveness of our
method.

TABLE VIII: Ablation studies on different temperature parameters $\tau$. Overall best results are in bold and the second best results are underlined. ($\uparrow$) represents a higher value indicating better performance. ID | $\tau$ | logit_scale |  5°2 _cm_ $\uparrow$ |  5°5 _cm_ $\uparrow$ |  10°2 _cm_ $\uparrow$ |  10°5 _cm_ $\uparrow$
---|---|---|---|---|---|---
0 | 0.2 | 5.0 | 48.8 | 58.9 |  69.4 |  83.8
1 | 0.07 | 14.3 |  48.7 |  58.3 | 70.4 | 85.2
2 | 0.03 | 33.3 | 46.3 | 55.9 | 68.2 | 83.6
3 | 0.02 | 50.0 | 46.8 | 56.5 | 67.2 | 81.8
4 | 0.001 | 100.0 | 45.7 | 54.6 | 67.4 | 81.4

5) Ablate on Temperature Parameter. The temperature $\tau$ is a vital hyper-
parameter in contrastive learning, as it influences model’s discriminative
ability towards negative samples. According to eq. 3, temperature parameter
$\tau$ will transform into the form of coefficient logit_scale as follows,

$logit\\_scale=log\ e^{\left(1/\tau\right)}=1/\tau$ (15)

The performance comparison of the different temperature parameters is shown in
table VIII. As can be observed, if $\tau$ is set too small, the data
distribution will become more concentrated, leading to the decline in model’s
generalization performance. Conversely, if we gradually increase $\tau$, the
distribution becomes smoother, which is more conducive to the model’s learning
process.

6) Ablate on Feature Fusion. We ultimately utilize point cloud features for
pose estimation. In practice, we also attempt to fuse image or text features
into point cloud features and then use the fused features for training.
Intuitively, the fused features contains more information and can achieve
better performance. We separately try several fusion manners among multi
modalities, e.g. direct concatenation, single cross-attention layer and
modified transformer encoder, to fuse the different features. However, the
performance of the network is not improved, but rather drops compared with the
baseline method.

TABLE IX: Ablation studies on the format of classification learning and pose information in textual descriptions. ID-0 denotes full version model the same as [$D_{0}$] in table III and is colored in gray. Other network settings remain default. ($\uparrow$) represents a higher value indicating better performance.  ID | Pose Information |  mAP$\uparrow$
---|---|---
_w_ | _w/o_ | _rand_ |  5°2 _cm_ |  5°5 _cm_ |  10°2 _cm_ |  10°5 _cm_
0 | ✓ | - | - | 48.7 | 58.2 | 70.4 | 85.2
1 | - | ✓ | - | 47.3 | 56.8 | 69.8 | 85.2
2 | - | - | ✓ | 47.7 | 57.0 | 68.3 | 83.3

In table VI, for the direct concatenation, we attempt two fusion ways
according to the steps of fusion and global pooling. If we fuse features
before global pooling, we replicate the pooled image/text features to the
number of sampled point clouds, then concatenate the image/text features after
point cloud features. If we fuse features after global pooling, we project the
pooled different features to the same dimension and then fuse to estimate the
pose. From the results, we can observe that a slight decrease in performance.
As for fusion using cross-attention layer, whose principle formula is as
follows,

$Attention\left(Q,K,V\right)=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$
(16)

where $d_{k}$ denotes the dimension of $K$. Typically, one modal features are
used as $Q$ and the other as $K$ and $V$. We can clearly see that $V$ is the
major part of fused features, and the $softmax$ function is use to calculate
the weights. We take the point cloud features as $Q$ and $KV$ respectively and
the results is presented in ID-3 to ID-6. The experiments reveal a significant
drop in performance when using the point cloud features as $Q$. When using
point cloud features as $KV$, _i.e._ as the main body of the fused features,
the results are slightly better than the former but still large lag behind the
baseline methods.

Finally, we also try to leverage the Transformer [37] encoder since the output
features of attention layer are skip connected with $Q$. Thus, $Q$ becomes the
major part of fused features. Hence, we introduce the cross-attention layer
into the Transformer encoder to replace the original self-attention layer and
fuse the features using the modified Transformer encoder, where we take the
point cloud features as $Q$. The results is shown in ID-7 and ID-8 of table
VI. Note that we make a further distinguish in modified Transformer encoder,
where Plus refers to regular skip connection. Concat indicates concatenation,
the fused features will be projected back to the original dimension. From the
results, we observe that the performance is better than using a single cross-
attention layer but slightly behind the direct concatenation. However, all
feature fusion manners result in a relative decrease in performance compared
to the baseline ID-0. Therefore, we directly utilize point cloud features for
recovering pose parameters.

7) Ablate on Pose Information in Textual Descriptions. Considering that we
introduce rotation and translation values in the textual descriptions to make
CLIP more adaptable to object’s pose information. It is necessary to validate
whether the CLIP text encoder can learn meaningful information from these
textual descriptions. To verify this, we employed three different settings:
one with correct pose information denoted as _w_ , one without pose
information, only containing object category descriptions denoted as _w/o_ and
one with random pose information denoted as _rand_. As shown in table IX,
introducing correct pose information (ID-0) proves to be effective in
enhancing prediction accuracy, especially on some rigorous metrics (e.g. 1.4%
on 5°5 _cm_). On the contrary, when random pose information (ID-2) is
introduced, the overall performance declines. In most metrics, the results are
even inferior compared to the scenario without pose information (ID-1). This
further indicates that guidance from the textual modality can effectively
assist in point cloud learning.

##  V Conclusion

In this paper, we propose a novel and effective category-level pose estimation
network, named CLIPose. CLIPose employs the pre-trained vision-language model
for the category-level pose estimation task. To fully leverage the abundant
semantic knowledge already captured in image and text modalities, CLIPose
aligns features from point clouds and two other modalities by means of
contrastive learning. The alignment operation allows the network learn more
robust and representative category-specific information. Furthermore, CLIPose
fine-tunes the pre-trained CLIP model by incorporating the prompt tokens and
pose information, which makes CLIP more suitable to the pose estimation,
resulting in better estimation performance. Extensive experiments on the
challenge benchmarks show the effectiveness of our method in both efficiency
and accuracy.

In the future, we consider exploring two aspects. First, building upon the
point cloud data, ewe aim to investigate leveraging RGB data to acquire
additional pose information, enabling the correction of poses predicted using
point cloud data. Second, the impressive capability of the CLIP pre-trained
model in zero-shot estimation tasks is impressive. We expect to achieve
promising zero-shot estimation performance in pose estimation tasks as well.
We intend to create large-scale point cloud datasets using generative models
and further explore the potential of multi-modal contrastive learning in the
context of 3D modality.

## References

  * [1] E. Marchand, H. Uchiyama, and F. Spindler, “Pose estimation for augmented reality: a hands-on survey,” _IEEE transactions on visualization and computer graphics_ , vol. 22, no. 12, pp. 2633–2651, 2015.
  * [2] X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, “Multi-view 3d object detection network for autonomous driving,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2017, pp. 1907–1915.
  * [3] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” _arXiv preprint arXiv:1809.10790_ , 2018.
  * [4] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese, “Densefusion: 6d object pose estimation by iterative dense fusion,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3343–3352.
  * [5] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao, “Pvnet: Pixel-wise voting network for 6dof pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4561–4570.
  * [6] G. Gao, M. Lauri, X. Hu, J. Zhang, and S. Frintrop, “Cloudaae: Learning 6d object pose regression with on-line data synthesis on point clouds,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 11 081–11 087.
  * [7] G. Zhou, D. Wang, Y. Yan, H. Chen, and Q. Chen, “Semi-supervised 6d object pose estimation without using real annotations,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 8, pp. 5163–5174, 2021\.
  * [8] Y. He, H. Huang, H. Fan, Q. Chen, and J. Sun, “Ffb6d: A full flow bidirectional fusion network for 6d pose estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 3003–3013.
  * [9] X. Lin, D. Wang, G. Zhou, C. Liu, and Q. Chen, “Transpose: 6d object pose estimation with geometry-aware transformer,” _arXiv preprint arXiv:2310.16279_ , 2023.
  * [10] T. Cao, W. Zhang, Y. Fu, S. Zheng, F. Luo, and C. Xiao, “Dgecn++: A depth-guided edge convolutional network for end-to-end 6d pose estimation via attention mechanism,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [11] S. Hinterstoisser, C. Cagniart, S. Ilic, P. Sturm, N. Navab, P. Fua, and V. Lepetit, “Gradient response maps for real-time detection of textureless objects,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 34, no. 5, pp. 876–888, 2011.
  * [12] E. Brachmann, F. Michel, A. Krull, M. Y. Yang, S. Gumhold _et al._ , “Uncertainty-driven 6d pose estimation of objects and scenes from a single rgb image,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3364–3372.
  * [13] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” _arXiv preprint arXiv:1711.00199_ , 2017.
  * [14] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas, “Normalized object coordinate space for category-level 6d object pose and size estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 2642–2651.
  * [15] M. Tian, M. H. Ang, and G. H. Lee, “Shape prior deformation for categorical 6d object pose and size estimation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_. Springer, 2020, pp. 530–546.
  * [16] W. Chen, X. Jia, H. J. Chang, J. Duan, L. Shen, and A. Leonardis, “Fs-net: Fast shape-based network for category-level 6d object pose estimation with decoupled rotation mechanism,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 1581–1590.
  * [17] Y. Di, R. Zhang, Z. Lou, F. Manhardt, X. Ji, N. Navab, and F. Tombari, “Gpv-pose: Category-level object pose estimation via geometry-guided point-wise voting,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 6781–6791.
  * [18] J. Liu, Y. Chen, X. Ye, and X. Qi, “Prior-free category-level pose estimation with implicit space transformation,” _arXiv preprint arXiv:2303.13479_ , 2023\.
  * [19] L. Zou, Z. Huang, N. Gu, and G. Wang, “Gpt-cope: A graph-guided point transformer for category-level object pose estimation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [20] A. Goyal, H. Law, B. Liu, A. Newell, and J. Deng, “Revisiting point cloud shape classification with a simple and effective baseline,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 3809–3820.
  * [21] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, “3d shapenets: A deep representation for volumetric shapes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1912–1920.
  * [22] X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, and J. Lu, “Point-bert: Pre-training 3d point cloud transformers with masked point modeling,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 19 313–19 322.
  * [23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [24] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” _Advances in neural information processing systems_ , vol. 34, pp. 9694–9705, 2021.
  * [25] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu, “Coca: Contrastive captioners are image-text foundation models,” _arXiv preprint arXiv:2205.01917_ , 2022.
  * [26] R. Dang, J. Feng, H. Zhang, C. Ge, L. Song, L. Gong, C. Liu, Q. Chen, F. Zhu, R. Zhao _et al._ , “Instructdet: Diversifying referring object detection with generalized instructions,” _arXiv preprint arXiv:2310.05136_ , 2023\.
  * [27] L. Qiu, R. Zhang, Z. Guo, Z. Zeng, Y. Li, and G. Zhang, “Vt-clip: Enhancing vision-language models with visual-guided texts,” _arXiv preprint arXiv:2112.02399_ , 2021.
  * [28] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang _et al._ , “Grounded language-image pre-training,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 10 965–10 975.
  * [29] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl, “Language-driven semantic segmentation,” _arXiv preprint arXiv:2201.03546_ , 2022.
  * [30] R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li, “Pointclip: Point cloud understanding by clip,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 8552–8562.
  * [31] L. Xue, M. Gao, C. Xing, R. Martín-Martín, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1179–1189.
  * [32] Z. Fan, Z. Song, J. Xu, Z. Wang, K. Wu, H. Liu, and J. He, “Acr-pose: Adversarial canonical representation reconstruction network for category level 6d object pose estimation,” _arXiv preprint arXiv:2111.10524_ , 2021\.
  * [33] J. Lin, Z. Wei, C. Ding, and K. Jia, “Category-level 6d object pose and size estimation using self-supervised deep prior deformation networks,” in _European Conference on Computer Vision_. Springer, 2022, pp. 19–34.
  * [34] K. Chen and Q. Dou, “Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2773–2782.
  * [35] L. Zheng, C. Wang, Y. Sun, E. Dasgupta, H. Chen, A. Leonardis, W. Zhang, and H. J. Chang, “Hs-pose: Hybrid scope feature extraction for category-level object pose estimation,” _arXiv preprint arXiv:2303.15743_ , 2023.
  * [36] S. Umeyama, “Least-squares estimation of transformation parameters between two point patterns,” _IEEE Transactions on Pattern Analysis & Machine Intelligence_, vol. 13, no. 04, pp. 376–380, 1991.
  * [37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [38] D. Chen, J. Li, Z. Wang, and K. Xu, “Learning canonical shape space for category-level 6d object pose and size estimation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 973–11 982.
  * [39] J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang, “Groupvit: Semantic segmentation emerges from text supervision,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 134–18 144.
  * [40] L. Wang, Z. He, R. Dang, H. Chen, C. Liu, and Q. Chen, “Res-sts: Referring expression speaker via self-training with scorer for goal-oriented vision-language navigation,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [41] Z. He, L. Wang, R. Dang, S. Li, Q. Yan, C. Liu, and Q. Chen, “Learning depth representation from rgb-d videos by time-aware contrastive pre-training,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [42] M. Zhu, X. Lin, R. Dang, C. Liu, and Q. Chen, “Fine-grained spatiotemporal motion alignment for contrastive video representation learning,” _arXiv preprint arXiv:2309.00297_ , 2023.
  * [43] X. Zhu, R. Zhang, B. He, Z. Zeng, S. Zhang, and P. Gao, “Pointclip v2: Adapting clip for powerful 3d open-world learning,” _arXiv preprint arXiv:2211.11682_ , 2022.
  * [44] R. Zhang, Z. Zeng, Z. Guo, and Y. Li, “Can language understand depth?” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 6868–6874.
  * [45] L. Xue, N. Yu, S. Zhang, J. Li, R. Martín-Martín, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, “Ulip-2: Towards scalable multimodal pre-training for 3d understanding,” _arXiv preprint arXiv:2305.08275_ , 2023\.
  * [46] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 2961–2969.
  * [47] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [48] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9729–9738.
  * [49] C. Dyer, “Notes on noise contrastive estimation and negative sampling,” _arXiv preprint arXiv:1410.8251_ , 2014.
  * [50] M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, “Visual prompt tuning,” in _European Conference on Computer Vision_. Springer, 2022, pp. 709–727.
  * [51] O. Rodrigues, “Des lois géométriques qui régissent les déplacements d’un système solide dans l’espace, et de la variation des coordonnées provenant de ces déplacements considérés indépendamment des causes qui peuvent les produire,” _Journal de mathématiques pures et appliquées_ , vol. 5, pp. 380–440, 1840.
  * [52] J. Lin, Z. Wei, Z. Li, S. Xu, K. Jia, and Y. Li, “Dualposenet: Category-level 6d object pose and size estimation using dual pose network with refined learning of pose consistency,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 3560–3569.
  * [53] R. Zhang, Y. Di, F. Manhardt, F. Tombari, and X. Ji, “Ssp-pose: Symmetry-aware shape prior deformation for direct category-level object pose estimation,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 7452–7459.
  * [54] L. Zhou, Z. Liu, R. Gan, H. Wang, and M. H. Ang Jr, “Dr-pose: A two-stage deformation-and-registration pipeline for category-level 6d object pose estimation,” _arXiv preprint arXiv:2309.01925_ , 2023.
  * [55] H. Lin, Z. Liu, C. Cheang, Y. Fu, G. Guo, and X. Xue, “Sar-net: Shape alignment and recovery network for category-level 6d object pose and size estimation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 6707–6717.
  * [56] R. Zhang, Y. Di, Z. Lou, F. Manhardt, F. Tombari, and X. Ji, “Rbp-pose: Residual bounding box projection for category-level pose estimation,” in _European Conference on Computer Vision_. Springer, 2022, pp. 655–672.
  * [57] Z.-H. Lin, S.-Y. Huang, and Y.-C. F. Wang, “Convolution in the cloud: Learning deformable kernels in 3d graph convolution networks for point cloud analysis,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 1800–1809.

|  Xiao Lin received his B.Eng. degree in Automation from Tongji University,
Shanghai, China, in 2021, where he is currently pursuing his direct Ph.D.
degree with the Robotics and Artificial Intelligence Laboratory. His research
interests are in visual perception for robotics, with a focus on 3D vision
detection and 6D object pose estimation.
---|---
|  Minghao Zhu received the B.Eng. degree from Tongji University in 2021 and
the B.Eng. degree from Polytechnic University of Turin in 2021. He is
currently pursuing the Ph.D. degree with Tongji University. His research
interests include computer vision, deep learning, with specific focus on
large-scale and multimodal video understanding.
---|---
|  Ronghao Dang received the B.Eng. degree from Tongji University in 2021. He
is currently pursuing the M.S. degree at the Department of Control Science and
Engineering, College of Electronics and Information Engineering, Tongji
University. His research interests include object navigation and Multimodal
large model.
---|---
|  Guangliang Zhou received his B.Eng. degree in Automation from Tongji
University, Shanghai, China, in 2017, where he is currently pursuing his Ph.D.
degree with the Robotics and Artificial Intelligence Laboratory. His research
interests are in visual perception for robotics, with a focus on 6D object
pose estimation and grasp detection.
---|---
|  Shaolong Shu (Senior Member, IEEE) received his B.Eng. degree in automatic
control, and his Ph.D. degree in control theory and control engineering from
Tongji University, Shanghai, China, in 2003 and 2008, respectively. Since
July, 2008, he has been with the School of Electronics and Information
Engineering, Tongji University, Shanghai, China, where he is currently a full
professor. From August, 2007 to February, 2008 and from April, 2014 to April,
2015, he was a visiting scholar in Wayne State University, Detroit, MI, USA.
His main research interests include state estimation and control of discrete
event systems and cyber–physical systems.
---|---
|  Feng Lin (Fellow, IEEE) received his B.Eng. degree in electrical
engineering from Shanghai Jiao-Tong University, Shanghai, China, in 1982, and
his M.A.Sc. and Ph.D. degrees in electrical engineering from the University of
Toronto, Toronto, Canada, in 1984 and 1988, respectively. From 1987 to 1988,
he was a postdoctoral fellow at Harvard University, Cambridge, MA. Since 1988,
he has been with the Department of Electrical and Computer Engineering, Wayne
State University, Detroit, Michigan, where he is currently a professor. His
research interests include discrete-event systems, hybrid systems, robust
control, and their applications in alternative energy, biomedical systems, and
automotive control. He is the author of a book entitled Robust Control Design:
An Optimal Control Approach. He was a consultant for GM, Ford, Hitachi and
other auto companies. He co-authored a paper that received a George Axelby
outstanding paper award from IEEE Control Systems Society. He was an associate
editor of IEEE Transactions on Automatic Control.
---|---
|  Chengju Liu received the Ph.D. degree in control theory and control
engineering from Tongji University, Shanghai, China, in 2011. From October
2011 to July 2012, she was with the BEACON Center, Michigan State University,
East Lansing, MI, USA, as a Research Associate. From March 2011 to June 2013,
she was a Postdoctoral Researcher with Tongji University, where she is
currently a Professor with the Department of Control Science and Engineering,
College of Electronics and Information Engineering, and a Chair Professor of
Tongji Artificial Intelligence (Suzhou) Research Institute. She is also a Team
Leader with the TJArk Robot Team, Tongji University. Her research interests
include intelligent control, motion control of legged robots, and evolutionary
computation.
---|---
|  Qijun Chen (Senior Member, IEEE) received the B.S. degree in automation
from Huazhong University of Science and Technology, Wuhan, China, in 1987, the
M.S. degree in information and control engineering from Xi’an Jiaotong
University, Xi’an, China, in 1990, and the Ph.D. degree in control theory and
control engineering from Tongji University, Shanghai, China, in 1999. He is
currently a Full Professor in the College of Electronics and Information
Engineering, Tongji University. His research interests include robotics
control, environmental perception, and understanding of mobile robots and
bioinspired control.
---|---