# Certifiable Object Pose Estimation:
Foundations, Learning Models, and Self-Training

Rajat Talak, Lisa Peng, and Luca Carlone The authors are with the Laboratory
of Information and Decision Systems (LIDS), Massachusetts Institute of
Technology, Cambridge, MA 02139, USA. Corresponding author: Rajat Talak
(email<EMAIL_ADDRESS>This work was partially funded by ARL DCIST CRA
W911NF-17-2-0181, ONR RAIDER N00014-18-1-2828, and NSF CAREER award
“Certifiable Perception for Autonomous Cyber-Physical Systems”.

###### Abstract

We consider a _certifiable_ object pose estimation problem, where —given a
partial point cloud of an object— the goal is to not only estimate the object
pose, but also to provide a certificate of correctness for the resulting
estimate. Our first contribution is a general theory of certification for end-
to-end perception models. In particular, we introduce the notion of _$\zeta$
-correctness_, which bounds the distance between an estimate and the ground
truth. We then show that $\zeta$-correctness can be assessed by implementing
two certificates: (i) a certificate of _observable correctness_ , that asserts
if the model output is consistent with the input data and prior information,
(ii) a certificate of _non-degeneracy_ , that asserts whether the input data
is sufficient to compute a unique estimate. Our second contribution is to
apply this theory and design a new learning-based certifiable pose estimator.
In particular, we propose C-3PO, a semantic-keypoint-based pose estimation
model, augmented with the two certificates, to solve the certifiable pose
estimation problem. C-3PO also includes a _keypoint corrector_ , implemented
as a differentiable optimization layer, that can correct large detection
errors (_e.g.,_ due to the sim-to-real gap). Our third contribution is a novel
self-supervised training approach that uses our certificate of observable
correctness to provide the supervisory signal to C-3PO during training. In it,
the model trains only on the observably correct input-output pairs produced in
each batch and at each iteration. As training progresses, we see that the
observably correct input-output pairs grow, eventually reaching near $100\%$
in many cases. We conduct extensive experiments to evaluate the performance of
the corrector, the certification, and the proposed self-supervised training
using the ShapeNet and YCB datasets. The experiments show that (i) standard
semantic-keypoint-based methods (which constitute the backbone of C-3PO)
outperform more recent alternatives in challenging problem instances, (ii)
C-3PO further improves performance and significantly outperforms all the
baselines, (iii) C-3PO’s certificates are able to discern correct pose
estimates. We release the implementation and an interactive visualization of
all the results presented in this paper at: https://github.com/MIT-SPARK/C-3PO
and https://github.com/MIT-SPARK/pose-baselines.

This paper has been accepted for publication in IEEE Trans. Robotics 2023.
Please cite the paper as: R. Talak, L. Peng, and L. Carlone, “Certifiable
Object Pose Estimation: Foundations, Learning Models, and Self-Training”,
_IEEE Trans. Robotics_ , 2023.

##  I Introduction

Object pose estimation is a crucial prerequisite for robots to understand and
interact with their surroundings. In this paper, we consider the problem of
object pose estimation using depth or partial point cloud data, which arises
when the robot is equipped with depth sensors (_e.g.,_ lidar, RBG-D cameras)
or has to postprocess a 3D point cloud (_e.g.,_ resulting from a 3D
reconstruction pipeline). Local-feature-based registration methods can be re-
purposed to solve the object pose estimation problem, see, _e.g.,_ [1]. Recent
work, however, argues that either a neural model that mimics the ICP algorithm
[2, 3], or approaches based on learned global representations [4, 5] yield
better performance. A known drawback of these methods is that they are not
expressive enough to yield high pose estimation accuracy when the object point
cloud suffers from occlusions. Object pose estimation from partial or depth
point clouds, therefore, remains an open problem [4, 3, 5].

When using a learning-based approach for pose estimation, another major
challenge is to guarantee —or at least assess— the trustworthiness of the
predictions of neural-network-based models. Robot perception requires models
that can be relied upon for the robot to act safely in the world. Perception
failures can lead to catastrophic consequences including loss of life [6].
Recent progress has been made towards certifying geometric optimization
problems [7, 1, 8], which form the back-end of common perception pipelines.
However, these approaches only produce certificates of optimality for the
back-end optimization, but might still fail without notice if the perception
front-end (_e.g.,_ the feature detector) produces largely incorrect results.
Here we argue that we need to go beyond certifiable optimization and need to
certify the _entire_ perception pipeline, _i.e.,_ we need to produce
certificates that can determine if the output produced by an end-to-end
(_e.g.,_ opaque box) perception pipeline is correct or not. To date, there is
no theory developed for this purpose for general perception tasks, let alone
for the case of object pose estimation. The current literature only offers
statistical guarantees for specific problems [9] or is restricted to data near
the training regime [10, 11, 12, 13, 14].

Finally, most of the recent progress on pose estimation has focused on
supervised learning and relies on large annotated datasets [15, 16, 17, 18,
19]. However, obtaining human-labeled data is not only costly, but also robot-
dependent: a change in placement of the sensor/camera (_e.g.,_ from the top of
a self-driving car to the head of a quadruped robot) is likely to distort the
predictions produced by the trained model. Though photo-realistic simulators
provide a partial answer to this problem [20, 21, 22], a sim-to-real gap is
likely going to remain and needs to be tackled. Here, we argue that a scalable
pose estimation method must rely on self-supervision rather than human-labeled
data. Very few works have tackled pose estimation in a self-supervised manner
[23, 24, 25, 26, 27, 28, 5].

###  I-A Contributions

We address the twin challenges of certifiability and self-supervised training
by noting that certification can provide a supervision signal for self-
training (_i.e.,_ if we can identify correct predictions, we can learn from
them). More specifically, we establish the _foundations_ of certifiable object
pose estimation using learning-based models, by defining a notion of
correctness and developing certificates that can assert correctness in
practice. We then propose a new _learning-based model_ , named C-3PO
(Certifiable 3D POse), for certifiable pose estimation from depth or point
cloud data. Finally, we show how to use the proposed certificates to enable
_self-supervised training_ of C-3PO. We unpack each contribution below.

(I) Theory of Certifiable Models and Certifiable Object Pose Estimation. We
start by defining the notion of _$\zeta$ -correctness_ that discerns whether
an estimate is correct (_i.e.,_ matches the ground truth object) or not
(Definition 1). Using this, we develop a general theory of certifying
learning-based models (Section III). We show that by implementing two
certificates, namely, a certificate of _observable correctness_ , and a
certificate of _non-degeneracy_ , we can infer $\zeta$-correctness (Theorem
4). The observable correctness certificate determines whether an output
produced by a model is consistent with the sensor data and prior knowledge
(_e.g.,_ knowledge of the object shape). Non-degeneracy, on the other hand,
ensures that for the provided input there exists a unique, correct solution. A
case of degeneracy arises often, for instance, in object pose estimation when
the partial point cloud is so occluded that there is more than one correct
pose fitting the data; see Fig. 1. Therefore, a _certifiable model_ is a
learning-based model that implements these two certificates —of observable
correctness and non-degeneracy— so that for every output produced by the
model, its $\zeta$-correctness may be checked.

We use this theory to introduce the problem of _certifiable object pose
estimation_ (1), where —given a partial point cloud of an object— we aim to
estimate the pose of the object in the point cloud and provide a certificate
that guarantees that the pose estimate is $\zeta$-correct (Section IV).

(II) C-3PO: a Certifiable Object Pose Estimation Model. We propose C-3PO
(Certifiable 3D POse) to solve the certifiable object pose estimation problem
(Section V). C-3PO first detects semantic keypoints from the input, partial
point cloud using a trainable regression model. To correct for potential
keypoint-detection errors (_e.g.,_ induced by a sim-to-real gap), C-3PO adds a
keypoint _corrector_ module that corrects some of the keypoint-detection
errors. The corrector module is implemented as a differentiable optimization
module [29, 30] (Section V-C): it takes in the detected keypoints and produces
a correction, by solving a non-linear, non-convex optimization problem.
Although the corrector optimization problem is hard to analyze, we show that
it can be differentiated through very easily. We provide an exact analytical
expression of the gradient of the output (the correction), given the input
(detected keypoints) to the corrector (Theorem 11), which we use to implement
back-propagation. We also implement a _batch gradient descent_ that solves a
batch of corrector optimization problems in parallel on GPU, thereby speeding
up the forward pass. Experimental analysis shows that the corrector is able to
correct high-variance perturbations to the semantic keypoints (Section VII-B).

Finally, using our theory of certifiable models, we derive two certificates,
namely, a certificate of observable correctness and a certificate of non-
degeneracy for C-3PO (Section V-D). We prove that this enables C-3PO to check
$\zeta$-correctness in the forward pass (Theorem 18). We also empirically
observe that the implemented certificates are able to identify correct
predictions in practice (Sections VII-C and VII-D).

(III) Certificate-based Self-supervised Training. C-3PO allows addressing the
twin problems of certification and self-supervision by noting the following:
if at test time we can use the proposed certificates to distinguish
$\zeta$-correct outputs, we can use the latter for further training. This
forms the basis of our new self-supervised training, which proceeds by
training only on the observably correct instances (_i.e.,_ input-output
pairs), in each batch and in each training iteration. As the training
progresses, the number of observably correct instances increases, eventually
resulting in a fully self-trained model.

(a)

(b)

(c)

Figure 1: Example of degenerate input for pose estimation. (a) Input: partial
point cloud of an object (gray), namely the “cap” in ShapeNet [31]; (b)
Estimated object pose shown as a posed object CAD model (blue); (c) Estimated
(blue) and ground-truth (gray) pose overlaid as two posed point clouds. Ground
truth semantic keypoints are in red, detected keypoints in green.

We test the self-supervised training and certification on depth-point-cloud
dataset generated using ShapeNet objects [31] and on the depth point clouds
extracted from RGB-D images in the YCB dataset [32] (Sections VII-C and
VII-D). In both cases, we observe that while the simulation-trained C-3PO
performs poorly (indicating a large sim-to-real gap), our self-supervised
training is able to successfully train C-3PO with self-supervision and
significantly outperforms existing baselines. As a further experiment, we show
that the proposed self-supervised training can be even used to self-train
C-3PO when the object category labels are not available, for instance C-3PO is
able to learn to detect a “chair” from a large unlabeled dataset containing
chairs, tables, cars, airplanes, etc. (Section VII-E).

An interactive visualization of all the results presented in this paper is
made available at: https://github.com/MIT-SPARK/C-3PO and
https://github.com/MIT-SPARK/pose-baselines.

##  II Related Work

Here we provide a non-exhaustive review about object pose estimation,
including recent image-based estimation methods.

Object Pose Estimation from RGB and RGB-D Inputs. Recent progress in object
pose estimation has been fueled by the availability of pose-annotated datasets
[33, 34, 35, 36]. State-of-the-art methods have evolved from correspondence-
based methods (which extract correspondences between the sensor data and the
object CAD model, and use them to register the model to the data) and
template-based methods (which attempt to match the sensor data to templates
consisting of the CAD model of the object rendered at various poses) to direct
regression, augmented with pose refinement. Xiang  _et al._ [17] regress the
object pose using a convolutional neural network (CNN) backbone. Wang  _et
al._ [37] propose GDR-Net, which instead extracts a $6$-dimensional rotation
representation, thus, avoiding discontinuity issues [38]. Nguyen  _et al._
[39] revive template-based methods by showing their generalization power. Li
_et al._ [40] propose DeepIM, which uses a pre-trained optical flow detection
model to guide and train object pose refinement. Labbe  _et al._ [41] propose
CosyPose, which showed state-of-the-art performance on the BOP’20 pose
estimation challenge [34], relying on the pose regression network in [17],
followed by the pose refinement proposed in [40]. The state-of-the-art
performance in the BOP’22 challenge was also attained by a model that uses a
pose regression network [42, 37]. Recent work also attempts object pose
estimation, without the availability of the object CAD model [43].

We note two gaps in the pose estimation literature. First, most of the
approaches above are RGB or RGB-D based, and investigating object pose
estimation from only depth or partial point clouds has received little
attention. Second, these methods rely on large pose-annotated datasets, with
only few attempts to develop self-supervised approaches [23, 24].

We next review different approaches for _point-cloud-only_ pose estimation,
including approaches based on local feature detection/description, global
features, and semantic keypoints.

Local Feature Detectors and Descriptors. Local-feature-based methods extract
local features (potentially with descriptors), determine point-to-point
correspondences, and solve a robust registration problem, which estimates the
object pose while filtering out incorrect correspondences. Significant
progress has been made in recent years towards (i) better features to extract
point-to-point correspondences [44, 45, 46, 47, 48, 49], (ii) robust
registration algorithms [50, 7, 1], as well as (iii) end-to-end registration
pipelines that simultaneously learn correspondences and obtain the relative
poses [51, 52, 53, 54, 55, 56, 57]. Modern robust registration methods have
been shown to compute accurate estimates even in the face of 99% random
outliers [1]. Several end-to-end learning-based registration approaches have
been proposed, but only a few can be trained in a self-supervised manner. A
weakly-supervised approach, using a triplet loss, is developed in [45]. Self-
supervised partial-to-partial scene registration is considered in [58, 59]. In
particular, Banani  _et al._ [59] use the fact that two successive frames
inherit some geometric and photometric consistency, and leverage
differentiable rendering to implement the training loss. A teacher-student-
verifier framework for joint feature learning and registration is considered
in [28].

Global Features. Methods that extract and use global features have been
proposed for point cloud alignment and object pose estimation. Sarode  _et
al._ [52] propose an architecture that uses PointNet encoding to extract
global features. Huang  _et al._ [60] extract global features from the point
clouds and then estimate the pose by minimizing a feature-metric projection
error. Li  _et al._ [3] propose a similar approach, but uses Lucas-Kanade-like
iterations for minimizing the feature-metric error. Yuan  _et al._ [4] propose
a probabilistic registration paradigm, which extracts pose-invariant point
correspondences, as latent Gaussian mixture model. Zhu  _et al._ [61]
addresses point cloud registration by extracting $\mathrm{SO}(3)$-equivariant
features. Sun  _et al._ [62] use a capsule network encoder. Among these
approaches PointNetLK [3] and DeepGMR [4] show the best results for object
pose estimation, albeit in different scenarios. Li  _et al._ [5] tackle
category-level object pose estimation from partial or complete object point
clouds, by extracting an $\mathrm{SE}(3)$-invariant shape feature and a
$\mathrm{SE}(3)$-equivariant pose feature.

Semantic Keypoints. Semantic keypoint detectors extract keypoints that
correspond to specific points on the object to detect (_e.g.,_ the wheels or
headlights of a car), hence circumventing the needs to compute local
descriptors. Semantic keypoint-based methods have been studied in the context
of human pose estimation [63, 64, 65]. Pavlakos  _et al._ [66, 67] use class-
specific semantic keypoints, extracted from CNN features, to estimate object
pose from an RGB image. Lin  _et al._ [68] propose an end-to-end,
differentiable pose estimation architecture using RGB images. Shi  _et al._
[69] propose an active shape model using semantic keypoints, and solve the
joint pose and shape estimation problem assuming detected semantic keypoints.
Yang and Pavone [9] obtain performance guarantees on keypoint detections and
pose estimates using conformal prediction. Zhou  _et al._ [70] propose
category-agnostic keypoint detection, and argue that using a fixed number of
keypoints per-category can be limiting (_e.g.,_ chair with many legs). Vecerik
_et al._ [71] advocate semantic keypoints to be the right visual
representation for object manipulation, and provide an efficient training
approach to learn instance and category-level keypoints using a small number
of annotated images.

Contrary to the works above, C-3PO does not rely on pose or keypoint
annotations for training. Instead, it only assumes the availability of the
object CAD model annotated with semantic keypoints. You  _et al._ [72] provide
a large-scale keypoint annotated dataset on ShapeNet CAD models. Suwajanakorn
_et al._ [73] show that semantic keypoint annotations can be learned in a
self-supervised manner, from just the object CAD models.

Self-Supervised Pose Estimation. Few works have tackled self-supervised pose
estimation. In [23, 24], a pose estimation model is first trained on synthetic
RGB-D data, and then refined further with self-supervised training on real,
unannotated data; differentiable rendering provides the required supervision
signal. Student-teacher iterative schemes are proposed in [27, 28] to bridge
the domain gap. Another approach is to extract a pose-invariant feature,
thereby canonizing only its shape [74], and using it for supervision. Zakharov
_et al._ [25] utilize differentiable rendering of signed distance fields of
objects, along with normalized object coordinate spaces [74], to learn 9D
cuboids in a self-supervised manner. Li  _et al._ [5] extract an
$\mathrm{SE}(3)$-invariant feature, which works as a canonical object, and
uses it to supervise training with a Chamfer loss. Sun  _et al._ [62] tackles
point cloud alignment in a self-supervised manner by extracting features using
capsule network. Deng  _et al._ [26] propose a way to self-supervise pose
estimation by interacting with the objects in the environment; the model gets
trained on the data collected autonomously by a manipulator.

Certification. Certifying correctness of the model output is crucial for
safety-critical and high-integrity applications. Certifiably optimal
algorithms return a solution and also provide a certificate of optimality [75,
7, 1] of the solution. Estimation contracts [1, 76], on the other hand,
determine if the input is reasonable enough for the optimal solution to be
indeed correct, _i.e.,_ close to the ground-truth. Certifiably optimal
algorithms have been devised for several geometric perception problems[7, 1,
8], where the duality gap serves as the certificate of optimality. These
notions of certification can be applied to optimization problems, which form
the back-end of any perception pipeline, but do not apply to the entire end-
to-end pipeline or learning-based models. In this work, we extend the notion
of certification to learning-based models.

Recent works have attempted to address the question of robustness and
uncertainty-quantification of learning-based models. Neural network
verification attempts to ensure that a learning-based model is robust to small
deviations in the input space [12, 13, 14]. Conformal prediction, on the other
hand, uses training data to safeguard a learning-based model [10, 11, 77, 78,
9]. Given a user-specified probability, it enables a trained model to predict
uncertainty sets/intervals, instead of a point estimate. The predicted set is
then provably correct, with the specified probability. Conformal prediction
works under the assumption that the test set and the dataset used to derive
thresholds for set-predictions have the same distribution. Providing
probabilistic guarantees under sim-to-real gap or covariate shifts is an
active research area [79, 80].

Differentiable Optimization. Learning-based models have traditionally relied
on simple feedfoward functions (_e.g.,_ linear, ReLU). Recent work has
proposed to embed differentiable optimization as a layer in learning-based
models [30, 81, 82]. A differentiable optimization layer is an optimization
problem for which the gradient of the optimal solution, with respect to the
input parameters, can be computed and back-propagated. Embedding an
optimization problem as a differentiable optimization layer enables a
learning-based model to explicitly take into account various geometric and
physical constraints. Gould  _et al._ [82] provide generic expressions to
differentiate any non-linear optimization problem. Agrawal  _et al._ [30]
provide a way to differentiate through convex programs in standard form. Amos
_et al._ [81] focus on quadratic optimization problems. Differentiating
combinatorial optimization problems is considered in [83, 84], and a
differentiable MAXSAT solver is proposed in [85]. Donti  _et al._ [86] propose
a way to differentiate a stochastic optimization problem. Teed  _et al._ [87]
implement differentiable bundle adjustment.

While developing generic techniques to implement derivatives of various
classes of optimization problems is useful, there is much scope for efficiency
if the problem structure can be exploited to implement simpler differentiation
rules [81, 88, 89]. C-3PO implements a corrector module, which is a
differentiable optimization layer. It solves a non-linear, non-convex
optimization problem, but also exploits its specific structure efficiently to
compute its derivative.

##  III A Theory of Certifiable Models

A learning-based model cannot guarantee _correctness_ of the output, in terms
of its closeness to the ground-truth. We first provide a general theory to
show that implementing two binary-valued certificates can determine the
correctness of the model estimate viz-a-viz the ground-truth. Our theory will
introduce the notions of $\zeta$-correctness, certificates of _observable
correctness_ and _non-degeneracy_ , and _certifiable models_. We apply this
theory to specify and solve the certifiable object pose estimation problem.

Setup. Let $\mathbb{{\bm{X}}}$ and $\mathbb{{\bm{Z}}}$ be the space of all
inputs and outputs, for a problem set that we are solving. We assume the
output space $\mathbb{{\bm{Z}}}$ is a metric space with a distance metric
$d_{\mathbb{{\bm{Z}}}}(\cdot,\cdot)$. Moreover, we assume any input
${\bm{X}}\in\mathbb{{\bm{X}}}$ is generated according to a given _generative
model_ , _i.e.,_ ${\bm{X}}=\phi({\bm{Z}}^{*})$, for some unknown
${\bm{Z}}^{*}\in\mathbb{{\bm{Z}}}$; hence, the goal is to estimate
${\bm{Z}}^{*}$ given ${\bm{X}}$. We make the following definition of
correctness:

######  Definition 1 ($\zeta$-correctness).

We say that the output ${\bm{Z}}$ produced by a model, for input ${\bm{X}}$,
is $\zeta$-correct if

$d_{\mathbb{{\bm{Z}}}}({\bm{Z}},{\bm{Z}}^{\ast})\leq\zeta.$ (1)

A learning-based model ${\cal M}$ maps inputs ${\bm{X}}\in\mathbb{{\bm{X}}}$
to outputs ${\bm{Z}}\in\mathbb{{\bm{Z}}}$, and is trained to solve a problem
${\cal P}({\bm{X}})$; instantiated by the input ${\bm{X}}$. Given this, we can
define the solution space ${\cal S}({\bm{X}})$ of the problem. It will play a
pivotal role in the definition and analysis of our certificates.

######  Definition 2 (Solution Space).

Given an input ${\bm{X}}$, the _solution space_ is the set of all outputs
${\bm{Z}}$ that can generate the input ${\bm{X}}$ using the generative model
$\phi(\cdot)$:

${\cal
S}({\bm{X}})\triangleq\left\\{{\bm{Z}}\in\mathbb{{\bm{Z}}}~{}|~{}\phi({\bm{Z}})={\bm{X}}\right\\}.$
(2)

The intuition behind our certification approach is as follows: the ground
truth ${\bm{Z}}^{*}$ is in the solution space since it generated the input
data, hence if we can (i) produce an estimate in the solution space, and (ii)
prove that the solution space is not large, we can conclude that the estimate
must be close to the ground truth. Our _certificates_ formalize these
intuitions.

Certificates. We define two certificates to check (i) the _observable
correctness_ of the output (_i.e.,_ whether the output ${\bm{Z}}$ is in the
solution space ${\cal S}({\bm{X}})$ or not) and (ii) the _non-degeneracy_
(_i.e.,_ whether ${\cal S}({\bm{X}})$ is small, or not); then we prove that
the two certificates imply correctness.

######  Definition 3 (Observable Correctness and Non-Degeneracy).

For model ${\cal M}$ and input ${\bm{X}}$, the _certificate of observable
correctness_ is a Boolean condition defined as

$\texttt{ObsCorrect}[{\cal M},{\bm{X}}]=\mathbb{I}\left\\{{\bm{Z}}={\cal
M}({\bm{X}})\in{\cal S}({\bm{X}})\right\\}.$ (3)

A _certificate of non-degeneracy_ is defined as

$\texttt{NonDegeneracy}[{\cal M},{\bm{X}}]=\mathbb{I}\left\\{\text{Diam}[{\cal
S}({\bm{X}})]<\delta\right\\},$ (4)

where $\text{Diam}[{\cal
S}({\bm{X}})]=\max_{{\bm{Z}},{\bm{Z}}^{\prime}\in{\cal
S}({\bm{X}})}d_{\mathbb{{\bm{Z}}}}({\bm{Z}},{\bm{Z}}^{\prime})$ denotes the
diameter of the solution space ${\cal S}({\bm{X}})$, and $\delta$ is a small
constant.

We now show that the two certificates enable us to determine when a model
produces an $\zeta$-correct output. The proof is given in Appendix -B.

######  Theorem 4.

For any input ${\bm{X}}$, the output produced by the model ${\bm{Z}}={\cal
M}({\bm{X}})$ is $\zeta$-certifiably correct, _i.e.,_
$d_{\mathbb{{\bm{Z}}}}({\bm{Z}},{\bm{Z}}^{\ast})<\zeta$, if the certificate of
observable correctness (3) and non-degeneracy (4) are both $1$, and
$\delta\leq\zeta$ in (4).

Implementing Certificates. Theorem 4 shows the importance of the certificates
in (3)-(4). However, in general it may be hard to implement (3)-(4) as the
solution space ${\cal S}({\bm{X}})$ might be hard to characterize. We next
prove that using an outer approximation of ${\cal S}({\bm{X}})$ (_i.e.,_
${\cal S}({\bm{X}})\subseteq\overline{{\cal S}}({\bm{X}})$), which is often
easier to obtain in practice, retains the correctness guarantees.

######  Theorem 5.

Define the certificates oc and nd as:

$\displaystyle\texttt{oc}({\cal M},{\bm{X}})$
$\displaystyle=\mathbb{I}\left\\{{\bm{Z}}={\cal M}({\bm{X}})\in\overline{{\cal
S}}({\bm{X}})\right\\},~{}\text{and}$ (5) $\displaystyle\texttt{nd}({\cal
M},{\bm{X}})$ $\displaystyle=\mathbb{I}\left\\{\text{Diam}[\overline{{\cal
S}}({\bm{X}})]<\delta\right\\},$ (6)

where ${\cal S}({\bm{X}})\subseteq\overline{{\cal S}}({\bm{X}})$. Then, the
output ${\bm{Z}}={\cal M}({\bm{X}})$ is $\zeta$-certifiably correct, _i.e.,_
$d_{\mathbb{{\bm{Z}}}}({\bm{Z}},{\bm{Z}}^{\ast})<\zeta$, if $\texttt{oc}({\cal
M},{\bm{X}})=1$ and $\texttt{nd}({\cal M},{\bm{X}})=1$, and $\delta\leq\zeta$
in (6).

In Section V, we provide practical implementations of the certificates in
Theorem 5 for the pose estimation problem.

Certifiable Model. We define a _certifiable model_ to be a triplet $({\cal
M},\texttt{oc},\texttt{nd})$, where ${\cal M}$ is a learning-based model, and
oc and nd are two certificates that (conservatively) approximate the
certificate of observable correctness (3) and non-degeneracy (4), in order to
establish $\zeta$-correctness of every output produced. In other words, a
certifiable model should always allow for a theorem, that is as follows:

######  Theorem 6 (Meta-Theorem for Certifiable Models).

The output ${\bm{Z}}={\cal M}({\bm{X}})$ is $\zeta$-certifiably correct,
_i.e.,_ $d_{\mathbb{{\bm{Z}}}}({\bm{Z}},{\bm{Z}}^{\ast})<\zeta$, if
$\texttt{oc}({\cal M},{\bm{X}})=1$ and $\texttt{nd}({\cal M},{\bm{X}})=1$.

######  Remark 7 (Relation to Certifiable Algorithms).

A parallel can be drawn between the two certificates in (3)-(4), and the
_certificate of optimality_ and the notion of _estimation contracts_ in
related work [7, 1, 76]. A certificate of optimality indicates whether the
solver to optimal estimation problem returns the optimal solution or not.
Estimation contracts, on the other hand, ensure that the input is “reasonable”
enough for the optimal solution to be indeed correct, _i.e.,_ close to the
ground-truth. While the former is analogous to our certificate of observable
correctness (which indicates whether the output of a learning-based model is
in the solution space or not), the latter is similar in spirit to the
certificate of non-degeneracy. The difference is that our approach, unlike [7,
1, 76], aims at guaranteeing correctness of learning-based models, rather than
optimization-based estimators.

We apply this general theory to specify and solve the certifiable object pose
estimation problem. In Section IV, we see how the definition of
$\zeta$-correctness translates to the object pose estimation problem, thereby
helping us to specify _correctness_ property. In Section V, we show how the
two certificates (3)-(4) can be implemented to establish $\zeta$-correctness.
Finally, in Section VI, we will see how we use these certificates to provide
the supervision in self-training.

##  IV Certifiable and Self-Supervised
Object Pose Estimation

Certifiable Object Pose Estimation. Let ${\bm{B}}$ be the CAD model of an
object, represented as the set of all points (expressed in homogeneous
coordinates) on the surface of the object. Let ${\bm{X}}^{*}$ be the
corresponding _posed_ model, _i.e.,_ the CAD model transformed according to
its ground-truth pose ${\bm{T}}^{\ast}\in\mathrm{SE}(3)$:

${\bm{X}}^{*}={\bm{T}}^{\ast}\cdot{\bm{B}}.$ (7)

Due to occlusions and sensor measurement noise, we only observe a partial and
noisy point cloud ${\bm{X}}$. This can be written as a function of the ground-
truth posed model ${\bm{X}}^{*}$:

${\bm{X}}=\Theta\left({\bm{X}}^{*}\right)+{\bm{n}}_{w},$ (8)

where $\Theta\left({\bm{X}}^{*}\right)$ denotes the sampling of $n$ points on
the object surface and deletion of occluded parts, and ${\bm{n}}_{w}$ is the
measurement noise.

The goal of certifiable pose estimation is to estimate the pose of the object,
given the partial point cloud ${\bm{X}}$, and also provide _correctness
guarantees_ of the resulting pose estimate —in terms of its closeness to the
ground-truth pose ${\bm{T}}^{\ast}$. Inspired by the theory of certifiable
models (Section III), we would like to implement a certifiable model $({\cal
M},\texttt{oc},\texttt{nd})$ that (i) estimates object pose
$\hat{{\bm{T}}}={\cal M}({\bm{X}})$, and (ii) determines if it is
$\zeta$-correct by checking whether $\texttt{oc}({\cal
M},{\bm{X}})=\texttt{nd}({\cal M},{\bm{X}})=1$.

In order to do this, we first instantiate the general definition of
$\zeta$-correctness to the problem of object pose estimation. Rather than
defining the output space to be the space of all poses
$\hat{{\bm{T}}}\in\mathrm{SE}(3)$, we consider the output space to be the
space of all posed CAD models, namely

$\mathbb{{\bm{Z}}}=\left\\{{\bm{T}}\cdot{\bm{B}}~{}\Big{|}~{}{\bm{T}}\in\mathrm{SE}(3)\right\\}.$
(9)

We define the distance between any two points in $\mathbb{{\bm{Z}}}$ to be the
Hausdorff distance:

$d_{H}({\bm{U}},{\bm{V}})=\max\left\\{\sup_{{\bm{x}}\in{\bm{U}}}D({\bm{x}},{\bm{V}}),\sup_{{\bm{x}}\in{\bm{V}}}D({\bm{x}},{\bm{U}})\right\\},$
(10)

where ${\bm{U}},{\bm{V}}\in\mathbb{{\bm{Z}}}$ and $D({\bm{x}},{\bm{U}})$
denotes the minimum distance from point ${\bm{x}}$ to the set ${\bm{U}}$.
Hausdorff distance is known to be a valid distance metric [90]. Thus, the
results in Section III hold when determining correctness of the pose estimate
$\hat{{\bm{T}}}$. The $\zeta$-correctness (Definition 3) translates to the
following definition for the object pose estimation problem:

######  Definition 8 ($\zeta$-correctness for Object Pose Estimation).

The pose estimate $\hat{{\bm{T}}}$, produced by a model, is $\zeta$-correct if

$d_{H}(\hat{{\bm{T}}}\cdot{\bm{B}},{\bm{T}}^{\ast}\cdot{\bm{B}})\leq\zeta,$
(11)

where $d_{H}(\cdot,\cdot)$ is the Hausdorff distance.

######  Remark 9 (Hausdorff Distance & Handling Symmetric Objects).

Intuitively, $\zeta$-correctness of the estimate $\hat{{\bm{T}}}$ ensures that
every point on the surface of the posed model $\hat{{\bm{T}}}\cdot{\bm{B}}$ is
at most $\zeta$ distance away from the surface of the ground-truth posed model
${\bm{T}}^{\ast}\cdot{\bm{B}}$. Unlike rotation and translation error, the
metric $d_{H}(\hat{{\bm{T}}}\cdot{\bm{B}},{\bm{T}}^{\ast}\cdot{\bm{B}})$
operates directly on the CAD model surfaces. This obviates the need to handle
symmetric objects separately: for instance, while two poses corresponding to
symmetries of an object would produce different rotation and translation
errors, the posed model associated to both poses will be the same, leading to
the same Hausdorff distance for both. The ADD-S metric [18] has also been
proposed to obviate the need for separately handling symmetric objects at
evaluation. However, it operates on sampled point clouds and cannot satisfy
the properties of a valid distance metric. In Section -A we discuss additional
reasons for using Hausdorff distance as opposed to pose error or ADD-S metric
[18] for theoretical analysis.

The $\zeta$-correctness of a pose estimate cannot be determined by directly
computing (11), as we do not have access to the ground-truth pose in practice.
A model solving the certifiable pose estimation problem should find a way to
assert if (11) is satisfied, without a genie-access to the ground truth,
leading to the following problem statement.

######  Problem 1 (Certifiable Object Pose Estimation).

Propose a pose estimation model that, along with producing an estimate, can
also provide a binary ($0/1$) _certificate_ , such that whenever a certificate
is $1$, the estimated pose is $\zeta$-correct (def. 8).

Unannotated Data and Self-Supervision. It is easy to realize that
certification is tightly coupled with self-supervision: if we are able to
discern —at test time— correct outputs, we can then use them for further
training. Therefore, in this paper we also study the twin problem of self-
supervising pose estimation models. In particular, we investigate learning
models that can self-train using an unannotated real-world dataset, which
consists of partial point clouds of objects, segmented from a scene; no pose
or any other annotation is assumed on the real-world dataset. We start by
noting that annotated simulation data is broadly available and can be used to
initialize weights of any learning-based pose estimation model. Therefore, we
consider the practical case where we are given a sim-trained learning-based
model, and our goal is to modify and further train the model on the
unannotated real-world data — especially, when the sim-to-real gap is
significant.

######  Problem 2 (Self-supervised Pose Estimation).

Propose a method to modify a sim-trained pose estimation model, and train it
with self-supervision on unannotated real-world data.

##  V C-3PO: a Certifiable Pose Estimation Model

This section provides a certifiable pose estimation model, named C-3PO
(Certifiable 3D POse), that includes a novel architecture (mixing learning-
based and optimization-based modules), and a practical implementation of the
certificates introduced in the previous section. Then, in Section VI we show
that the use of the proposed certificates naturally enables C-3PO to self-
train on unannotated data.

Figure 2: KeyPO: Semantic keypoint-based pose estimation. Figure 3: Annotation
of semantic keypoints (red) on CAD models of (a) table, (b) motorcycle, and
(c) car in KeypointNet [72].

###  V-A Preliminaries: Semantic-Keypoint-based Pose Estimation (KeyPO)

We start by briefly reviewing a standard pose estimation method based on
semantic keypoints (Fig. 2), which constitutes the backbone of C-3PO. Semantic
keypoints ${\bm{b}}[i],i=1,\ldots,N$, are specific points annotated on the CAD
model ${\bm{B}}$, and are fixed in number for a given object (see, _e.g.,_
Fig. 3). A standard semantic-keypoint-based pose estimator (KeyPO) first
detects semantic keypoints $\tilde{{\bm{y}}}[i],i=1,\ldots,N$, from the input
point cloud using a neural network. Then, it estimates the object pose by
solving an outlier-free registration problem:

$\displaystyle\hat{{\bm{T}}}=$
$\displaystyle\operatorname*{arg\,min}_{{\bm{T}}\in\mathrm{SE}(3)}$
$\displaystyle\textstyle\sum_{i=1}^{N}~{}~{}\lVert\tilde{{\bm{y}}}[i]-{\bm{T}}\cdot{\bm{b}}[i]\rVert_{2}^{2},$
(12)

that computes the object pose
${\bm{T}}\triangleq\left[\begin{array}[]{cc}{\bm{R}}&{\bm{t}}\\\
{\mathbf{0}}&1\end{array}\right]$ (where ${\bm{R}}$ is the object rotation and
${\bm{t}}$ is its translation) by aligning the detected keypoints
$\tilde{{\bm{y}}}[i]$ and the model keypoints ${\bm{b}}[i]$, annotated on the
CAD model. After computing the pose estimate $\hat{{\bm{T}}}$, we can also
compute the _posed_ CAD model and keypoints, namely

$(\hat{{\bm{y}}},\hat{{\bm{X}}})=(\hat{{\bm{T}}}\cdot{\bm{b}},\hat{{\bm{T}}}\cdot\hat{{\bm{B}}})$
(13)

where $\hat{{\bm{B}}}$ denotes a point cloud obtained by densely sampling the
CAD model ${\bm{B}}$.

######  Remark 10 (Limitations of Standard Semantic-Keypoints-based Models and
Novel Insights).

In the presence of keypoint detection errors (_e.g.,_ caused by sim-to-real
gap), the pose computed by KeyPO might be inaccurate. Moreover, using only a
sparse number of keypoints (typically in the order of tens) for pose
estimation might lead to less accurate results. For this reason, modern
approaches either resort to neural models that mimic ICP [2, 3], approaches
based on dense correspondences [51, 91, 53], or approaches based on learned
global representations [4, 5]. In this paper we provide two novel insights:
(i) despite common beliefs, standard semantic-keypoint-based methods still
outperform newer alternatives in the presence of occlusions and large object
displacements (Section VII-A); (ii) we can correct detection errors and factor
dense information into the registration phase of semantic-keypoint methods by
introducing a _corrector_ module, as discussed in the following section.

###  V-B Overview of C-3PO

C-3PO takes a partial or occluded object point cloud ${\bm{X}}$ as input and
estimates the object pose $\hat{{\bm{T}}}$, as well as a posed point cloud
$\hat{{\bm{X}}}$ and keypoints $\hat{{\bm{y}}}$ as in (13). C-3PO uses a
standard semantic keypoint-based pose estimation architecture as the backbone,
which first uses a neural network to detect semantic keypoints, and then
estimates the 3D pose via registration to the corresponding CAD model.
However, contrary to standard semantic keypoint-based methods, C-3PO adds a
keypoint _corrector module_ that corrects some of the keypoint-detection
errors (_cf._ Figs. 2 and 4). Finally, C-3PO implements two certificates to
check $\zeta$-correctness of the estimate (Fig. 5). Below, we provide an
overview of the key components of C-3PO, while we postpone the details to the
following sections.

Semantic Keypoint-based Pose Estimation. A semantic keypoint detection network
first detects the semantic keypoints $\tilde{{\bm{y}}}[i],i=1,\ldots,N$, from
the input point cloud ${\bm{X}}$ (Fig. 4). We implement the keypoint detector
in KeyPO as a trainable regression model; in our tests, we use PointNet++ [92]
or point transformer [93] as a neural architecture that operates on point
clouds. A regression model enables detecting semantic keypoints even in the
occluded regions of the input ${\bm{X}}$.

Corrector. The semantic keypoint detector might produce perturbed keypoints
when tested on real data (_e.g.,_ if the detector is trained on a simulation
dataset and there is a sim-to-real gap). In C-3PO, we add a _corrector_ module
that takes the estimated semantic keypoints $\tilde{{\bm{y}}}$ —produced by
the keypoint detector— as input, and outputs a correction term
$\Delta{\bm{y}}^{\ast}$ to the keypoints. The corrected keypoints become:

${\bm{y}}=\tilde{{\bm{y}}}+\Delta{\bm{y}}^{\ast}.$ (14)

The resulting architecture is shown in Fig. 4. The correction term
$\Delta{\bm{y}}^{\ast}$ is obtained as a solution to the _corrector
optimization problem_ , whose details are given in Section V-C. We implement
the corrector as a differentiable optimization module, by explicitly computing
its gradient. For faster training, we implement batch gradient descent that
solves a batch of corrector optimization problems, in parallel, on GPU.

Figure 4: C-3PO: Proposed semantic-keypoint-based pose estimation and model
fitting architecture with _corrector_.

Certification. C-3PO implements the two certificates introduced in Section
III, a certificate of observable correctness and a certificate of non-
degeneracy, which together determine $\zeta$-correctness of a pose estimate.
While Theorem 5 provides a mathematical justification of implementing the two
certificates, we describe what they intuitively mean in the case of object
pose estimation. The certificate of observable correctness assesses whether
the output of C-3PO fits the input data; if it does then it is equivalent to
saying that the “output is in the solution space”. The non-degeneracy
certificate, on the other hand, ensures that the input point cloud ${\bm{X}}$
provides sufficient information on the pose of the object to allow us to
unambiguously estimate it. Figure 1 shows an instance in which using just the
input point cloud (shown in (a)), it is impossible to estimate the ground-
truth pose. The key intuition in C-3PO is that we can use the _keypoints_ to
infer non-degeneracy: if we choose keypoints to cover the different parts of
the object (_e.g.,_ in Fig. 1, the keypoints capture the visor, the top
button, and other parts of the cap), then as long as we detect specific
subsets of keypoints on ${\bm{X}}$ (_e.g.,_ including the ones on the visor
for the cap in Fig. 1), we can unambiguously estimate the object pose. Since
we design the two certificates to be conservative, we then use Theorem 5 to
conclude that, when the certificates are $1$, the resulting estimate is
$\zeta$-correct.

Figure 5: C-3PO: Certificate of observable correctness and non-Degeneracy.
Here, $\left\\{\textsf{G}_{l}\right\\}_{l=1}^{g}$ denote the collection of
_indicator sets_ that are used to determine non-degeneracy; see (4).

###  V-C Keypoint Corrector

We now discuss the semantic keypoint corrector in the C-3PO. The keypoint
corrector adds a correction $\Delta{\bm{y}}^{\ast}$ to the detected keypoints
$\tilde{{\bm{y}}}$. This is intended to correct for any keypoint detection
errors. This section describes how we compute $\Delta{\bm{y}}^{\ast}$, from
the detected keypoints $\tilde{{\bm{y}}}$, and the implementation of the
corrector as a differentiable optimization module.

First, we introduce some notation. Let $\hat{{\bm{T}}}(\Delta{\bm{y}})$ denote
the optimal solution to the outlier-free registration problem in (12), but
with the keypoints ${\bm{y}}=\tilde{{\bm{y}}}+\Delta{\bm{y}}$, instead of
$\tilde{{\bm{y}}}$ in (12). For a given $\hat{{\bm{T}}}(\Delta{\bm{y}})$, let

$(\hat{{\bm{y}}}(\Delta{\bm{y}}),\hat{{\bm{X}}}(\Delta{\bm{y}}))\in\hat{\mathbb{{\bm{Z}}}},$
(15)

be the posed keypoints and CAD model, computed using (13). Namely,
$\hat{{\bm{y}}}(\Delta{\bm{y}})$ are the keypoints ${\bm{b}}$ transformed
according to the pose $\hat{{\bm{T}}}(\Delta{\bm{y}})$ and
$\hat{{\bm{X}}}(\Delta{\bm{y}})$ is the dense object point cloud
$\hat{{\bm{B}}}$ transformed by the same transformation. In (15), we denoted
with $\hat{\mathbb{{\bm{Z}}}}$ the output space, _i.e.,_ the set of all posed
keypoints and sample CAD models.

Corrector Optimization Problem. The correction $\Delta{\bm{y}}$ is optimized
so that the predicted object model $\hat{{\bm{X}}}(\Delta{\bm{y}})$ and model
keypoints $\hat{{\bm{y}}}(\Delta{\bm{y}})$ are aligned to the input point
cloud ${\bm{X}}$ and the corrected keypoints ${\bm{y}}$:

$\displaystyle\underset{\Delta{\bm{y}}\in{{\mathbb{R}}^{3\times
N}}}{\text{Minimize}}$
$\displaystyle\text{ch}_{1/2}({\bm{X}},\hat{{\bm{X}}}(\Delta{\bm{y}}))+\gamma||{\bm{y}}-\hat{{\bm{y}}}(\Delta{\bm{y}})||^{2}_{2},$
(16)

where $\gamma$ is a positive constant and
$\text{ch}_{1/2}({\bm{X}},\hat{{\bm{X}}})$ is the half-Chamfer loss given by

$\text{ch}_{1/2}({\bm{X}},\hat{{\bm{X}}})=\frac{1}{n}\sum_{i\in[n]}\min_{j\in[m]}\lVert{\bm{X}}[i]-\hat{{\bm{X}}}[j]\rVert^{2}_{2}.$
(17)

The half-Chamfer loss measures the average squared distance between each point
in the input ${\bm{X}}$ and the closest point in the estimated model
$\hat{{\bm{X}}}$. The use of the half-Chamfer loss is motivated by the fact
that the input point cloud ${\bm{X}}$ is typically occluded, hence we are only
interested in assessing how well the visible part of the object fits the posed
model.

Forward Pass: Solving (16). The corrector output $\Delta{\bm{y}}^{\ast}$ is
the solution of the optimization problem (16). The optimization problem in
(16) is non-linear and non-convex. A number of open-source solvers can be used
to solve (16) in the forward pass. We tested different solvers and observed
that two solvers work well, _i.e.,_ they are able correct large errors in the
keypoints. These are a simple constant-step-size gradient descent, which we
implement in PyTorch, and a trust-region method [94], implemented in the SciPy
library [95]. In implementing these solvers, we need to obtain gradients of
the objective function in (16), which depends on the solution to the outlier-
free registration problem (12). The solution to (12) involves linear
operations and SVD computation [96, 97, 98], the derivative of which is
computed using the autograd functionality in PyTorch. More specifically, we
implement _batch gradient descent_ – a simple, fixed-step-size gradient
descent method implemented in PyTorch that solves (16) for a batch input on
the GPU, and this leads to much faster compute times (Remark 23).

Gradient Computation for Back-Propagation. Implementing the corrector as a
block in an end-to-end differentiable pipeline requires (16) to be
differentiable, _i.e.,_ there must be a way to compute
$\partial\Delta{\bm{y}}^{\ast}/\partial\tilde{{\bm{y}}}$. We now show that the
corrector optimization problem (16), although being non-linear and non-convex,
heeds to a very simple derivative (proof in Appendix 23). This allows us to
implement the corrector as a differentiable optimization module in the C-3PO
architecture.

######  Theorem 11.

The gradient of the correction $\Delta{\bm{y}}^{\ast}$ with respect to the
estimated keypoints $\tilde{{\bm{y}}}$ is the negative identity, i.e.,

$\partial\Delta{\bm{y}}^{\ast}/\partial\tilde{{\bm{y}}}=-\mathbf{I}.$ (18)

######  Remark 12 (Corrector vs. ICP).

The corrector is similar in spirit but computationally different from an ICP-
based pose refinement. ICP-based methods tend to correct/refine the pose by
optimizing over the object rotation or translation. The corrector, on the
other hand, does so by optimizing over the keypoint space; see (16). In our
experiments, we observe that correcting for pose over the space of keypoints
is more powerful (_i.e.,_ corrects larger errors) compared to doing so over
the space of rotation and poses.

###  V-D Certificates for Pose Estimation

We now use the framework in Section III to derive two certificates for C-3PO.
C-3PO uses a dense point cloud representation $\hat{{\bm{B}}}$ of the object
${\bm{B}}$, to obtain the posed output $\hat{{\bm{X}}}$; see (13). While such
a sampling is necessary for implementation (_e.g.,_ to make it easy to compute
distances between two posed models), the definition of $\zeta$-correctness
(Definition 3) requires closeness between the CAD models
$\hat{{\bm{T}}}\cdot{\bm{B}}$ and ${\bm{T}}^{\ast}\cdot{\bm{B}}$.

To bridge this gap, we use two output spaces: (i) $\hat{\mathbb{{\bm{Z}}}}$:
the space of all outputs $(\hat{{\bm{y}}},\hat{{\bm{X}}})$ produced by C-3PO;
see (13). This space is nothing but the space of all posed tuple
$({\bm{b}},\hat{{\bm{B}}})$ of model keypoints ${\bm{b}}$ and sampled object
CAD model $\hat{{\bm{B}}}$. (ii) $\mathbb{{\bm{Z}}}$: the space of all posed
tuple $({\bm{b}},{\bm{B}})$:

$\mathbb{{\bm{Z}}}=\left\\{({\bm{T}}\cdot{\bm{b}},{\bm{T}}\cdot{\bm{B}})~{}\Big{|}~{}{\bm{T}}\in\mathrm{SE}(3)\right\\},$
(19)

where now ${\bm{B}}$ is the full CAD model.

######  Remark 13.

The choice of $\mathbb{{\bm{Z}}}$ (or $\hat{\mathbb{{\bm{Z}}}}$) to be the
space of all posed keypoints and models, and not just the space of all poses
(_i.e.,_ $\mathrm{SE}(3)$) circumvents the need to consider symmetric objects
as a special case, and helps our analysis (Appendix -D).

Certificate of Observable Correctness. We conservatively approximate the
observable correctness certificate by checking the geometric consistency
between the input point cloud ${\bm{X}}$ and the output
$\hat{{\bm{Z}}}=(\hat{{\bm{y}}},\hat{{\bm{X}}})$ in (13). Since the input
point cloud is partial, we only check if every point in the partial input
${\bm{X}}$ is close to a point in $\hat{{\bm{X}}}$:

$\texttt{oc}(\hat{{\bm{Z}}},{\bm{X}})=\mathbb{I}\left\\{\max_{i\in[n]}\min_{j\in[m]}\lVert{\bm{X}}[i]-\hat{{\bm{X}}}[j]\rVert_{2}<\epsilon_{\texttt{oc}}\right\\},$
(20)

where $\epsilon_{\texttt{oc}}$ is a small positive constant. We show that this
certificate can be derived as an outer approximation certificate when the
noise ${\bm{n}}_{w}$ in (8) is bounded, and the point cloud representation
$\hat{{\bm{B}}}$ of ${\bm{B}}$ is sufficiently dense (Appendix -C).

We note that while the certificate (20) is written as a function of the output
$\hat{{\bm{Z}}}$ and the input ${\bm{X}}$, it can repurposed to the form
$\texttt{oc}({\cal M},{\bm{X}})$ (in Section III) by setting
$\hat{{\bm{Z}}}={\cal M}({\bm{X}})$.

Certificate of Non-degeneracy. Non-degeneracy depends on the size of the
solution space ${\cal S}({\bm{X}})$ (see (4)). Intuitively, the solution space
is likely to be large if the input point cloud is highly occluded or is
missing important features or parts of the object. Consider the case of an
otherwise symmetric mug, with a handle. If the input point cloud ${\bm{X}}$
misses the handle entirely, there will be multiple ways to register the CAD
model to the input – thereby, making ${\cal S}({\bm{X}})$ large.

Motivated by this observation, we define the notion of a _cover set_ of points
on a CAD model.

######  Definition 14 (Cover Set).

A subset of points ${\bm{A}}$ on the surface of a CAD model ${\bm{B}}$ is a
_cover set_ if, given the 3D positions of points in ${\bm{A}}$, we can
unambiguously estimate (up to noise) the pose of ${\bm{B}}$:

$\sup_{{\bm{x}}\in{\bm{T}}^{\prime}\cdot{\bm{A}}}D({\bm{x}},{\bm{T}}\cdot{\bm{B}})<\delta\implies
d_{H}({\bm{T}}^{\prime}\cdot{\bm{B}},{\bm{T}}\cdot{\bm{B}})<\delta,$ (21)

for any ${\bm{T}},{\bm{T}}^{\prime}\in\mathrm{SE}(3)$ and where $\delta$ is a
positive constant.

Eq. (21) states that if the partial point cloud ${\bm{A}}$ of the object
${\bm{B}}$ is such that every point in ${\bm{T}}^{\prime}\cdot{\bm{A}}$ is
close in distance to the posed model ${\bm{T}}\cdot{\bm{B}}$, then it will
necessitate that that two posed models ${\bm{T}}\cdot{\bm{B}}$ and
${\bm{T}}^{\prime}\cdot{\bm{B}}$, are in fact very closely aligned. This
implies that if our input point cloud ${\bm{X}}$ is large enough to encompass
at least a cover set of the object, the size of the solution space must be
small; implying non-degeneracy (4).

Our insight is that we can use semantic keypoints, intelligently annotated, to
check for the size of the solution space, and if the observed ${\bm{X}}$ does
indeed encompass a cover set. In our mug example, if we are able to detect a
few keypoints on the mug handle, such that those points are also close to the
input point cloud, then we can conclude that the input is non-degenerate. This
leads to the notion of _indicator set_.

######  Definition 15 (Indicator Set).

Let $\Theta\left({\bm{B}}\right)$ be a sampled and visible portion of an
object ${\bm{B}}$ as in (8). A subset of keypoints $\textsf{G}\subset[N]$ is
an _indicator set_ if when all keypoints $\\{{\bm{b}}[i]\\}_{i\in\textsf{G}}$
are close to $\Theta\left({\bm{B}}\right)$ by distance $\delta_{ind}$, _i.e.,_

$\min_{j\in[n]}\lVert{\bm{b}}[i]-\Theta\left({\bm{B}}\right)[j]\rVert_{2}<\delta_{ind}~{}\forall~{}i\in\textsf{G},$
(22)

then $\Theta\left({\bm{B}}\right)$ is a cover set for ${\bm{B}}$.

Let $\left\\{\textsf{G}_{l}\right\\}_{l=1}^{g}$ be a collection of such
indicator sets for the object. Given an input point cloud ${\bm{X}}$ and an
output
$\hat{{\bm{Z}}}=(\hat{{\bm{y}}},\hat{{\bm{X}}})\in\hat{\mathbb{{\bm{Z}}}}$, we
declare non-degeneracy if —for at least one set $\textsf{G}_{l}$, $l\in[g]$—
every keypoint in $\hat{{\bm{y}}}[i]$ in $\textsf{G}_{l}$ is close to a point
in the input ${\bm{X}}$. Mathematically, this is given by:

$\texttt{nd}(\hat{{\bm{Z}}},{\bm{X}})=\mathbb{I}\left\\{\bigcup_{l=1}^{g}\bigcap_{i\in\textsf{G}_{l}}\left\\{\min_{j\in[n]}\lVert\hat{{\bm{y}}}[i]-{\bm{X}}[j]\rVert_{2}<\delta_{\texttt{nd}}\right\\}\right\\},$
(23)

where $\delta_{\texttt{nd}}$ is a small positive constant.

Certification Guarantees. We now prove that using the two certificates
(20)-(23), we can determine whether an estimate produced by the model ${\cal
M}$ is $\zeta$-correct (Definition 8), under some reasonable assumptions. The
proof is given in Appendix -D.

######  Assumption 16 (Bounded Noise).

The noise (8) is bounded, namely
$\max_{i}\lVert{\bm{n}}_{w}[i]\rVert_{2}<\epsilon_{w}$.

######  Assumption 17 (Dense CAD Model Sampling).

The sampled point cloud $\hat{{\bm{B}}}$ is such that every point on
${\bm{B}}$ is at most a distance $\epsilon_{s}$ away from a point in
$\hat{{\bm{B}}}$, namely
$\min_{j\in[m]}\lVert{\bm{x}}-\hat{{\bm{B}}}[j]\rVert_{2}<\epsilon_{s}$ for
all ${\bm{x}}\in{\bm{B}}$.

######  Theorem 18.

Assume bounded noise, dense CAD model sampling (Assumptions 16-17), and that
the sets $\\{\textsf{G}_{l}\\}_{l=1}^{g}$ in (23) are indicator sets. Then,
the output produced by a pose estimator ${\cal M}$:
$\hat{{\bm{Z}}}=(\hat{{\bm{y}}},\hat{{\bm{X}}})={\cal M}({\bm{X}})$ is
$\zeta$-correct (Definition 8), if
$\texttt{oc}(\hat{{\bm{Z}}},{\bm{X}})=\texttt{nd}(\hat{{\bm{Z}}},{\bm{X}})=1$
in (20), (23) for any $\zeta>\epsilon_{\texttt{oc}}+\epsilon_{w}$, provided
$\epsilon_{\texttt{oc}}+\delta_{\texttt{nd}}+2\epsilon_{w}<\delta_{ind}$.

######  Remark 19 (Choosing Indicator Sets).

The non-degeneracy certificate in (23) provably works (Theorem 18) when there
exists indicator sets, among the annotated keypoints. Deriving indicator sets
to provably meet its definition is hard and we do not choose that route. We,
instead, hand-craft the indicator sets and show its usefulness towards
correctly certifying C-3PO’s pose estimates. We show, in our experiments, that
a simple choice of subsets (listed in Appendix -G) suffices to determine non-
degeneracy.

##  VI Self-Supervised Training

The certificates implemented in C-3PO enable a simple yet effective self-
supervised training procedure. According to the setup described in 2, we
assume that the real-world, training dataset ${\cal D}$ consists of a
collection of input point clouds ${\bm{X}}$, bearing no pose or keypoint
annotation. The only trainable part in our C-3PO architecture is the keypoint
detector, which is initialized to a sim-trained model. In the self-supervised
training, it is the keypoint detector that gets trained to better detect
semantic keypoint on real-world data. The other components in C-3PO (_i.e.,_
corrector and certificates) enable this self-supervised training.

Certificate-based Self-Supervised Training. Our self-supervised training is
the same as a standard supervised training using stochastic gradient descent
(SGD), except from two changes. First, at each iteration, when we see a batch
of inputs $\\{{\bm{X}}^{i}\\}$, we compute the output
$\\{(\hat{{\bm{y}}}^{i},\hat{{\bm{X}}}^{i})\\}$ using the current model
weights. We then compute the loss:

$\mathcal{L}_{i}=\text{ch}_{1/2}({\bm{X}}^{i},\hat{{\bm{X}}}^{i})+\theta\sum_{j=1}^{N}\lVert{\bm{y}}^{i}[j]-\hat{{\bm{y}}}^{i}[j]\rVert^{2}_{2},$
(24)

for each input-output pair $i$ in the batch; where $\text{ch}_{1/2}$ is the
half-Chamfer loss, $\\{{\bm{y}}^{i}\\}_{i}$ are the corrected keypoints, and
$\theta$ is a positive constant.

Second, at each iteration, we also determine the observable correctness of
each input-output pair, by computing certificates in (20):
$\texttt{oc}_{i}=\texttt{oc}((\hat{{\bm{y}}}^{i},\hat{{\bm{X}}}^{i}),{\bm{X}}^{i}).$

We then compute the total training loss for the batch —that we use for back-
propagation— as

$\mathcal{L}=\sum_{i}\texttt{oc}_{i}\cdot\mathcal{L}_{i},$ (25)

which is nothing but the sum-total of loss (24) computed only for the
observably correct input-output pairs, in the batch.

######  Remark 20 (Role of Certification).

Self-supervised training using the total loss ${\cal L}=\sum_{i}{\cal L}_{i}$,
_i.e.,_ using all and not just the observably correct instances, would not
work well in practice. This is because, some of the predicted models
$\hat{{\bm{X}}}$ are not correctly registered to the input point clouds
${\bm{X}}$. Including them in the loss function (25) induces incorrect
supervision causing the self-supervised training to fail. Certification, on
the other hand, weeds out incorrectly registered object models during the
self-supervised training and provides correct supervision to the keypoint
detector during training. In all our experiments we observe that as our self-
supervised training progresses, the fraction of observably correct instances
increases, eventually converging to nearly 100% of the model outputs being
observably correct (see Fig. 9, in Section VII-D).

(a) Easy + Full PC

(b) Hard + Full PC

(c) Hard + Depth PC

Figure 6: Shows the distribution of ADD-S scores [18] for object pose
estimation of a ShapeNet object car. Shows various baselines on three
datasets: (a) Easy + Full PC, (b) Hard + Full PC, and (c) Hard + Full PC.

##  VII Experiments

We present five experiments. We first show that a standard semantic-keypoint-
based architecture outperforms more recent alternatives in problems with
partial point clouds and large object displacements (Section VII-A). We then
analyze the ability of the corrector to correct errors in the keypoint
detections (Section VII-B). We show the effectiveness of our self-supervised
training and certification on a depth-point-cloud dataset generated using
ShapeNet [31] objects (Section VII-C) and on the YCB dataset [32] (Section
VII-D); here we observe that C-3PO significantly outperforms all the baselines
and state-of-the-art approaches. Finally, we show that our self-supervised
training method can work even when the training data has no object category
labels on the input point clouds (Section VII-E).

(a) Rotations

(b) Translations

Figure 7: Cumulative distribution function of object rotations and
translations for easy/hard test dataset. The object rotations are in radians
(rotation angle around a randomly chosen rotation axis), while the
translations is measured as the norm of the translation vector normalized by
the object diameter.

###  VII-A Why Do We Use Semantic Keypoints?

We start by providing an experimental analysis that justifies our choice of
KeyPO as the backbone for C-3PO. This choice seems to go against the
commonplace belief that neural models that mimic the ICP algorithm [2, 3], or
based on learned global representations [4, 5] outperform keypoint-based
approaches. We do not consider dense correspondence-based methods like [91,
53, 51] as they are shown to be worse than PointNetLK [2, 3] and DeepGMR [4].
This section empirically shows that this conclusion is only true in relatively
easy problem instances, while keypoint-based approaches still constitute the
go-to solution for hard problems with occlusions and arbitrary object poses.

Setup. We consider the following approaches: (i) FPFH + TEASER++ [44, 1]: a
local-feature-based pose estimation method, that extracts local features and
computes a pose estimate via robust registration; (ii) PointNetLK [3]: a
learning-based model that attempts to “neuralize” iterative closest point and
is considered a state-of-the-art approach for point cloud alignment and pose
estimation; (iii) DeepGMR [4]: a learning-based model that attempts to extract
deep features, by modeling them as latent Gaussian variables; the method is
also known to be competitive to PointNetLK; (iv) EquiPose [5]: a learning-
based model that attempts to solve shape completion, pose estimation, and in-
category generalization; and (v) KeyPO: a standard semantic keypoint-based
pose estimation architecture (Fig. 2) with a point transformer [93] as a
keypoint detector.

We analyze these models under two experimental settings, named _easy_ and
_hard_. In the _easy_ case, we consider relatively small rotations and
translations of the objects, while the _hard_ case we induce larger rotations
and translations. The cumulative distributions of the object rotations and
translations are visualized in Fig. 7. Since several methods tend to perform
well when the input ${\bm{X}}$ is a full point cloud, as opposed to a depth or
a partial point cloud, we consider the following scenarios: (i) Easy + Full
PC, where the object displacement is small and the input point cloud is the
full point cloud, (ii) Hard + Full PC, where the object displacement is large
and the input point cloud is the full point cloud, and (iii) Hard + Depth PC,
where the object displacement is large, and the input point cloud is partial
and computed from the rendered depth image of the object.

Results. Figure 6 shows the distribution of the ADD-S scores [18] for object
pose estimation of a ShapeNet car object and for all the techniques above.
From the figure, we note that the performance of all the approaches, except
KeyPO, degrades as we go from easy to hard test setups and from full point
clouds to depth point clouds. PointNetLK, while performing extremely well on
the Easy + Full PC dataset degrades significantly when the object rotations
and translations are large. DeepGMR, while still performing well with large
pose magnitudes, shows significant performance degradation when the input
point cloud is a depth point cloud. EquiPose, while generalizing better, shows
consistent sub-optimal performance. In particular, EquiPose seems to trade off
its pose estimation accuracy for the objective of in-category generalization,
and we consistently see the object shape/size not exactly matching the input.
Like EquiPose, the local feature-based method FPFH + TEASER++ also generalizes
well, but are still outperformed by KeyPO on the Hard + DepthPC dataset.

Insights. We gather the following insights from the above analysis. Methods
attempting to “neuralize” ICP, for solving object pose estimation, can yield
optimal performance under small rotations and translation, while yielding sub-
optimal performance when the object displacement is large; this is not
dissimilar from ICP, which works well when initialized close to the ground-
truth pose, but is prone to converge to local minima otherwise. Methods like
DeepGMR, that extract learned representations, while succeeding when the input
is a full point cloud, fail when the input is occluded. The performance gap
between KeyPO and DeepGMR (on Hard + Depth PC) indicates that we are better
off using semantic keypoints as a learned representation for the task of pose
estimation.

|  |
---|---|---
Figure 8: Rotation error (left), translation error (mid), and fraction of
_not_ observably correct instances ($\texttt{oc}=0$) (right), as a function of
the noise parameter $\sigma$ for the “chair”object in ShapeNet. The rotation
error is in radians and the translation error is normalized by the object
diameter.

###  VII-B Keypoint Corrector Analysis

This section shows that the keypoint corrector module is able to correct large
errors in the keypoint detections.

Setup. We consider ShapeNet objects [31] and use the semantic keypoints
labeled by the KeypointNet dataset [72]. Give an object model and semantic
keypoints $({\bm{b}},{\bm{B}})$, we extract depth point cloud of ${\bm{B}}$
from a certain viewing angle, transform $({\bm{b}},{\bm{B}})$ by a random
pose, and add perturbations to the keypoints. The induced pose error is the
same as in the _hard_ case described in Section VII-A. For each keypoint
${\bm{b}}[i]$, with probability $f$, we add uniform noise distributed in
$[-\sigma d/2,\sigma d/2]$, and with probability $1-f$, keep ${\bm{b}}[i]$
unperturbed. Here, $d$ is the diameter of the object. We set $f=0.8$. We study
the performance of the corrector as a function of the noise variance parameter
$\sigma$.

TABLE I: Evaluation of C-3PO and baselines for the ShapeNet experiment. ADD-S ADD-S (AUC) | car | chair | helmet | laptop | skateboard | table
---|---|---|---|---|---|---
KeyPo (sim) | $0.00$ | $0.00$ | $0.00$ | $12.01$ | $0.00$ | $2.00$ | $0.00$ | $0.00$ | $4.00$ | $11.30$ | $0.00$ | $3.13$
KeyPo (sim) + ICP | $0.00$ | $0.00$ | $0.00$ | $11.56$ | $0.00$ | $2.00$ | $2.00$ | $2.00$ | $2.00$ | $9.80$ | $0.00$ | $0.00$
KeyPo (sim) + Corr. | $0.00$ | $13.69$ | $68.00$ | $57.16$ | $10.00$ | $20.14$ | $26.00$ | $20.57$ | $70.00$ | $59.17$ | $80.00$ | $53.54$
C-3PO | $80.00$ | $64.00$ | $100.00$ | $79.06$ | $100.00$ | $70.49$ | $\mathbf{100.00}$ | $\underline{64.70}$ | $\mathbf{100.00}$ | $\underline{82.90}$ | $\underline{98.00}$ | $\underline{67.69}$
C-3PO (oc=1, nd=1) | $\mathbf{100.00}$ | $\mathbf{84.59}$ | $\mathbf{100.00}$ | $\mathbf{90.13}$ | $\mathbf{100.00}$ | $\underline{71.53}$ | $\mathbf{100.00}$ | $\underline{64.70}$ | $\mathbf{100.00}$ | $\underline{82.90}$ | $\mathbf{100.00}$ | $\mathbf{69.07}$
DeepGMR | $0.00$ | $2.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
PointNetLK | $0.00$ | $2.76$ | $6.00$ | $7.06$ | $0.00$ | $0.00$ | $0.00$ | $0.00$ | $2.00$ | $5.14$ | $4.00$ | $5.27$
FPFH + TEASER++ | $26.00$ | $25.03$ | $26.00$ | $25.65$ | $50.00$ | $46.29$ | $42.00$ | $32.32$ | $38.00$ | $40.33$ | $48.00$ | $39.23$
EquiPose | $16.00$ | $19.69$ | $14.00$ | $20.33$ | – | – | – | – | – | – | – | –
KeyPo (real) | $\mathbf{100.00}$ | $\underline{79.08}$ | $\mathbf{100.00}$ | $\underline{79.92}$ | $\mathbf{100.00}$ | $\mathbf{80.28}$ | $\mathbf{100.00}$ | $\mathbf{83.11}$ | $\mathbf{100.00}$ | $\mathbf{84.02}$ | $54.00$ | $44.67$

Results. Figure 8(left, mid) show the rotation and translation error as a
function of the noise parameter $\sigma$; for the ShapeNet object chair. We
compare the output of the corrector + registration with a naive method that
applies only the registration block (eq. (12)) to the detected/perturbed
keypoints $\tilde{{\bm{y}}}$. Not all outputs produced by these two models
—naive and corrector— are observably correct per eq. (20). The figures also
plot the rotation and translation errors for the observably correct instances.
Figure 8(right) shows the fraction of _non_ -observably correct outputs
produced by the two methods. We conduct similar analysis for several other
ShapeNet objects in Section -F.

Insights. We observe that the corrector significantly improves the rotation
and translation error, viz-a-viz the keypoint registration used in standard
keypoint-based methods. A further performance boost is seen when we inspect
the observably correct, _i.e.,_ $\texttt{oc}=1$, instances produced by the
corrector. This boost is not only in the absolute reduction of the two error
metrics, but also in the reduction of the error variance. We observe a near-
constant error, w.r.t. increasing keypoint noise $\sigma$, for the observably
correct instances produced by the corrector. Finally, we note that the
fraction observably correct remains significantly high, when using the
corrector. For instance, more than $90\%$ of the outputs produced by the
corrector remain observably correct, for $\sigma<1$, whereas the naive method
fails to produce any observably correct output for $\sigma\geq 0.4$.

Note that the purpose of the corrector module in C-3PO is to ensure that there
is a non-negligible fraction of input-output instances that are observably
correct for self-supervised training, thus surmounting the sim-to-real gap.
This analysis indicates that the corrector can indeed help bridge the sim-to-
real gap.

###  VII-C The ShapeNet Experiment

This section demonstrates our self-supervised training on depth-point-cloud
data generated using ShapeNet [31] objects.

Setup. ShapeNet [31] classifies objects into 16 categories. We select one
object in each category and use the semantic keypoints labeled by the
KeypointNet dataset [72]. We use uniformly sampled point clouds of these 16
objects as the simulation dataset, and a collection of depth point clouds,
rendered using Open3D [100], as the real-world dataset. This choice ensures a
large sim-to-real gap, and enables us to showcase the utility of our self-
supervised training. We consider object displacements corresponding to the
_hard_ setup described in Section VII-A. We initially train the detector using
the simulation dataset, and self-train it on the depth point cloud dataset, as
discussed in Section VI. Hyper-parameter tuning and the implementation of the
non-degeneracy certificates is discussed in Section -G.

We report the following baselines in Table I: (i) KeyPO (sim): a simulation-
trained keypoint detector KeyPO; (ii) KeyPO (sim) + ICP: the simulation-
trained keypoint detector, with iterative closest point refinement using the
input ${\bm{X}}$ and the predicted point cloud $\hat{{\bm{X}}}$; (iii) KeyPO
(sim) + corrector, and (iv) KeyPO (real): the keypoint detector trained on the
real-wold dataset of depth point clouds, with full supervision. KeyPO (real),
therefore, marks an upper-bound on the performance of any self-supervised
method. C-3PO denotes the proposed method, obtained after the self-supervised
training on real-data. We use the point transformer regression model [93] for
the keypoint detector in KeyPO and C-3PO. We also compare the performance of
C-3PO against: (i) DeepGMR, (ii) PointNetLK, (iii) EquiPose: only in cases
where the trained model is available, and (iv) FPFH + TEASER++. Table I shows
the performance on 6 of the 16 objects. The remaining objects are reported in
Section -G. We evaluate the performance of each model using threshold ADD-S
and ADD-S (AUC) score [18]. We choose thresholds for the ADD-S and ADD-S (AUC)
to be $5\%$ and $10\%$ of the object diameter $d$.

Figure 9: Percentage of observably correct instances (20) vs. number of SGD
iterations during the proposed self-supervised training (Section VI).

Results. Figure 9 sheds light on the performance of the proposed self-
supervised approach and shows how the number of observably correct instances
in C-3PO increases with the number of SGD iterations during self-supervised
training, reaching close to 100% in most cases. In Fig. 9, there is a
noticeable difference in convergence time when the keypoint detector is
modeled as a point transformer regression model vs. PointNet++ regression
model. The point transformer regression model is consistently better than the
PointNet++ across all objects (see [99]).

Table I shows that KeyPO (sim) performs very poorly and is not helped much by
ICP; thereby, confirming the large sim-to-real gap in the experiment. KeyPO
(sim) + Corr., on the other hand, shows a significant performance improvement
in Table I; confirming that the corrector indeed helps bridge the sim-to-real
gap, to an extent. We, therefore, deduce that correcting for pose over the
space of keypoints is more powerful than doing so over the space of rotation
and poses, which is what the ICP attempts, thus validating Remark 12.

We observe that C-3PO is able to significantly outperform all the baselines,
and match the performance of the fully supervised KeyPO (real). A further
performance boost is attained by evaluating only the observably correct and/or
non-degenerate outputs produced. We see this in Table I, as well as in Figure
10, which shows the distribution of ADD-S scores, over the test dataset. The
reasons for very low scores of the baselines such as DeepGMR, PointNetLK, FPFH
+ TEASER++, and EquiPose remain the same as discussed in Section VII-A.

Degeneracy. Degeneracy arises when the depth point cloud of an object is
severely occluded, and there are multiple ways to fit the object model to it.
We observe this specifically in the case of two objects in the ShapeNet
dataset: mug and cap. For the mug, degeneracy occurs when the handle of the
mug is not visible, whereas for the cap, when the visor of the cap is not
visible (Fig. 1). Table II shows the ADD-S and ADD-S (AUC) scores attained by
C-3PO on these two objects (cap and mug). We observe that, while the self-
supervised training works correctly to reach $100\%$ and $82\%$ observably
correct instances for cap and mug, respectively, the ADD-S and ADD-S (AUC)
scores remain low. This is because the observably correct instances include
degenerate cases. This causes the predicted output to deviate from the ground
truth. Table II also shows the ADD-S and ADD-S AUC scores for $\texttt{oc}=1$
and $\texttt{oc}=\texttt{nd}=1$ instances produced by C-3PO; see Section -G
for implementation details of the non-degeneracy certificate. We observe that
the non-degeneracy check (_i.e.,_ $\texttt{nd}=1$) on observably correct
instances (_i.e.,_ $\texttt{oc}=1$) significantly improves the performance,
empirically validating Theorem 18.

###  VII-D The YCB Experiment

Figure 10: Distribution of ADD-S score for C-3PO and a few baselines in the ShapeNet Experiment. TABLE II: C-3PO w/wo observable correctness (20) and non-degeneracy certificates (23). object | C-3PO | ADD-S | ADD-S AUC | percent
---|---|---|---|---
cap | all | 78.00 | 63.87 | 100.00
oc =1 | 78.00 | 63.87 | 100.00
oc, nd =1 | 100.00 | 80.11 | 32.00
mug | all | 68.00 | 53.55 | 100.00
oc =1 | 82.93 | 60.63 | 82.00
oc, nd =1 | 100.00 | 70.06 | 22.00
cracker box | all | 68.00 | 54.24 | 100.00
oc =1 | 69.39 | 55.35 | 98.00
oc, nd =1 | 100.00 | 81.08 | 42.00
TABLE III: Evaluation of C-3PO and baselines for the YCB experiment. ADD-S ADD-S (AUC) | chips can | mustard bottle | banana | scissors | extra large clamp | cracker box
---|---|---|---|---|---|---
KeyPo (sim) | $0.00$ | $3.84$ | $0.00$ | $15.15$ | $2.00$ | $27.29$ | $34.00$ | $53.22$ | $0.00$ | $11.73$ | $0.00$ | $2.00$
KeyPo (sim) + ICP | $0.00$ | $4.47$ | $0.00$ | $13.97$ | $4.00$ | $26.09$ | $36.00$ | $52.55$ | $2.00$ | $10.79$ | $0.00$ | $2.00$
KeyPo (sim) + Corr. | $50.00$ | $41.68$ | $40.00$ | $48.63$ | $72.00$ | $68.06$ | $92.00$ | $80.59$ | $54.00$ | $53.14$ | $38.00$ | $36.18$
C-3PO | $98.00$ | $78.48$ | 100.00 | $\underline{84.04}$ | 100.00 | $85.97$ | 100.00 | $84.42$ | 100.00 | 84.22 | $68.00$ | $54.24$
C-3PO (oc=1, nd=1) | 100.00 | $\underline{78.78}$ | 100.00 | 85.05 | 100.00 | $\underline{87.34}$ | 100.00 | $\underline{86.77}$ | 100.00 | 84.80 | 100.00 | 81.08
DeepGMR | $0.00$ | $7.62$ | $0.00$ | $16.51$ | $2.50$ | $30.49$ | $14.14$ | $35.64$ | $0.76$ | $7.06$ | $0.00$ | $0.00$
PointNetLK | $0.00$ | $0.00$ | $0.00$ | $3.72$ | $22.17$ | $28.37$ | $7.07$ | $8.08$ | $0.00$ | $0.00$ | $0.00$ | $0.00$
FPFH + TEASER++ | $0.17$ | $5.90$ | $0.33$ | $13.23$ | $4.83$ | $24.43$ | $47.81$ | $50.07$ | $5.29$ | $13.20$ | $0.17$ | $4.91$
KeyPo (real) | 100.00 | 83.67 | 100.00 | $74.12$ | 100.00 | 89.92 | 100.00 | 88.51 | $98.00$ | $82.06$ | $\underline{98.00}$ | $\underline{79.34}$

This section shows that the proposed self-training method allows bridging the
sim-to-real gap in the YCB dataset [32].

Setup. The YCB dataset [32] includes RGB-D images of household objects on a
tabletop. The dataset provides Poisson-reconstructed meshes, which we use as
object models ${\bm{B}}$. We manually annotate semantic keypoints on each
model. For the simulation data, we use uniformly sampled point clouds on
${\bm{B}}$, and for the real-world data we use the segmented depth point
clouds extracted from the RGB-D images in the dataset.

We use the same baselines as in the ShapeNet experiment, except EquiPose. This
is because a trained EquiPose models are not available for the category-less
YCB objects. We evaluate the performance of each model using threshold ADD-S
and ADD-S (AUC) score [18]. We choose thresholds for the ADD-S and ADD-S (AUC)
to be $2$cm and $5$cm.

Results and Insights. Table III shows the results for 6 of the 21 YCB objects.
The remaining objects are reported in Section -G. Even though the YCB dataset
is generated using real objects and sensors (unlike the dataset in the
ShapeNet experiment), the key results and insights in Section VII-C hold. We
also see a large sim-to-real gap, and that the KeyPO (sim) and KeyPO (sim) +
ICP do not yield good performance. The corrector helps to an extent to bridge
the sim-to-real gap, and provide a respectable, initial fraction of observably
correct instances for self-supervised training. We observe that the self-
supervised training works, as the observable correct instances increase with
training iterations. C-3PO significantly outperforms all the baseline
approaches, and attains performance close to the supervised baseline KeyPO
(real). We observe several degenerate cases (see cracker box in Table III),
but our implemented non-degeneracy certificate (see Section -G) is able to
filter them and boost performance: see Table II. This, again, validates
Theorem 18.

###  VII-E Learning without Object Category Labels

As a final result, we show that C-3PO’s self-training works even when the data
does not contain object category labels.

Setup. We use the depth point clouds from the ShapeNet and YCB experiments
(see Sections VII-C and VII-D). We omit the object category labels and create
two mixed datasets. The first one contains five ShapeNet objects (table,
chair, bottle, laptop, skateboard) and the second contains five YCB objects
(master chef can, mustard bottle, banana, scissors, extra large clamp). We
train C-3PO for each object in the dataset. We set
$\epsilon_{\texttt{oc}}=0.0316d$ where $d$ is the object diameter.

TABLE IV: Percentage of observably correct instances (_i.e.,_ $\%\texttt{oc}=1$) for the sim-trained C-3PO, for object A (row), evaluated on the depth point cloud dataset for object B (column) $\%$ obs. correct | table | chair | bottle | laptop | skateboard
---|---|---|---|---|---
C-3PO (table) | 74.00 | $00.00$ | $00.00$ | $00.00$ | $00.00$
C-3PO (chair) | $00.00$ | 58.00 | $00.00$ | $00.00$ | $00.00$
C-3PO (bottle) | $00.00$ | $00.00$ | 98.00 | $00.00$ | $00.00$
C-3PO (laptop) | $00.00$ | $00.00$ | $00.00$ | 06.00 | $00.00$
C-3PO (skateboard) | $00.00$ | $00.00$ | $00.00$ | $00.00$ | 52.00

Results and Insights. A typical pose estimation method produces object poses
without any certificates. The use of certificates in C-3PO helps it
distinguish not only if the estimated pose is correct, but also if the input
is a partial point cloud of the same object. Table IV shows C-3PO (A) (for
instance, “C-3PO (table)”), trained to estimates pose of object A in sim,
produces $\texttt{oc}=0$ when it sees other objects (_i.e.,_ chair, bottle,
laptop, skateboard). Since we use observably correct instances to self-
supervise (Section VI), this indicates that the training will ignore pose
estimates produced on other objects.

Table V indeed shows that also in this case the self-supervised training in
Section VI succeeds. In particular, Table V evaluates C-3PO (chair), trained
to estimate the pose of a chair, on a mixed dataset containing depth point
clouds of tables, chairs, bottles, laptop, and skateboard. We use the same
ADD-S and ADD-S (AUC) thresholds as in Section VII-C. We see that C-3PO
(chair) is able to attain performance close to C-3PO and the supervised
baseline KeyPO (real) in Table I. We get similar success when training and
evaluating C-3PO for other ShapeNet and YCB objects.

TABLE V:  Cross evaluation of C-3PO (chair) trained on the mixed ShapeNet dataset to estimate pose of the chair object.  | table | chair | bottle | laptop | skateboard
---|---|---|---|---|---
ADD-S | $00.00$ | 100.00 | $00.00$ | $00.00$ | $00.00$
ADD-S (AUC) | $00.00$ | 83.73 | $00.00$ | $00.00$ | $00.00$

##  VIII Conclusion

We introduced the problem of certifiable pose estimation from partial point
clouds, and defined the notion of $\zeta$-correctness. We developed a theory
to certify end-to-end perception models, and showed that implementing two
certificates —observable correctness and non-degeneracy— provides a way to
ascertain $\zeta$-correctness of the resulting pose estimates. We proposed
C-3PO, a certifiable model, that implements the two certificates derived using
our theory. C-3PO relies on a semantic keypoint-based architecture, and also
implements a keypoint corrector module, which we saw to be more effective at
correcting pose errors than traditional methods like ICP. Finally, we
introduced a self-supervised training procedure, that leverages the
certificate of observable correctness to provide a supervisory signal and
enable self-training on unlabeled real data. Our experiments show that (i)
standard semantic-keypoint-based methods (which constitute the backbone of
C-3PO) outperform more recent alternatives in challenging problem instances,
(ii) C-3PO further improves performance and significantly outperforms all the
baselines, (iii) C-3PO’s certificates are able to discern correct pose
estimates.

This work opens many avenues for future investigation. The success of the
semantic-keypoint-based architecture for pose estimation indicates the need to
explore in-category generalization power of a semantic keypoint detector. We
note here that self-supervised discovery of semantic keypoints on CAD models
has been shown in [73], thus obviating the need for hand-annotation of
semantic keypoints. Self-supervised, category-level keypoint discovery is a
problem, which, once solved, can extend C-3PO to solve category-level object
pose estimation. Secondly, the theory of certifiable models developed in this
work remains to be applied to other problems. We believe this can pave the way
towards certifying end-to-end perception pipelines. Finally, we believe that
two other ideas presented in this paper, namely the corrector and the self-
supervised training approach, can be extended and applied to other perception
problems and deserve further investigation.

###  -A Appendix: Hausdorff Metric for Certification

The $\zeta$-correctness of a pose estimate is defined in terms of the
Hausdorff distance between the estimated $\hat{{\bm{T}}}\cdot{\bm{B}}$ and the
ground-truth ${\bm{T}}^{\ast}\cdot{\bm{B}}$ posed CAD models. Here we address
the question of why we chose the Hausdorff distance, as opposed to other
metrics that are more common in the pose estimation literature, like the pose
error or the ADD-S score [18].

Pose Error. A common choice of metrics is to use rotation and translation
error between the estimated pose $\hat{{\bm{T}}}$ and the ground truth
${\bm{T}}^{\ast}$. The translation error is typically computed as the
Euclidean distance between the estimated and ground-truth translation. Several
alternatives exist for measuring the rotation error [101, 102]. Two popular
options are (i) the angular distance, (ii) the chordal distance, which
corresponds to the Frobenius norm of
${\bm{I}}-\hat{{\bm{R}}}^{\mathsf{T}}{\bm{R}}^{\ast}$. While these metrics are
appealing, they cause problems for symmetric objects. For a symmetric object,
there exist at least two different poses from which the object is identical,
_i.e.,_ ${\bm{T}}\cdot{\bm{B}}={\bm{B}}$ for some ${\bm{T}}\in\mathrm{SE}(3)$.
Using rotation and translation error, therefore, necessitates us to first
address the question of such equivalent poses, for each object, thus adding
complexity and getting in the way of our analysis.

ADD-S Metric. A popular metric for evaluating pose estimation is the ADD-S
metric [18]. It is defined as the Chamfer loss between the estimated
$\hat{{\bm{T}}}\cdot\hat{{\bm{B}}}$ and the ground-truth
${\bm{T}}^{\ast}\cdot\hat{{\bm{B}}}$ posed CAD models. Unlike the metric in
(10), ADD-S is computed on sampled point clouds. The difference between such a
metric and the Hausdorff distance metric in (10) is that the latter computes
the worst-case distance from a point on $\hat{{\bm{T}}}\cdot{\bm{B}}$ to the
surface ${\bm{T}}^{\ast}\cdot{\bm{B}}$, and vice-versa. The ADD-S metric, on
the other hand, operates on averages, which can be problematic. An average
metric, like ADD-S, tends to average out and potentially miss high errors in a
few parts of the object, which gets in the way of certification. The Hausdorff
distance metric, using “max” instead of “average”, is able to single out such
instances.

###  -B Appendix: Certifiable Models

Proof of Theorem 4. The result follows from the definitions. Recall that the
ground truth ${\bm{Z}}^{\ast}$ generates the input data ${\bm{X}}$, hence
—according to (2)— it belongs to the solution space: ${\bm{Z}}^{\ast}\in{\cal
S}({\bm{X}})$. Now assume that, for an estimate ${\bm{Z}}={\cal M}({\bm{X}})$,
both the certificate of observable correctness (3) and non-degeneracy (4) are
equal to $1$. This implies (i) ${\bm{Z}}\in{\cal S}({\bm{X}})$ and (ii)
$\text{Diam}[{\cal S}({\bm{X}})]<\delta$. Now, (ii) implies that
$d_{\mathbb{{\bm{Z}}}}({\bm{Z}}^{\prime},{\bm{Z}}^{\prime\prime})<\delta$ for
any ${\bm{Z}}^{\prime},{\bm{Z}}^{\prime\prime}\in{\cal S}({\bm{X}})$. Setting
${\bm{Z}}^{\prime}={\bm{Z}}$ and ${\bm{Z}}^{\prime\prime}={\bm{Z}}^{\ast}$,
and using the assumption $\delta\leq\zeta$, we obtain the desired result.

Proof of Theorem 5. Since $\overline{{\cal S}}({\bm{X}})$ is an outer
approximation of the solution set (_i.e.,_ ${\cal
S}({\bm{X}})\subseteq\overline{{\cal S}}({\bm{X}})$) and
${\bm{Z}}^{\ast}\in{\cal S}({\bm{X}})$, it follows
${\bm{Z}}^{\ast}\in\overline{{\cal S}}({\bm{X}})$. Moreover, since
$\texttt{oc}({\cal M},{\bm{X}})=1$, we also know that
${\bm{Z}}\in\overline{{\cal S}}({\bm{X}})$. Now, $\texttt{nd}({\cal
M},{\bm{X}})=1$ implies that
$d_{\mathbb{{\bm{Z}}}}({\bm{Z}}^{\prime},{\bm{Z}}^{\prime\prime})<\delta$ for
all ${\bm{Z}}^{\prime},{\bm{Z}}^{\prime\prime}\in\overline{{\cal
S}}({\bm{X}})$. Putting these three conclusion together, and using the
assumption $\delta\leq\zeta$, we obtain the desired result.

###  -C Appendix: Observable Correctness Certificate (20) as an Outer
Approximation

The observable correctness certificate (20) is given by

$\texttt{oc}(\hat{{\bm{Z}}},{\bm{X}})=\mathbb{I}\left\\{\max_{i\in[n]}\min_{j\in[m]}\lVert{\bm{X}}[i]-\hat{{\bm{T}}}\cdot\hat{{\bm{B}}}[j]\rVert_{2}<\epsilon_{\texttt{oc}}\right\\},$
(26)

where we have used the fact that
$\hat{{\bm{X}}}=\hat{{\bm{T}}}\cdot\hat{{\bm{B}}}$ in (20). Note that we use
the densely sampled CAD model $\hat{{\bm{B}}}$ (instead of ${\bm{B}}$) to
implement the certificate, while the definition of $\zeta$-correctness is with
respect to the posed CAD model ${\bm{B}}$. For mathematical analysis, we also
consider the following certificate

$\texttt{oc}^{\prime}(\hat{{\bm{Z}}},{\bm{X}})=\mathbb{I}\left\\{\max_{i\in[n]}D({\bm{X}}[i],\hat{{\bm{T}}}\cdot{\bm{B}})<\epsilon_{\texttt{oc}^{\prime}}\right\\}.$
(27)

Note that (27) is an “idealized” version of the certificate in (20), as it
computes the exact distance $D({\bm{X}}[i],{\bm{T}}\cdot{\bm{B}})$ from any
point in the point cloud ${\bm{X}}$ to the posed CAD model
${\bm{T}}\cdot{\bm{B}}$. We prove the following.

######  Theorem 21 (Outer Approximation).

Let $\overline{{\cal S}}({\bm{X}})$ be given by

$\overline{{\cal
S}}({\bm{X}})=\left\\{({\bm{T}}\cdot{\bm{b}},{\bm{T}}\cdot{\bm{B}})~{}\Big{|}~{}\max_{i\in[n]}D({\bm{X}}[i],{\bm{T}}\cdot{\bm{B}})<\epsilon_{\texttt{oc}^{\prime}}\right\\}.$

Assuming bounded noise (Assumption 16) and dense CAD model sampling
(Assumption 17), we have

  1. (i)

$\overline{{\cal S}}({\bm{X}})$ is an outer approximation of the solution
space ${\cal S}({\bm{X}})$, provided
$\epsilon_{w}<\epsilon_{\texttt{oc}^{\prime}}$.

  2. (ii)

$\texttt{oc}^{\prime}({\bm{Z}},{\bm{X}})\leq\texttt{oc}({\bm{Z}},{\bm{X}})$,
provided $\epsilon_{w}+\epsilon_{s}<\epsilon_{\texttt{oc}}$.

###### Proof:

(i) We first show that $\overline{{\cal S}}({\bm{X}})$ is an outer
approximation to the solution space ${\cal S}({\bm{X}})$. Recall that, given a
generative model $\phi:\mathbb{{\bm{Z}}}\rightarrow\mathbb{{\bm{X}}}$, the
solution space ${\cal S}({\bm{X}})$ is given by

${\cal
S}({\bm{X}})=\left\\{{\bm{Z}}\in\mathbb{{\bm{Z}}}~{}|~{}\phi({\bm{Z}})={\bm{X}}\right\\}.$
(28)

For the certifiable pose estimation problem, we have the generative model
(13):

${\bm{X}}=\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)+{\bm{n}}_{w}.$ (29)

Let a tuple
$({\bm{T}}\cdot{\bm{b}},{\bm{T}}\cdot{\bm{B}})\in\mathbb{{\bm{Z}}}$ be in the
solution space ${\cal S}({\bm{X}})$. This implies that there exists a
realization of the noise ${\bm{n}}^{{}^{\prime}}_{w}$ such that
${\bm{X}}-{\bm{n}}^{{}^{\prime}}_{w}=\Theta\left({\bm{T}}\cdot{\bm{B}}\right)$,
which is equivalent to saying that
${\bm{X}}-{\bm{n}}^{{}^{\prime}}_{w}\subseteq{\bm{T}}\cdot{\bm{B}}$. We can
write this as

$D({\bm{X}}[i]-{\bm{n}}^{{}^{\prime}}_{w}[i],{\bm{T}}\cdot{\bm{B}})=0,~{}\forall~{}i\in[n]$
(30)

The equation simply states that the output, up to noise, is a rigid
transformation of points in the CAD model. Since we assume bounded noise (16),
we can write (30) as

$\max_{i\in[n]}D({\bm{X}}[i],{\bm{T}}\cdot{\bm{B}})<\epsilon_{w},$ (31)

after making use of the triangle inequality. From this, we know that whenever
$({\bm{T}}\cdot{\bm{b}},{\bm{T}}\cdot{\bm{B}})\in{\cal S}({\bm{X}})$ we must
have (31), which is equivalent to saying
$({\bm{T}}\cdot{\bm{b}},{\bm{T}}\cdot{\bm{B}})\in\overline{{\cal
S}}({\bm{X}})$ with $\epsilon_{w}<\epsilon_{\texttt{oc}^{\prime}}$.

(ii) We now show
$\texttt{oc}^{\prime}({\bm{Z}},{\bm{X}})\leq\texttt{oc}({\bm{Z}},{\bm{X}})$,
provided $\epsilon_{w}+\epsilon_{s}<\epsilon_{\texttt{oc}}$. It suffices to
prove that whenever $\texttt{oc}^{\prime}({\bm{Z}},{\bm{X}})=1$ we have
$\texttt{oc}({\bm{Z}},{\bm{X}})=1$. We do this by showing

$\max_{i\in[n]}D({\bm{X}}[i],{\bm{T}}\cdot{\bm{B}})<\epsilon_{w}\\\
\implies\max_{i\in[n]}\min_{j\in[m]}\lVert{\bm{X}}[i]-\hat{{\bm{X}}}[j]\rVert_{2}<\epsilon_{\texttt{oc}},$
(32)

provided $\epsilon_{w}+\epsilon_{s}<\epsilon_{\texttt{oc}}$.

Let the left-hand side (LHS) of (32) hold for a ${\bm{T}}$. We note that

$\lVert{\bm{X}}[i]-\hat{{\bm{X}}}[j]\rVert_{2}\leq\lVert{\bm{X}}[i]-{\bm{x}}\rVert_{2}+\lVert{\bm{x}}-\hat{{\bm{X}}}[j]\rVert_{2},$
(33)

for a ${\bm{x}}\in{\bm{T}}\cdot{\bm{B}}$ such that
$\lVert{\bm{X}}[i]-{\bm{x}}\rVert_{2}<\epsilon_{w}$; which is possible due the
LHS of (32). Taking minimum over $j\in[m]$, we obtain

$\min_{j\in[m]}\lVert{\bm{X}}[i]-\hat{{\bm{X}}}[j]\rVert_{2}<\epsilon_{w}+\epsilon_{s},$
(34)

after applying the CAD model sampling 17. This is nothing but the right-hand
side of (32), provided $\epsilon_{w}+\epsilon_{s}<\epsilon_{\texttt{oc}}$. ∎

######  Remark 22.

We remark here that Theorems 21 and 5 imply that the implemented observable
correctness certificate is an upper-bound, _i.e.,_

$\texttt{ObsCorrect}[{\cal M},{\bm{X}}]\leq\texttt{oc}({\bm{Z}}={\cal
M}({\bm{X}}),{\bm{X}}),$ (35)

under Assumptions 16-17 and
$\epsilon_{\texttt{oc}}>\epsilon_{w}+\epsilon_{s}$.

###  -D Appendix: Proof of Theorem 18

We prove the result in three parts: (i) we first prove that if the certificate
of non-degeneracy holds, then the posed keypoints cannot be far from the posed
ground-truth CAD model; (ii) if the certificate of observable correctness
holds, then the posed CAD model cannot be far away from the posed and occluded
ground-truth CAD model; and finally (iii) we use these two to show that the
estimated pose $\hat{{\bm{T}}}$ is $\zeta$-correct.

(i): Let $\hat{{\bm{Z}}}=(\hat{{\bm{y}}},\hat{{\bm{X}}})={\cal M}({\bm{X}})$
be the model output as in (13). Let $\texttt{nd}(\hat{{\bm{Z}}},{\bm{X}})=1$.
Then, there exists a $\textsf{G}_{l}$ such that

$\min_{j\in[n]}\lVert\hat{{\bm{y}}}[i]-{\bm{X}}[j]\rVert_{2}<\delta_{\texttt{nd}},~{}\forall~{}i\in\textsf{G}_{l}.$
(36)

Now, ${\bm{X}}=\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)+{\bm{n}}_{w}$.
This implies

$\displaystyle\lVert\hat{{\bm{y}}}[i]-\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)[j]\rVert_{2}$
$\displaystyle\leq\lVert\hat{{\bm{y}}}[i]-{\bm{X}}[j]\rVert_{2}+\lVert{\bm{n}}_{w}[j]\rVert_{2},$
(37)
$\displaystyle\leq\lVert\hat{{\bm{y}}}[i]-{\bm{X}}[j]\rVert_{2}+\epsilon_{w},$
(38)

where we used the bounded noise 16. Now, eqs. (38) and (36) imply

$\min_{j\in[n]}\lVert\hat{{\bm{y}}}[i]-\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)[j]\rVert_{2}<\delta_{\texttt{nd}}+\epsilon_{w},~{}\forall~{}i\in\textsf{G}_{l}.$
(39)

This proves that the posed keypoints
$\hat{{\bm{y}}}=\hat{{\bm{T}}}\cdot{\bm{b}}$, _cf._ (13), are not far away
from the posed ground-truth CAD model. We use
$\hat{{\bm{y}}}=\hat{{\bm{T}}}\cdot{\bm{b}}$, _cf._ (13), to write (39) as

$\min_{j\in[n]}\lVert{\bm{b}}[i]-{\bm{C}}[j]\rVert_{2}<\delta_{\texttt{nd}}+\epsilon_{w},~{}\forall~{}i\in\textsf{G}_{l},$
(40)

where
${\bm{C}}=\hat{{\bm{T}}}^{-1}\cdot\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)$.

(ii): We also have $\texttt{oc}(\hat{{\bm{Z}}},{\bm{X}})=1$. This implies

$\min_{i\in[m]}\lVert{\bm{X}}[j]-\hat{{\bm{X}}}[i]\rVert_{2}<\epsilon_{\texttt{oc}},~{}\forall~{}j\in[n].$
(41)

Again, using the fact that
${\bm{X}}=\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)+{\bm{n}}_{w}$, and
that the noise ${\bm{n}}_{w}$ is bounded (16), we can show

$\min_{i\in[m]}\lVert\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)[j]-\hat{{\bm{X}}}[i]\rVert_{2}<\epsilon_{\texttt{oc}}+\epsilon_{w},~{}\forall~{}j\in[n].$
(42)

Recall that $\hat{{\bm{X}}}=\hat{{\bm{T}}}\cdot\hat{{\bm{B}}}$ and consider
the posed CAD model $\hat{{\bm{T}}}\cdot{\bm{B}}$. Using (42), and the fact
that every point in $\hat{{\bm{X}}}$ is in the set
$\hat{{\bm{T}}}\cdot{\bm{B}}$, we get

$D(\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)[j],\hat{{\bm{T}}}\cdot{\bm{B}})<\epsilon_{\texttt{oc}}+\epsilon_{w},~{}\forall~{}j\in[n].$
(43)

Substituting
${\bm{C}}=\hat{{\bm{T}}}^{-1}\cdot\Theta\left({\bm{T}}^{\ast}\cdot{\bm{B}}\right)$
in (43) we get

$D({\bm{C}},{\bm{B}})<\epsilon_{\texttt{oc}}+\epsilon_{w},~{}\forall~{}j\in[n].$
(44)

This implies that there exists an occlusion and sampling function
$\Theta^{\prime}$ such that

$d_{H}({\bm{C}},\Theta^{\prime}\left({\bm{B}}\right))<\epsilon_{\texttt{oc}}+\epsilon_{w}.$
(45)

Such a point cloud $\Theta^{\prime}\left({\bm{B}}\right)$ can be obtained by
only sampling points on ${\bm{B}}$ that are closest to every point in
${\bm{C}}$.

(iii): From (40) and (45), we obtain

$\min_{j}\lVert{\bm{b}}[i]-\Theta^{\prime}\left({\bm{B}}\right)[j]\rVert_{2}<\epsilon_{\texttt{oc}}+\delta_{\texttt{nd}}+2\epsilon_{w},$
(46)

for all $i\in\textsf{G}_{l}$, using the triangle inequality. This means that
all the keypoints in the indicator set $\textsf{G}_{l}$ are close to the non-
occluded part of $\Theta^{\prime}\left({\bm{B}}\right)$. Since
$\textsf{G}_{l}$ is an indicator set (Definition 15), we have that
$\Theta^{\prime}\left({\bm{B}}\right)$ is a cover set of ${\bm{B}}$, provided

$\epsilon_{\texttt{oc}}+\delta_{\texttt{nd}}+2\epsilon_{w}<\delta_{ind}.$ (47)

Now, (45) also implies that
$D(\Theta^{\prime}\left({\bm{B}}\right)[j],{\bm{C}})<\epsilon_{\texttt{oc}}+\epsilon_{w}$,
from which we can deduce

$D(\Theta^{\prime}\left({\bm{B}}\right)[j],\hat{{\bm{T}}}^{-1}\cdot{\bm{T}}^{\ast}\cdot{\bm{B}})<\epsilon_{\texttt{oc}}+\epsilon_{w},$
(48)

for all $j$. Knowing that $\Theta^{\prime}\left({\bm{B}}\right)$ is a cover
set, this gives us
$d_{H}({\bm{B}},\hat{{\bm{T}}}^{-1}\cdot{\bm{T}}^{\ast}\cdot{\bm{B}})<\epsilon_{\texttt{oc}}+\epsilon_{w}$,
and therefore,

$d_{H}(\hat{{\bm{T}}}\cdot{\bm{B}},{\bm{T}}^{\ast}\cdot{\bm{B}})<\epsilon_{\texttt{oc}}+\epsilon_{w},$
(49)

which proves the result. This proves that the estimate produced
$\hat{{\bm{T}}}$ is $\zeta$-correct for any
$\zeta>\epsilon_{\texttt{oc}}+\epsilon_{w}$.

###  -E Appendix: Corrector

Batch Gradient Descent.

Figure 11: Compute time (sec) for solving (16) per data point in a batched
input, as a function of the batch size.

Figure 11 shows a comparison of the compute time advantage of batch gradient
descent, as opposed to using the SciPy optimization solver. The figure plots
the time taken by a solver to output the solution to (16), per data point in
the batch, and a function of the total batch size. We see that with increasing
batch size, the implemented batch gradient descent yields a compute time
advantage over the SciPy solver and over using batch size of one.

######  Remark 23 (Batch Solvers for Differentiable Optimization Layers).

We remark here that any differentiable optimization layer require batch
solvers to enable faster training. We observe that implementations using
existing solvers, and iterating over data points in the batch, results in
slower training times; whereas PyTorch solvers (_e.g.,_ SDG, ADAM) are not
suited to solve non-linear optimization problems by design.

Proof of Theorem 11. Note that the only places where $\Delta{\bm{y}}$ appears
in the corrector is in conjunction with $\tilde{{\bm{y}}}$, and that too, in
the form $\tilde{{\bm{y}}}+\Delta{\bm{y}}$ (see (12), (16)). The corrector
optimization problem can therefore be written as:

$\displaystyle\min_{\Delta{\bm{y}}}$ $\displaystyle
f(\tilde{{\bm{y}}}+\Delta{\bm{y}}),$ (50)

for some function $f$, obtained by subsuming the constraints in (16) into the
objective function and realizing that ${\bm{R}}(\Delta{\bm{y}})$ and
${\bm{t}}(\Delta{\bm{y}})$, can in fact be written as
${\bm{R}}(\tilde{{\bm{y}}}+\Delta{\bm{y}})$ and
${\bm{t}}(\tilde{{\bm{y}}}+\Delta{\bm{y}})$.

Let ${\bm{y}}^{\ast}$ be a solution of the optimization problem
$\min_{{\bm{y}}}f({\bm{y}})$, for $f$ in (50). Then, the optimal correction
will be $\Delta{\bm{y}}^{\ast}={\bm{y}}^{\ast}-\tilde{{\bm{y}}}$. Taking
derivative with respect to $\tilde{{\bm{y}}}$ gives the result.

TABLE VI: Comparing the keypoint detector architectures used in C-3PO: PointNet++ vs Point Transformer. Object | ADD-S | ADD-S AUC | % oc=1 | keypoint detector
---|---|---|---|---
airplane | 100.00 | 87.76 | 100.00 | Point Transformer
58.00 | 63.35 | 54.00 | PointNet++
bathtub | 72.00 | 57.71 | 60.00 | Point Transformer
54.00 | 50.40 | 50.00 | PointNet++
bed | 56.00 | 52.68 | 0.00 | Point Transformer
34.00 | 38.14 | 0.00 | PointNet++
bottle | 100.00 | 70.35 | 100.00 | Point Transformer
100.00 | 68.41 | 100.00 | PointNet++
cap | 78.00 | 63.87 | 100.00 | Point Transformer
60.00 | 47.55 | 82.00 | PointNet++
car | 80.00 | 64.00 | 6.00 | Point Transformer
46.00 | 44.53 | 0.00 | PointNet++
chair | 100.00 | 79.06 | 12.00 | Point Transformer
100.00 | 73.44 | 2.00 | PointNet++
guitar | 100.00 | 83.77 | 28.00 | Point Transformer
100.00 | 80.03 | 0.00 | PointNet++
helmet | 100.00 | 70.49 | 92.00 | Point Transformer
60.00 | 47.78 | 32.00 | PointNet++
knife | 100.00 | 86.94 | 86.00 | Point Transformer
100.00 | 80.28 | 20.00 | PointNet++
laptop | 100.00 | 64.70 | 100.00 | Point Transformer
82.00 | 56.67 | 86.00 | PointNet++
motorcycle | 100.00 | 75.02 | 18.00 | Point Transformer
98.00 | 71.33 | 0.00 | PointNet++
mug | 68.00 | 53.55 | 82.00 | Point Transformer
32.00 | 37.38 | 52.00 | PointNet++
skateboard | 100.00 | 82.90 | 100.00 | Point Transformer
96.00 | 79.66 | 96.00 | PointNet++
table | 98.00 | 67.69 | 98.00 | Point Transformer
100.00 | 64.61 | 100.00 | PointNet++
vessel | 100.00 | 78.21 | 80.00 | Point Transformer
80.00 | 63.23 | 20.00 | PointNet++
TABLE VII: C-3PO w/wo observable correctness and non-degeneracy certificates. object | C-3PO | ADD-S | ADD-S AUC | percent
---|---|---|---|---
sugar box | all | 58.00 | 52.03 | $\mathbf{100.00}$
oc =1 | 58.33 | 52.57 | 96.00
oc, nd =1 | 100.00 | 87.43 | 50.00
pudding box | all | 50.00 | 53.82 | 100.00
oc =1 | 26.09 | 36.52 | 46.00
oc, nd =1 | 100.00 | 89.73 | 8.00
gelatin box | all | 82.00 | 78.07 | 100.00
oc =1 | 73.53 | 74.41 | 68.00
oc, nd =1 | 100.00 | 89.26 | 42.00
potted meat can | all | 54.00 | 72.17 | 100.00
oc =1 | 61.90 | 75.86 | 84.00
oc, nd =1 | 100.00 | 88.81 | 36.00
power drill | all | 86.00 | 81.42 | 100.00
oc =1 | 97.22 | 85.53 | 72.00
oc, nd =1 | 100.00 | 87.13 | 56.00
wood block | all | 62.00 | 50.17 | 100.00
oc =1 | 48.28 | 39.85 | 58.00
oc, nd =1 | $\mathbf{100.00}$ | 83.11 | 22.00
foam brick | all | 98.00 | 85.33 | 100.00
oc =1 | 96.67 | 85.74 | 60.00
oc, nd =1 | 100.00 | 89.68 | 42.00

###  -F Appendix: Keypoint Corrector Analysis Experiment

We study the performance of the corrector across 16 object categories in the
ShapeNet dataset [31]. We randomly pick an object in each category and perform
the ablation study described in Section VII-B. Figures 13, 14, 15, and 16 plot
ADD-S, normalized to the object diameter, and percent _not_ observably
correct, as a function of the noise variance parameter $\sigma$. The sub-
figures (c) visualizes an instance of keypoint perturbation at $\sigma=0.8$.
In all cases we observe that the corrector is able to correct added noise to
the keypoints, even when $80\%$ of the keypoints are randomly perturbed by a
measure proportional to the objects’ diameter. We observe a reduction in the
variance of the ADD-S score and improved fraction of observaly correct
instances, across all object categories. The insights noted in Section VII-B
hold across all the objects we analyzed.

###  -G Appendix: The ShapeNet and YCB Experiment

ShapeNet Data Preparation. ShapeNet [31] classifies objects into 16
categories. We select one object in each category and use the semantic
keypoints labeled by the KeypointNet dataset [72]. We use uniformly sampled
point clouds of these 16 objects as the simulation dataset and a collection of
depth point clouds, rendered using Open3D [100], as the real-world dataset.
This choice ensures a large sim-to-real gap, and enables us to showcase the
utility of our self-supervised training. We initially train the detector using
the simulation dataset, and self-train it on the depth point cloud dataset, as
proposed in Section VI. For the real-world, depth point cloud data, we
generate $5000$ point clouds for training, $50$ point clouds each for
validation and testing, respectively. We note that the pose error induced is
equal to the _hard_ case described in Section VII-A.

YCB Data Preparation. The YCB dataset [32] includes RGB-D images of household
objects on a tabletop. The dataset provides Poisson-reconstructed meshes,
which we use as object models ${\bm{B}}$. We manually annotate semantic
keypoints on each model. We choose a subset of 19 objects that span across
varied shapes and sizes within the dataset, and use the Poisson reconstructed
meshes as the object model ${\bm{B}}$. We manually annotated semantic
keypoints on each of the 19 objects, using Open3D’s visualizer interface
[100]. For the simulation data, we use uniformly sampled point clouds on
${\bm{B}}$, and for the real-world data we use the segmented depth point
clouds extracted from the RGB-D images in the dataset. The number of depth
point clouds extracted varies across the YCB objects. We generate the
train/test/val data using a $80/20/20$ split.

To increase the train dataset size and to ensure that the learned model is
robust to small changes in the point cloud, we perform data augmentation on
the train and validation data. For each depth point cloud, in the train and
validation set, we remove a randomly selected $10\%$ of the points, and
perturb the rest with Gaussian noise, with a standard deviation of 0.001m. We
also discarded point clouds with less than 500 points.

(a) cap

(b) mug

(c) 004 sugar box

(d) 019 pitcher base

(e) 035 power drill

Figure 12: Annotated keypoints and keypoint indices for ShapeNet objects (a)
cap, (b) mug and for YCB objects (c) 004 sugar box, (d) 019 pitcher base, and
(e) 035 power drill.

Non-degeneracy Certificate. In this appendix, we discuss the implementation of
the non-degeneracy certificates and the choice of indicator sets, used in the
experiments. We note that deriving indicator sets to provably meet the
specifications in Definition 15 is hard and we do not choose that route. We,
instead, hand-craft the indicator sets and show its usefulness towards
correctly certifying C-3PO’s pose estimates.

For most of the ShapeNet and YCB objects, we choose the indicator set to be
empty, _i.e.,_ $\textsf{G}_{l}=\emptyset$, implying that
$\texttt{nd}(\hat{{\bm{Z}}},{\bm{X}})=1$ always. This is because any depth
point cloud of the object turns out to be a valid cover set, and therefore,
can uniquely identify the object pose (see (21)). However, for some objects,
this turns out not to be the case (_e.g.,_ cap in Figure 1). We implement non-
trivial non-degeneracy certificates for ShapeNet object’s cap and mug, and for
YCB object’s boxes (of any kind), pitcher base, and power drill.

Let ${\cal G}=\left\\{\textsf{G}_{l}\right\\}_{l=1}^{g}$ denote the collection
of all the indicator sets. We describe our implementation of ${\cal G}$ for
the few cases in which we implement a non-trivial non-degeneracy certificate.
Figure 12 shows a list of five ShapeNet and YCB objets, along with annotated
semantic keypoints and their indices. For these objects we implement a non-
trivial, non-degeneracy certificate. For the cap and mug, we choose ${\cal G}$
= {set([1])} and ${\cal G}$ = {set([9])}, respectively. Note that [1] is the
keypoint on the flap of the cap, while [9] is the keypoint on the handle of
the mug. This choice requires that the input point cloud ${\bm{X}}$ have
points near the flap of the cap and the handle of the mug, to be deemed as a
cover set, and thereby, declare the input ${\bm{X}}$ as non-degenerate. For
019 pitcher base we choose ${\cal G}$ = { set([8]), set([10])} as non-
visibility of the handle causes degeneracy, and for 035 power drill we choose
${\cal G}$ = {set([9]), set([10]), set([11]), set([12])} as non-visibility of
the base of the power drill leads to degeneracy.

We implement non-trivial non-degeneracy certificate for all the boxes in the
YCB dataset. This is because it is impossible to determine the correct pose of
the box by seeing the depth point cloud of only one side. We therefore
implement ${\cal G}$ = {set([0,1,3,4]), set([0,1,2,5]), set([1,2,3,7]),
set([0,2,3,6])set([4,5,6,0]), set([4,5,7,1]), set([5,6,7,2]), set([4,6,7,3])},
which requires visibility of at least three sides of the box to be declared
non-degenerate in (4). We choose $\delta_{\texttt{nd}}$ in (23) to be $1.5\%$
of the object diameter for ShapeNet objects and $1.5$cm for the YCB objects.

######  Remark 24 (Learning Indicator Sets).

While the implementation of the non-degeneracy certificate is hand-crafted, we
only use it at test time and on a few objects. For most objects, evaluations
show that observable correctness imply certifiable correctness. We leave it to
future work to implement a more exact and systematic non-degeneracy
certificate –probably a learning-based model, that does not require hand-
crafting.

Hyper-parameters. We describe our choice of learning rate, number of epochs,
the $\epsilon_{\texttt{oc}}$ used in certification (Definition 3), and other
parameters used during the self-supervised training. We set the maximum number
of epochs to be $20$ for ShapeNet and $50$ for YCB objects. We made two
exceptions in the case of ShapeNet, where we observed that the models for
“car” and “helmet” were still training (_i.e.,_ observed continuing decrease
in the train and validation loss, and improvement in the percent certifiable).
For these two models we set the maximum number of epochs to $40$. For the
learning rate, we used $0.02$, momentum to be $0.9$, while the training batch
size was $50$. We did not optimize much over these parameters, but only found
ones that worked. We believe some improvement can be expected by exhaustively
optimizing these training hyper-parameters.

We observe that correctly setting up the certification parameter
$\epsilon_{\texttt{oc}}$ in (20) plays a crucial role in efficiently training
the model in a self-supervised manner. Setting $\epsilon_{\texttt{oc}}$ too
high allows for the model to make and accept errors in training, while setting
it to too tight leads to increased training time, as a very small number of
input-output pairs remain certifiable, for the simulation-trained model with
the corrector, at the beginning of the self-supervised training. A good
$\epsilon_{\texttt{oc}}$ can be obtained either by grid search or by visually
inspecting the simulation-trained KeyPO, applied on the real-world data, with
the corrector. For instance, if all the declared $\texttt{oc}=1$ instances are
also visually correct (_i.e.,_ posed CAD model $\hat{{\bm{X}}}$ is correctly
aligned with the input point cloud ${\bm{X}}$), and the $\texttt{oc}=0$
instances are not, then the chosen $\epsilon_{\texttt{oc}}$ is correct. Table
VIII lists the $\epsilon_{\texttt{oc}}$ used for various objects in the
ShapeNet and YCB dataset.

######  Remark 25 (Automatic Tuning of $\epsilon_{\texttt{oc}}$).

The parameter $\epsilon_{\texttt{oc}}$ in (20) (i) determines the observable
correctness of the input-output pairs and (ii) influences the proposed self-
supervised training (see Section VI). This dual purpose makes
$\epsilon_{\texttt{oc}}$ amenable for automatic tuning during self-supervised
training. We think that it is possible to use the number of observably correct
instances, during training, as an indicator to automatically adjust
$\epsilon_{\texttt{oc}}$. However, we leave this thread of inquiry for future
work.

TABLE VIII: $\epsilon_{\texttt{oc}}$ for ShapeNet (in % of object diameter
$d$) and YCB objects (in cm).

object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$
---|---|---|---|---|---|---|---|---|---|---|---
airplane | 3.16 | bathtub | 4.47 | bed | 4.47 | bottle | 10.03 | cap | 10.03 | car | 3.16
chair | 3.16 | guitar | 2.24 | helmet | 4.47 | knife | 2.24 | laptop | 7.08 | motorcycle | 3.16
mug | 7.08 | skateboard | 3.16 | table | 10.03 | vessel | 3.16 |  |  |  |

object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$ | object | $\epsilon_{\texttt{oc}}$
---|---|---|---|---|---|---|---|---|---
chips can | 1.00 | master chef can | 1.00 | cracker box | 1.41 | sugar box | 1.00 | tomato soup can | 0.71
mustard bottle | 1.00 | tuna fish can | 0.77 | pudding box | 0.71 | gelatin box | 0.71 | potted meat can | 1.00
banana | 0.89 | pitcher base | 1.58 | bleach cleanser | 1.14 | power drill | 0.89 | wood block | 1.00
scissors | 0.95 | large marker | 1.00 | large clamp | 0.71 | extra large clamp | 0.89 | foam brick | 0.71

PointNet++ vs Point Transformer. In Section VII-C, we noted that the C-3PO
works better when we use point transformer as a keypoint detector, as opposed
to PointNet++, _i.e.,_ the former converges to 100% certifiability much sooner
during training (see Fig. 9). Table VI reports the ADD-S, ADD-S (AUC), and
percentage certifiable for models trained using the point transformer and the
PointNet++ architecture. We observe that the point transformer performs better
in all the cases.

Extended Results. For the ShapeNet dataset, we provide the results for 16
randomly selected objects, one in each object category, in Table IX. We
observe similar trends as noted in Section VII-C across all the object
categories. We observe that additions such as ICP or outlier rejection of
perturbed keypoints using RANSAC does not help much. The performance of C-3PO
reaches close to that of the supervised baseline KeyPO (real) in most cases.
We note that mug and cap remain the only two degenerate cases, for which we
report the results using the certificate of non-degeneracy in Table II in
Section VII-C.

For the YCB experiment, we present the extended set of results, for all
baselines and 18 YCB object models, in Table X. We observe a similar trend as
in Table III. We, however, observe quite a few degenerate cases, as opposed to
the ShapeNet dataset. These cases arise for different types of boxes, blocks,
brick, can and a power drill. For the boxes, degeneracy arises when only one
face of the box is visible. Similar degnerate cases are observed for other
box-like objects such as brick and blocks. Degeneracy arises in the power
drill when only the drill portion is visible, but not its handle. We report
our evaluation for these cases in Table VII, and show the performance boost in
implementing the certificate of non-degeneracy. We observe that while the
ADD-S and ADD-S (AUC) numbers are low over all, non-certified instances, we
see high ADD-S and ADD-S (AUC) scores for the observably correct and non-
degenerate instances produced by C-3PO. This empirically validates our
implemented non-degeneracy certificates and Theorem 18.

## References

  * [1] H. Yang, J. Shi, and L. Carlone, “TEASER: Fast and Certifiable Point Cloud Registration,” IEEE Trans. Robotics, vol. 37, no. 2, pp. 314–333, 2020\.  extended arXiv version 2001.07715 (pdf).
  * [2] Y. Aoki, H. Goforth, R. A. Srivatsan, and S. Lucey, “PointNetLK: robust & efficient point cloud registration using PointNet,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 7163–7172, 2019.
  * [3] X. Li, J. K. Pontes, and S. Lucey, “Pointnetlk revisited,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 12758–12767, 2021.
  * [4] W. Yuan, B. Eckart, K. Kim, V. Jampani, D. Fox, and J. Kautz, “Deepgmr: Learning latent gaussian mixture models for registration,” in European Conf. on Computer Vision (ECCV), p. 733–750, Aug. 2020.
  * [5] X. Li, Y. Weng, L. Yi, L. Guibas, A. L. Abbott, S. Song, and H. Wang, “Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation,” in Advances in Neural Information Processing Systems (NIPS), Dec. 2021.
  * [6] The New York Times, “Self-Driving Uber Car Kills Pedestrian in Arizona, Where Robots Roam.” https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html, 2018\.
  * [7] H. Yang and L. Carlone, “Certifiably optimal outlier-robust geometric perception: Semidefinite relaxations and scalable global optimization,” IEEE Trans. Pattern Anal. Machine Intell., 2022.  (pdf).
  * [8] M. Garcia-Salguero, J. Briales, and J. Gonzalez-Jimenez, “Certifiable relative pose estimation,” Image and Vision Computing, vol. 109, p. 104142, 2021\.
  * [9] H. Yang and M. Pavone, “Conformal Semantic Keypoint Detection with Statistical Guarantees,” in NeurIPS Workshop on Robot Learning: Trustworthy Robotics, Nov. 2022.
  * [10] G. Shafer and V. Vovk, “A Tutorial on Conformal Prediction,” J. of Machine Learning Research, p. 51, 2008.
  * [11] A. N. Angelopoulos and S. Bates, “A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification,” arXiv, Sep. 2022.
  * [12] K. Xu, H. Zhang, S. Wang, Y. Wang, S. Jana, X. Lin, and C.-J. Hsieh, “Fast and complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers,” in Intl. Conf. on Learning Representations (ICLR), Sep. 2021.
  * [13] S. Wang, H. Zhang, K. Xu, X. Lin, S. Jana, C.-J. Hsieh, and J. Z. Kolter, “Beta-CROWN: Efficient bound propagation with per-neuron split constraints for neural network robustness verification,” in Advances in Neural Information Processing Systems (NIPS), Nov. 2021.
  * [14] A. Albarghouthi, “Introduction to Neural Network Verification,” arXiv, Oct. 2021.
  * [15] C. Gümeli, A. Dai, and M. Nießner, “ROCA: Robust CAD Model Retrieval and alignment from a single image,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [16] K. Park, T. Patten, and M. Vincze, “Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation,” Intl. Conf. on Computer Vision (ICCV), pp. 7667–7676, Oct. 2019.
  * [17] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “PoseCNN: A convolutional neural network for 6D object pose estimation in cluttered scenes,” in Robotics: Science and Systems (RSS), 2018.
  * [18] C. Wang, D. Xu, Y. Zhu, R. Martin-Martin, C. Lu, L. Fei-Fei, and S. Savarese, “Densefusion: 6d object pose estimation by iterative dense fusion,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 3338–3347, Jun. 2019.
  * [19] A. Kundu, Y. Li, and J. M. Rehg, “3d-rcnn: Instance-level 3d object reconstruction via render-and-compare,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 3559–3568, 2018.
  * [20] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield, “Deep object pose estimation for semantic robotic grasping of household objects,” in Conference on Robot Learning (CoRL), pp. 306–316, Oct. 2018\.
  * [21] S. Höfer, K. Bekris, A. Handa, J. C. Gamboa, M. Mozifian, F. Golemo, C. Atkeson, D. Fox, K. Goldberg, J. Leonard, C. Karen Liu, J. Peters, S. Song, P. Welinder, and M. White, “Sim2Real in Robotics and Automation: Applications and Challenges,” IEEE Trans. Autom. Sci. Eng., vol. 18, pp. 398–400, Apr. 2021.
  * [22] X. Li, R. Cao, Y. Feng, K. Chen, B. Yang, C.-W. Fu, Y. Li, Q. Dou, Y.-H. Liu, and P.-A. Heng, “A Sim-to-Real Object Recognition and Localization Framework for Industrial Robotic Bin Picking,” IEEE Robotics and Automation Letters, vol. 7, pp. 3961–3968, Apr. 2022.
  * [23] G. Wang, F. Manhardt, J. Shao, X. Ji, N. Navab, and F. Tombari, “Self6D: Self-supervised monocular 6D object pose estimation,” in European Conf. on Computer Vision (ECCV) (A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds.), pp. 108–125, Nov. 2020.
  * [24] G. Wang, F. Manhardt, X. Liu, X. Ji, and F. Tombari, “Occlusion-Aware Self-Supervised Monocular 6D Object Pose Estimation,” IEEE Trans. Pattern Anal. Machine Intell., pp. 1–1, 2022.
  * [25] S. Zakharov, W. Kehl, A. Bhargava, and A. Gaidon, “Autolabeling 3D Objects with Differentiable Rendering of SDF Shape Priors,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 12224–12233, Jun. 2020.
  * [26] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox, “Self-supervised 6D Object Pose Estimation for Robot Manipulation,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), pp. 3665–3671, May 2020.
  * [27] K. Chen, R. Cao, S. James, Y. Li, Y.-H. Liu, P. Abbeel, and Q. Dou, “Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin-picking,” arXiv:2204.07049 [cs], Apr. 2022.
  * [28] H. Yang, W. Dong, L. Carlone, and V. Koltun, “Self-supervised geometric perception,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.  Oral Presentation, arXiv preprint: 2103.03114, (pdf), (code).
  * [29] S. Gould, R. Hartley, and D. Campbell, “Deep declarative networks: A new hope,” arXiv preprint arXiv:1909.04866, 2019.
  * [30] A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter, “Differentiable convex optimization layers,” in Advances in Neural Information Processing Systems (NIPS), pp. 9558–9570, 2019.
  * [31] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al., “Shapenet: An information-rich 3d model repository,” arXiv preprint arXiv:1512.03012, 2015.
  * [32] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar, “The YCB object and Model set: Towards common benchmarks for manipulation research,” in Intl. Conf. on Advanced Robotics (ICAR), pp. 510–517, Jul. 2015.
  * [33] T. Hodaň, F. Michel, E. Brachmann, W. Kehl, A. G. Buch, D. Kraft, B. Drost, J. Vidal, S. Ihrke, X. Zabulis, C. Sahin, F. Manhardt, F. Tombari, T.-K. Kim, J. Matas, and C. Rother, “BOP: Benchmark for 6D Object Pose Estimation,” in European Conf. on Computer Vision (ECCV), pp. 19–35, 2018.
  * [34] T. Hodaň, M. Sundermeyer, B. Drost, Y. Labbé, E. Brachmann, F. Michel, C. Rother, and J. Matas, “BOP challenge 2020 on 6D object localization,” European Conference on Computer Vision Workshops (ECCVW), 2020.
  * [35] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set,” IEEE Robotics & Automation Magazine, vol. 22, pp. 36–52, Sep. 2015.
  * [36] T. Hodan, P. Haluza, S. Obdrzalek, J. Matas, M. Lourakis, and X. Zabulis, “T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-less Objects,” in IEEE Winter Conf. Appl. Computer Vision (WACV), Mar. 2017.
  * [37] G. Wang, F. Manhardt, F. Tombari, and X. Ji, “GDR-Net: Geometry-Guided Direct Regression Network for Monocular 6D Object Pose Estimation,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 16611–16621, 2021.
  * [38] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the Continuity of Rotation Representations in Neural Networks,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 5738–5746, Jin. 2019.
  * [39] V. N. Nguyen, Y. Hu, Y. Xiao, M. Salzmann, and V. Lepetit, “Templates for 3d object pose estimation revisited: Generalization to new objects and robustness to occlusions,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Jun. 2022.
  * [40] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox, “DeepIM: Deep Iterative Matching for 6D Pose Estimation,” in European Conf. on Computer Vision (ECCV), pp. 683–698, 2018.
  * [41] Y. Labbe, J. Carpentier, M. Aubry, and J. Sivic, “CosyPose: Consistent multi-view multi-object 6D pose estimation,” in European Conf. on Computer Vision (ECCV), 2020.
  * [42] X. Liu, R. Zhang, C. Zhang, B. Fu, J. Tang, X. Liang, J. Tang, X. Cheng, Y. Zhang, G. Wang, and X. Ji, “Gdrnpp.” https://github.com/shanice-l/gdrnpp_bop2022, 2022.
  * [43] X. He, J. Sun, Y. Wang, D. Huang, H. Bao, and X. Zhou, “Onepose++: Keypoint-free one-shot object pose estimation without CAD models,” in Advances in Neural Information Processing Systems (NIPS), 2022.
  * [44] R. Rusu, N. Blodow, and M. Beetz, “Fast point feature histograms (fpfh) for 3d registration,” in IEEE Intl. Conf. on Robotics and Automation (ICRA), pp. 3212–3217, Citeseer, 2009.
  * [45] Z. Yew and G. Lee, “3dfeat-net: Weakly supervised local 3d features for point cloud registration,” in European Conf. on Computer Vision (ECCV), 2018\.
  * [46] H. Deng, T. Birdal, and S. Ilic, “PPFNet: Global Context Aware Local Features for Robust 3D Point Matching,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 195–205, Jun. 2018.
  * [47] H. Deng, T. Birdal, and S. Ilic, “PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors,” in European Conf. on Computer Vision (ECCV) (V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, eds.), pp. 620–638, 2018.
  * [48] C. Choy, J. Park, and V. Koltun, “Fully convolutional geometric features,” in Intl. Conf. on Computer Vision (ICCV), pp. 8958–8966, 2019.
  * [49] A. Zeng, S. Song, M. Nießner, M. Fisher, J. Xiao, and T. Funkhouser, “3dmatch: Learning the matching of local 3d geometry in range scans,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, vol. 1, p. 4, 2017.
  * [50] J. Yang, H. Li, D. Campbell, and Y. Jia, “Go-ICP: A globally optimal solution to 3D ICP point-set registration,” IEEE Trans. Pattern Anal. Machine Intell., vol. 38, pp. 2241–2254, Nov. 2016.
  * [51] Q. Zhou, J. Park, and V. Koltun, “Fast global registration,” in European Conf. on Computer Vision (ECCV), pp. 766–782, Springer, 2016.
  * [52] V. Sarode, X. Li, H. Goforth, Y. Aoki, R. A. Srivatsan, S. Lucey, and H. Choset, “PCRNet: Point Cloud Registration Network using PointNet Encoding,” arXiv:1908.07906 [cs], Nov. 2019.
  * [53] Y. Wang and J. M. Solomon, “PRNet: Self-Supervised Learning for Partial-to-Partial Registration,” in Advances in Neural Information Processing Systems (NIPS), pp. 8812–8824, 2019.
  * [54] C. Choy, W. Dong, and V. Koltun, “Deep global registration,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [55] S. Huang, Z. Gojcic, M. Usvyatsov, A. Wieser, and K. Schindler, “PREDATOR: Registration of 3D Point Clouds with Low Overlap,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 4265–4274, Jun. 2021.
  * [56] Z. J. Yew and G. H. Lee, “RPM-Net: Robust Point Matching using Learned Features,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 11821–11830, Jun. 2020.
  * [57] Z. Gojcic, C. Zhou, J. D. Wegner, and A. Wieser, “The perfect match: 3d point cloud matching with smoothed densities,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5545–5554, 2019.
  * [58] Y. Yuan, D. Borrmann, J. Hou, Y. Ma, A. Nüchter, and S. Schwertfeger, “Self-supervised point set local descriptors for point cloud registration,” Sensors, vol. 21, no. 2, 2021.
  * [59] M. E. Banani, L. Gao, and J. Johnson, “UnsupervisedR&R: Unsupervised Point Cloud Registration via Differentiable Rendering,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 7125–7135, Jun. 2021.
  * [60] X. Huang, G. Mei, and J. Zhang, “Feature-Metric Registration: A Fast Semi-Supervised Approach for Robust Point Cloud Registration Without Correspondences,” in IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 11363–11371, Jun. 2020.
  * [61] M. Zhu, M. Ghaffari, and H. Peng, “Correspondence-Free Point Cloud Registration with SO(3)-Equivariant Implicit Shape Representations,” in Conference on Robot Learning (CoRL), pp. 1412–1422, Jan. 2022.