# COPE: End-to-end trainable Constant Runtime Object Pose Estimation

Stefan Thalhammer
Automation and Control Institute, TU Wien
<EMAIL_ADDRESS>Timothy Patten
Robotics Institute, UTS Sydney
<EMAIL_ADDRESS>Markus Vincze
Automation and Control Institute, TU Wien
<EMAIL_ADDRESS>
###### Abstract

State-of-the-art object pose estimation handles multiple instances in a test
image by using multi-model formulations: detection as a first stage and then
separately trained networks per object for 2D-3D geometric correspondence
prediction as a second stage. Poses are subsequently estimated using the
Perspective-$n$-Points algorithm at runtime. Unfortunately, multi-model
formulations are slow and do not scale well with the number of object
instances involved. Recent approaches show that direct 6D object pose
estimation is feasible when derived from the aforementioned geometric
correspondences. We present an approach that learns an intermediate geometric
representation of multiple objects to directly regress 6D poses of all
instances in a test image. The inherent end-to-end trainability overcomes the
requirement of separately processing individual object instances. By
calculating the mutual Intersection-over-Unions, pose hypotheses are clustered
into distinct instances, which achieves negligible runtime overhead with
respect to the number of object instances. Results on multiple challenging
standard datasets show that the pose estimation performance is superior to
single-model state-of-the-art approaches despite being more than $\sim$35
times faster. We additionally provide an analysis showing real-time
applicability ($>24$ fps) for images where more than $90$ object instances are
present. Further results show the advantage of supervising geometric
correspondence-based object pose estimation with the 6D pose.

##  1 Introduction

Object pose estimation is a challenging problem for monocular computer vision
despite being essential for many tasks such as augmented reality, object
manipulation, scene understanding, autonomous driving and industrial
inspection [20, 38, 42]. Learning-based object pose estimation research
focuses on maximizing the performance under challenging conditions like domain
shift, object occlusion and object symmetries by tendentiously separating the
detection from the pose correspondence estimation stage [12, 26, 29, 41, 48]
then deriving the 6D pose with the Perspective-$n$-Points (P$n$P) algorithm
[10] using the estimated geometric correspondences. This leads to shortcomings
because a) adopting surrogate training targets decouples pose estimation from
the training process and thus limits learning [7] and b) running inference for
multi-instance scenarios leads to a computational complexity of at least
$\mathcal{O}\left(n\right)$ with respect to the number of objects ($n$) for
the pose estimation stage. Thus, this type of approach has severely
diminishing applicability for realistic scenarios.

Figure 1: COPE: Multi Object Instance 6D Pose Estimation with Constant
Runtime. Our end-to-end trainable pose estimator uses intermediate geometric
correspondences to directly estimate 6D object poses from a single RGB image,
thus processing multiple instances in parallel.

Recent object pose estimation research trends recognize those shortcomings and
partially alleviate them by directly regressing the 6D pose from the
intermediate pose correspondences to achieve tremendous results [4, 7, 22,
53]. In [53] and [7], detection is separated from the pose estimation stage,
which makes them not end-to-end trainable because they require an object
detector. The work in [22] is end-to-end trainable but separate networks need
to be trained for each object and pooling geometric correspondences means
multiple objects and instances cannot be handled simultaneously. We improve
over these findings by proposing a natural extension to efficiently handle
multi-object multi-instance scenarios.

In this work we propose a solution to the aforementioned shortcomings by
sharing the latent representation as well as the direct pose regressor over
objects and their instances; see Figure 1 for a high-level overview. We
classify image locations in the feature maps, regress bounding box and view-
dependent object geometry correspondences and regress the direct 6D pose.
While the first three of these tasks are intermediate representations, the
direct 6D pose head is an up-stream task shared over those intermediate
outputs of the network. Consequently, the loss related to the 6D pose is also
backpropagated to the down-stream task of geometric correspondence estimation.
This design also allows further guidance of the learning process by enforcing
consistency between these consecutive tasks, which additionally improves each
of them. We propose a concurrent solution to anchors [43] for true location
sampling during training that does not require manually choosing
hyperparameters based on the expected test data distribution. True locations
are sampled and regression targets are standardized from a scalar shape prior
derived from the respective object mesh and the backpropagated loss is
normalized for each object class. Thus, training is not biased towards larger
objects and no prior assumptions need to be made in contrast to the case for
anchors.

In summary, our contributions are:

  * •

A simple and efficient solution for multi-object multi-instance object pose
estimation that improves over the state of the art.

  * •

A training target sampling scheme that requires no assumptions about the test
data distribution.

Efficiently sharing internal representations over objects and instances
enables end-to-end trainability that requires only one forward pass through
the network to process all object instances in a single input image. We show
that processing more than $90$ object instances in a single image with more
than $24fps$ on a modern consumer GPU, our method’s performance is competitive
to similar state-of-the-art approaches but up to $35$ times faster.

The remainder of the paper discusses related work in Section 2, followed by a
description of our proposed approach in Section 3 and evaluations in Section
4. Lastly, Section 5 concludes the paper.

##  2 Related Work

In this section we present the state of the art for monocular 6D object pose
estimation with a focus on work that directly regress the object pose. This is
followed by reviewing training target sampling for object detection.

Object Pose Estimation Since direct pose regression from feature space leads
to inferior performance, the dominant monocular object pose estimation
approaches leverage geometric correspondences as regression targets [17, 29,
39, 41, 40, 23, 54]. Poses are derived for each estimated set of object
correspondences using variants of P$n$P [10, 28, 49]. Recent trends replace
the classical solver with trainable versions [4, 7, 22, 46, 53] to infer the
6D pose directly from the intermediate geometric correspondences. This enables
end-to-end trainable object pose estimation as it provides the additional
supervision for the down-stream network parts with the 6D pose. Their findings
indicate that direct 6D pose estimation also results in state-of-the-art
performance by sharing the pose regressor over objects [7, 53]. However,
efficient and simultaneous single-stage multi-object instances handling is a
problem that still remains [17].

Considering the top performing approaches in the BOP challenge [18], a
benchmark that aims to provide a standardized protocol for an unbiased
comparison of object pose estimation, a frequently used technique to handle
multiple object instances in an image is to separate object detection from
pose estimation [29, 26, 17, 34, 41, 12]. In the first stage, 2D location
hypotheses are provided using common object detectors such as Faster-RCNN
[43], RetinaNet [31] or FCOS [51]. In the second stage, object crops are
passed to the pose estimator but this leads to considerable temporal and
computational cost. An exception is EPOS [17] where multi-instance handling is
facilitated by using Graph-Cut RANSAC [1] to cluster the predicted geometric
correspondences to individual instances. Despite providing a sophisticated
approach for addressing object symmetries and multiple object instances with
one forward pass through a network, their multi-instance fitting of poses
using [1] is computationally very demanding. In our work, we alleviate this
issue by adopting ideas from object detectors and incorporate direct pose
regression into the detection stage.

Detecting Objects in Images Single-stage object detectors offer efficient
solutions for multi-object and multi-instance object localization in 2D [2, 9,
27, 35, 31, 51]. Object detection uses anchors [31, 35, 43] to sample bounding
box priors with different sizes and aspect ratios in the multi-scale feature
map of feature pyramids [30]. For training, foreground image locations are
chosen based on the Intersection-over-Union (IoU) between ground truth
bounding boxes and the anchor boxes. As such, training locations are
correlated with the projected object shape in the image space. This leads to
effective encoding and handling of objects with different scales.

Using anchors has two downsides. Firstly, it requires the manual specification
of $16$ hyperparameters that reflect the expected training and test dataset
statistics. Secondly, the size of the output space depends on the number of
anchors sampled for each image location. Recent approaches propose alternative
formulations to circumvent these shortcomings while retaining the advantages
of anchors [9, 27, 51, 56]. The authors of [51] choose the respective feature
map resolution for training explicitly by using the bounding box size to
overcome the necessity for sampling anchor boxes. True image locations of the
feature map, for loss backpropagation, are assigned based on the respective
pixel’s centerness with respect to the ground truth bounding box.
Alternatively, [27] models objects as paired-keypoints: the top-left and the
bottom-right corner of the bounding box. Similarly, [9] addresses the
inefficiency of anchor-based object detection by modeling objects as their
center points and estimating the bounding box relative to it. The authors of
[56] overcome the requirement of hyperparameters for assigning objects to
anchors by designing a flexible maximum likelihood estimation assignment for
network training.

We propose to sample training locations based on the visible object mask and
the 3D object dimensions. We also use the 3D object dimension to effectively
replace the anchor-based target annotation standardization. As such, we encode
objects of different sizes and eccentricities more effectively, while also
reducing the size of the output space and the number of required
hyperparameters.

##  3 Constant Runtime Object Pose Estimation

This section provides an overview of the proposed direct 6D pose estimation
framework, which we name Constant Runtime Object Pose Estimation, abbreviated
as COPE. We start with a high-level overview of the method. Afterwards, we
detail the approaches for deriving anchor-free training targets, true image
location sampling and geometry correspondence standardization during training.
This is followed by an explanation of how the direct 6D pose is parameterized
and how symmetries are handled. We conclude with a description of multi-
instance clustering and hypotheses filtering during testing.

###  3.1 Constant Runtime via Direct-pose regression

Our aim is to classify and estimate the poses of all object instances in a
single RGB input image. The 6D pose is defined as $\hat{P}\in SE(3)$, which
represents the object’s rotation $R\in\mathbb{R}^{3}$ and translation
$t\in\mathbb{R}^{3}$ with respect to the camera’s coordinate frame. Object
meshes are considered to be known in advance but no additional information
regarding the test scene is required. We define the corner points of the
smallest cuboid enclosing the respective object mesh in its coordinate frame
as the geometric correspondences ($G_{3D}$). COPE, outlined in Figure 2,
outputs the set of object instances visible in the image, parameterized with
object type and 6D pose.

Figure 2: Constant Runtime Object Pose Estimation. Given an input image and a
3D model, image locations are classified while bounding boxes and geometric
correspondences are regressed. A direct pose regression module slides over the
image locations and regresses the 6D pose from the geometric correspondences.
Training is supervised with losses for each module ($L_{cls}$, $L_{box}$,
$L_{key}$, $L_{tra}$ and $L_{rot}$) as well as auxiliary losses ($L_{proj}$
and $L_{cons}$) to enforce consistency between estimated correspondences and
direct poses. During testing, instances are efficiently clustered using their
2D IoU then the $n$ hypotheses with the highest consistency generate the 6D
output.

COPE builds upon the success of recent efficient object detection approaches
[31, 51, 55]. The RGB input image is first processed with a CNN backbone and
then multi-scale features are computed using a feature pyramid to estimate the
intermediate object representation. Three modules shared over feature maps of
sizes [$s/8$, $s/16$, $s/32$], with $s$ being the input image resolution,
generate the intermediate outputs $\hat{O}$, $\hat{B}$ and $\hat{G}$. The
first module predicts the set of object class probabilities
$\hat{O}\coloneqq\\{\ \hat{o}_{0},...,\hat{o}_{k}\\}$, where $k$ is the number
of image locations in the multi-scale feature map and
$\hat{o}_{k}\in\mathbb{R}^{a}$ is the Bernoulli-distributed object class
prediction. We denote the number of object classes in the dataset with $a$.
The second module predicts the amodal bounding boxes $\hat{B}\coloneqq\\{\
\hat{b}_{0},...,\hat{b}_{k}\\}$, where $\hat{b}_{k}\in\mathbb{R}^{4}$. The
third module predicts the projection of $G_{3D}$ in the image space
$\hat{G}\coloneqq\\{\ \hat{g}_{0},...,\hat{g}_{k}\\}$, where
$\hat{g}_{k}\in\mathbb{R}^{16}$ represents the 2D coordinates of the $8$
corner points of the smallest cuboid enclosing the respective object mesh. A
shared direct pose module slides over the set $\hat{G}$, estimating direct
pose hypotheses $\hat{P}\coloneqq\\{\ \hat{p}_{0},...,\hat{p}_{k}\\}$, where
$\hat{p}_{k}\in\mathbb{R}^{9}$. The pose output is parameterized by $3$ values
for translation in $\mathbb{R}^{3}$ and the first two basis vectors of the
rotation matrix in $\mathbb{R}^{3}$, thus $6$ values in total [57]. A set
$\hat{C}\coloneqq\\{\ \hat{c}_{0},...,\hat{c}_{k}\\}$ is computed to quantify
the consistency $\hat{c}_{k}\in\mathbb{R}^{1}$ between $\hat{g}_{k}$ and
$proj_{3D\to 2D}(G_{3D}\cdot\hat{p}_{k})$ for each image location separately.

During inference, the network predicts $H=\\{\hat{O}$, $\hat{B}$, $\hat{G}$,
$\hat{P}$, $\hat{C}\\}$ with constant runtime for a given query image.
Corresponding elements of $H$ with an image location $k$ of maximum class
probability $\hat{o}_{k}$ below the detection threshold are discarded. The
resulting subsets are clustered into object instances using the IoU between
elements of $\hat{B}$. We define a hyperparameter $n$ for the highest number
of consistencies in $\hat{C}$. This parameter is set to $10$ for the presented
experiments; see the ablation in Table 3. Finally, the detected object classes
and the mean of the $n$ poses with the highest consistencies per instance are
returned with negligible increase in runtime with respect to the number of
object instances. Through this procedure, our method can estimate the poses of
a large number of object instances in a single test image in real-time
($>24fps$) on an Nvidia Geforce 3090 GPU.

###  3.2 Training Target Sampling

Effective assignment of true image locations for updating network weights
during training is an ongoing research problem [9, 27, 31, 51, 55, 56]. These
true image locations are often sampled in the output feature maps of feature
pyramids [30], which are a great tool to efficiently encode scale information
in the feature space. Anchors [43] are the standard representation for
providing bounding box priors to sample true image location based on the IoU
with the ground truth during training [43, 35, 31]. For each image location in
the multi-scale feature map, $9$ differently shaped and sized bounding box
priors are sampled. This requires $16$ hyperparameters: $5$ each for base
sizes and strides and $3$ each for ratios and scales [31]. This results in two
convenient traits since anchor locations used for updating the network’s
weights are chosen based on a threshold parameter for the IoU with the ground
truth:

  * •

Sampling anchors leads to a uniform scale space for the expected bounding
boxes. As such, a similar amount of training locations are sampled per object,
independent of the object’s size in the image space.

  * •

Regression targets are standardized using the respective anchor’s center,
width and height. This means that the regression target space has similar
statistics for differently sized objects.

Despite these convenient traits, training target sampling can still be
improved since anchors require a) choosing $16$ hyperparameters depending on
the expected object scales in image space and b) generating $9$ anchors per
feature map location, which results in a large output space that slows down
convergence. We overcome these shortcomings by using a regression target
standardization scheme that reflects the object’s geometry and scale.

####  3.2.1 True Location Sampling

Object masks are used for true training location sampling as in [23]. However,
instead of predicting object masks and correspondences from a single feature
map resolution, our work adopts the divide-and-conquer strategy of feature
pyramids to make predictions from multiple feature map resolutions. To
overcome the necessity of requiring hyperparameters [43, 51] for choosing the
best suited feature map resolution for locating an object, we propose a
geometry-based approach to assign true training locations. We supplement true
location sampling with a scalar shape prior:

$\displaystyle\delta_{o}={max}||(m_{i}-m_{j})||_{2}\hskip
4.30554pt\forall\hskip 4.30554ptm_{i},m_{j}\in M,i\neq j$ (1)

where $M$ is the set of object model vertices. Since the spatial downsampling
of the input image through the backbone follows an exponential function, it is
intuitive to explicitly choose pyramid levels using a logarithmic function. As
such, we choose the respective feature pyramid level with:

$\displaystyle level=f+log_{d}(\delta_{o}/t_{z}),$ (2)

where $f$ depends on the number of pyramid levels used, $t_{z}$ corresponds to
the object’s distance from the camera and $d$ is the only remaining
hyperparameter. Since we use three pyramid levels, as in [50], this requires
choosing only $6$ hyperparameters for FCOS and $12$ when using anchors. An
additional advantage is that $\delta_{o}$ better reflects the object shape in
all three spatial dimensions and thus also the visible object surface in the
image space compared to using the bounding boxes for the assignment of true
training locations. As a consequence, elongated objects are tendentiously
sampled in higher resolved feature pyramid levels than boxy shaped objects.
Despite needing fewer hyperparameters, we retain a similar amount of true
locations used for training. Classifying true image locations ($L_{cls}$) is
supervised using the focal loss [31].

####  3.2.2 Geometric Correspondence Standardization

Instead of standardizing the projected object correspondences $G$ using anchor
priors or with a scalar value agnostic to object shape [51], we directly
incorporate $\delta_{o}$ to scale regression targets of different objects to a
similar magnitude:

$\displaystyle y_{G}=(c-G)/\delta_{o},$ (3)

where $c$ is the center of the respective feature map location, $G$ are the
image locations of the geometric correspondences and $y_{G}$ are the
standardized regression targets. As such, regression targets are encoded
similarly as with anchors (with similar $\sigma$ for $G$ for all objects
independent of their scale or shape eccentricity). Thus, the computed error is
independent of the object’s scale in the image space and the training process
is not biased for larger objects. Our approach needs no hyperparameters for
standardization and convergence is improved since $9$ times fewer network
output parameters per feature map location are required compared to anchors.

####  3.2.3 Imbalance Problem of Target Locations

Choosing training target locations based on the object mask leads to a
training process that is biased towards objects with a larger projected image
surface. For classification this is commonly circumvented using the focal loss
[43]. Using anchors as location priors alleviates the issue since anchors are
sampled uniformly over the expected object scale space. We define a concurrent
solution by normalizing over the number of true training locations $l$ and
accumulating the gradient afterwards. The regression loss is:

$\displaystyle
L_{reg}(\hat{y},y)=\dfrac{1}{a}\cdot\sum_{i=0}^{a}\dfrac{1}{l_{i}}\cdot\sum_{j=0}^{l_{i}}huber(\hat{y}_{j},y_{j}),$
(4)

where $huber$ is the augmented $l_{1}$ loss used in RetinaNet [11, 31] and $y$
and $\hat{y}$ are the ground truth and estimate, respectively. This procedure
requires no additional trainable parameters and only leads to minor
computational overhead during training time and to none during test time
despite improving multi-object handling.

####  3.2.4 Direct Pose Regression

The direct pose is regressed using the output $\hat{y}$ of the module
estimating intermediate geometric correspondences as done in [7, 22, 46, 53].
The 6D pose is parameterized as $P\in SE(3)$, with $t\in\mathbb{R}^{3}$ being
the 3D translation vector and $R\in\mathbb{R}^{6}$ the first two base vectors
of the $SO(3)$ rotation matrix as done in [7, 25, 53].

Prior methods perform pose estimation on zoomed crops of the detected objects
of interest. Using the object rotation in the camera coordinate system, i.e.
the allocentric rotation [25], leads to mapping different object views to the
same rotation. To alleviate that problem, these approaches use the rotation of
the camera in the object coordinate frame, i.e. the egocentric rotation.

In contrast, we learn to predict geometric correspondences directly in the
image space. These correspondences are destandardized with the inversion of
Equation (3) and fed to the direct pose estimation module. As such, our
approach correlates object rotation with its image location. This means that
we are able to directly regress the allocentric rotation since we require no
zooming or cropping. Additionally, we can directly regress the 3D translation
without requiring a scale-invariant translation representation as used in [7,
29, 53]. The network training is supervised using the image locations sampled
with Equation (2).

###  3.3 Symmetry-aware Loss

Objects exhibiting discrete or continuous symmetries, i.e. similar views that
correspond to different ground truth poses P, are detrimental to the
convergence of the network training [36, 41, 45]. We adopt the transformer
loss of [41] since symmetries are efficiently handled during loss computation
and require no additional trainable weights. We define our keypoint estimation
loss for supervising the training of the geometric correspondence learning
with:

$\displaystyle L_{key}=\underset{s\in S_{i}}{min}\ L_{reg}(\hat{y},hy),$ (5)

where $S_{i}$ is a set of symmetry transformations that depend on the visual
ambiguities of the object. We observed that separately choosing hypotheses
with Lkey, and substituting the direct pose losses for Lrot and Ltra with
Equation (5) introduces ambiguities since the 6D pose is directly derived from
the estimated intermediate geometric correspondences. To alleviate this issue
we define an indicator function $\mathbb{I}$, indicating the symmetry that
minimizes $L_{key}$. As such, we supervise the direct pose regression with:

$\displaystyle L_{rot/tra}=L_{reg}(\hat{y},\mathbb{I}(S)y).$ (6)

Since only one set of $\hat{G}$ is predicted per image location and Equations
(5) and (6) sufficiently account for object symmetries, $L_{proj}$ and
$L_{cons}$ can be directly computed with $L_{key}$. The projection and the
consistency loss are thus defined as:

$\displaystyle L_{proj}=L_{reg}(G_{3D}\hat{P},G),$ (7) $\displaystyle
L_{cons}=L_{reg}(G_{3D}\hat{P},\hat{G}).$ (8)

The overall loss is:

$L=\alpha\cdot L_{cls}+\beta\cdot L_{box}+\gamma\cdot L_{key}+\delta\cdot
L_{rot}\\\ +\epsilon\cdot L_{tra}+\zeta\cdot L_{proj}+\eta\cdot L_{cons},$ (9)

where $\alpha$, $\beta$, $\gamma$, $\delta$, $\epsilon$, $\zeta$ and $\eta$
are loss weights. The bounding box estimation, $L_{box}$, is supervised using
Equation (4).

###  3.4 Multi-instance Handling

Commonly, multiple instances of the same object in a single image are handled
before correspondence estimation by non-maximum suppression of the detection
stage [29, 52, 26, 41] or by clustering correspondences afterwards [17]. The
first family of methods individually process each instance’s image crop to
estimate the 6D pose. The second family of methods is more advantageous
because the network is shared over all objects of interest. Unfortunately,
since [17] predicts dense geometric correspondences the method has a high
runtime. This is due to the clustering of correspondences to object instances
using [1], which is computationally demanding.

In our work, $H$ is first filtered by discarding the non-maximally scoring
object classes for each image location $k$. Subsequently, image locations with
a detection score below the detection threshold are pruned. The remaining
hypotheses correspond to detected objects. The 2D bounding boxes, $\hat{B}$,
are used to cluster object instances based on the respective IoU between the
outputs of different image locations. Ultimately, using the computed
consistency $\hat{C}$, the pose is averaged over the $n$ hypotheses of
$\hat{P}$ with the highest consistency for each object instance.

##  4 Experiments

This section provides quantitative and qualitative evaluations of COPE on
several datasets. After introducing the experimental setup, we proceed with
comparisons to the state of the art on two challenging datasets using the BOP
protocol [18]. In addition, ablation studies are presented to quantify the
influence of direct pose supervision on an additional dataset. To further
validate and thoroughly test the capabilities of our method, we present
results on a synthetic dataset with up to $100$ object instances per image.

###  4.1 Datasets

Evaluation is provided on three standard datasets: LM [15], LM-O [3] and IC-
BIN [8]. For evaluation, we use the subsets provided with the BOP challenge.
LM provides $200$ test images for each of the $13$ objects that come with
watertight object models. LM constitutes a common benchmark for object pose
estimation in cluttered environments. LM-O consists of $200$ test images of
LM’s second test sequence with all eight objects annotated in each. LM-O
provides test images with challenging object occlusions. IC-BIN presents $150$
test images of up to $21$ instances of two objects with heavy occlusion.

For training we use the $50$k photorealistic renderings for each dataset
available through the BOP challenge [6]. These are generated using physically
based rendering (pbr) [19]. Results on LM and LM-O are provided with the same
models trained on all $13$ objects of LM. No annotated real images are used
for training.

###  4.2 Evaluation Metrics

Comparison to the state of the art is provided using the performance score of
the BOP challenge [18]. Results for pose estimation are reported using the
Average Recall: $AR=(AR_{VSD}+AR_{MSSD}+AR_{MSPD})/3$. Ablations are evaluated
using the ADD recall, or ADD-S recall for objects exhibiting symmetries [15].
We report the fraction of poses below the commonly used error threshold of
$10\%$ of the object diameter. Results for object detection are reported using
the the mean Average Precision (mAP) of the Microsoft COCO object detection
challenge [32]. The results are those for the IoU values from $0.5$ to $0.95$
in $0.05$ steps. Please refer to the supplementary material for details.

###  4.3 Implementation Details

The weights of the backbone are pre-trained on ImageNet [44] and fine-tuned
for $120$ epochs using the Adam [24] optimizer with a learning rate of
$10^{-5}$ and a batch size of $8$. Previous work suggests overcoming the
domain gap between training on synthetic and testing on real images by not
updating certain network weights during optimization [16, 54]. Similarly, we
do not update parameters of batch normalizations and the convolutions of the
first two stages of the backbone during fine-tuning. We also apply image
augmentations as described in [50]. The parameter $d$ in Equation (2) is set
to $3$ for all experiments.

###  4.4 Comparison to the State of the Art

Object Pose Estimation This section compares the performance of COPE to the
state of the art on IC-BIN and LM-O. Results using the BOP setting reporting
the AR are provided in Table 1. The bottom section compares single-model
methods, i.e approaches that produce estimates for all object classes and
their instances in a single forward pass. Both DPOD [54] and EPOS [17] require
PnP for deriving the 6D pose from the predicted geometric correspondences,
while COPE directly outputs the 6D pose. COPE improves over both methods in AR
on average. Compared to the previous single-model state of the art, EPOS, COPE
achieves similar AR on LM-O $0.543$ as compared to $0.547$ but improves to
$0.440$ in comparison to $0.363$ on IC-BIN. More remarkable, however, is that
the runtime of COPE is $37$ times faster using the inference speed calculated
by the BOP toolkit111https://github.com/thodan/bop_toolkit.

The top section of Table 1 presents the results of multi-model methods. These
multi-model methods use an object detector to sample sparse location priors
then separately trained networks per object class for correspondence
prediction and pose estimation. For the methods [7, 21, 33, 47] no results on
IC-BIN are available. Compared to the best individually performing methods on
both datasets, CosyPose on IC-BIN and ZebraPose on LM-O, COPE results in $\sim
24\%$ relative performance decrease. This is in the expected range due to the
known performance decrease for single-staged approaches [31].

Runtime Figure 3 presents the average runtime and standard deviation on IC-BIN
for five test runs of COPE, CDPNv2 and CosyPose on an Intel CPU with 3.6GHz
and an Nvidia Geforce 3090 GPU. The times reported for CDPNv2 exclude the time
required for detecting objects. Despite omitting the runtime of CDPNv2’s first
stage, our method is more than $12$ times faster and $7$ times faster than
CosyPose when processing $15$ object instances. Most notably, in contrast to
multi-model approaches, COPE is capable of directly providing 6D poses for
multi-object multi-instance cases at almost constant runtime, which makes it
highly suitable for real-time scenarios.

Figure 3: Runtime Comparison to the State of the Art on IC-BIN. Provided are the times it takes to estimate poses for all object instances in a single image. Table 1: Comparison to the State of the Art for Pose Estimation. Presented are the Average Recall on IC-BIN and LM-O, the average over both and the inference speed using the BOP toolkit. Method | IC-BIN | LM-O | Avg. | Time
---|---|---|---|---
Multi-model
AAE [48] | 0.217 | 0.146 | 0.182 | 0.199
Pix2Pose [41] | 0.226 | 0.363 | 0.295 | 1.230
2Dto3D [34] | 0.342 | 0.525 | 0.434 | 0.546
CDPNv2 [29] | 0.473 | 0.624 | 0.549 | 1.010
SurfEmb [12] | 0.550 | 0.623 | 0.587 | 6.296
CosyPose [26] | 0.574 | 0.618 | 0.596 | 0.227
SO-Pose [7] | - | 0.613 | - | -
CIR [33] | - | 0.655 | - | -
PFA [21] | - | 0.683 | - | -
ZebraPose [47] | - | 0.718 | - | 0.250
Single-model
DPOD [54] | 0.169 | 0.130 | 0.150 | 0.211
EPOS [17] | 0.363 | 0.547 | 0.455 | 2.804
Ours | 0.440 | 0.543 | 0.492 | 0.075

Object Detection Table 2 compares the object detection accuracy (mAP [32]) of
COPE to the state of the art on IC-BIN and LM-O using the same training data.
On average, COPE outperforms both MaskRCNN [13] and FCOS [51], achieving the
highest average mAP over both datasets. On LM-O, COPE is superior to MaskRCNN,
achieving $0.532$ as compared to $0.375$ but slightly inferior to FCOS that
reaches $0.622$. This is partly due to the lean network design of COPE. Please
refer to the supplementary material for more details. COPE’s detected bounding
boxes are more precise than both standard detectors used by many multi-model
methods on IC-BIN, achieving $0.431$ as compared to $0.323$ and $0.316$. As
such our method provides excellent location priors for pose refinement.

Table 2: Comparison to the State of the Art for Object Detection. Presented are the Average Recall on IC-BIN and LM-O and the average using the BOP toolkit. Method | IC-BIN | LM-O | Avg.
---|---|---|---
MaskRCNN [13, 26, 12] | 0.316 | 0.375 | 0.346
FCOS [51, 29, 47] | 0.323 | 0.622 | 0.473
Ours | 0.431 | 0.532 | 0.482

###  4.5 Ablation Studies

Runtime Evaluation In order to exhaustively test the runtime and scalability
of COPE, we create a synthetic test dataset using the IC-BIN objects and
OpenGL222https://github.com/thodan/bop_renderer rendering, named IC-BIN syn.
The number of object instances to render per image is sampled from a uniform
distribution with a lower bound of $10$ and upper bound of $100$. We render
the sampled number of object instances randomly from the IC-BIN objects onto
the test images of IC-BIN. Results are again provided for processing one test
image on an Intel CPU with 3.6GHz and an Nvidia Geforce 3090 GPU.

Figure 4: Runtime Evaluation of COPE on IC-BIN and IC-BIN syn. (a) COPE
provides negligible runtime increase up to $70$ object instances per image.
The black line indicates the threshold for real-time processing. (b) An
example of a test image of IC-BIN syn and a visualization of the estimated
poses.

Figure 4 (a) presents the runtime of our method for detecting and estimating
the poses of up to $100$ object instances in a single image. We report the
average runtime and standard deviation for five test runs. The runtime
increases negligibly up to $70$ detected object instances. For more than $90$
instances per image, our method exceeds real-time processing. As such, it
provides quantitative proof of the tremendous scalability of the presented
approach and the constancy of the runtime with respect to the number of object
instances in a single test image. Figure 4 (b) provides a rendered synthetic
test image (top) and projected object models based on the estimated poses
(bottom).

Direct-pose Regression Table 3 displays the influence of direct pose
regression on the end-to-end architecture on the LM [15] and LM-O [3] datasets
using the ADD/(-S) recall. The column Voting indicates the pose voting
procedure using RANSAC-EPnP, an average of all direct pose hypotheses, or an
average of the direct pose estimates with the best n hypotheses in terms of
$\hat{C}$.

Table 3: Ablation Study for Pose Supervision. Provided is the average ADD/(-S) recall. The objects Eggbox and Glue are considered as symmetric objects. Supervision | Voting | LM | LM-O
---|---|---|---
IM | 2D | PnP | 0.654 | 0.280
DR | 2D | PnP | 0.712 | 0.330
| 6D | all | 0.715 | 0.342
DR-P | 2D | PnP | 0.672 | 0.341
| 6D | all | 0.672 | 0.345
DR-PC | 2D | PnP | 0.712 | 0.348
| 6D | n=1 | 0.722 | 0.338
| 6D | n=5 | 0.724 | 0.346
| 6D | n=10 | 0.732 | 0.350
| 6D | all | 0.738 | 0.349
Figure 5: Pose Supervision Comparison on LM-O’s Cat. From left to right: raw
image, pose obtained from geometric correspondences and RANSAC-EPnP, and
direct pose regression. Blue, red and green meshes indicate ground truth,
false positive and true positive pose (as measured by ADD).

The results show that supervising the training process with direct pose
regression (DR) improves the quality of the intermediate representation (IM)
tremendously. The improvement is from $0.654$ to $0.715$ on LM and from
$0.280$ to $0.342$ on LM-O. Using DR direct pose estimates is superior to
using those estimated by RANSAC-EPnP. Providing additional guidance with Lproj
(DR-P) improves for the occluded scenario of LM-O but is detrimental for LM.
Ultimately, enforcing consistency between the internal representation and the
correspondences projected to 2D using the regressed 6D pose with Lcons (DR-PC)
leads to good results for direct regression and when using the intermediate
representation on both datasets. Figure 5 shows an example of LM-O’s cat under
occlusion. The re-projected model using the ground truth is colored blue and
the wrong estimate based on ADD, using the intermediate representation and
RANSAC-EPnP, is colored red (middle image). Direct pose regression recovers
from the incorrect intermediate representation, which is displayed in green
(right image).

###  4.6 Qualitative Evaluation

Figure 6 visualizes results on LM-O, LM and IC-BIN. Displayed are projected
object meshes based on the estimated pose in the top row and estimated
bounding boxes in comparison to the respective ground truth in the bottom row.
Green and red bounding boxes portray estimates and ground truth, respectively.
The left image pair indicates a common error for LM-O: a false negative
detection of the Eggbox. The right image pair shows that some of IC-BIN’s
instances of Juice are difficult to detect while detecting Coffeecup works
well even under heavy occlusion if more than the lid is visible.

Figure 6: Qualitative results on LM-O, LM and IC-BIN. Top row shows
reprojected object meshes based on the estimated poses, bottom row shows
bounding box estimates. Blue boxes indicate the ground truth while green boxes
indicate estimates.

##  5 Conclusion

This paper presented a framework for pose estimation that processes up to $70$
object instances in a single image with effectively constant runtime and up to
$90$ in real-time ($>24$ fps). Our end-to-end trainable, single-staged
approach achieves up to $35$ times faster runtime than state-of-the-art
approaches with similar formulation while also generating similar pose
estimation accuracy. Directly regressing the 6D pose from sparse intermediate
geometric correspondences in the image space enables efficient network scaling
with respect to the amount of object classes and instances. During test time,
multiple instances are handled based on their 2D overlaps, which results in
negligible runtime increase with respect to the number of instances. As such,
we have developed an object pose estimator that is applicable and useful for a
broad variety of real-time tasks. In the future, we plan to improve our work
by learning the intermediate representation in a self-supervised manner in
order to overcome the necessity of defining geometric correspondences.

## Acknowledgement

We gratefully acknowledge the support of the EU-program EC Horizon 2020 for
Research and Innovation under grant agreement No. 101017089, project TraceBot,
the support by the Austrian Research Promotion Agency (FFG) under grant
agreement No. 879878, project K4R, the Austrian Science Fund (FWF) under grant
agreement No. I3969-N30, project InDex, and the NVIDIA Corporation for the
donation of a GPU used for this research.

## References

  * [1] Daniel Barath and Jiří Matas.  Graph-cut ransac.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6733–6741, 2018.
  * [2] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [3] Eric Brachmann, Alexander Krull, Frank Michel, Stefan Gumhold, Jamie Shotton, and Carsten Rother.  Learning 6D object pose estimation using 3D object coordinates.  In Proceedings of the European Conference on Computer Vision, pages 536–551, 2014.
  * [4] Bo Chen, Alvaro Parra, Jiewei Cao, Nan Li, and Tat-Jun Chin.  End-to-end learnable geometric vision by backpropagating pnp optimization.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8100–8109, 2020.
  * [5] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1251–1258, 2017.
  * [6] Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam.  Blenderproc.  CoRR, abs/1911.01911, 2019.
  * [7] Yan Di, Fabian Manhardt, Gu Wang, Xiangyang Ji, Nassir Navab, and Federico Tombari.  So-pose: Exploiting self-occlusion for direct 6d pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12396–12405, 2021.
  * [8] Andreas Doumanoglou, Rigas Kouskouridas, Sotiris Malassiotis, and Tae-Kyun Kim.  Recovering 6d object pose and predicting next-best-view in the crowd.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3583–3592, 2016.
  * [9] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian.  Centernet: Keypoint triplets for object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6569–6578, 2019.
  * [10] Richard Hartley and Andrew Zisserman.  Multiple view geometry in computer vision.  Cambridge university press, 2003.
  * [11] Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman.  The elements of statistical learning: data mining, inference, and prediction, volume 2.  Springer, 2009.
  * [12] Rasmus Laurvig Haugaard and Anders Glent Buch.  Surfemb: Dense and continuous correspondence distributions for object pose estimation with learnt surface embeddings.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6749–6758, 2022.
  * [13] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2961–2969, 2017.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Stefan Hinterstoisser, Vincent Lepetit, Slobodan Ilic, Stefan Holzer, Gary Bradski, Kurt Konolige, and Nassir Navab.  Model based training, detection and pose estimation of texture-less 3D objects in heavily cluttered scenes.  In Proceedings of the Asian Conference on Computer Vision, pages 548–562, 2012.
  * [16] Stefan Hinterstoisser, Vincent Lepetit, Paul Wohlhart, and Kurt Konolige.  On pre-trained image features and synthetic images for deep learning.  In Proceedings of the European Conference on Computer Vision Workshops, pages 0–0, 2018.
  * [17] Tomas Hodan, Daniel Barath, and Jiri Matas.  Epos: Estimating 6d pose of objects with symmetries.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11703–11712, 2020.
  * [18] Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas.  BOP challenge 2020 on 6D object localization.  Proceedings of the European Conference on Computer Vision Workshops, 2020.
  * [19] Tomáš Hodaň, Vibhav Vineet, Ran Gal, Emanuel Shalev, Jon Hanzelka, Treb Connell, Pedro Urbina, Sudipta N Sinha, and Brian Guenter.  Photorealistic image synthesis for object instance detection.  In 2019 IEEE international conference on image processing, pages 66–70. IEEE, 2019.
  * [20] Tingbo Hou, Adel Ahmadyan, Liangkai Zhang, Jianing Wei, and Matthias Grundmann.  Mobilepose: Real-time pose estimation for unseen objects with weak shape supervision.  arXiv preprint arXiv:2003.03522, 2020.
  * [21] Yinlin Hu, Pascal Fua, and Mathieu Salzmann.  Perspective flow aggregation for data-limited 6d object pose estimation.  arXiv preprint arXiv:2203.09836, 2022.
  * [22] Yinlin Hu, Pascal Fua, Wei Wang, and Mathieu Salzmann.  Single-stage 6d object pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2930–2939, 2020.
  * [23] Yinlin Hu, Joachim Hugonot, Pascal Fua, and Mathieu Salzmann.  Segmentation-driven 6d object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  * [24] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [25] Abhijit Kundu, Yin Li, and James M Rehg.  3d-rcnn: Instance-level 3d object reconstruction via render-and-compare.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3559–3568, 2018.
  * [26] Yann Labbé, Justin Carpentier, Mathieu Aubry, and Josef Sivic.  Cosypose: Consistent multi-view multi-object 6d pose estimation.  In European Conference on Computer Vision, pages 574–591. Springer, 2020.
  * [27] Hei Law and Jia Deng.  Cornernet: Detecting objects as paired keypoints.  In Proceedings of the European conference on computer vision, pages 734–750, 2018.
  * [28] Vincent Lepetit, Francesc Moreno-Noguer, and Pascal Fua.  Epnp: An accurate o (n) solution to the pnp problem.  International journal of computer vision, 81(2):155–166, 2009.
  * [29] Zhigang Li, Gu Wang, and Xiangyang Ji.  Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7678–7687, 2019.
  * [30] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2117–2125, 2017.
  * [31] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2980–2988, 2017.
  * [32] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [33] Lahav Lipson, Zachary Teed, Ankit Goyal, and Jia Deng.  Coupled iterative refinement for 6d multi-object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6728–6737, 2022.
  * [34] Jinhui Liu, Zhikang Zou, Xiaoqing Ye, Xiao Tan, Errui Ding, Feng Xu, and Xin Yu.  Leaping from 2d detection to efficient 6dof object pose estimation.  In Proceedings of the European Conference on Computer Vision, pages 707–714. Springer, 2020.
  * [35] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In Proceedings of the European conference on computer vision, pages 21–37. Springer, 2016.
  * [36] Fabian Manhardt, Diego Martin Arroyo, Christian Rupprecht, Benjamin Busam, Tolga Birdal, Nassir Navab, and Federico Tombari.  Explaining the ambiguity of object detection and 6d pose from visual data.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6841–6850, 2019.
  * [37] Diganta Misra.  Mish: A self regularized non-monotonic activation function.  arXiv preprint arXiv:1908.08681, 2019.
  * [38] Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang.  Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2020.
  * [39] Markus Oberweger, Mahdi Rad, and Vincent Lepetit.  Making deep heatmaps robust to partial occlusions for 3d object pose estimation.  In Proceedings of the European Conference on Computer Vision, pages 119–134, 2018.
  * [40] Keunhong Park, Arsalan Mousavian, Yu Xiang, and Dieter Fox.  Latentfusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10710–10719, 2020.
  * [41] Kiru Park, Timothy Patten, and Markus Vincze.  Pix2pose: Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, Oct 2019.
  * [42] Timothy Patten, Kiru Park, and Markus Vincze.  Dgcm-net: dense geometrical correspondence matching network for incremental experience-based robotic grasping.  Frontiers in Robotics and AI, page 120, 2020.
  * [43] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28, 2015.
  * [44] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115(3):211–252, 2015\.
  * [45] Yifei Shi, Junwen Huang, Xin Xu, Yifan Zhang, and Kai Xu.  Stablepose: Learning 6d object poses from geometrically stable patches.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15222–15231, 2021.
  * [46] Chen Song, Jiaru Song, and Qixing Huang.  Hybridpose: 6d object pose estimation under hybrid representations.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 431–440, 2020.
  * [47] Yongzhi Su, Mahdi Saleh, Torben Fetzer, Jason Rambach, Nassir Navab, Benjamin Busam, Didier Stricker, and Federico Tombari.  Zebrapose: Coarse to fine surface encoding for 6dof object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6738–6748, 2022.
  * [48] Martin Sundermeyer, Zoltan-Csaba Marton, Maximilian Durner, Manuel Brucker, and Rudolph Triebel.  Implicit 3d orientation learning for 6d object detection from rgb images.  In Proceedings of the european conference on computer vision, pages 699–715, 2018.
  * [49] George Terzakis and Manolis Lourakis.  A consistently fast and globally optimal solution to the perspective-n-point problem.  In Proceedings of the European Conference on Computer Vision, pages 478–494. Springer, 2020.
  * [50] Stefan Thalhammer, Markus Leitner, Timothy Patten, and Markus Vincze.  Pyrapose: Feature pyramids for fast and accurate object pose estimation under domain shift.  Proceedings of the IEEE International Conference on Robotics and Automation, 2021.
  * [51] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627–9636, 2019.
  * [52] Gu Wang, Fabian Manhardt, Jianzhun Shao, Xiangyang Ji, Nassir Navab, and Federico Tombari.  Self6d: Self-supervised monocular 6d object pose estimation.  In Proceedings of the European Conference on Computer Vision, pages 108–125. Springer, 2020.
  * [53] Gu Wang, Fabian Manhardt, Federico Tombari, and Xiangyang Ji.  Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16611–16621, 2021.
  * [54] Sergey Zakharov, Ivan Shugurov, and Slobodan Ilic.  Dpod: 6d pose object detector and refiner.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1941–1950, 2019.
  * [55] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li.  Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
  * [56] Xiaosong Zhang, Fang Wan, Chang Liu, Rongrong Ji, and Qixiang Ye.  FreeAnchor: Learning to match anchors for visual object detection.  In Neural Information Processing Systems, 2019.
  * [57] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745–5753, 2019.

## Appendix

This supplementary manuscript provides the reader with an overview of the
network layout in Section A, additional training details in Section B,
discussion of the performance in comparison to the state of the art in Section
C, a highlight of the capabilities of COPE in Section D and concludes with a
presentation of error cases in Section E.

###  A Network Design

ResNet101 [14] in conjunction with PFPN [50] is used for multi-scale feature
extraction. This is followed by three modules, one for location
classification, bounding box estimation and keypoint estimation that are
shared over feature pyramid levels. Each of the modules consist of four
convolution layers with Swish [37] activation. The convolution layers of the
classification module have 256 feature channels and those of the bounding box
and the keypoint estimation module have 512 feature channels. Linearly
activated output layers convolve the feature maps to a one-hot encoding for
the object class, bounding box corners and the number of keypoints. Feature
maps are not spatially reduced when passing through these modules. The
estimated keypoints are fed to a module that directly learns 6D pose
estimation, consisting of two convolution layers with 512 and 256 feature
channels with Mish activation as well as a linearly activated output
convolution with $a\cdot 9$ output parameters, where $a$ is the number of
classes. Thus, the outputs are $3$ for translation and $6$ for rotation for
each dataset object separately. The parameter $d$ in Equation 2 of the
submitted manuscript is set to $3$ in all experiments.

###  B Training Details

#### Backbone

The performance reduction occurring when estimating poses under domain shift
is partially alleviated by setting low-level stages of the backbone to non-
trainable [17, 16, 54]. EPOS [17] freezes the majority of the backbone, i.e.
early and middle flow of Xception-65 [5] when training exclusively on
synthetic data. We experienced this strategy to be infeasible for feature
pyramid-based approaches since-prediction are made from different feature map
resolutions taken also from early and intermediate feature maps of the
backbone. Thus, freezing the weights of all layers up to the output layers
reduces the pose estimation performance since feature learning for the present
task is limited. Hence, further investigating the adaption of the backbone to
have deeper early stages might lead to improved domain transfer.

#### Hyperparameters

All results are presented with the same set of hyperparameters, the only
exception is Table 2 in the submitted manuscript. For the ablations, the
parameter settings are mentioned in the table.

Figure 7: Comparison of Training Location Sampling: Utilizing the object
dimensions for sampling the pyramid level to train on (top) leads to a similar
amount of true locations per object and prevents ambiguous training locations.
Sampling small objects in coarse feature map resolutions results in aliasing
effects that are detrimental to convergence (bottom). Images are cropped to
improve visibility. Best viewed on a screen.

Training: The relevant hyperparameters are, apart from the network layout and
optimization itself, the base of the logarithm for location sampling, the
visibility threshold for foreground samples and the color and image space
augmentation hyperparameters. A visual comparison for location sampling on
physically-based rendering images of the LM [15] objects is provided in Figure
7. The top row shows our physics-based scheme, the bottom shows using all
pyramid levels for training. Without providing quantitative analysis it is
already visible that using all feature pyramid levels leads to aliasing
effects in the lower resolved pyramid levels, while explicitly using object
depth and dimension samples a similar amount of true locations per object.

Table 4: Color space augmentations applied during training. Augmentation | Chance (per channel) | Range
---|---|---
gaussian blur | 0.2 | $\sigma\sim\mathcal{U}(0.0,2.0)$
average/median/motion blur | 0.2 | $\sigma\sim\mathcal{U}(3,7)$
bilateral blur | 0.2 | $\sigma\sim\mathcal{U}(1,7)$
hue/saturation | 0.5 | $\mathcal{U}(-15,15)$
grayscale | 0.5 | $\mathcal{U}(0.0,0.2)$
add | 0.5 (0.5) | $\mathcal{U}(-0.04,0.04)$
multiply | 0.5 (0.5) | $\mathcal{U}(0.75,1.25)$
gamma contrast | 0.5 (0.5) | $\mathcal{U}(0.75,1.25)$
sigmoid contrast | 0.5 (0.5) | $\mathcal{U}(0,10)$
logarithmic contrast | 0.5 (0.5) | $\mathcal{U}(0.75,1.0)$
linear contrast | 0.5 (0.5) | $\mathcal{U}(0.7,1.3)$

Training on objects with too much occlusion is detrimental to pose estimation
performance. To overcome this issue we set the threshold for foreground
samples to $0.25$ of the object visibility when computing $L_{cls}$; all other
losses are computed for objects with at least $0.5$ object visibility. We
apply affine color space transformations to improve the domain transfer, as
also mentioned in the manuscript, parameters and ranges are provided in Table
4. Additionally we randomly scale training images by $5\%$ to improve
translation equivariance of our trained models.

Inference: Parameters that require manual assignment are the detection
threshold, the Intersection-over-Union (IoU) for clustering hypotheses, number
of hypotheses to use for the pose computation per instance and the maximal
number of instances per image. All image locations with a detection threshold
above $0.5$ are considered as foreground, thus as true locations containing
objects of interest and are consequently used for hypotheses clustering and
pruning. An IoU of $0.5$ is used for clustering instance hypotheses. An
ablation for the amount of hypotheses to derive the final pose is presented in
Table 3 in the submitted manuscript. The hyperparameter is set to $10$ for all
experiments apart from those ablating its influence. For the setting of the
manuscript, the number of maximal instances to detect is set to $100$. Yet,
despite there being little restrictions for that parameter, increasing this
threshold contributes little to nothing since $100$ instances to detect per
image is already a considerably large number. After clustering and pruning
hypotheses, those that are not supported by another hypothesis are discarded.

#### Evaluation Metrics

Comparison to the state of the art is provided using the performance score of
the BOP challenge [18]. The deviation of the estimated pose $\hat{P}$ to the
ground truth $P$ is projected to a scalar value using the average recall of
three error metrics. These are the Visual Surface Discrepancy, the Maximum
Symmetry-Aware Surface Distance and the Maximum Symmetry-Aware Projection
Distance:

$e_{VSD}=\underset{p\in\hat{V}\cup V}{avg}\begin{cases}0&\text{if
$p\in\hat{V}\cap V\wedge|\hat{D}(p)-D(p)|<\tau$ },\\\
1&\text{otherwise}\end{cases}\\\ e_{MSSD}=\underset{s\in S_{i}}{min}\
\underset{m\in M_{i}}{max}||\hat{P}m-Ps||_{2},\\\ e_{MSPD}=\underset{s\in
S_{i}}{min}\ \underset{m\in M_{i}}{max}||proj_{3D\to 2D}(\hat{P}m)\\\
-proj_{3D\to 2D}(Psm)||_{2},$ (10)

where $\hat{V}$ and $V$ are sets of image pixels; $\hat{D}$ and $D$ are
distance maps and $\tau$ is a misalignment tolerance. Distance maps are
rendered and compared to the distance map of the test image to derive
$\hat{V}$ and $V$. $S_{i}$ is a set of symmetry transformations that depend on
the visual ambiguities of the object mesh. $M_{i}$ is a subset of the mesh
vertices and $proj_{3D\to 2D}(.)$ denotes the projection to the image space.
For each of these metrics the average recall ($AR$) is measured when comparing
errors to multiple error thresholds (and $\tau$ in the case of $e_{VSD}$).
Results are then reported as the Average Recall:
$AR=(AR_{VSD}+AR_{MSSD}+AR_{MSPD})/3$.

Ablations are evaluated using the ADD(-S) recall [15]:

$\displaystyle e_{ADD}$ $\displaystyle=\underset{m\in
M_{i}}{avg}||\hat{P}m-Pm||,$ (11) $\displaystyle e_{ADDS}$
$\displaystyle=\underset{m_{1}\in M_{i}}{avg}\ \underset{m_{2}\in
M_{i}}{min}||\hat{P}m_{1}-Pm_{2}||.$ (12)

ADD measures the average deviation of models points using the corresponding
point distance. For objects exhibiting symmetric transformations, the ADD-S
error, using the closest point distance, is calculated. We report the fraction
of poses below the commonly used error threshold of $10\%$ of the object
diameter.

Results for object detection are reported using the the mean Average Precision
(mAP) of the Microsoft COCO object detection challenge [32].

###  C Comparing Detection Performance

The results reported in Table 2 in the submitted manuscript indicate that the
detection performance of COPE is inferior to that of FCOS [51]. However, this
conclusion has to be drawn with caution since FCOS only performs 2D Detection.
The network size of FCOS is $\sim 50$ million parameters just for object
detection while COPE additionally predict geometric correspondences and direct
6D poses with only $\sim 17$ million parameters more. Additionally, FCOS uses
an input image resolution with up to $1333$ pixels for the larger image side
while COPE uses $640\times 480$ input images. Thus, COPE solves twice as many
tasks with higher complexity from images with half of the input resolution.

###  D Highlights of COPE

Handling Multiple Mutually Occluding Objects The top row of Figure 8 shows
accurate bounding box and pose estimates on IC-BIN’s [8] Juice. Due to the
end-to-end trainability and the parallel learning of detection and pose
estimation, COPE learns to effectively handle multiple mutually occluding
instances of the same object. Increased mutual occlusion of instances of
Coffeecup is displayed in the middle row, which again shows accurate bounding
box and pose estimates for all the visible object instances. The bottom row
shows a scenario where both Juice and Coffeecup are present. Ultimately, a
false positive detection of Juice occurs in the center of the bulk due to the
heavy mutual occlusion of multiple instances.

Figure 8: Multiple Object Instances: Columns are, from left to right, input
image, 2D detections and reprojected object mehses based on the estimated
poses. Each instance is indicated with a specific color. Green and blue
bounding boxes correspond to estimates and ground truth, respective. Best
viewed on screen.

Handling Occlusion in Clutter Figure 9 displays accurate bounding box and pose
estimates on occluded examples of LM-O’s [3] Ape, Can and Eggbox. The middle
row shows similarly occluded examples of Drill, Holepunch and Glue and the
bottom row for Cat and Duck. Despite only training one model for all of LM’s
objects COPE is robustly handling each object, even under occlusion.

Figure 9: Occlusion Handling on LM-O: Columns are, from left to right, input
image, 2D detections and reprojected object mehses based on the estimated
poses. Each instance is indicated with a specific color. Green and blue
bounding boxes correspond to estimates and ground truth, respective. Best
viewed on screen.

###  E Error Cases

Figure 10 presents recurring errors on IC-BIN. The top row shows an instance
of Juice in top-view not being detected, indicated with a red and white
circle. Despite the high relative visibility of Juice these reduced views are
not often sampled during training data generation and are thus difficult to
detect during runtime. The middle row displays a similar error occuring for
Coffeecup, again indicated with red and white circles. Multiple top-view
orientated instances are not detected and thus result in false negative
detections. The bottom row shows one instance of each Coffeecup and Juice not
being detected despite providing rich visual features. Assigning true training
locations on objects with too low visibility leads to reduced performance
during inference. As such we treat objects that are largely occluded, i.e.
with less than $25\%$ relative object visibility, as background during
training. However, this leads to cases where discriminative object portions
are visible in the image, yet are treated as background, as can be seen here.
Further investigation is required to consider the abundance of features during
training target sampling to overcome such issues.

Figure 10: Error Cases on IC-BIN: Columns are, from left to right, input
image, 2D detections and reprojected object mehses based on the estimated
poses. Each instance is indicated with a specific color. Green and blue
bounding boxes correspond to estimates and ground truth, respective. Errors
are indicated with a red and white circle. Best viewed on screen.

Figure 11 presents common error cases on LM-O, again indicated with red and
white circles. The top row displays a false positive detection of the
Holepunch on a toy car with the same color and very similar material
properties as the object of interest. The middle row shows a similar false-
positive detection of the object Duck. In the bottom row an example of Eggbox
with an occlusion pattern that is unlikely to be similarly sampled when
randomizing object placements using physical modelling is displayed. Since it
is unlikely that objects of roughly the same size end up being stacked, these
cases are not experienced during training [6].

Figure 11: Error Cases on LM-O: Columns are, from left to right, input image,
2D detections and reprojected object mehses based on the estimated poses. Each
instance is indicated with a specific color. Green and blue bounding boxes
correspond to estimates and ground truth, respective. Errors are indicated
with a red and white circle. Best viewed on screen.