Florian<EMAIL_ADDRESS>Gwangbin<EMAIL_ADDRESS>Ignas
<EMAIL_ADDRESS>Roberto<EMAIL_ADDRESS>Department of
Engineering
University of Cambridge
Cambridge, UK SPARC: Sparse Render-and-Compare for CAD alignment

# SPARC: Sparse Render-and-Compare for CAD model alignment in a single RGB
Image

###### Abstract

Estimating 3D shapes and poses of static objects from a single image has
important applications for robotics, augmented reality and digital content
creation. Often this is done through direct mesh predictions [Gkioxari et
al.(2019)Gkioxari, Malik, and Johnson, Wang et al.(2018)Wang, Zhang, Li, Fu,
Liu, and Jiang, Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang] which
produces unrealistic, overly tessellated shapes or by formulating shape
prediction as a retrieval task followed by CAD model alignment [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai, Kuo et al.(2021)Kuo, Angelova, Lin, and
Dai, Grabner et al.(2020)Grabner, Wang, Zhang, Guo, Xiao, Vajda, Roth, and
Lepetit, Gümeli et al.(2022)Gümeli, Dai, and Nießner]. Directly predicting CAD
model poses from 2D image features is difficult and inaccurate [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai, Kuo et al.(2021)Kuo, Angelova, Lin, and
Dai]. Some works, such as ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner],
regress normalised object coordinates and use those for computing poses. While
this can produce more accurate pose estimates, predicting normalised object
coordinates is susceptible to systematic failure. Leveraging efficient
transformer architectures [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Vinyals,
Zisserman, and Carreira] we demonstrate that a sparse, iterative, render-and-
compare approach is more accurate and robust than relying on normalised object
coordinates. For this we combine 2D image information including sparse depth
and surface normal values which we estimate directly from the image with 3D
CAD model information in early fusion. In particular, we reproject points
sampled from the CAD model in an initial, random pose and compute their depth
and surface normal values. This combined information is the input to a pose
prediction network, SPARC-Net which we train to predict a 9 DoF CAD model pose
update. The CAD model is reprojected again and the next pose update is
predicted. Our alignment procedure converges after just 3 iterations,
improving the state-of-the-art performance on the challenging real-world
dataset ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and
Nießner] from $25.0\%$ [Gümeli et al.(2022)Gümeli, Dai, and Nießner] to
$31.8\%$ instance alignment accuracy. Code will be released under
https://github.com/florianlanger/SPARC.

##  1 Introduction

Figure 1: Comparison of our CAD model alignment approach to ROCA [Gümeli et
al.(2022)Gümeli, Dai, and Nießner]. For each pixel of the detected object ROCA
predicts the 3D normalised object coordinates (NOCs) in a canonical,
normalised frame. However, those predictions are susceptible to systematic
offsets (see Front View and Side View). While the reprojected NOCs match the
image, the corresponding CAD model alignment does not. Our approach in
contrast reprojects points and surface normals sampled from the CAD model in
an initial pose into the image and uses those to predict pose updates. By
iteratively updating the pose and reprojecting our system achieves precise CAD
model alignments.

Previous work on shape and pose prediction can be classified into two
different types of methods relying either on shape generation [Gkioxari et
al.(2019)Gkioxari, Malik, and Johnson, Wang et al.(2018)Wang, Zhang, Li, Fu,
Liu, and Jiang, Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang] or
shape retrieval [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai, Kuo et
al.(2021)Kuo, Angelova, Lin, and Dai, Grabner et al.(2020)Grabner, Wang,
Zhang, Guo, Xiao, Vajda, Roth, and Lepetit, Gümeli et al.(2022)Gümeli, Dai,
and Nießner]. Generative approaches usually struggle to produce realistic
object shapes. For methods relying on shape retrieval one of the key
challenges is to align the retrieved CAD model to the object detected in the
image [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai, Kuo et al.(2021)Kuo,
Angelova, Lin, and Dai, Grabner et al.(2020)Grabner, Wang, Zhang, Guo, Xiao,
Vajda, Roth, and Lepetit, Gümeli et al.(2022)Gümeli, Dai, and Nießner]. Many
existing approaches directly regress object poses from the 2D features of the
image [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai, Kuo et al.(2021)Kuo,
Angelova, Lin, and Dai, Maninis et al.(2020)Maninis, Popov, Nießner, and
Ferrari, Engelmann et al.(2021)Engelmann, Rematas, Leibe, and Ferrari].
However, this produces approximate poses rather than accurate alignments. ROCA
[Gümeli et al.(2022)Gümeli, Dai, and Nießner] follows a more geometric
approach in which they predict normalised object coordinates (NOCs) [Taylor et
al.(2012)Taylor, Shotton, Sharp, and Fitzgibbon], dense correspondences from
2D pixel to 3D points in a canonical object space, that are used to compute
the object pose. The fundamental issue with learning NOCs is that it is
unclear how different shapes should be registered with each other. This means
that the NOCs do not generalise well between CAD models and predicting them
often fails with a systematic offset leading to a displacement in the final
alignment (see Figure 1). Rather than using NOCs we propose a sparse, render-
and-compare approach. We combine 2D image information including sparse depth
and surface normals along with RGB colors with 3D CAD model information.
Specifically, we initialise a CAD model in a generic pose and reproject points
and surface normals sampled uniformly from its surface onto the image plane.
This combined information is used by our pose update prediction network,
SPARC-Net, to estimate a 9-DoF pose update. After adding the predicted pose
update to the initial pose we reproject the CAD model again and estimate the
next pose update step. Repeating this procedure we obtain the final pose in
just three iterations.
Having access to estimated normal and depth values from the image and
reprojected normals and depth from the CAD model allows the network to
evaluate the current pose and predict an accurate pose update by easily
comparing observed (image) and projected (CAD model) information. This is in
contrast to other approaches that do not make use of any shape information
when predicting object poses [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai, Kuo
et al.(2021)Kuo, Angelova, Lin, and Dai] or that rely on shape encodings
[Gümeli et al.(2022)Gümeli, Dai, and Nießner] which seems to be a more
difficult learning task than using render-and-compare. Note also that because
of the render-and-compare our approach does not require the ability to
register different CAD models with each other nor for the network to memorise
3D shapes.
We choose the Perceiver [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Vinyals,
Zisserman, and Carreira] architecture over traditional CNNs for the pose
prediction network as Perceivers [Jaegle et al.(2021)Jaegle, Gimeno, Brock,
Vinyals, Zisserman, and Carreira] allow for an efficient processing of the
sparse object reprojection as they do not require a full, image-sized input.
Also Perceivers [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Vinyals, Zisserman,
and Carreira] have linear time and memory complexity in terms of input as
opposed to traditional transformers [Vaswani et al.(2017)Vaswani, Shazeer,
Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin]. Not only do sparse
inputs reduce the memory and network complexity, but we also show that they
improve the alignment accuracy by avoiding overfitting. Using our approach we
improve upon the state-of-the art performance on the challenging real-world
dataset ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and
Nießner] from $25.0\%$ instance alignment accuracy to $31.8\%$ instance
alignment accuracy.
Our contributions include:

  * •

a novel, sparse, render-and-compare method that achieves state-of-the-art
results in CAD model alignment from a single RGB image

  * •

a demonstration of how 3D CAD model information and 2D image information can
be effectively combined with early-fusion

  * •

highlighting that sparse inputs improve alignment accuracy while at the same
time significantly decreasing memory and compute complexity.

##  2 Related Work

In this section we discuss relevant works including shape estimation from a
single image, CAD model retrieval and alignment and efficient transformer
architectures.
Shape estimation from a single RGB image. Most works on shape estimation from
a single image are structured such as to use an RGB image as input to a neural
network and directly predict a 3D shape using some particular representation.
Various representations have been explored ranging from voxels [Choy et
al.(2016)Choy, Xu, Gwak, Chen, and Savarese] to point clouds [Fan et
al.(2017)Fan, Su, and Guibas, Zeng et al.(2020)Zeng, Karaoglu, and Gevers],
meshes [Wang et al.(2018)Wang, Zhang, Li, Fu, Liu, and Jiang, Gkioxari et
al.(2019)Gkioxari, Malik, and Johnson, Pan et al.(2019)Pan, Han, Chen, Tang,
and Jia, Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang], packed
spheres [Fan et al.(2017)Fan, Su, and Guibas], binary space partitioning [Chen
et al.(2020)Chen, Tagliasacchi, and Zhang], convex polytopes [Deng et
al.(2020)Deng, Genova, Yazdani, Bouaziz, Hinton, and Tagliasacchi], signed
distance fields [Park et al.(2019)Park, Florence, Straub, Newcombe, and
Lovegrove] to other implicit representations [Mildenhall et
al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng].
However, this is a difficult learning task and regardless of the
representation chosen most approaches struggle to predict realistic shapes.
Because of these issues we use a retrieval-based method.
CAD model retrieval and alignment. Rather than generating shapes a second
class of approaches estimates 3D shapes by retrieving from large-scale CAD
model databases [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang,
Li, Savarese, Savva, Song, Su, Xiao, Yi, and Yu]. The large number of
available CAD models ensures that appropriate CAD models can be retrieved
which can reconstruct the scene in a clean and compact representation that can
be easily consumed for downstream applications. Existing approaches differ
significantly in their procedures for aligning the retrieved CAD models to the
image. Mask2CAD [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] as well as [Kuo
et al.(2021)Kuo, Angelova, Lin, and Dai, Maninis et al.(2020)Maninis, Popov,
Nießner, and Ferrari] simply regress the object pose from image features.
However, this produces inaccurate pose estimates and requires the network to
memorize every CAD model shape, consequently performing poorly for unseen CAD
models. [Langer et al.(2021)Langer, Budvytis, and Cipolla] explicitly
leverages the geometry of the retrieved CAD model by estimating 2D-3D keypoint
matches but depend on accurate segmentation masks and require exact
correspondence between CAD model and the object observed in the image. ROCA
[Gümeli et al.(2022)Gümeli, Dai, and Nießner] learns dense correspondences
between 2D pixels and 3D object coordinates in a normalised, canonical frame
and use those for computing the object pose. While these dense correspondences
are more robust than sparse ones, they often fail systematically, leading to
constant offsets in the pose alignments (see Supp. mat.).
Another class of methods uses render-and-compare to iteratively update pose
estimates [Grabner et al.(2020)Grabner, Wang, Zhang, Guo, Xiao, Vajda, Roth,
and Lepetit, Izadinia et al.(2017)Izadinia, Shan, and Seitz]. These tend to be
more precise, but rely on good pose initialisation [Grabner et
al.(2020)Grabner, Wang, Zhang, Guo, Xiao, Vajda, Roth, and Lepetit, Izadinia
et al.(2017)Izadinia, Shan, and Seitz]. Traditionally these have been very
slow due to the time consuming rendering process and the large number of
renderings required. The large number of renderings is needed as existing
render-and-compare works for shape and pose estimation learn a comparison
function which they directly minimise at test time using gradient descent
which requires a large number of steps [Grabner et al.(2020)Grabner, Wang,
Zhang, Guo, Xiao, Vajda, Roth, and Lepetit, Izadinia et al.(2017)Izadinia,
Shan, and Seitz] (on the order of 100s to 1000). In spirit our approach is
most similar to the render-and-compare approaches. However, the advantage of
our method is that it does not require rendering of the full object, but just
a simple reprojection of a small amount of 3D points (e.g. 100,see 5.2), and
it directly learns pose updates rather than a comparison function, making our
approach a lot faster than [Izadinia et al.(2017)Izadinia, Shan, and Seitz,
Grabner et al.(2020)Grabner, Wang, Zhang, Guo, Xiao, Vajda, Roth, and
Lepetit].
Efficient transformer architectures. In recent years Transformers [Vaswani et
al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and
Polosukhin] have been used for a range of different computer vision tasks
[Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai,
Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby, Sun
et al.(2019)Sun, Myers, Vondrick, Murphy, and Schmid, Parmar et
al.(2018)Parmar, Vaswani, Uszkoreit, Kaiser, Shazeer, Ku, and Tran, Wang et
al.(2021)Wang, Yeshwanth, and Nießner]. However, the all-to-all attention
mechanism used in classical transforms suffers from a quadratic scaling
problem. [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Vinyals, Zisserman, and
Carreira, Jaegle et al.(2022)Jaegle, Borgeaud, Alayrac, Doersch, Ionescu,
Ding, Koppula, Zoran, Brock, Shelhamer, Hénaff, Botvinick, Zisserman, Vinyals,
and Carreira, Carreira et al.(2022)Carreira, Koppula, Zoran, Recasens,
Ionescu, Hénaff, Shelhamer, Arandjelovic, Botvinick, Vinyals, Simonyan,
Zisserman, and Jaegle] are one line of work aiming to make transformers more
efficient and enabling them to deal with larger inputs. They achieve this by
using cross-attention from the inputs to a small set of latent units and
subsequently only perform all-to-all attention in this smaller latent space.

##  3 Method

Figure 2: Method. (i) Given an RGB image we predict per pixel surface normals
$\mathbf{N}_{Img}$ and depth $\mathbf{D}_{Img}$ (2D image information) and use
bounding box and CAD model retrievals from [Gümeli et al.(2022)Gümeli, Dai,
and Nießner] (see Sec. 3.1). (ii) We sparsely sample RGB colors,
$\mathbf{N}_{Img}$ and $\mathbf{D}_{Img}$ from three different regions (the
reprojected CAD model points, the bounding box and the whole image), stack
them and add pixel coordinates $(u,v)$ and a token $\tau$ allowing the network
to distinguish between different input types. 3D points and surface normals
are sampled from the CAD model in an initial canonical pose and reprojected
into the image plane (see Sec. 3.2). 2D image information, 3D CAD model
information and extra information is combined to form the input to the pose
prediction network, SPARC-Net, predicting pose update steps
$\Delta\textbf{T},\Delta\textbf{R},\Delta\textbf{S}$ and rotation
classification score $c$. Based on the prediction the pose is updated, the CAD
model sparsely rendered and the next pose update is predicted (see Sec. 3.3).

This section explains the key steps of our method. In a first step, we perform
object detection as well as surface normal and depth estimation in an image.
In a second step, we sample depth and normal values from an image and combine
them with reprojected depth and surface normals sampled uniformly from a CAD
model. Finally, in a third step, we iterate the refinements of the 9-DoF
initial pose of the CAD model, partially recomputing sparse inputs of step two
for every refinement.

###  3.1 Object Detection, Normal and Depth Prediction

As a first step we perform 2D Object detection, including bounding box and
category prediction. The category prediction determines the class of CAD model
that is retrieved while the bounding box allows us to initialise the CAD model
pose (see Sec. 4) and guide the point sampling (see Sec. 3.2). We use the same
object detections and CAD model retrievals as ROCA [Gümeli et al.(2022)Gümeli,
Dai, and Nießner] for exact comparability. However, any other method could be
used as well. Further we estimate per pixel surface normal
$\mathbf{N}_{\text{Img}}$ and depth values $\mathbf{D}_{\text{Img}}$ as these
contain crucial geometric information that can be used for precise pose
predictions. For both surface normal and depth estimation, we use a light-
weight convolutional encoder-decoder architecture from [Alhashim and
Wonka(2018)]. The training losses are the same as the state-of-the-art works
[Bae et al.(2021)Bae, Budvytis, and Cipolla] for surface normal estimation and
[Bae et al.(2022)Bae, Budvytis, and Cipolla] for depth estimation. We use
ground truth surface normals provided by [Huang et al.(2019)Huang, Zhou,
Funkhouser, and Guibas] and ground truth depth as provided by ScanNet [Dai et
al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner] (for details see
Supp. mat.). When training the surface normal and depth network the train and
test split used for evaluating SPARC-Net is respected.

###  3.2 Input Fusion

When attempting to estimate CAD model pose one is presented with two different
kinds of information: 2D image information and 3D CAD model information. We
fuse the two by sampling 3D points and corresponding surface normals from the
CAD model and reprojecting those into the image plane. Comparing the
reprojected surface normals and the depth associated with the reprojected
point to depth and surface normals estimated from the 2D image allows a
network to easily evaluate the current pose and predict a pose update based on
this comparison.
2D image information. We stack RGB colors and predicted surface normals
$\mathbf{N}_{\text{Img}}$ and depth values $\mathbf{D}_{\text{Img}}$. However,
rather than using the entire image as input we sparsely sample pixels from
three selected regions. We sample $N_{\text{reproj}}$ pixels onto which CAD
model points are reprojected (see paragraph “3D CAD model information”),
$N_{\text{bbox}}$ pixels inside the bounding box and $N_{\text{context}}$
pixels from the entire image. Respectively, these provide information on
whether the current pose matches the image, what the pose update should be and
global context. In Fig. 4b and Tab. 2 we demonstrate that sampling image
information sparsely reduces overfitting, leading to a better alignment
accuracy, and greatly reducing memory consumption and network complexity (see
Sec. 5.2). For each of the sampled pixels containing color, depth and surface
normal values we add the pixel coordinates $(u,v)$ and a token $\tau$ (to
allow the network to distinguish between different types of inputs) and stack
them to a
$(N_{\text{reproj}}+N_{\text{bbox}}+N_{\text{context}})\times(3+3+1+2+1)$
dimensional tensor as seen in Fig. 2.
3D CAD model information. We represent the CAD model as a collection of
$N_{\text{CAD}}$ 3D points sampled uniformly from the object surface. We
simply reproject 3D object points and their surface normals into the image
plane which reduces the rendering operation to a perspective projection using
a single matrix multiplication. The intuition behind this idea is that for
texture-less CAD models almost all information is contained in the 3D shape.
Hence there is no need for a full rendering pipeline that takes into account
material, lighting, textures or visibility111While reprojecting all CAD model
points into the image plane ignores potentially important occlusion effects we
found SPARC-Net to perform well regardless. We have also tried to add
information about whether a 3D object point is occluded or not, but did not
observe any improvement in performance.. We stack reprojected surface normals
$\mathbf{N}_{\text{CAD}}$ and the computed depth values
$\mathbf{D}_{\text{CAD}}$ with their pixel coordinates and a token $\tau=3$.
Padding this tensor with zeros as RGB values we obtain a $N_{\text{CAD}}\times
10$ shaped tensor.
Extra information. Additionally, we encode the bounding box information by
sampling 10 points from each of the sides of the bounding box and inputting
their pixel values with a different token $\tau=4$ (padding extra dimensions
with 0’s). We also directly input the initialised pose consisting of
translation T, rotation parameterised as quaternion Q, scale S and the CAD
model ID encoded as a binary vector, each padded with 0’s and a unique token.
Finally, we concatenate the three different blocks containing image,
reprojected CAD model and extra information to form a
($N_{\text{reproj}}+N_{\text{bbox}}+N_{\text{context}}+N_{\text{CAD}}+N_{\text{extra}})\times
10$ = ($N_{\text{input}}\times 10)$ dimensional tensor. This tensor is Fourier
encoded with 64 frequency bands with a maximum frequency of 1120 (same as
original Perceiver [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Vinyals,
Zisserman, and Carreira] applied to point clouds) resulting in a
$N_{\text{input}}\times(10+64*2+1)=N_{\text{input}}\times C_{\text{input}}$
tensor which serves as the network input described in the next section.

###  3.3 Pose updates and Iterative Refinement

Combining image information and CAD model information in the image plane
allows the network to easily evaluate a given pose and predict a refinement
based on this evaluation. We repeat this process $N_{\text{iter}}$ times
allowing the network to refine its own predictions. Specifically, given an
initial pose ($\textbf{T},\textbf{Q},\textbf{S}$) and all combined information
explained in Sec. 3.2 we use a Perceiver network [Jaegle et al.(2021)Jaegle,
Gimeno, Brock, Vinyals, Zisserman, and Carreira] to predict refinement steps
($\Delta\textbf{T},\Delta\textbf{Q},\Delta\textbf{S}$) such that
($\textbf{T}+\Delta\textbf{T},\textbf{Q}\cdot\Delta\textbf{Q},\textbf{S}+\Delta\textbf{S}$)
is close to the correct pose
$(\textbf{T}_{\text{gt}},\textbf{Q}_{\text{gt}},\textbf{S}_{\text{gt}})$ =
($\textbf{T}+{\Delta\textbf{T}}_{\text{gt}},\textbf{Q}\cdot{\Delta\textbf{Q}}_{\text{gt}},\textbf{S}+{\Delta\textbf{S}}_{\text{gt}}$).
Similar to previous works [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai, Kuo et
al.(2021)Kuo, Angelova, Lin, and Dai] we find that learning rotations over the
full, non-euclidean rotation space is difficult. We therefore follow a coarse-
to-fine approach where we simultaneously train the network to predict a binary
classification $c$ whether an initial rotation lies within the correct
$90^{\circ}$ rotation bin $c_{\text{gt}}$ around the vertical. For classifying
rotation we use a standard binary cross-entropy loss $L_{\mathrm{BCE}}$ and
for learning the offsets we use the L2 loss such that our loss function is
given by:

$L_{\text{align
}}=w_{\text{c}}L_{\mathrm{BCE}}(c_{\text{gt}},c)+w_{\text{t}}L_{\mathrm{L2}}(\Delta\mathbf{T}_{\text{gt}},\Delta\mathbf{T})+w_{\text{s}}L_{\mathrm{L2}}(\Delta\mathbf{S}_{\text{gt}},\Delta\mathbf{S})+w_{\text{q}}L_{\mathrm{L2}}(\Delta\mathbf{Q}_{\text{gt}},\Delta\mathbf{Q}).$
(1)

At test time we apply SPARC-Net to four rotation initialisation that are
$90^{\circ}$ rotated around the vertical. For each of those we predict the
probability $c$ indicating whether the correct rotation lies within $\pm
45^{\circ}$ of the tested initialisation. For the initialisation with the
highest estimated probability $c$ we subsequently predict the 9 DoF pose
updates and iteratively rerender and refine the pose.

##  4 Experimental Setup

This section briefly describes the dataset used for training and testing our
method, as well as the evaluation metrics used and the hyperparameters chosen.
ScanNet dataset. Similar to [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai, Kuo
et al.(2021)Kuo, Angelova, Lin, and Dai, Nie et al.(2020)Nie, Han, Guo, Zheng,
Chang, and Zhang, Gümeli et al.(2022)Gümeli, Dai, and Nießner] we train and
test our approach on the ScanNet25k image data [Dai et al.(2017)Dai, Chang,
Savva, Halber, Funkhouser, and Nießner] for which [Avetisyan et
al.(2019)Avetisyan, Dahnert, Dai, Savva, Chang, and Nießner] provide CAD model
annotation for a wide range of objects. This dataset contains 20k training
images representing 1200 train scenes and 5k validation images from 300
different evaluation scenes. We train and test our method on the 9 categories
with the most CAD model annotations covering more than 2500 distinct shapes.
Evaluation metric. We follow the original evaluation protocol introduced by
Scan2CAD [Avetisyan et al.(2019)Avetisyan, Dahnert, Dai, Savva, Chang, and
Nießner] which evaluates CAD model alignments per-scene. In the same way as
ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] we transform predicted CAD
model poses into ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber,
Funkhouser, and Nießner] world coordinates and apply 3D non-maximum
suppression to eliminate multiple detections of the same object from different
images. A CAD model prediction is considered correct if the object class
prediction is correct, the translation error is less than 20 cm, the rotation
error is less than $20^{\circ}$ and the scale ratio is less than $20\%$. We
found that there was a bug in the original evaluation code introduced by
[Avetisyan et al.(2019)Avetisyan, Dahnert, Dai, Savva, Chang, and Nießner]
which was subsequently used for evaluating [Gümeli et al.(2022)Gümeli, Dai,
and Nießner, Kuo et al.(2020)Kuo, Angelova, Lin, and Dai]. When computing the
scale error the formula
$\text{s}_{\text{error}}=|\sum_{i=x,y,z}(S_{i}/S^{\text{gt}}_{i}-1)|$ was used
instead of
$\text{s}_{\text{error}}=\sum_{i=x,y,z}|(S_{i}/S^{\text{gt}}_{i})-1|$ which
allowed scale errors in different directions to cancel each other out. We
corrected this mistake and reevaluated [Kuo et al.(2020)Kuo, Angelova, Lin,
and Dai, Gümeli et al.(2022)Gümeli, Dai, and Nießner] (see Supp. mat.).
Input details. For the main experiment we choose $N_{\text{CAD}}=1000$ and use
the reprojected points as samples from the 2D image such that
$N_{\text{reproj}}=1000$. Further we use $N_{\text{bbox}}=1000$ and
$N_{\text{context}}=5000$ as also shown in Fig. 2.
Perceiver architecture. For the Perceiver [Jaegle et al.(2021)Jaegle, Gimeno,
Brock, Vinyals, Zisserman, and Carreira] we set $N_{\text{latent}}=128$ and
$C_{\text{latent}}=256$ and as depicted in Fig. 2 repeat three blocks of one
cross attention layer followed by two self-attention layers with weight
sharing between each of the layers in the three blocks.
Train details. We train the SPARC-Net for 300 epochs using the LAMB [You et
al.(2020)You, Li, Reddi, Hseu, Kumar, Bhojanapalli, Song, Demmel, Keutzer, and
Hsieh] optimiser (as used by the original Perceiver [Jaegle et
al.(2021)Jaegle, Gimeno, Brock, Vinyals, Zisserman, and Carreira]) with
learning rate 0.001. For the loss function in Eq. 1 we set $w_{c}=0.5$,
$w_{t}=0.5$, $w_{s}=0.5$ and $w_{q}=1$. We use a batchsize of 80. For more
information on sampling and initialising poses at train and test time see the
Supp. mat.
Implementation Details. All code is implemented in PyTorch [Paszke et
al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin,
Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison, Tejani,
Chilamkurthy, Steiner, Fang, Bai, and Chintala]. SPARC-Net was trained on a
single TitanXp for 36 hours.

##  5 Experimental Results

In the first part of this section we compare ourselves to the state-of-the-art
approaches [Gümeli et al.(2022)Gümeli, Dai, and Nießner, Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai, Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and
Zhang]. In the second part, we investigate and ablate different key aspects of
our system, including the 2D pixel sampling, the sparse 3D object
representation, render-and-compare, the number of refinements used and the
relevance of different input information.

Figure 3: Qualitative results on ScanNet [Dai et al.(2017)Dai, Chang, Savva,
Halber, Funkhouser, and Nießner]. Our sparse, render-and-compare approach
allows for more precise CAD model alignment compared to ROCA [Gümeli et
al.(2022)Gümeli, Dai, and Nießner]. Typical failure cases of ROCA include
systematic failure when predicting normalised object coordinates leading to
wrong object translations (row 3) and wrong scale predictions (row 4). Row 5
shows failure cases for both methods in complex scenes (left) and when object
boundaries are not clearly identifiable (right).

###  5.1 Main Results

Tab. 1 shows that our approach outperforms all competing approaches in all
object classes (except for ROCA on the class “display”). Further we
significantly improve both the class average alignment accuracy from 18.4% to
24.9% as well as the instance alignment accuracy from 25.0% to 31.8%. Visually
comparing our predictions to ROCA [Gümeli et al.(2022)Gümeli, Dai, and
Nießner] in Fig. 3 we note that the improvements are mainly due to better
translation and scale predictions. ROCA [Gümeli et al.(2022)Gümeli, Dai, and
Nießner] relies on predicting normalised object coordinates for estimating
translation. However, due to difficulties in aligning and registering
different CAD models to a canonical frame, predicting NOCs often fails with a
systematic offset leading to an offset in the final alignment (see Supp.
mat.). In contrast, our approach is independent of the existence of a
canonical object frame as it directly compares a reprojected object to an
image. Further, ROCA decouples the scale predictions from predicting rotation
and translation which can produce very wrong alignments for bad scale
predictions (see Fig. 3). We also evaluate our system when using ROCA’s
rotation predictions as an initialisation as opposed to our own classified
rotation bins. Here we observe a further improvement of our alignment
accuracy. The reason why ROCA’s rotation predictions are good is that they are
computed geometrically from correspondences and are largely unaffected by
systematic offsets in NOCs predictions and wrong scale predictions.

###  5.2 Ablation

Note that in addition to average class alignment accuracy and instance
alignment accuracy we report $\mathbf{T},\mathbf{R},\mathbf{S}$ and $c$
accuracy in Tab. 2 where the same evaluation protocol is used as described in
Sec. 4, but rather than requiring $\mathbf{T}$, $\mathbf{R}$ and $\mathbf{S}$
to be correct at the same time only the quantity of interest is required to be
correct ($c$ is considered correct if the binary rotation classification is
correct). If not otherwise specified in the following “accuracy” or
“performance” refers to instance alignment accuracy.

Method | bathtub | bed | bin | bkshlf | cabinet | chair | display | sofa | table | class | instance
---|---|---|---|---|---|---|---|---|---|---|---
Number of Instances # | 120 | 70 | 232 | 212 | 260 | 1093 | 191 | 113 | 553 | 9 | 2844
Total3D-ODN [Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang] | 10.0 | 2.9 | 16.8 | 2.8 | 4.2 | 14.4 | 13.1 | 5.3 | 6.7 | 8.5 | 10.4
Mask2CAD-b5 [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] | 7.5 | 2.9 | 24.6 | 1.4 | 5.0 | 29.9 | 13.1 | 5.3 | 5.6 | 10.6 | 16.7
ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] | 20.8 | 8.6 | 26.3 | 9.0 | 13.1 | 39.9 | 24.6 | 10.6 | 12.7 | 18.4 | 25.0
SPARC-Net (ours) | 25.8 | 25.7 | 24.6 | 14.2 | 20.8 | 51.5 | 17.8 | 28.3 | 15.4 | 24.9 | 31.8
SPARC-Net + ROCA rot init | 25.0 | 30.0 | 36.2 | 14.2 | 19.2 | 52.3 | 20.4 | 28.3 | 20.1 | 27.3 | 34.1

Table 1: Alignment Accuracy on ScanNet [Dai et al.(2017)Dai, Chang, Savva,
Halber, Funkhouser, and Nießner, Avetisyan et al.(2019)Avetisyan, Dahnert,
Dai, Savva, Chang, and Nießner] in comparison to the state-of-the-art.

Image sampling. We compare sampling the entire image to sparse sampling and
observe worse alignment accuracy when using the entire image ($25.0\%$ vs.
$31.8\%)$ due to faster overfitting at train time (see Fig. 4b). We further
simplify our sampling scheme and instead of sampling the image globally, the
bounding box and the reprojected query, we use samples from just the bounding
box, $N_{\text{bbox}}=1000$. Surprisingly, we observe that we can recover
almost the same alignment accuracy $30.1\%$ compared to $31.8\%$ before. By
traininig and testing with less than 1% of the original number of pixel (image
resolution 480 x 360) we reduce the memory by a factor of 100. The complexity
of the Perceiver [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Vinyals,
Zisserman, and Carreira] is given by
$\mathcal{O}(N_{\text{input}}N_{\text{latent}})+\mathcal{O}(LN_{\text{latent}}^{2})\approx\mathcal{O}(N_{\text{input}}N_{\text{latent}})$
where $N_{\text{input}}>>N_{\text{latent}}$, $N_{\text{input}}>>L$. When using
the whole image as input $N_{\text{input}}$ is dominated by the number of
pixels $N_{\text{input}}\approx 480\times 360$. Therefore as
$N_{\text{latent}}=128$ and $L=2$ (two self-attention layers following the
cross-attention layer) sparse inputs also reduce the compute complexity by a
factor of 100, greatly speeding up training and testing.
Sparse CAD model representation. When reducing the number of points
$N_{\text{CAD}}$ we observe that remarkably with just 100 points we obtain
similar performance as with 1000 points (30.0% vs. 31.8%), showing that 3D
shapes can indeed be represented very sparsely. However, as we further
decrease the number of points we observe a rapid decline in performance (20.9%
for $N_{\text{CAD}}=50$ and $4.2\%$ for $N_{\text{CAD}}=50$. Interestingly,
this decrease is almost entirely due to worse rotation predictions (see
$\mathbf{R}$ accuracy $24.8\%$ compared to $67.9\%$). Intuitively, this makes
sense as an extremely sparse point cloud has very little structure from which
to predict rotation while translation and scale can still be predicted.
Initialising those extremely sparse models with ROCA [Gümeli et
al.(2022)Gümeli, Dai, and Nießner] rotation predictions we can still recover
reasonable alignment accuracy of $24.0\%$ and $29.9\%$ for $N_{\text{CAD}}=20$
and $N_{\text{CAD}}=50$.
Render-and-Compare. We test the assumption that Render-and-Compare is
beneficial for precise pose predictions by considering two networks which are
trained either on no 3D shape information but just the CAD model ID or for
which the CAD model is always presented in the same canonical pose. The first
is motivated by works (e.g. [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai, Kuo
et al.(2021)Kuo, Angelova, Lin, and Dai]) that contain no explicit 3D shape
information and the second by [Kanazawa et al.(2018)Kanazawa, Black, Jacobs,
and Malik] that perform iterative pose updates without explicit rerendering.
Similar to before both of these networks fail to learn accurate rotation
predictions (both for the classification and the refinement) therefore we only
show results where we initialise poses with ROCA [Gümeli et al.(2022)Gümeli,
Dai, and Nießner] rotation predictions. Here we note that using just the CAD
ID results in a significant decrease in performance (21.0%), whereas using 3D
information but presenting the CAD model in the same canonical pose achieves
$25.9\%$ instance alignment accuracy.
Number of refinements. Training and evaluating our network for different
number of iterations we observe a significant improvement when increasing the
number of refinements from 1 ($26.4\%$) to 3 ($31.8\%$) and no more
improvement for 5 refinements ($31.3\%$).

Figure 4:  a) Visualisation of refinements in iterative render-and-compare. b)
Train and validation loss as well as validation accuracy on the ScanNet
dataset [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner,
Avetisyan et al.(2019)Avetisyan, Dahnert, Dai, Savva, Chang, and Nießner].
When training on the whole image SPARC-Net overfits the training data more
easily resulting in higher validation loss and worse instance alignment
accuracy.

Accuracy | 2D Image Sampling | 3D CAD model points | R. and C. | N Refinement | Input Information
---|---|---|---|---|---
|  $N_{\text{reproj}}=100$ $N_{\text{bbox}}=100$ $N_{\text{context}}=0$ $N_{\text{CAD}}=100$ |  $N_{\text{reproj}}=0$ $N_{\text{bbox}}=200$ $N_{\text{context}}=0$ $N_{\text{CAD}}=100$ |  $N_{\text{reproj}}=0$ $N_{\text{bbox}}=1000$ $N_{\text{context}}=0$ $N_{\text{CAD}}=100$ |  $N_{\text{reproj}}=0$ $N_{\text{bbox}}=0$ $N_{\text{context}}=5000$ $N_{\text{CAD}}=1000$ |  Whole img $N_{\text{CAD}}=1000$ |  20 |  20* |  50 |  50* |  100 | 1000 |  Just CAD ID* |  Canon- ical pose* | 1 | 3 | 5 |  No D |  No N |  No RGB |  No RTS or CAD info
Class | 12.9 | 12.7 | 23.9 | 14.8 | 20.3 | 3.5 | 17.7 | 15.9 | 24.9 | 23.1 | 24.9 | 15.0 | 20.2 | 16.6 | 24.9 | 25.2 | 15.0 | 16.6 | 24.7 | 22.2
Instances | 18.5 | 19.7 | 30.1 | 20.1 | 25.0 | 4.2 | 24.0 | 20.9 | 29.9 | 30.0 | 31.8 | 21.0 | 25.9 | 26.4 | 31.8 | 31.3 | 18.8 | 20.7 | 30.5 | 28.9
T | 40.8 | 42.1 | 42.4 | 37.1 | 38.7 | 38.0 | 39.9 | 38.2 | 42.5 | 42.2 | 45.4 | 36.4 | 41.4 | 42.3 | 45.4 | 43.7 | 30.7 | 40.2 | 42.6 | 42.2
R | 49.1 | 48.7 | 70.8 | 61.4 | 68.2 | 24.8 | 63.7 | 58.4 | 70.9 | 71.1 | 67.9 | 63.8 | 63.7 | 65.5 | 67.9 | 71.5 | 68.2 | 56.8 | 72.3 | 66.3
S | 61.0 | 62.4 | 69.0 | 61.3 | 66.8 | 63.7 | 63.3 | 66.4 | 67.7 | 67.8 | 68.4 | 62.9 | 68.9 | 62.8 | 68.4 | 66.4 | 64.5 | 63.7 | 68.5 | 67.2
R class | 70.7 | 71.3 | 75.2 | 69.7 | 72.9 | 53.4 | 76.4 | 68.9 | 77.4 | 75.6 | 75.9 | 77.0 | 76.3 | 74.8 | 75.9 | 75.6 | 73.0 | 74.5 | 76.1 | 74.4

Table 2: Ablation. We perform a number of different ablations as explained in
detail in Sec. 5.2. Bold numbers are the ones referred to in Sec. 5.2. Note
that a “*” indicates that ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner]
rotation predictions are used for initialising rotations. Our main experiment
is listed twice under “3D CAD model points $N_{\text{CAD}}=1000$” and “N
Refinements, $N=3$” for convenience. “R. and C.” stands for Render-and-
Compare.

Input modalities. Furthermore, we investigate the importance of the different
information that is provided as input to the network. Here we find that both
depth and normal predictions are crucial for SPARC-Net. Not providing depth
predictions from the 2D image reduces the accuracy to $18.8\%$ while not
providing normal predictions reduces it to $20.7\%$. We find that for depth
this drop in performance is mainly due to worse $\mathbf{T}$ predictions
($30.7\%$ compared to $45.4\%$) whereas for the normals it is due to worse
rotation predictions ($56.8\%$ compared to $67.9\%$) which is exactly as one
would expect. In contrast, not providing RGB color only reduces the accuracy
marginally to $30.5\%$. This makes sense as given depth and surface normal
estimates color only adds very little information for shape alignments. When
no extra information is provided the accuracy is reduced to $28.9\%$. The
individual accuracies suggest that the network uses extra information to learn
the distribution of object positions as we observe the largest decrease for
$\mathbf{T}$ accuracy to $42.2\%$.

##  6 Conclusion

We have proposed SPARC, a sparse render-and-compare approach for CAD model
alignment from a single RGB image. We show how to effectively combine 3D CAD
model information and 2D image information with early-fusion which helps our
pose prediction network to estimate accurate pose updates. In this way we
improve the state-of-the-art on ScanNet [Dai et al.(2017)Dai, Chang, Savva,
Halber, Funkhouser, and Nießner] from $25.0\%$ to $31.8\%$ instance alignment
accuracy. We demonstrate that using sparse 2D image information (less than 1%
of available pixels) reduces overfitting leading to better pose predictions
and greatly reducing memory and network complexity. Furthermore, sparse input
processing may bring even more benefits when dealing with sensory input that
is naturally sparse such as LIDAR or event cameras. In the future, we plan to
expand this work to updating not only the estimated pose, but also the CAD
model shape in order to alleviate the requirement of CAD models that are very
similar to the objects in the image.

## References

  * [Alhashim and Wonka(2018)] Ibraheem Alhashim and Peter Wonka.  High quality monocular depth estimation via transfer learning.  _arXiv preprint arXiv:1812.11941_ , 2018.
  * [Avetisyan et al.(2019)Avetisyan, Dahnert, Dai, Savva, Chang, and Nießner] Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, and Matthias Nießner.  Scan2cad: Learning cad model alignment in rgb-d scans.  In _Proc. Computer Vision and Pattern Recognition (CVPR), IEEE_ , 2019\.
  * [Avetisyan et al.(2020)Avetisyan, Khanova, Choy, Dash, Dai, and Nießner] Armen Avetisyan, Tatiana Khanova, Christopher Choy, Denver Dash, Angela Dai, and Matthias Nießner.  Scenecad: Predicting object alignments and layouts in rgb-d scans.  In _The European Conference on Computer Vision (ECCV)_ , August 2020\.
  * [Bae et al.(2021)Bae, Budvytis, and Cipolla] Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla.  Estimating and exploiting the aleatoric uncertainty in surface normal estimation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 13137–13146, 2021.
  * [Bae et al.(2022)Bae, Budvytis, and Cipolla] Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla.  Multi-view depth estimation by fusing single-view depth probability with multi-view geometry.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2842–2851, June 2022.
  * [Carreira et al.(2022)Carreira, Koppula, Zoran, Recasens, Ionescu, Hénaff, Shelhamer, Arandjelovic, Botvinick, Vinyals, Simonyan, Zisserman, and Jaegle] João Carreira, Skanda Koppula, Daniel Zoran, Adrià Recasens, Catalin Ionescu, Olivier J. Hénaff, Evan Shelhamer, Relja Arandjelovic, Matthew M. Botvinick, Oriol Vinyals, Karen Simonyan, Andrew Zisserman, and Andrew Jaegle.  Hierarchical perceiver.  _CoRR_ , abs/2202.10890, 2022.
  * [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva, Song, Su, Xiao, Yi, and Yu] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [Chen et al.(2020)Chen, Tagliasacchi, and Zhang] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang.  BSP-Net: Generating Compact Meshes via Binary Space Partitioning.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Virtual, June 2020.
  * [Choy et al.(2016)Choy, Xu, Gwak, Chen, and Savarese] Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction.  In _Proc. 14th European Conference on Computer Vision_ , Amsterdam, NL, October 2016.
  * [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  Scannet: Richly-annotated 3d reconstructions of indoor scenes.  In _Proc. Computer Vision and Pattern Recognition (CVPR), IEEE_ , 2017\.
  * [Deng et al.(2020)Deng, Genova, Yazdani, Bouaziz, Hinton, and Tagliasacchi] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.  CvxNet: Learnable Convex Decomposition.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Virtual, June 2020.
  * [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  _ICLR_ , 2021.
  * [Engelmann et al.(2021)Engelmann, Rematas, Leibe, and Ferrari] Francis Engelmann, Konstantinos Rematas, Bastian Leibe, and Vittorio Ferrari.  From Points to Multi-Object 3D Reconstruction.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021.
  * [Fan et al.(2017)Fan, Su, and Guibas] Haoqiang Fan, Hao Su, and Leonidas Guibas.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Honolulu, USA, July 2017.
  * [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh R-CNN.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Seoul, Korea, October 2019.
  * [Grabner et al.(2020)Grabner, Wang, Zhang, Guo, Xiao, Vajda, Roth, and Lepetit] Alexander Grabner, Yaming Wang, Peizhao Zhang, Peihong Guo, Tong Xiao, Peter Vajda, Peter M. Roth, and Vincent Lepetit.  Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2020.
  * [Gümeli et al.(2022)Gümeli, Dai, and Nießner] Can Gümeli, Angela Dai, and Matthias Nießner.  Roca: Robust cad model retrieval and alignment from a single image.  In _Proc. Computer Vision and Pattern Recognition (CVPR), IEEE_ , 2022\.
  * [Huang et al.(2019)Huang, Zhou, Funkhouser, and Guibas] Jingwei Huang, Yichao Zhou, Thomas Funkhouser, and Leonidas J Guibas.  Framenet: Learning local canonical frames of 3d surfaces from a single rgb image.  In _Proc. of IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * [Izadinia et al.(2017)Izadinia, Shan, and Seitz] Hamid Izadinia, Qi Shan, and Steven M. Seitz.  IM2CAD.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Honolulu, USA, July 2017.
  * [Jaegle et al.(2021)Jaegle, Gimeno, Brock, Vinyals, Zisserman, and Carreira] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira.  Perceiver: General perception with iterative attention.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 4651–4664. PMLR, 18–24 Jul 2021.
  * [Jaegle et al.(2022)Jaegle, Borgeaud, Alayrac, Doersch, Ionescu, Ding, Koppula, Zoran, Brock, Shelhamer, Hénaff, Botvinick, Zisserman, Vinyals, and Carreira] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier J. Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira.  Perceiver IO: A general architecture for structured inputs & outputs.  _ICLR_ , 2022.
  * [Kanazawa et al.(2018)Kanazawa, Black, Jacobs, and Malik] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In _Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] Weicheng Kuo, Anelia Angelova, Tsung-Yi Lin, and Angela Dai.  Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve.  In _Proc. 16th European Conference on Computer Vision_ , Glasgow, UK, August 2020.
  * [Kuo et al.(2021)Kuo, Angelova, Lin, and Dai] Weicheng Kuo, Anelia Angelova, Tsung-yi Lin, and Angela Dai.  Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Montreal (Virtual), October 2021.
  * [Langer et al.(2021)Langer, Budvytis, and Cipolla] F. Langer, I. Budvytis, and R. Cipolla.  Leveraging geometry for shape estimation from a single rgb image.  In _Proc. British Machine Vision Conference_ , (Virtual), November 2021.
  * [Lichtsteiner et al.(2008)Lichtsteiner, Posch, and Delbruck] Patrick Lichtsteiner, Christoph Posch, and Tobi Delbruck.  A 128$\times$ 128 120 db 15 $\mu$s latency asynchronous temporal contrast vision sensor.  _IEEE Journal of Solid-State Circuits_ , 43(2):566–576, 2008.  10.1109/JSSC.2007.914337.
  * [Loshchilov and Hutter(2019)] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _ICLR_ , 2019.
  * [Maninis et al.(2020)Maninis, Popov, Nießner, and Ferrari] Kevis-Kokitsi Maninis, Stefan Popov, Matthias Nießner, and Vittorio Ferrari.  Vid2cad: Cad model alignment using multi-view constraints from videos.  _arXiv_ , 2020.
  * [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _Proc. 16th European Conference on Computer Vision_ , Glasgow, UK (Virtual), August 2020.
  * [Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang] Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang.  Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Virtual, June 2020.
  * [Pan et al.(2019)Pan, Han, Chen, Tang, and Jia] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia.  Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Seoul, Korea, October 2019.
  * [Park et al.(2019)Park, Florence, Straub, Newcombe, and Lovegrove] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Long Beach, CA, USA, June 2019.
  * [Parmar et al.(2018)Parmar, Vaswani, Uszkoreit, Kaiser, Shazeer, Ku, and Tran] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In Jennifer Dy and Andreas Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4055–4064. PMLR, 10–15 Jul 2018.
  * [Paszke et al.(2019)Paszke, Gross, Massa, Lerer, Bradbury, Chanan, Killeen, Lin, Gimelshein, Antiga, Desmaison, Kopf, Yang, DeVito, Raison, Tejani, Chilamkurthy, Steiner, Fang, Bai, and Chintala] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.
  * [Smith and Topin(2018)] Leslie Smith and Nicholay Topin.  Super-convergence: Very fast training of residual networks using large learning rates.  _ICLR_ , 2018.
  * [Sun et al.(2019)Sun, Myers, Vondrick, Murphy, and Schmid] Chen Sun, Austin Myers, Carl Vondrick, Kevin P. Murphy, and Cordelia Schmid.  Videobert: A joint model for video and language representation learning.  _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 7463–7472, 2019.
  * [Taylor et al.(2012)Taylor, Shotton, Sharp, and Fitzgibbon] Jonathan Taylor, Jamie Shotton, Toby Sharp, and Andrew Fitzgibbon.  The vitruvian manifold: Inferring dense correspondences for one-shot human pose estimation.  In _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 103–110, 2012.  10.1109/CVPR.2012.6247664.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , pages 5998–6008, 2017.
  * [Wang et al.(2018)Wang, Zhang, Li, Fu, Liu, and Jiang] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images.  In _Proc. 15th European Conference on Computer Vision_ , Munich, Germany, September 2018.
  * [Wang et al.(2021)Wang, Yeshwanth, and Nießner] Xinpeng Wang, Chandan Yeshwanth, and Matthias Nießner.  Sceneformer: Indoor scene generation with transformers.  In _2021 International Conference on 3D Vision (3DV)_ , pages 106–115. IEEE, 2021.
  * [Yifan et al.(2020)Yifan, Aigerman, Kim, Chaudhuri, and Sorkine-Hornung] Wang Yifan, Noam Aigerman, Vladimir G. Kim, Siddhartha Chaudhuri, and Olga Sorkine-Hornung.  Neural cages for detail-preserving 3d deformations.  In _CVPR_ , 2020.
  * [You et al.(2020)You, Li, Reddi, Hseu, Kumar, Bhojanapalli, Song, Demmel, Keutzer, and Hsieh] Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh.  Large batch optimization for deep learning: Training bert in 76 minutes.  In _International Conference on Learning Representations_ , 2020.
  * [Zeng et al.(2020)Zeng, Karaoglu, and Gevers] Wei Zeng, Sezer Karaoglu, and Theo Gevers.  Inferring Point Clouds from Single Monocular Images by Depth Intermediation.  _arXiv_ , 2020.

## Supplementary Material

We provide additional information for various aspects of our main work. In
Sec. A we present alignment accuracies evaluated with the orignal (wrong)
evaluation code and the corrected one. In Sec. B and Sec. C we give additional
information for training our pose prediction network, SPARC-Net, and the
networks used for depth and surface normal estimation. In Sec. D we highlight
issues when trying to align differently shaped CAD models with each other and
the resulting systematic offsets that appear in the predictions of the
normalised object coordinates. We quantitatively support this section by
ablating our system with ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner]
predictions in Sec. E. Finally, we discuss limitations and possibility for
future works in Sec. F and explain the provided video showing qualitative
results on ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and
Nießner] in Sec. G.

##  Appendix A Correction to Evaluation Script

Scan2CAD [Avetisyan et al.(2019)Avetisyan, Dahnert, Dai, Savva, Chang, and
Nießner] proposed to consider a CAD alignment correct if the object class
prediction is correct, the translation error is less than 20 cm, the rotation
error is less than 20 degrees and the scale ratio is less than 20 %. We found
that there was a bug in the original evaluation code which was subsequently
used to evaluate ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] and
Mask2CAD [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai]. When computing the
scale ratio the formula
$\text{s}_{\text{error}}=|\sum_{i=x,y,z}(s^{\text{pred}}_{i}/s^{\text{gt}}_{i}-1)|$
was used instead of
$\text{s}_{\text{error}}=\sum_{i=x,y,z}|(s^{\text{pred}}_{i}/s^{\text{gt}}_{i})-1|$
which allowed scale errors in different directions to cancel each other out.
We correct for this mistake and reevaluate [Kuo et al.(2020)Kuo, Angelova,
Lin, and Dai, Gümeli et al.(2022)Gümeli, Dai, and Nießner]. The accuracy
computed with the evaluation code containing the mistake are presented in
Table 3 and their corrected counterparts are presented in Table 4.

##  Appendix B Additional Train and Test Information

Train data generation. We generate per-image CAD model pose annotations for
all images in ScanNet25k [Dai et al.(2017)Dai, Chang, Savva, Halber,
Funkhouser, and Nießner] (which is the original ScanNet[Dai et al.(2017)Dai,
Chang, Savva, Halber, Funkhouser, and Nießner] dataset sampled for every 100th
frame) by transforming CAD model pose annotations from [Avetisyan et
al.(2019)Avetisyan, Dahnert, Dai, Savva, Chang, and Nießner] from ScanNet [Dai
et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner] world
coordinates into camera coordinates. ScanNet25k [Dai et al.(2017)Dai, Chang,
Savva, Halber, Funkhouser, and Nießner] contains ca. 20K train images from
1200 different scenes. Note that for a given image we only train our CAD model
to align CAD models whose center is reprojected into the image (such as to
avoid training on objects that are barely visible). Further, we filter objects
which do not have at least 50% of their reprojected depth values within 30 cm
of the ground truth depth values. This avoids training on objects that are
hidden behind walls or in other way strongly occluded.
Pose sampling at train time. When initialising poses for training we uniformly
sample translations T to be between 1 and 5 m along the $Z$-axis and sample
$X$ and $Y$ by uniformly sampling pixel coordinates in the predicted bounding
box and mulitplying the corresponding pixel bearing by the sampled $Z$ to
obtain $X$ and $Y$ values. Scale values S are sampled uniformly in the range
of all observed CAD model scales for the detected category. 25% of train
examples are uniformly sampled in random rotation (any azimuthal angle,
$20^{\circ}$ tilt range and $40^{\circ}$ elevation range around
$\textbf{R}^{\text{gt}}$) to learn to classify rotations, whereas 75% of train
examples are sampled in the correct rotation bin ($90^{\circ}$ azimuthal angle
range, $20^{\circ}$ tilt range and $40^{\circ}$ elevation range around
$\textbf{R}^{\text{gt}}$) to learn to regress pose offsets.
At test time the pose prediction network is used to iteratively refine its own
predictions. During training it is therefore crucial to not just generate
random CAD model poses, but also poses based on the networks predictions. This
ensures that the poses sampled during training are as similar to the ones the
network sees at test time as possible. For every image and corresponding CAD
model annotation we therefore use a randomly initialised pose as well as the
two subsequent refinements predicted by the network as training data.
Specifically, for a batch of $N$ train examples, each containing image
information and CAD model information sampled in a random initial pose, we
predict the pose updates and apply losses. Based on the predicted pose
updates, we update the CAD model poses for every train example and recompute
the inputs. The recomputed inputs are fed through the network again, pose
updates are predicted and losses applied. This process is repeated once more,
after which new training images with new CAD models in random initial poses
are sampled as the next training examples.
Test details. At test time R is initialised with four rotations at 0, 90, 180
and 270 degrees around the vertical axis, 0 degrees tilt and 20 elevation
angle (such that the camera is looking slightly down at an object that is
standing straight upright). T is initialised to have $z=3$ m and x and y such
that the reprojected T lies at the bounding box center. The scale S is
initialised with the median value of all CAD models for the given category.
For the rotation with the highest classification score $c$ we predict
($\Delta\textbf{T},\Delta\textbf{R},\Delta\textbf{S}$) and iteratively refine
the pose 3 times. Note that our pose prediction network is very robust to poor
initialisation for scale and translation (see the video explained in Sec. G)
but can not reorient CAD models if their are initialised within the wrong
$90^{\circ}$ rotation bin.

##  Appendix C Details for Training Surface Normal and Depth Networks

For both surface normal $\mathbf{N}_{\text{Img}}$ and depth estimation
$\mathbf{D}_{\text{Img}}$, we use a light-weight convolutional encoder-decoder
architecture from [Alhashim and Wonka(2018)]. For both tasks, we predict the
per-pixel probability distribution for the output and supervise the network by
minimizing the negative log-likelihood (NLL) of the ground truth. For surface
normal, we parameterize the distribution using the Angular vonMF distribution,
proposed in [Bae et al.(2021)Bae, Budvytis, and Cipolla], while we
parameterise the depth distribution with a gaussian distribution. After
training, we discard the uncertainty and only use the predicted mean values.
We use ground truth surface normals provided by [Huang et al.(2019)Huang,
Zhou, Funkhouser, and Guibas] and ground truth depth as provided by ScanNet
[Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner],
respecting the train/test split. For depth we train on all two million
available train images, while for surface normals we train on all images for
which [Huang et al.(2019)Huang, Zhou, Funkhouser, and Guibas] provide
annotations and that are within the set of train images which results in ca.
200K train images. We train both networks for ten epochs using the AdamW
optimiser [Loshchilov and Hutter(2019)] and schedule the learning rate using
1cycle policy [Smith and Topin(2018)] with $lr_{\text{max}}=3.5\times 10^{-4}$
(same as [Bae et al.(2022)Bae, Budvytis, and Cipolla]). We use a batch size of
four for both surface normal and depth training.

Method | bathtub | bed | bin | bkshlf | cabinet | chair | display | sofa | table | class | instance
---|---|---|---|---|---|---|---|---|---|---|---
Mask2CAD-b5 [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] | 8.3 | 2.9 | 25.9 | 3.8 | 5.4 | 30.9 | 17.3 | 5.3 | 7.1 | 11.9 | 17.9
ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] | 22.5 | 10.0 | 29.3 | 14.2 | 15.8 | 41.0 | 30.4 | 15.9 | 14.6 | 21.5 | 27.4
Ours | 26.7 | 25.7 | 26.7 | 17.5 | 23.8 | 52.6 | 22.5 | 32.7 | 17.7 | 27.3 | 33.9
Ours + ROCA rot init | 27.5 | 30.0 | 41.4 | 17.5 | 23.5 | 53.7 | 26.2 | 32.7 | 22.6 | 30.6 | 36.8

Table 3: Alignment accuracy with original (incorrect) treatment of scale
predictions on ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser,
and Nießner].

Method | bathtub | bed | bin | bkshlf | cabinet | chair | display | sofa | table | class | instance
---|---|---|---|---|---|---|---|---|---|---|---
Mask2CAD-b5 [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] | 7.5 | 2.9 | 24.6 | 1.4 | 5.0 | 29.9 | 13.1 | 5.3 | 5.6 | 10.6 | 16.7
ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] | 20.8 | 8.6 | 26.3 | 9.0 | 13.1 | 39.9 | 24.6 | 10.6 | 12.7 | 18.4 | 25.0
Ours | 25.8 | 25.7 | 24.6 | 14.2 | 20.8 | 51.5 | 17.8 | 28.3 | 15.4 | 24.9 | 31.8
Ours + ROCA rot init | 25.0 | 30.0 | 36.2 | 14.2 | 19.2 | 52.3 | 20.4 | 28.3 | 20.1 | 27.3 | 34.1

Table 4: Alignment accuracy with correct treatment of scale predictions on
ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner].

##  Appendix D Aligning CAD Models for Normalised Object Coordinates

ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] relies on predicting
Normalised Object Coordinates (NOCs) for each pixel of the detected objects.
NOCs are 3D object coordinates in a canonical frame in which objects have been
aligned. However, here we demonstrate that aligning different object shapes
with each other is not trivial. Figure 6 shows that even for two very similar
shapes different alignments are possible depending on which object parts one
wishes to align. This means that NOCs learned for one shape do not generalise
well to other shapes. When attempting to predict NOCs we observe that ROCA
[Gümeli et al.(2022)Gümeli, Dai, and Nießner] often predicts NOCs with a
systematic offset (see Figure 6). Here we show the NOCs predicted overlayed in
the canonical object frame. One can see that the 3D coordinates predicted are
often systematically above or below the actual object. This means that even
though the reprojected NOCs roughly match the objects in the image the
corresponding object alignment is very wrong.

Figure 5: Issues with shape alignment. When trying attempting to align Shape 1
and Shape 2 with each other different alignments are possible. Alignment 1
aligns the top and the bottom of the chairs. This ensures that both shapes fit
into the same normalised 3D bounding box and is the alignment used for NOCs.
Alignment 2 aligns the chairs by their seating area. While now both shapes do
not lie in the same normalised 3D bounding box, their seating areas align
which is useful when trying to predict their coordinates. The point is that
for different shapes different alignments are possible depending on which
object parts one wishes to align. The more different the shapes are the harder
it is to find some global alignment that aligns all different object parts
with each other. This means that NOCs learned for one shape do not generalise
well to NOCs learned for other shapes.

Figure 6: Visualisation of ROCA’s NOCs predictions. For different inputs we
show the GT CAD model alignment, our prediction and ROCA’s prediction. Further
we show the estimated NOCs overlayed in the canonical object frame from
different views for relevant objects. We also show the NOCs reprojected back
into the image under the predicted pose. One can observe that the predicted
NOCs are often systematically offset from the actual 3D shape. Therefore even
though the NOCs reprojected under the estimated pose roughly match the images
the corresponding CAD alignments can be very wrong.

##  Appendix E Ablating our system with ROCA predictions

The previous section showed that ambiguities in aligning different shapes with
each other can lead to systematic offsets when predicting NOCs. A second issue
with ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] is that object scale
is directly regressed which is inaccurate and can produce very wrong scale
estimates subsequently leading to poor translation estimates. We demonstrate
both of these issues quantitatively by replacing either the scale, translation
or rotation prediction of our system with ROCA’s [Gümeli et al.(2022)Gümeli,
Dai, and Nießner] prediction (see Table 5, lower half). The top half of the
table is a copy of our main results table and is for reference only. Note that
the row “Ours + ROCA rot init” uses the ROCA rotation predictions as an
initialisation which is subsequently refined by our own predictions. In
contrast the rows “Ours + ROCA rotation/translation/scale” show results when
replacing the respective predictions with ROCA’s predictions and keeping them
fixed during the refinement process. We observe a noticeable drop in alignment
accuracy when replacing our translation or scale predictions with ROCA’s
confirming the issues presented above. ROCA’s rotation predictions are largely
unaffected by inaccurate scale predictions and systematic offsets in NOCS and
we observe that they perform similarly to ours.

Method | bathtub | bed | bin | bkshlf | cabinet | chair | display | sofa | table | class | instance
---|---|---|---|---|---|---|---|---|---|---|---
Number of Instances # | 120 | 70 | 232 | 212 | 260 | 1093 | 191 | 113 | 553 | 9 | 2844
ROCA [Gümeli et al.(2022)Gümeli, Dai, and Nießner] | 20.8 | 8.6 | 26.3 | 9.0 | 13.1 | 39.9 | 24.6 | 10.6 | 12.7 | 18.4 | 25.0
Ours | 25.8 | 25.7 | 24.6 | 14.2 | 20.8 | 51.5 | 17.8 | 28.3 | 15.4 | 24.9 | 31.8
Ours + ROCA rot init | 25.0 | 30.0 | 36.2 | 14.2 | 19.2 | 52.3 | 20.4 | 28.3 | 20.1 | 27.3 | 34.1
Ours + ROCA rotation | 19.2 | 22.9 | 29.3 | 13.2 | 20.0 | 46.1 | 16.8 | 27.4 | 15.9 | 23.4 | 29.6
Ours + ROCA translation | 20.8 | 10.0 | 22.8 | 9.0 | 11.5 | 40.7 | 14.1 | 17.7 | 11.4 | 17.6 | 24.2
Ours + ROCA scale | 1.7 | 24.3 | 22.8 | 6.6 | 12.7 | 43.9 | 17.3 | 21.2 | 9.6 | 17.8 | 24.9

Table 5: Ablation of SPARC-Net with ROCA predictions. The top half repeats the
results presented in the main paper. The lower half shows results when we
replace one of our predictions (rotation, translation or scale) with ROCAs
[Gümeli et al.(2022)Gümeli, Dai, and Nießner] prediction. Note that those
predictions are not refined with our own predictions. Row 3 “Ours + ROCA rot
init” in contrast uses ROCA’s rotation prediction as an initialisation which
is subsequently refined with our own predictions.

##  Appendix F Limitations

This section lists limitations of the current approach which we plan to
address in future works.
Dense depth and surface normal predictions. Currently our method uses dense
depth and surface normal estimates that are precomputed. If depth and surface
normals are computed in realtime, predicting them sparsely only for relevant
pixel coordinates may reduce inference time. Further for real applications our
method could make use of additional sensory information such as LiDAR which
provides naturally sparse depth inputs that can easily replace predicted depth
values in our pipeline. Further for video applications event cameras
[Lichtsteiner et al.(2008)Lichtsteiner, Posch, and Delbruck] may be of
particular interest as these are extremely fast and energy efficient as they
only react to changes in light intensity, therefore providing sparse image
data containing object edge information that is crucial for 3D shape
estimation or CAD model alignment.
Rotation predictions. Currently our handling of rotation is not elegant as at
test time it requires four extra forward passes through our network to
determine the initialisation of the rotation. This aspect could be improved by
reprojecting the four different rotation initialisation simultaneously while
at the same time predicting pose updates for all of them, but then only
applying those pose updates to the rotation initialisation with the highest
estimated probability.
CAD model refinements. Currently, our approach is limited to refining an
initial object pose, but not the initial object shape. In general all
retrieval-based approaches for shape estimation are limited by the
availability of a fitting CAD model. However, even with growing CAD model
databases it is unrealistic that every object in the real world will have a
precisely fitting CAD model in the database. Therefore it is important to
deform a retrieved CAD model to better fit an observed object. This could be
nicely achieved with the presented framework by in addition to 9 DoF pose
updates predicting $N$-dim shape updates where $N$ are the numbers of
parameters of some shape transformation. One possibiliy for such a shape
transformation function are neural cages [Yifan et al.(2020)Yifan, Aigerman,
Kim, Chaudhuri, and Sorkine-Hornung].
Joint shape and pose predictions. Currently for reconstructing scenes every
object is treated individually. However, this neglects important information
contained within object-object relationships. These can contain information
about the pose (e.g. two tables that are standing next to each other are
likely to be aligned with each other) or the shape (e.g. chairs around a table
are likely to have the same shape which can be a very important signal when
dealing with strong occlusion.). Taking into account such information (e.g. by
modelling it as a scene-graph [Avetisyan et al.(2020)Avetisyan, Khanova, Choy,
Dash, Dai, and Nießner] will further improve our shape and pose predictions.

##  Appendix G Visualisation Video

Figure 7: Visualisation Video. We visualise intermediate refinement steps on
ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner]
in a video (https://youtu.be/eVVW__0QGnM).

In the visualisation video (https://youtu.be/eVVW__0QGnM) one can see nicely
that our pose prediction network, SPARC-Net, is very robust to rough
initialisation and able to significantly translate, rotate and scale CAD
models to fit the objects observed in the image. The results demonstrate the
advantage of an iterative procedure: the first refinement is usually a large
pose update, transforming the often very bad initialisation to roughly match
the pose of the object in the image. The second and third refinement in
contrast are smaller pose updates that really align the CAD model with the
objects in the image.

Note that for some images in the video the number of ground truth CAD models
and the number of CAD models for which the pose is predicted do not correspond
exactly, either due to missing 2D object detections or because of missing
ground truth annotation as [Avetisyan et al.(2019)Avetisyan, Dahnert, Dai,
Savva, Chang, and Nießner] did not provide exhaustive CAD model annotation for
ScanNet [Dai et al.(2017)Dai, Chang, Savva, Halber, Funkhouser, and Nießner].