# 3D Surface Reconstruction in the Wild by Deforming Shape Priors from
Synthetic Data

Nicolai Häni  University of Minnesota
Minneapolis, Minnesota 55455
Email<EMAIL_ADDRESS>Jun-Jee Chao  University of Minnesota
Minneapolis, Minnesota 55455
Email<EMAIL_ADDRESS>Volkan Isler  University of Minnesota
Minneapolis, Minnesota 55455
Email<EMAIL_ADDRESS>
###### Abstract

Reconstructing the underlying 3D surface of an object from a single image is a
challenging problem that has received extensive attention from the computer
vision community. Many learning-based approaches tackle this problem by
learning a 3D shape prior from either ground truth 3D data or multi-view
observations. To achieve state-of-the-art results, these methods assume that
the objects are specified with respect to a fixed canonical coordinate frame,
where instances of the same category are perfectly aligned. In this work, we
present a new method for joint category-specific 3D reconstruction and object
pose estimation from a single image. We show that one can leverage shape
priors learned on purely synthetic 3D data together with a point cloud pose
canonicalization method to achieve high quality 3D reconstruction in the wild.
Given a single depth image at test time, we first transform this partial point
cloud into a learned canonical frame. Then, we use a neural deformation field
in to reconstruct the 3D surface of the object. Finally, we jointly optimize
object pose and 3D shape to fit the partial depth observation. Our approach
achieves state-of-the-art reconstruction performance across several real-world
datasets, even when trained only on synthetic data. We further show that our
method generalizes to different input modalities, from dense depth images to
sparse and noisy LIDAR scans.

##  I Introduction

Surface reconstruction of a 3D object from a partial observation, such as a
depth image or a LIDAR scan, is a longstanding problem in computer vision [61,
65, 38, 46]. Discovering the full shape of an object from a partial input has
many applications, including in visual servoing [28], robotic manipulation [3,
39, 46], autonomous driving [4, 67] and content creation [20].

Every computational approach aimed at 3D reconstruction must choose a
representation for the D model. An increasingly popular choice is to use
neural fields [42, 35] for this task. These neural fields, trained on 3D
ground truth data, represent the de-facto gold standard regarding
reconstruction quality. At inference time, the learned 3D shape prior is
adapted to the partial observation. However, these methods suffer from two
major limitations: i) they require 3D ground truth data in the form of
occupancy values or signed distance functions and ii) these models expect
shapes to be aligned and normalized in a fixed canonical coordinate frame - a
frame of reference that is shared between all instances in the shape category.
These two limitations have for now, limited these approaches to synthetic
data, such as Shapenet [5].

To remove the reliance on 3D data, the community has shifted to dense [36], or
sparse [68] multi-view supervision with known camera poses, which can be
estimated using Structure from Motion (SfM). Similarly, single-view 3D
reconstruction methods have also made considerable progress by using neural
fields as their shape representation [31, 13]. While these single-view methods
can be trained from unconstrained image collections, they have not achieved
the high quality of multi-view or 3D ground truth supervised models. In this
work, we aim to answer the question: How can we achieve the reconstruction
quality of 3D supervised methods from single view observations in the wild?

Figure 1: We pretrain a 3D shape prior on synthetic 3D data. Using the Equi-
pose [29] canonicalization algorithm, we register the depth image to the
canonical coordinate frame and use a finetuning scheme to jointly estimate the
surface reconstruction and object pose from a single observation, leading to a
model that generates diverse 3D reconstructions and object poses.

We propose to use a single depth image from a calibrated camera together with
a pretrained canonicalization network to register the partial point clouds to
the canonical coordinate space. We reduce the effect of errors in the
canonicalization process by jointly fine-tuning the latent shape descriptor
and object pose using only the partial observation as input (Figure 1). We
achieve 3D reconstruction results on synthetic data close to or better than
the state-of-the-art. Furthermore, we show that using depth images as input
allows for generalization across various datasets, from dense depth in
synthetic and natural images to sparse depth inputs from LIDAR scans.

##  II Related Work

3D object reconstruction based on a conditional input, such as images or depth
is an active research area [7, 35, 36, 47, 52, 22]. The defacto gold standard
in terms of reconstruction quality uses 3D ground truth data [42, 35]. These
approaches are largely limited to synthetic data, such as Shapenet [5] as they
require shapes that are aligned in a common canonical coordinate frame.
Reconstruction of real-world shapes has been performed by transferring the
learned representation across domains [14, 1] or with the use of special depth
sensors [40, 9]. However, collecting 3D ground truth data in the real world
can be difficult. With the development of neural rendering and inverse
graphics methods, the requirement for 3D ground truth has been relaxed in
favor of dense multi-view supervision [59, 36, 18, 68] or single view methods
that require ground truth camera poses for training [31, 13]. However, not all
applications allow for the collection of multi-view images, and estimating
camera poses from images remains challenging. With the advent of generative
models for 3D shapes [17], using 3D supervision has become an interesting
prospect once more. Our work shows how we can leverage shape canonicalization
for shape reconstruction in the wild.

###  II-A Pose Registration and 3D Shape Canonicalization

Reliance on camera poses is an issue for many real-world datasets but a
necessary step for neural rendering or deformation-based models. Point cloud
registration can estimate the object pose directly and has achieved good
performance when matching point clouds of the same object; however, these
methods are unsuitable for single view pose estimation without a ground truth
3D model [23, 54, 43]. Category-level object pose estimation methods achieve
tremendous results, for supervised training mechanisms [44, 41, 53, 6], and
using only self-supervision [49, 50, 29, 45, 26]. For example, Canonical
Capsules [50] learn to represent object parts with pose-invariant capsules by
training a Siamese network in a self-supervised manner. Although the learned
capsules can reconstruct the input point cloud in the learned canonical frame,
Canonical Capsules only works on complete point clouds. In contrast, Equi-pose
[29] can canonicalize both complete and partial point clouds. By leveraging an
SE(3) equivariant network, Equi-pose simultaneously learns to estimate object
pose and canonical point cloud completion. Our work shows that one can
leverage Equi-pose with test time pose refinement to get accurate shape
reconstructions in canonical space.

###  II-B Pointcloud Completion

Instead of relying on ground truth camera poses, we use depth images to
register the partial 3D point cloud into a canonical frame. As we use depth
images as input, our method closely relates to point cloud completion
algorithms. Early work on point cloud completion used 3D convolutions to learn
shape completion [10, 21]. However, 3D convolutions are costly and operate on
a canonical voxel grid. More recently, PointNet encoders were used for shape
completion [32, 66]. Transformers have also been shown to work well on this
task [65, 61]. However, these methods rely on points already in a canonical
coordinate frame. Further, these methods do not reconstruct the underlying
surface of the object but output a limited number of points. In contrast, out
method does not rely on canonical input points and reconstructs the underlying
object surface with high fidelity.

###  II-C Surface Reconstruction from a Single View

There have been extensive studies on 3D reconstruction from single view images
using various 3D representations, such as voxels [60, 52, 56, 62, 56, 55],
points [16, 63], primitives [11, 8] or meshes [24, 18]. Most of the methods
above use explicit representations, which suffer from limited resolution or
fixed topology. Neural rendering and neural fields provide an alternative
representation to overcome these limitations. Recent methods showed how to
learn Signed Distance Functions (SDFs) [59, 31, 13] or volumetric
representations such as occupancy [64], which have shown great promise in
learning category-specific 3D reconstructions from unstructured image
collections. However, these methods usually require additional information,
such as ground truth camera poses or aligned 3D shapes, which limits their
applicability. In our work, we propose a method that does not require ground
truth camera poses or aligned 3D data and leverages widely available synthetic
data to learn a category-specific 3D prior model.

###  II-D Learning Shape Reconstruction through Deformation

Learning a generalizable model that maps a low-dimensional latent code to 3D
surfaces can suffer from low-quality reconstructions. Category-specific
deformable shape priors are useful to improve the quality of the
reconstruction [2, 15, 24, 25, 33, 37]. These methods generally learn the
deformation to an initial base shape. More recent work has used neural
rendering together with SDFs [31, 13] to learn 3D shape priors from image
collections and their associated camera poses. Other methods [12, 69] jointly
learn the deformation and the template shape in a canonical frame. In this
work, we go one step further and show how we can leverage template shape and
deformation models for incomplete observations registered to the template
coordinate frame.

##  III Method

Given a single segmented RGB-D image of an object, our goal is to jointly
estimate the object pose and reconstruct the underlying 3D surface. To do so,
we first learn a category-specific 3D template in the canonical coordinate
frame together with an instance specific deformation field by leveraging
synthetic 3D data. During test time, rather than directly reconstructing the
shape in the camera coordinate frame, we use recent advances in point cloud
canonicalization to transform a partial depth scan to the canonical space for
surface reconstruction. Next, we describe first how we learn the 3D shape
prior purely on synthetic data. Then we discuss how we reconstruct the surface
of an observed depth image by deforming the learned canonical template shape.

Figure 2: We train a DIF-Net as our 3D representation on purely synthetic data
in an auto-decoder fashion. During inference, we lift the depth image onto 3D
using the known camera intrinsics and estimate an initial transformation
between the camera frame and the canonical frame. We jointly optimize the
object pose and 3D shape to fit the partial observation. Trainable
parameters/network parts are marked in red.

###  III-A 3D Shape Prior

Given a set of 3D objects ${\mathcal{O}_{i}}$, our goal is to learn a
category-specific 3D shape prior together with a latent space describing the
variation in shapes. Instead of directly mapping a low dimensional latent code
$z_{i}\in\mathbb{R}^{n}$ to the 3D shape, we follow recent advances in
learning 3D shape priors through deformation of a canonical template [69, 12].
We jointly learn our 3D shape prior, represented as a neural network, and
latent codes $z_{i}$ through the auto-decoder framework presented in [42]. To
generate high-quality 3D reconstructions, we use signed distance fields
(SDFs). SDF is a function that assigns each point $x_{j}\in\mathbb{R}^{3}$ a
scalar value $s_{j}\in\mathbb{R}$

$SDF(x_{j})=s_{j},$ (1)

representing the distance to the closest object surface. The sign of $s_{j}$
indicates whether a point is inside (negative) or outside (positive) of the
object, and the surface can be extracted as a mesh through marching cubes
[34]. We use a DIF-Net as our 3D shape prior network [12]. The 3D
representation network consists of a neural template field and a deformation
field. We use the template field to capture common structures among a category
of shapes by keeping the weights shared across all instances in the training
set. The template field takes a 3D coordinate $x_{j}$ as input and predicts
the signed distance to the closest surface $\hat{s}$:

$T:x_{j}\in\mathbb{R}^{3}\rightarrow\hat{s}\in\mathbb{R}$ (2)

To deform the template to a specific object instance, we use a deformation
field together with a structural correction field

$D:x_{j}\in\mathbb{R}^{3}\rightarrow(v,\Delta s)\in\mathbb{R}^{4}.$ (3)

The vector $v$ deforms a point in the instance space to the template space,
and the correction factor $\Delta s$ modifies the SDF value of point $x_{j}$
if it still differs from the ground truth value. The correction factor has
been shown to be beneficial for categories with significant shape variations.
For example, for chairs, there exist instances with and without armrests. We
use a Hyper-Network [47, 37] to condition the deformation field on a latent
code. With a learned template field $T$ and deformation field $D$, the SDF
value of a point $x_{j}$ can be obtained with

$s_{j}=T(x_{j}+v)+\Delta s=T(x_{j}+D_{v}(x_{j}))+D_{\Delta}(x_{j}).$ (4)

###  III-B Training DIF

During training we use the auto-decoder framework [47, 42] to jointly learn
latent codes $z_{i}$ and the weights of the DIF network that predicts SDF
values $\hat{s}=\Psi(x)$. Given a collection of shapes with ground truth SDF
values on the object surface and in free space, we first apply an SDF
regression loss from [48] as

$\mathcal{L}_{sdf}=\sum_{i}(\sum_{x\in\Omega}|\Psi_{i}(x)-s|+\sum_{x\in
S_{i}}(1-\langle\nabla\Psi_{i}(x),n\rangle)\\\
+\sum_{x\in\Omega}|||\nabla\Psi_{i}(x)|-1||+\sum_{p\in\Omega\setminus
S_{i}}\rho(\Psi_{i}(x))),$ (5)

where $s$ and $n$ denote the ground truth SDF value and normal, $\nabla$ is
the spatial gradient of the neural field, $\Omega$ is the 3D space in which
values are sampled, and $s_{j}$ is the shape surface. We select an equal
number of surface and free space points uniformly at random to compute this
loss. The first term in Equ. 5 regresses the SDF value; the second term learns
consistent normals on the shape surface, the third term is the eikonal
equation that enforces unit norm or the spatial gradients, and the last term
penalizes SDF values close to $0$ which are far away from the object surface
with $\rho(s)=\exp(-\delta\cdot|s|),\delta>>1$. For more details on this loss,
check [47]. We further apply multiple regularization terms to help learn
smooth deformations and consistent latent space. The first regularization term
applies $L_{2}$ regularization on the embeddings as
$\mathcal{L}_{z}=\sum_{i}||z_{i}||_{2}$. Prior work by [12] showed that
learning a template shape that captures common attributes across a category is
improved by enforcing normal consistency across all shapes by regularizing the
normals of the template networks with

$\mathcal{L}_{normal}=\sum_{i}\sum_{x\in S_{i}}(1-\langle\nabla
T(x+D_{v}(x)),n\rangle).$ (6)

We further want deformations to be smooth and the optional corrections to the
SDF field to be small, which is enforced with the following two loss terms
$\mathcal{L}_{smooth}=\sum_{i}\sum_{x\in\Omega}||\nabla D_{v}(x)||_{2}$ and
$\mathcal{L}_{c}=\sum_{i}\sum_{x\in\Omega}|D_{\Delta s}(x)|$. The overall loss
to training the 3D shape prior is

$\mathcal{L}=\mathcal{L}_{sdf}+\lambda_{1}{L}_{normal}+\lambda_{2}\mathcal{L}_{z}+\lambda_{3}\mathcal{L}_{smooth}+\lambda_{3}\mathcal{L}_{c},$
(7)

with the $\lambda$ terms weighing the relative importance of each loss term.

###  III-C Point Cloud Lifting and Canonicalization during testing

During inference, we use a single RGB-D image and known camera intrinsic
parameters to lift the depth image to a partial 3D point cloud. In order to
predict the deformation field, we first transform the partial point cloud to
the canonical coordinate frame using Equi-pose [29] as our pose estimation
module. Equi-pose is a SE(3)-equivariant network that learns category-specific
canonical shape reconstruction and pose estimation in a self-supervised
manner. By enforcing consistency between the invariant shape reconstruction
and the input point cloud transformed by the estimated pose, Equi-pose can
estimate the pose of the input point cloud with respect to the learned
canonical frame. Therefore, we first input a complete template shape in our
canonical frame to Equi-pose such that the transformation between our
canonical frame and Equi-pose’s canonical frame can be obtained. This way, we
can transform any observed partial point cloud to our canonical frame using
Equi-pose as a pose estimator. However, the estimated pose from Equi-pose can
only serve as a noisy initialization. We show in Section III-D how our method
further finetunes the pose to achieve accurate shape reconstruction.

###  III-D Jointly Optimizing Shape and Pose

Once we train the 3D shape prior network and the partial input point cloud is
roughly aligned in the canonical space, we reconstruct the object surface by
optimizing the latent code and the object pose while keeping the SDF network
weights fixed. As canonical 3D reconstruction methods are sensitive to minor
deviations between estimated and canonical coordinate frames, we jointly
optimize the latent code $z_{i}$ and the initial transformation by minimizing
the SDF values at the observed depth points. At the same time, we sample
random points in free space for the Eikonal term to ensure that the neural
field is an SDF. We represent the translation as a three-dimensional vector
initialized to zero and use the continuous 6D rotation parametrization from
[70] for rotations. We choose a random latent code from the learned latent
space as our initialization $z_{i}$ and optimize

$\text{min}_{z_{i},R,t}\mathcal{L}_{sdf}+\lambda_{2}\mathcal{L}_{z}.$ (8)

##  IV Experiments

Datasets According with other works in the literature, we include three
categories in our experiments: car, chair, and airplane. These categories are
present across multiple datasets, facilitating comparison between approaches
and enabling evaluation of a methods generalization capabilities. We use
synthetic data from the ShapeNet dataset [5] to train our deformation and
template networks using 3D ground truth. Then our method trained on Shapenet
is directly evaluated on the following datasets: ShapeNet [5], Pix3D chairs
[30], Pascal3D+ [58] and the DDAD [19], without retraining. Since Pascal3D+
and Pix3D do not contain depth scans, we generate the partial point clouds by
removing invisible points of the CAD models using the ground truth camera
poses. As DDAD does not provide reconstruction ground truth, we show the
performance of our method on real-world noisy scans qualitatively only. See
the appendix for additional information on datasets, baselines, and
implementation details.

Implementation Details In line with prior work we train the 3D shape network
on three categories in the Shapenet [5] dataset, namely car, chair and plane.
The networks are trained using the Adam optimizer [27]. We use batch size
$128$ shapes per iteration and use $4000$ points on the surface, and $4000$
randomly sampled points in free space per object. Training takes $10$ hours on
four NVIDIA V100 GPUs.

Baselines We compare against the state-of-the-art in single view, category-
specific 3D object reconstruction: i) SDF-SRN [31], a neural field method that
represents the object in camera coordinate frame and uses a neural renderer
with silhouette supervision. ii) TARS-3D [13] is a method that uses ground
truth camera poses to render a deformed template shape in canonical space to
the image coordinate frame. Note that TARS-3D does not require ground truth
camera poses during inference, but also does not estimate the object pose.
Rather, it outputs the estimates surface reconstruction in the canonical
coordinate frame. Instead, both TARS and SDF-SRN output the 3D surface
reconstruction in the canonical coordinate frame.

As our method is closely related to point cloud completion, we further compare
our method against a transformer-based point cloud completion method, PoinTr
[65]. In contrast to the baselines, our model is only trained on Shapenet and
does not require camera pose information during both training and testing
time. Only a partial depth scan is needed during inference.

Figure 3: Qualitative result on the occluded Shapenet dataset. Our model
outputs more accurate 3D shapes, as we do not condition on the input image,
but can finetune the latent shape based on the partial depth observation.

Evaluation Metrics In this work, we follow [51] and report the F1-score at
threshold $1\%$ as our primary evaluation metric. Tatarchenko et. al showed,
that common metrics, such as chamfer distance and IoU allow for large
variations from the ground truth model. In contrast, F1 score with a tight
threshold requires the prediction to closely follow the ground truth to
achieve high score. We additionally report the bidirectional chamfer distance
(CD), multiplied by a factor of $1e4$ for readability.

###  IV-A 3D Reconstruction on synthetic Shapenet data

Table I shows quantitative results of testing all approaches on the holdout
test set of Shapenet. Our method outperforms the baselines in the car and
plane categories with ground truth camera poses and is competitive in the
chair category. We investigate cases where no ground truth camera poses are
available and initialize our method and PoinTr with the Equi-pose estimates.
Even without access to ground truth camera poses, our method performs
comparable to or better than the baseline methods. We can see that PoinTr
suffers greatly when the coordinates are not in the canonical coordinate
system, showing that our approach of combining canonicalization with shape
reconstruction is necessary for 3D shape reconstruction on real-world depth
data. Our method’s 3D reconstructions are more faithful to the underlying
ground truth mesh, shown by the fact that we outperform all other methods on
the F1 metric. PoinTr outputs only a limited number of points and does not
reconstruct the underlying surface, nor does it give us correspondences
between shapes in a category.

TABLE I: 3D reconstruction results on synthetic test data. We report chamfer
distance (CD) $\downarrow$ and F-score at threshold $0.01$
(F@$1\%$)$\uparrow$. † with ground truth camera poses.

Methods Car Chair Plane CD $(\downarrow)$ F@1 $(\uparrow)$ CD $(\downarrow)$
F@1 $(\uparrow)$ CD $(\downarrow)$ F@1 $(\uparrow)$ SDF-SRN 9.965 0.404 27.562
0.283 12.374 0.459 TARS-3D 10.175 0.412 28.823 0.272 11.302 0.418 PoinTr
56.435 0.125 36.714 0.230 23.713 0.302 Ours 9.371 0.439 32.138 0.334 16.562
0.631 PoinTr † 13.249 0.264 12.834 0.352 3.637 0.685 Ours † 6.181 0.497 27.292
0.343 4.495 0.768

Figure 4: Qualitative results on the Pascal3D+ (top) and Pix3D (bottom)
datasets.

###  IV-B 3D reconstruction on Shapenet with Occlusion

This section investigates how our method performs when the observed object is
occluded. For this experiment, we generate occluded area on the RGB-D images
and remove the occluded points from the Shapenet test data as presented in
Figure 3. To study the influence of a wide variety of occlusions, we use
rectangular overlap regions and generate overlap ratios from 5 - 85%. We apply
models trained on Shapenet on the occlusion task for all methods. We remove
the occluded region from the RGB-D images and canonicalize the occluded,
partial pointclouds, before finetuning the surface reconstruction to fit the
available points. As shown in Table II, our method outperforms all the
baselines without access to ground truth camera poses despite not seeing any
occluded data during training. Compared to the other baselines, PoinTr has a
higher tolerance to occlusion. However, point cloud completion methods cannot
predict the object’s surface. Moreover, these methods usually predict the
complete point cloud by adding predicted points to the input. Therefore, the
predicted point clouds are not guaranteed to be uniformly distributed and can
have a higher density around the input points, resulting in lower chamfer
distance, as shown in Figure 3. In contrast, our method does not have these
drawbacks and can reconstruct the occluded surface by leveraging the learned
category-level prior with high fidelity in terms of F-score.

TABLE II: 3D reconstruction results on occluded data from the synthetic test
set. We report chamfer distance (CD) $\downarrow$ and F-score at threshold
$0.01$ (F@$1\%$)$\uparrow$. † with ground truth camera poses.

Methods Car Chair Plane CD $(\downarrow)$ F@1 $(\uparrow)$ CD $(\downarrow)$
F@1 $(\uparrow)$ CD $(\downarrow)$ F@1 $(\uparrow)$ SDF-SRN 25.863 0.243
156.825 0.122 72.255 0.256 TARS-3D 23.806 0.241 108.976 0.131 58.998 0.263
PoinTr 59.292 0.1225 54.327 0.184 37.425 0.322 Ours 11.155 0.390 45.886 0.273
22.223 0.580 PoinTr † 22.110 0.208 19.079 0.297 6.168 0.601 Ours† 6.457 0.497
35.972 0.297 15.725 0.654

###  IV-C 3D reconstruction on Pascal3D+ and Pix3D

For this experiment, we test the generalization capabilities of our approach.
We directly apply our model trained on Shapenet for reconstructing Pascal3D+
objects. TARS-3D and PoinTr also apply network weights trained on Shapenet to
this task, while SDF-SRN is trained directly on the Pascal3D+ dataset. As
shown in Table III, our method again outperforms the other surface
reconstruction methods without access to ground truth camera poses. Figure 4
shows that our method generates reasonable outputs.

TABLE III: 3D reconstruction results on the Pascal3D+ and Pix3D dataset. We
report chamfer distance (CD) $\downarrow$ and F-score at threshold $1\%$
(F@$1\%$)$\uparrow$. † with ground truth camera poses.

Pascal3D+ Pix3D Methods Car Chair Plane Chair CD $(\downarrow)$ F@1
$(\uparrow)$ CD $(\downarrow)$ F@1 $(\uparrow)$ CD $(\downarrow)$ F@1
$(\uparrow)$ CD $(\downarrow)$ F@1 $(\uparrow)$ SDF-SRN 16.740 0.245 24.374
0.216 29.7457 0.169 60.432 0.158 TARS-3D 19.129 0.2427 79.8922 0.1577 83.866
0.140 55.555 0.197 PoinTr 80.896 0.064 21.251 0.216 35.095 0.231 35.527 0.283
Ours 15.516 0.283 17.328 0.275 54.849 0.276 35.729 0.335 PoinTr† 17.859 0.221
11.769 0.302 4.701 0.525 13.092 0.377 Ours† 13.370 0.302 16.804 0.274 54.286
0.289 31.165 0.335

We further test our method on another chair dataset, namely the chair category
of the Pix3D dataset. As shown in Table III, our method outperforms the
baselines. PoinTr again achieves a lower chamfer distance, which does not
fully represent the reconstruction quality. As shown in Figure 4, PoinTr
generates point clouds that are not uniformly distributed while our method
predicts smooth surfaces.

Figure 5: Qualitative results on the DDAD dataset.

###  IV-D Ablation Study and Failure Cases

In this section, we conduct an ablation study to asses the importance of
optimizing the pose during inference. The main results are shown in Table IV.
On the left, we show the ground truth mesh overlaid with the partial input
points with pose optimization (blue) and without (green). We can see a
noticeable reduction in F1 scores and a significant reduction in the
reconstruction quality. The chamfer score difference between reconstruction
methods across the dataset is slight, though, confirming [51] in that chamfer
distance is not an ideal metric for 3D reconstruction. This ablation study
shows that canonical reconstruction methods are sensitive to deviations in the
estimated canonical coordinate frame. This result is also confirmed by the
poor performance of PoinTr on input points where the estimated canonical
coordinate frame is off (see Tables I and III).

TABLE IV: Ablation of our method with and without pose optimization during
inference. Left, we show the ground truth mesh overlayed with the optimized
and non-optimized pose. We show the resulting optimized surfaces with and
without pose optimization in the middle and right.

Pose optim. ✓ CD 23.008 22.223 F1 0.313 0.580 Shape

###  IV-E 3D reconstruction on real-world noisy scans

Finally, we apply our method trained on Shapenet directly to real-world noisy
LIDAR scans. To demonstrate our tolerance to noise in the point clouds, we
test our method on an autonomous driving benchmark DDAD [19]. DDAD contains
urban scenes scanned using LiDARs mounted on self-driving cars. To showcase
our method, we extract frames that include other driving cars and crop the
LiDAR scans of other cars with masked images. Finally, these noisy LiDAR scans
are fed to the pose estimation module and our deformation field to reconstruct
the surfaces. Since DDAD does not contain ground truth CAD models, we present
the qualitative results in Figure 5. Note that our method does not have access
to the image but only the noisy LiDAR point clouds. Despite large portions of
missing parts and the noise in the LiDAR scans, our method can still
reconstruct reasonable car surfaces without access to ground truth camera
poses.

##  V Conclusion and future work

We introduced a new method for complete 3D surface reconstruction of an object
from real-world depth images. Our method relies on a representation obtained
solely by training on synthetic data, which allows for extracting high-
quality, category-specific geometry. We showed that even small errors in pose
estimation lead to significant errors in 3D reconstruction. Therefore a simple
method which uses an independently trained pose estimator followed by
reconstruction in the object frame does not yield good reconstruction results.
Instead, we presented a finetuning scheme to optimize the object surface and
pose jointly during inference. We also showed that learning strong 3D priors
benefits the 3D reconstruction of occluded objects. Our method generalizes
across datasets and input modalities, from dense depth images to sparse LIDAR
point clouds. While our process still exhibits failure modes when the error in
the estimated pose is large, this could be alleviated by combining pose
estimation and 3D reconstruction in an end-to-end trainable manner. We hope
our work will inspire further work in this direction.

## References

  * Bechtold et al. [2021] Jan Bechtold, Maxim Tatarchenko, Volker Fischer, and Thomas Brox.  Fostering Generalization in Single-view 3D Reconstruction by Learning a Hierarchy of Local and Global Shape Priors.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 15875–15884, Nashville, TN, USA, June 2021. IEEE.
  * Blanz and Vetter [1999] Volker Blanz and Thomas Vetter.  A morphable model for the synthesis of 3D faces.  In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_ , pages 187–194, 1999.
  * Bylow et al. [2013] Erik Bylow, Jürgen Sturm, Christian Kerl, Fredrik Kahl, and Daniel Cremers.  Real-time camera tracking and 3D reconstruction using signed distance functions.  In _Robotics: Science and Systems_ , volume 2, page 2, 2013.
  * Campbell et al. [2010] Mark Campbell, Magnus Egerstedt, Jonathan P How, and Richard M Murray.  Autonomous driving in urban environments: approaches, lessons and challenges.  _Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences_ , 368(1928):4649–4672, 2010.  Publisher: The Royal Society Publishing.
  * Chang et al. [2015] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * Chao et al. [2022] Jun-Jee Chao, Selim Engin, Nicolai Häni, and Volkan Isler.  Category-Level Global Camera Pose Estimation with Multi-Hypothesis Point Cloud Correspondences.  _arXiv preprint arXiv:2209.14419_ , 2022.
  * Chen et al. [2019] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler.  Learning to predict 3d objects with an interpolation-based differentiable renderer.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Chen et al. [2020] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang.  Bsp-net: Generating compact meshes via binary space partitioning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 45–54, 2020.
  * Choe et al. [2021] Jaesung Choe, Sunghoon Im, Francois Rameau, Minjun Kang, and In So Kweon.  Volumefusion: Deep depth fusion for 3d scene reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 16086–16095, 2021.
  * Dai et al. [2017] Angela Dai, Charles Ruizhongtai Qi, and Matthias Nießner.  Shape completion using 3d-encoder-predictor cnns and shape synthesis.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5868–5877, 2017.
  * Deng et al. [2020] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.  Cvxnet: Learnable convex decomposition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 31–44, 2020.
  * Deng et al. [2021] Yu Deng, Jiaolong Yang, and Xin Tong.  Deformed implicit field: Modeling 3d shapes with learned dense correspondence.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10286–10296, 2021.
  * Duggal and Pathak [2022] Shivam Duggal and Deepak Pathak.  Topologically-Aware Deformation Fields for Single-View 3D Reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1536–1546, 2022.
  * Duggal et al. [2022] Shivam Duggal, Zihao Wang, Wei-Chiu Ma, Sivabalan Manivasagam, Justin Liang, Shenlong Wang, and Raquel Urtasun.  Mending Neural Implicit Modeling for 3D Vehicle Reconstruction in the Wild.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1900–1909, 2022.
  * Engelmann et al. [2017] Francis Engelmann, Jörg Stückler, and Bastian Leibe.  SAMP: shape and motion priors for 4d vehicle reconstruction.  In _2017 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pages 400–408. IEEE, 2017.
  * Fan et al. [2017] Haoqiang Fan, Hao Su, and Leonidas J Guibas.  A point set generation network for 3d object reconstruction from a single image.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 605–613, 2017.
  * Gao et al. [2022] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler.  GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images.  In _Advances In Neural Information Processing Systems_ , 2022.
  * Goel et al. [2022] Shubham Goel, Georgia Gkioxari, and Jitendra Malik.  Differentiable Stereopsis: Meshes from multiple views using differentiable rendering.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8635–8644, 2022.
  * Guizilini et al. [2020] Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon.  3D Packing for Self-Supervised Monocular Depth Estimation.  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Huang et al. [2017] Jingwei Huang, Angela Dai, Leonidas Guibas, and Matthias Nießner.  3DLite: Towards Commodity 3D Scanning for Content Creation.  _ACM Transactions on Graphics 2017 (TOG)_ , 2017.
  * Huang et al. [2020] Zitian Huang, Yikuan Yu, Jiawen Xu, Feng Ni, and Xinyi Le.  Pf-net: Point fractal network for 3d point cloud completion.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 7662–7670, 2020.
  * Häni et al. [2020] Nicolai Häni, Selim Engin, Jun-Jee Chao, and Volkan Isler.  Continuous Object Representation Networks: Novel View Synthesis without Target View Supervision.  In _Advances in Neural Information Processing Systems_ , 2020\.
  * Jiang et al. [2021] Haobo Jiang, Yaqi Shen, Jin Xie, Jun Li, Jianjun Qian, and Jian Yang.  Sampling network guided cross-entropy method for unsupervised point cloud registration.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6128–6137, 2021.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik.  Learning category-specific mesh reconstruction from image collections.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 371–386, 2018.
  * Kar et al. [2015] Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik.  Category-Specific Object Reconstruction From a Single Image.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2015.
  * Katzir et al. [2022] Oren Katzir, Dani Lischinski, and Daniel Cohen-Or.  Shape-Pose Disentanglement using SE (3)-equivariant Vector Neurons.  _arXiv preprint arXiv:2204.01159_ , 2022.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kumar et al. [2017] Gourav Kumar, Harit Pandya, Ayush Gaud, and K Madhava Krishna.  Pose induction for visual servoing to a novel object instance.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 2953–2959. IEEE, 2017.
  * Li et al. [2021] Xiaolong Li, Yijia Weng, Li Yi, Leonidas Guibas, A Lynn Abbott, Shuran Song, and He Wang.  Leveraging SE(3) Equivariance for Self-Supervised Category-Level Object Pose Estimation.  _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021.
  * Lim et al. [2013] Joseph J Lim, Hamed Pirsiavash, and Antonio Torralba.  Parsing ikea objects: Fine pose estimation.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2992–2999, 2013.
  * Lin et al. [2020] Chen-Hsuan Lin, Chaoyang Wang, and Simon Lucey.  Sdf-srn: Learning signed distance 3d object reconstruction from static images.  _Advances in Neural Information Processing Systems_ , 33:11453–11464, 2020.
  * Liu et al. [2020] Minghua Liu, Lu Sheng, Sheng Yang, Jing Shao, and Shi-Min Hu.  Morphing and sampling network for dense point cloud completion.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 11596–11603, 2020.  Issue: 07.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  SMPL: A skinned multi-person linear model.  _ACM transactions on graphics (TOG)_ , 34(6):1–16, 2015.  Publisher: ACM New York, NY, USA.
  * Lorensen and Cline [1987] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3D surface construction algorithm.  _ACM siggraph computer graphics_ , 21(4):163–169, 1987.  Publisher: ACM New York, NY, USA.
  * Mescheder et al. [2019] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3d reconstruction in function space.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4460–4470, 2019.
  * Mildenhall et al. [2020] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _Proceedings of the European Conference on Computer Vision_ , pages 405–421. Springer, 2020.
  * Mitchell et al. [2019] Eric Mitchell, Selim Engin, Volkan Isler, and Daniel D Lee.  Higher-Order Function Networks for Learning Composable 3D Object Representations.  In _International Conference on Learning Representations_ , 2019\.
  * Mittal et al. [2022] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani.  AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation.  In _CVPR_ , 2022.
  * Mousavian et al. [2019] Arsalan Mousavian, Clemens Eppner, and Dieter Fox.  6-dof graspnet: Variational grasp generation for object manipulation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2901–2910, 2019.
  * Newcombe et al. [2011] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In _2011 10th IEEE international symposium on mixed and augmented reality_ , pages 127–136. Ieee, 2011.
  * Novotny et al. [2019] David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, and Andrea Vedaldi.  C3dpo: Canonical 3d pose networks for non-rigid structure from motion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7688–7697, 2019.
  * Park et al. [2019] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  Deepsdf: Learning continuous signed distance functions for shape representation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 165–174, 2019.
  * Qin et al. [2022] Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu.  Geometric Transformer for Fast and Robust Point Cloud Registration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11143–11152, June 2022\.
  * Rempe et al. [2020] Davis Rempe, Tolga Birdal, Yongheng Zhao, Zan Gojcic, Srinath Sridhar, and Leonidas J Guibas.  Caspr: Learning canonical spatiotemporal point cloud representations.  _Advances in neural information processing systems_ , 33:13688–13701, 2020.
  * Sajnani et al. [2022] Rahul Sajnani, Adrien Poulenard, Jivitesh Jain, Radhika Dua, Leonidas J. Guibas, and Srinath Sridhar.  ConDor: Self-Supervised Canonicalization of 3D Pose for Partial Shapes.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2022.
  * Sen et al. [2023] Bipasha Sen, Aditya Agarwal, Gaurav Singh, Srinath Sridhar, Madhava Krishna, and others.  SCARP: 3D Shape Completion in ARbitrary Poses for Improved Grasping.  _arXiv preprint arXiv:2301.07213_ , 2023.
  * Sitzmann et al. [2019] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein.  Scene representation networks: Continuous 3D-structure-aware neural scene representations.  In _Advances in Neural Information Processing Systems_ , volume 32, pages 1119–1130, 2019.
  * Sitzmann et al. [2020] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein.  Implicit neural representations with periodic activation functions.  _Advances in Neural Information Processing Systems_ , 33:7462–7473, 2020.
  * Spezialetti et al. [2020] Riccardo Spezialetti, Federico Stella, Marlon Marcon, Luciano Silva, Samuele Salti, and Luigi Di Stefano.  Learning to orient surfaces by self-supervised spherical cnns.  _Advances in Neural Information Processing Systems_ , 33:5381–5392, 2020.
  * Sun et al. [2021] Weiwei Sun, Andrea Tagliasacchi, Boyang Deng, Sara Sabour, Soroosh Yazdani, Geoffrey Hinton, and Kwang Moo Yi.  Canonical Capsules: Self-Supervised Capsules in Canonical Pose.  In _Neural Information Processing Systems_ , 2021.
  * Tatarchenko et al. [2019] Maxim Tatarchenko, Stephan R Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What do single-view 3d reconstruction networks learn?  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3405–3414, 2019.
  * Tulsiani et al. [2017] Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik.  Multi-View Supervision for Single-View Reconstruction via Differentiable Ray Consistency.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * Wang et al. [2019] He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas.  Normalized object coordinate space for category-level 6d object pose and size estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2642–2651, 2019.
  * Wu et al. [2021] Bingli Wu, Jie Ma, Gaojie Chen, and Pei An.  Feature interactive representation for point cloud registration.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5530–5539, 2021.
  * Wu et al. [2017] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum.  Marrnet: 3d shape reconstruction via 2.5 d sketches.  _Advances in neural information processing systems_ , 30, 2017.
  * Wu et al. [2018] Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T Freeman, and Joshua B Tenenbaum.  Learning shape priors for single-view 3d completion and reconstruction.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 646–662, 2018.
  * Wu et al. [2015] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.  3d shapenets: A deep representation for volumetric shapes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1912–1920, 2015.
  * Xiang et al. [2014] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese.  Beyond pascal: A benchmark for 3d object detection in the wild.  In _IEEE winter conference on applications of computer vision_ , pages 75–82. IEEE, 2014.
  * Xu et al. [2019] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann.  Disn: Deep implicit surface network for high-quality single-view 3d reconstruction.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * Yan et al. [2016] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee.  Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision.  _Advances in neural information processing systems_ , 29, 2016.
  * Yan et al. [2022] Xingguang Yan, Liqiang Lin, Niloy J. Mitra, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  ShapeFormer: Transformer-Based Shape Completion via Sparse Representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6239–6249, June 2022.
  * Yang et al. [2018] Guandao Yang, Yin Cui, Serge Belongie, and Bharath Hariharan.  Learning single-view 3d reconstruction with limited pose supervision.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 86–101, 2018.
  * Yang et al. [2019] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu, Serge Belongie, and Bharath Hariharan.  Pointflow: 3d point cloud generation with continuous normalizing flows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4541–4550, 2019.
  * Ye et al. [2021] Yufei Ye, Shubham Tulsiani, and Abhinav Gupta.  Shelf-Supervised Mesh Prediction in the Wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8843–8852, June 2021.
  * Yu et al. [2021] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou.  Pointr: Diverse point cloud completion with geometry-aware transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 12498–12507, 2021.
  * Yuan et al. [2018] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert.  Pcn: Point completion network.  In _2018 International Conference on 3D Vision (3DV)_ , pages 728–737. IEEE, 2018.
  * Zeng et al. [2020] Wenyuan Zeng, Shenlong Wang, Renjie Liao, Yun Chen, Bin Yang, and Raquel Urtasun.  Dsdnet: Deep structured self-driving network.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16_ , pages 156–172. Springer, 2020.
  * Zhang et al. [2021] Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan.  NeRS: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild.  _Advances in Neural Information Processing Systems_ , 34:29835–29847, 2021.
  * Zheng et al. [2021] Zerong Zheng, Tao Yu, Qionghai Dai, and Yebin Liu.  Deep implicit templates for 3d shape representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1429–1439, 2021.
  * Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5745–5753, 2019.

###  -A Dataset Details

Shapenet [5]: For training on Shapenet, we follow [12] and extract 500,000 SDF
values on the object’s surface and 500,000 randomly sampled in a cube of side
length 2. We also render a single RGB-D image with a camera sampled on the
viewing hemisphere for testing according to [13]. We use the same train/test
split for each category as DIF-Net [12], but only keep the intersection of
objects in both ours and the baselines test sets.

Pascal3D+ [58]: Pascal3D+ is a real-world dataset containing camera images.
The dataset provides object silhouettes, camera poses, and the CAD models used
to annotate the camera poses. Since the same set of CAD models is used to
annotate both the training and testing set, the dataset is considered to
possess a bias [52]. However, unlike the other baselines, our method is only
trained on Shapenet and therefore has not seen any of the CAD models from
Pascal3D+ during training. We evaluate our method on the car, chair, and
airplane category of Pascal3D+. We follow the testing split that is used by
SDF-SRN [31]. To generate partial point clouds in the camera frame as the
input for our method, we first transform the CAD models into the camera frame
using the ground truth camera poses. We remove points that are invisible from
the camera origin. We do not have access to either ground truth poses or the
complete CAD model during training and testing.

Pix3D [30]: Like Pascal3D+, Pix3D contains real-world 2D images annotated with
3D CAD models. Unlike Pascal3D+, Pix3D uses a variety of CAD models that align
better with the images. We randomly select 200 images from the Pix3D chair
dataset as the test set. We follow the same process as Pascal3D+ to generate
the partial point clouds in the camera frame. Again, our method does not
access the complete CAD models, and ground truth camera poses.

DDAD [19]: DDAD is an autonomous driving benchmark containing diverse urban
scenes captured using car sensors. The dataset includes RGB videos captured
using six cameras covering the 360-degree surrounding of the vehicle. DDAD
also provides depth data across an entire 360-degree field of view scanned
using long-range LiDAR sensors. We test our method on one scene with 100
frames. We extract only the frames that contain other cars in the camera’s
field of view. To generate the partial point clouds of the observed cars, we
crop the LiDAR data with the ground truth poses and the masked images. The
cropped LiDAR scans in the camera frame serve as our method’s input. The DDAD
dataset does not provide ground truth models of the observed cars, and our
method does not have access to the ground truth camera poses. We therefore
show only qualitative results.

###  -B Implementation Details

DIF-Net Architecture DIF-Net uses a SIREN [48] network as the MLP backbone for
both, the deformation and template networks. The Hyper-network is a ReLU
network, where each MLP predicts the weights of one of the layers in the
deformation network $D$.

Training Details We initialize the latent codes to small values from
$\mathcal{N}(0,0.01)$. Each model is optimized for 60 epochs and during each
epoch, the model has access to 200,000 free and 200,000 surface points. We set
the weights for the SDF loss $\mathcal{L}_{sdf}$ to 3e3, 1e2, 5e1 and 5e2
according to [48]. We follow [12] in choosing the weighting parameters as
follows. $\lambda_{1}$ and $\lambda_{4}$ are 1e2 and 1e6 for all categories.
$lambda_{2}$ is 5, 2, 5 for car, plane and chair. $\lambda_{3}$ is 1e2, 1e2
and 5e1 for each of the above categories.

Inference Details We jointly optimize the object pose and shape using the Adam
optmizer with a learning rate of $0.001$ for the shape and $0.01$ for the
pose. We optimize each shape for a total of 30 iterations, taking roughly 4
seconds.

Equi-pose [29]: In this paper, we directly apply Equi-pose as an off-the-shelf
pose estimation module. Since our method does not require accurate camera
poses but rough initialization, we use the model weights trained on ModelNet40
[57] provided by the authors for all the experiments in this paper.

###  -C Baselines

SDF-SRN [31]: We directly use the source code and pre-trained weights for
Pascal3D+ and Shapenet provided by the original authors. In this paper, we
follow SDF-SRN’s test split for Pascal3D+ dataset. As for Shapenet, we take
the union between our test set and SDF-SRN’s test set as the Shapenet test
set.

TARS-3D [13]: We use the original implementation of TARS-3D from the authors.
Since TARS-3D follows the same dataset setup as SDF-SRN, we use the network
weights trained on Shapenet provided by the authors.

PoinTr [65]: We directly adopt the original code and network weights trained
on Shapenet provided by the authors. For fair comparison, we transform the
input partial point clouds into their coordinate frame with both ground truth
and estimated camera poses. Furthermore, we follow their original training
setup where the input point clouds are downsampled to 2048 points using
farthest point sampling, and predict 8192 points as the complete point cloud.

###  -D Additional Results

Extracted Template Shapes Here we show the extracted template shapes from our
pretrained 3D prior network. We extract the zero-level set of the neural field
using marching cubes. We can observe that the template represents shapes close
to the ”mean” object for cars and chairs. For planes however, the neural field
does not represent an object, but fuses common aspects of different shapes
together.

Figure 6: Extracted learned template shapes Figure 7: Additional results on
the Pascal3D+ dataset Figure 8: Additional results on the Pix3D dataset