Florian<EMAIL_ADDRESS>Ignas<EMAIL_ADDRESS>Roberto
<EMAIL_ADDRESS>Department of Engineering
University of Cambridge
Cambridge, UK Leveraging Geometry for Single Image Shape Estimation

# Leveraging Geometry for Shape Estimation from a Single RGB Image

###### Abstract

Predicting 3D shapes and poses of static objects from a single RGB image is an
important research area in modern computer vision. Its applications range from
augmented reality to robotics and digital content creation. Typically this
task is performed through direct object shape and pose predictions [Gkioxari
et al.(2019)Gkioxari, Malik, and Johnson, Wang et al.(2018)Wang, Zhang, Li,
Fu, Liu, and Jiang, Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang]
which is inaccurate. A promising research direction [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai, Izadinia et al.(2017)Izadinia, Shan, and Seitz,
Grabner et al.(2019)Grabner, Roth, and Lepetit] ensures meaningful shape
predictions by retrieving CAD models from large scale databases [Chang et
al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva,
Song, Su, Xiao, Yi, and Yu, Sun et al.(2018)Sun, Wu, Zhang, Zhang, Zhang, Xue,
Tenenbaum, and Freeman] and aligning them to the objects observed in the
image. However, existing work [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai]
does not take the object geometry into account, leading to inaccurate object
pose predictions, especially for unseen objects. In this work we demonstrate
how cross-domain keypoint matches from an RGB image to a rendered CAD model
allow for more precise object pose predictions compared to ones obtained
through direct predictions. We further show that keypoint matches can not only
be used to estimate the pose of an object, but also to modify the shape of the
object itself. This is important as the accuracy that can be achieved with
object retrieval alone is inherently limited to the available CAD models.
Allowing shape adaptation bridges the gap between the retrieved CAD model and
the observed shape. We demonstrate our approach on the challenging Pix3D [Sun
et al.(2018)Sun, Wu, Zhang, Zhang, Zhang, Xue, Tenenbaum, and Freeman]
dataset. The proposed geometric shape prediction improves the
$\mathrm{AP}^{\mathrm{mesh}}$ [Gkioxari et al.(2019)Gkioxari, Malik, and
Johnson] over the state-of-the-art [Kuo et al.(2020)Kuo, Angelova, Lin, and
Dai] from 33.2 to 37.8 on seen objects and from 8.2 to 17.1 on unseen objects.
Furthermore, we demonstrate more accurate shape predictions without closely
matching CAD models when following the proposed shape adaptation. Code is
publicly available at
https://github.com/florianlanger/leveraging_geometry_for_shape_estimation.

##  1 Introduction

Figure 1:  Example result of our approach. Given an input image we retrieve a
CAD model rendering and perform key-point matching with the masked input
image. Without shape adaptation the retrieved CAD model prediction is limited
by the availability of similar CAD models in the database. When shape
adaptation is performed target object shapes and their poses can be predicted
precisely.

The past few years have seen rapid advances in object recognition in RGB
images [Krizhevsky et al.(2017)Krizhevsky, Sutskever, and Hinton, He et
al.(2017)He, Gkioxari, Dollár, and Girshick, Liu et al.(2021)Liu, Lin, Cao,
Hu, Wei, Zhang, Lin, and Guo]. However, for many applications reasoning not in
the 2-dimensional image but in the 3-dimensional world is crucial. One
important task is 3D shape estimation from a single image, with applications
ranging from robotics to augmented reality and digital content creation.
Current research on this task can be categorised into two different streams:
direct shape prediction and shape estimation via object retrieval.
For direct object shape predictions different representations are used ranging
from voxels [Choy et al.(2016)Choy, Xu, Gwak, Chen, and Savarese], point
clouds [Fan et al.(2017)Fan, Su, and Guibas, Zeng et al.(2020)Zeng, Karaoglu,
and Gevers], meshes [Wang et al.(2018)Wang, Zhang, Li, Fu, Liu, and Jiang,
Gkioxari et al.(2019)Gkioxari, Malik, and Johnson, Pan et al.(2019)Pan, Han,
Chen, Tang, and Jia, Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang],
packed spheres [Fan et al.(2017)Fan, Su, and Guibas], binary space
partitioning [Chen et al.(2020)Chen, Tagliasacchi, and Zhang], convex
polytopes [Deng et al.(2020)Deng, Genova, Yazdani, Bouaziz, Hinton, and
Tagliasacchi], signed distance fields [Park et al.(2019)Park, Florence,
Straub, Newcombe, and Lovegrove] to other implicit representations [Mildenhall
et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng]. Most
of these approaches suffer either from a lack of precision [Choy et
al.(2016)Choy, Xu, Gwak, Chen, and Savarese, Fan et al.(2017)Fan, Su, and
Guibas, Zeng et al.(2020)Zeng, Karaoglu, and Gevers, Wang et al.(2018)Wang,
Zhang, Li, Fu, Liu, and Jiang, Gkioxari et al.(2019)Gkioxari, Malik, and
Johnson, Pan et al.(2019)Pan, Han, Chen, Tang, and Jia, Nie et al.(2020)Nie,
Han, Guo, Zheng, Chang, and Zhang, Deng et al.(2020)Deng, Genova, Yazdani,
Bouaziz, Hinton, and Tagliasacchi] or lack applicability to a wide range of
object classes [Chen et al.(2020)Chen, Tagliasacchi, and Zhang, Park et
al.(2019)Park, Florence, Straub, Newcombe, and Lovegrove, Mildenhall et
al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng].
Regardless of the representation chosen, directly predicting the shape and
pose of an object from a single image is a difficult learning task. It is
difficult as crucial information about the parts of an object without direct
line of sight to the camera is absent at train and test time. The absence of
complete shape information leads to an inherent ambiguity, as multiple
consistent shape predictions may be possible; this ambiguity prevents the
networks from learning effectively, causing them to perform class-averaged
object shape predictions [Tatarchenko et al.(2019)Tatarchenko, Richter,
Ranftl, Li, Koltun, and Brox]. A natural way to deal with the ill-posed nature
of the problem lies in retrieving CAD models from existing large-scale
databases [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li,
Savarese, Savva, Song, Su, Xiao, Yi, and Yu]. [Li et al.(2015)Li, Su, Qi,
Fish, Cohen-Or, and Guibas] and IM2CAD [Izadinia et al.(2017)Izadinia, Shan,
and Seitz] show how CAD models and real images can be effectively encoded into
a joint embedding space. At test time given a target RGB image multiple
candidate CAD models can be retrieved. In order to estimate the object pose
[Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] regress rotation parameters and
use the ground truth $z$-coordinate for predicting object translation (see
Section 2). While in this manner a neural network is able to store and
interpolate between poses for objects that were seen during training, it fails
to generalise to unseen objects. We avoid this problem by using a deep network
to perform the simpler task of matching keypoints between the real image and
the retrieved CAD model render [Georgakis et al.(2019)Georgakis, Karanam, Wu,
and Kosecka], and using these matches as constraints to calculate the object
pose analytically. More importantly we show that we can use keypoint matches
to modify the shape of the retrieved CAD model to better fit the object
observed (see Figure 1). We modify CAD model shapes by stretching them along
the normal vector of 3D-planes. Stretching can be seen as a local operation
which in contrast to a global scaling operation can modify proportions of
objects within a single dimension (e.g. adjusting the height of the sitting
area of a chair as shown in Figure 3). Currently our approach uses stretching
along the three principal object axis. However, in the future predicting
additional stretch planes and limiting stretch extents to variable 3D-boxes
will allow for even more fine-grained shape adaptations.
We evaluate our approach on the Pix3D [Sun et al.(2018)Sun, Wu, Zhang, Zhang,
Zhang, Xue, Tenenbaum, and Freeman] dataset. When combining object retrieval
with a geometric pose prediction we outperform existing work [Gkioxari et
al.(2019)Gkioxari, Malik, and Johnson, Kuo et al.(2020)Kuo, Angelova, Lin, and
Dai] on splits containing both seen and unseen CAD models at train time (see
Table 1). We evaluate the proposed object adaptation on a range of adaptation
experiments. Here we observe that dynamic fitting improves the shape
predictions when no access to correct CAD models is given at test time and
retrieved models have to be adapted (see Figure 6).

##  2 Related Work

Related literature on shape estimation for static objects from single images
can be categorised into direct shape prediction methods and retrieval based
methods.
Direct Prediction Methods. Current direct prediction methods differ greatly in
the choice of object representations that are used. In the voxel
representation [Choy et al.(2016)Choy, Xu, Gwak, Chen, and Savarese] the 3D
world is discretised into cubes and object shapes are encoded as binary
occupancies of these cubes. This allows the shape prediction task to be
formulated as a binary classification task which is usually simpler to learn
compared to a regression task. However, the voxel representation suffers from
an inherent trade off between accuracy and storage space due to the cubic
scaling. Meshes [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] alleviate
the storage-accuracy trade-off by encapsulating information about the 2D
object surface rather than its 3D volume. An object is represented as a set of
vertices which are interconnected to form (usually triangular) mesh faces. The
difficulties associated with meshes are twofold; predicting the correct object
topology and predicting precise vertex positions. Early work [Wang et
al.(2018)Wang, Zhang, Li, Fu, Liu, and Jiang] simply deformed an original
ellipsoid mesh and was therefore not able to predict shapes with complex
topologies. In contrast [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson]
first predict a rough shape estimate in the voxel representation. This serves
as the initialisation for the topology of the shape and is subsequently
refined as a mesh. Recently Topology Modification Networks [Pan et
al.(2019)Pan, Han, Chen, Tang, and Jia] aim to directly predict varying object
topologies. Other works decompose objects into a set of convex polytopes [Deng
et al.(2020)Deng, Genova, Yazdani, Bouaziz, Hinton, and Tagliasacchi] or
assemble 3D shapes from geometric primitives [Tulsiani et al.(2018)Tulsiani,
Su, Guibas, Efros, and Malik]. Besides these explicit object encodings other
implicit representations exist for defining the object boundary. [Chen et
al.(2020)Chen, Tagliasacchi, and Zhang] approximate an object by predicting a
set of 3D planes and the corresponding side on which the object lies on. Given
a 3D point signed distance fields [Park et al.(2019)Park, Florence, Straub,
Newcombe, and Lovegrove] predict the distance of the point from the object
surface and whether it resides within or outside of the object. Neural
radiance fields [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik,
Barron, Ramamoorthi, and Ng] encode information not just about the object
shape but also its texture. While these implicit representations are promising
research directions, they require vast amounts of training and have not been
shown yet to work on a large number of different shapes in realistic settings.
Object Retrieval. Almost all existing work on object retrieval constructs an
embedding space of CAD model renderings. At test time a real image is embedded
into this space and its nearest neighbours are retrieved. While early work
[Aubry et al.(2014)Aubry, Maturana, Efros, Russell, and Sivic] construct and
map into this space based on the Histogram-of-Gradient [Dalal and
Triggs(2005)] (HoG) descriptor, [Li et al.(2015)Li, Su, Qi, Fish, Cohen-Or,
and Guibas] learn real image embeddings using a convolutional neural network.
Instead of constructing the embedding space from HoG-descriptors [Izadinia et
al.(2017)Izadinia, Shan, and Seitz] learns a joint embedding space between CAD
model renderings and real images. In order to align retrieved CAD models with
the objects in the image [Izadinia et al.(2017)Izadinia, Shan, and Seitz]
follows a Render-and-Compare approach in which they optimise for an object
pose by iteratively comparing the re-rendered CAD model embedding to the
original image embedding and update the pose accordingly. While [Izadinia et
al.(2017)Izadinia, Shan, and Seitz] can successfully predict rough room
layouts and retrieve similar CAD models to the ones observed, their approach
is significantly less accurate in terms of object retrieval and pose
prediction than ours. The work most similar to ours is Mask2CAD [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai]. In contrast to our method [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai] estimates the object pose by regressing
the rotation parameters as well as the offset between the reprojected object
center and the 2D bounding box centers. Using the ground truth $z$-coordinate
[Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] can estimate the object pose.
Unlike our system [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] is limited to
pure shape retrieval as it can not perform shape adaptations. Patch2CAD [Kuo
et al.(2021)Kuo, Angelova, Lin, and Dai] is similar to [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai] but instead of retrieving a single CAD model based on
the entire image, [Kuo et al.(2021)Kuo, Angelova, Lin, and Dai] retrieves many
CAD models for different image regions and perform majority voting to obtain
the final prediction. [Grabner et al.(2019)Grabner, Roth, and Lepetit] learn
an embedding space and estimate poses using a Location Field Descriptor
containing the estimated 3D object coordinate for every pixel. However, their
method struggles to predict correct location fields from which the pose can be
accurately computed and do not work for occluded objects. Recently [Maninis et
al.(2020)Maninis, Popov, Nießner, and Ferrari] have extended [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai] to temporally combine predictions from
individual video frames into more accurate CAD model and pose predictions. We
differ from all existing work on object retrieval in the usage of keypoint
matches for precise pose and shape estimation.

##  3 Method

Our method consists of four steps: (i) object detection and instance
segmentation, (ii) CAD model retrieval, (iii) keypoint matching and finally,
(iv) pose and shape optimisation.

Figure 2: Method: Given an RGB image we perform object detection and instance
segmentation (step 1). We then retrieve a set (e.g. 10) of the nearest
neighbour CAD model renderings (step 2) and perform keypoint matching between
the retrieved image and the rendered image using SuperPoint [DeTone et
al.(2018)DeTone, Malisiewicz, and Rabinovich] (step 3). The keypoint matches
are subsequently used to jointly optimise over the shape and pose of the
object (step 4).

Object Detection and Instance Segmentation. For object detection and instance
segmentation we train a Swin-Transformer [Liu et al.(2021)Liu, Lin, Cao, Hu,
Wei, Zhang, Lin, and Guo] network on the Pix3D [Sun et al.(2018)Sun, Wu,
Zhang, Zhang, Zhang, Xue, Tenenbaum, and Freeman] dataset. During training we
employ standard image augmentation techniques including random crops, scaling,
rotations, horizontal flipping, and random brightness and contrast
adjustments. We also report results on segmentation masks obtained using Mask
R-CNN [He et al.(2017)He, Gkioxari, Dollár, and Girshick] trained by [Gkioxari
et al.(2019)Gkioxari, Malik, and Johnson] as our approach is sensitive to the
quality of segmentation predictions. The Swin-Transformer network provides
more accurate segmentations of objects with hard edges (e.g. chairs, tables or
wardrobes), while Mask R-CNN is superior on objects from categories with soft
edges such as beds and sofas.
Learning a Joint Embedding Space. In order to map CAD models and masked RGB
images into a joint embedding space, we first render a CAD model in regular
intervals and represent it as a collection of these renderings [Chen et
al.(2003)Chen, Tian, Shen, and Ouhyoung]. This step is important as it bridges
the domain gap from CAD models to RGB images and increases the similarity of
the two inputs therefore simplifying the matching task. We use a single VGG
[Simonyan and Zisserman(2015)] encoder for encoding both real masked RGB
images and rendered inputs. This encoder is trained using a triplet-loss
[Chechik et al.(2010)Chechik, Sharma, Shalit, and Bengio]:

$\mathcal{L}=\sum_{i=1}^{16}\mathrm{max}(0,d^{2}(A,P)-d^{2}(A,N_{i})+m)\,.$
(1)

Here $A$ is the encoding of an anchor RGB image, $P$ is the encoding of the
positive example and $N_{i}$ is the encoding of a negative example. Only the
rendering of the corresponding ground truth CAD model in the most similar
orientation to the object is considered to be the positive example. A random
subset of 16 renderings of different CAD models are used as negative examples
$N_{i}$. Finally, $m$ is the margin and $d(x,y)$ is the Euclidean distance
over the $128$-dimensional encodings. As in [Kuo et al.(2020)Kuo, Angelova,
Lin, and Dai] we employ hard example mining during training. Only renderings
of CAD models of the same category as the query image are considered for hard-
negative mining. Unlike [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai], no hard
example mining is applied to positive examples. Doing hard positive mining
forces the network to match very different views, often not sharing any
features with each other, therefore necessarily leading to poor performance
via over-fitting. Instead, as mentioned above, the CAD model rendering in the
most similar orientation is used as a positive example. At test time we use an
embedding of a masked RGB image and retrieve the nearest neighbour CAD model
renderings which are then passed on for keypoint matching and joint pose and
shape estimation.

Key-Point Matching. In order to precisely estimate poses, keypoint matching is
performed between the masked RGB image and its retrieved CAD model rendering
[Georgakis et al.(2019)Georgakis, Karanam, Wu, and Kosecka]. Finding and
matching valid keypoints across different domains (RGB and rendered image) may
seem initially a difficult task due to the contrast of cleanly (sharp
boundaries, perfect segmentation masks, no occlusions) rendered images and
varying appearances of objects in RGB images (variety of textures, lighting
conditions and imperfect segmentation masks). However, we found a SuperPoint
[DeTone et al.(2018)DeTone, Malisiewicz, and Rabinovich] keypoint detection
and matching network to be well suited for this task. Its robustness stems
from the fact that it was initially trained to detect corners of triangles,
quadrilaterals, lines, cubes, checkerboards and stars in synthetic images
which was followed by fine-tuning on real images, hence performing well on
both domains. Crucially man-made furniture objects contain many of the
aforementioned primitives. We used the trained off-the-shelf network
implementation from [DeTone et al.(2018)DeTone, Malisiewicz, and Rabinovich]
to avoid over-fitting as Pix3D [Sun et al.(2018)Sun, Wu, Zhang, Zhang, Zhang,
Xue, Tenenbaum, and Freeman] is considerably smaller than typical datasets
used for keypoint detector training. SuperPoint [DeTone et al.(2018)DeTone,
Malisiewicz, and Rabinovich] returns approximately $25$ keypoints (each
described with a 256-D vector) on average per real RGB image ($15$ on rendered
image) using a default confidence threshold of $0.015$. Nearest neighbour
matches are found for each keypoint using the L2-distance. Cross-checking is
used to eliminate one-sided matches producing on average $12$ matches per RGB
and rendered image pair.

Pose Estimation.

Figure 3: Left: Pose Selection. Green points are reprojected points from the
CAD model under the current pose estimate. Purple points are sampled inside
the predicted segmentation mask. Selecting a pose based on the approximated
silhouette overlap yields precise poses (top) while using the minimum distance
of the reprojected keypoints selects inaccurate poses (bottom). Right:
Stretching Procedure. A CAD model can be stretched along different stretch
planes to generate new shapes. The plane normal $\mathbf{n}$ and the distance
$d$ of the plane from the object center are highlighted for one of the stretch
planes.

The keypoint matches provide correspondences between real image pixel
coordinates (2D) and 3D world coordinates in CAD model space. This allows us
to formulate the pose estimation as a PnP-problem. Since the obtained matches
are often noisy (only 4-5 matches out of 12 are correct on average) we
estimate poses for all available quadruplets111On average we obtain 12
keypoint matches per image leading to $12\times 11\times 10\times 9/(4\times
3\times 2\times 1)=495$ possible quadruplets and poses. of matches using the
UPnP [Kneip et al.(2014)Kneip, Li, and Seo] algorithm. Instead of using an
inlier scoring as is typically done in robust pose estimation, we select a
final pose from the computed poses by approximating the Intersection-over-
Union (IoU) silhouette overlap of the reprojected CAD model and the predicted
segmentation mask. For this purpose we sample 1000 points from the CAD model
and reproject them into the RGB image. Additionally we sample 1000 points from
within the predicted segmentation mask. For each reprojected point we compute
the distance to the closest points sampled from within the predicted
segmentation mask and vice versa (see Figure 3 on the left). For each object
pose the average of the largest $20\%$ of pairwise distances is computed and
the pose with the smallest average is selected as the final pose. We found
that selecting the final pose based on the average of all pairwise distances
was less robust in pose estimation (see supplementary material).

Pose and Shape Estimation. In order to perform joint pose and shape adaptation
the standard PnP-formulation is not sufficient. In particular, to enable shape
adaptation, we reparametrise previously fixed object coordinates $x$ (now
$x_{stretch}$) in the following way. We allow stretching of the object by an
amount $\tau_{i}$ along the normal $\mathbf{n}_{i}$ of a plane $P_{i}$ defined
by $\mathbf{n}_{i}\cdot\mathbf{x}_{i}=d_{i}$. Here $d_{i}$ is the distance of
the plane $P_{i}$ from the object center (see Figure 3 on the right).
Repeating this stretching for $N_{\mathrm{planes}}$ orthogonal planes the
stretched world coordinates become

$\mathbf{x}_{\mathrm{stretch}}=\mathbf{x}+\sum_{i=1}^{N_{\mathrm{planes}}}s_{i}\cdot\frac{\tau_{i}}{2}\cdot\mathbf{n}_{i}\quad\quad\mathrm{where}\,\,s_{i}=\begin{cases}1,&\text{if
}\mathbf{x}\cdot\mathbf{n}_{i}\geq d_{i}\\\ 0,&\text{if
}\mathbf{x}\cdot\mathbf{n}_{i}=d_{i}\\\ -1,&\text{if
}\mathbf{x}\cdot\mathbf{n}_{i}\leq d_{i}\end{cases}\,.$ (2)

From the stretched world coordinates one obtains reprojected pixel
$\mathbf{v}\in\mathcal{R}^{2}$ under the perspective camera model
$(s\,v_{x},s\,v_{y},s)=\mathbf{K}\left[\mathbf{R}|\mathbf{T}\right]\left[\mathbf{x}_{\mathrm{stretch}},1\right]^{T}$
for camera calibration matrix $\mathbf{K}\in\mathcal{R}^{3\times 3}$, rotation
matrix $\mathbf{R}\in\mathcal{R}^{3\times 3}$ and translation vector
$\mathbf{T}\in\mathcal{R}^{3\times 1}$. The rotation matrix is parameterised
in terms of Euler angles $\mathbf{\theta}\in\mathcal{R}^{3}$. For known camera
intrinsics the objective function to be minimised is therefore
$f=\sum_{j=1}^{N_{\mathrm{matches}}}(\mathbf{u}_{j}-\mathbf{v}_{j})^{2}$ with
respect to $(\mathbf{\theta},\mathbf{T},\mathbf{\tau})$ where $\mathbf{u}_{j}$
are pixel coordinates of the j-th match in the RGB image. L-BFGS [Liu and
Nocedal(1989)] minimiser, instead of UPnP is used for the minimisation of this
non-linear objective function. It is initialised with the original pose of the
retrieved CAD model and no stretching $\mathbf{\tau}=0$. Note that in this
case we sample sets of 6 matches instead of 4 to cover enough degrees of
freedom for 3 deformations.

##  4 Experimental Setup

This section briefly describes the Pix3D [Sun et al.(2018)Sun, Wu, Zhang,
Zhang, Zhang, Xue, Tenenbaum, and Freeman] dataset that was used for training
and evaluation, the $\mathrm{AP}^{mesh}$ metric we adopted for evaluation as
well as the hyperparameters chosen.
Pix3D Dataset. The Pix3D [Sun et al.(2018)Sun, Wu, Zhang, Zhang, Zhang, Xue,
Tenenbaum, and Freeman] dataset consists of 10,069 RGB images annotated with
aligned 3D CAD models (one per image). There are a total of 395 different CAD
models from 9 categories (chair, sofa, table, bed, desk, bookcase, wardrobe,
tool and miscellaneous). For our experiments we consider two splits originally
proposed by [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson].
S1 split. The $S1$ split randomly splits the 10,069 images into 7539 train
images and 2530 test images. In this split all CAD models are seen during
training and the challenge is to retrieve and align the correct CAD model from
images containing different scenes where (possibly occluded) objects appear
with new textures under varying lighting conditions.
S2 split. Under the S2 split train and test images are split such that the CAD
models that have to be retrieved at test time were unseen during training.
This split is more difficult as it prohibits the embedding network to simply
remember CAD models and truly tests its ability to learn meaningful
embeddings.
Evaluation metric. We adopt the commonly used $\mathrm{AP}^{mesh}$ metric
[Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] for evaluating the
retrieved object shapes. Following the standard COCO [Lin et al.(2015)Lin,
Maire, Belongie, Bourdev, Girshick, Hays, Perona, Ramanan, Zitnick, and
Dollár] object detection protocol of AP50-AP95 (denoted AP), we average over
10 IoU thresholds ranging from 0.50 to 0.95 in 0.05 intervals. For a given
threshold the $\mathrm{AP}^{mesh}$ score is defined as the mean area under the
per-category precision-recall curve where a shape prediction is considered a
true-positive if its predicted category label is correct, it is not a
duplicate detection, and its $\mathrm{F1}^{\tau}$ is greater than the IoU
threshold. For a given predicted shape the $\mathrm{F1}^{\tau}$ score is the
harmonic mean of the fraction of predicted points within $\tau$ of a ground-
truth point and the fraction of ground-truth points within $\tau$ of a
predicted point. We follow [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson,
Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] in choosing $\tau=0.3$. For fair
comparison across different object sizes we rescale all objects such that the
longest edge of the ground truth model’s bounding box has length 10 before
computing the $\mathrm{F1}$ score.
Hyperparameter settings. CAD models are rendered at $16$ regularly sampled
azimuthal angles spanning $360^{\circ}$ and $4$ different elevation angles
between $0^{\circ}$ and $45^{\circ}$. The VGG encoder is trained with a
batchsize of $8$ real images as each example requires $16$ negative anchors
and one positive anchor leading to a total of $144$ images per batch. We use a
learning rate of $2\times 10^{-6}$ and set the margin of the triplet-loss in
Equation 1 to $m=0.1$. We use off-the-shelf object detection and segmentation
networks [Liu et al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo, He et
al.(2017)He, Gkioxari, Dollár, and Girshick] as well as keypoint matching
network - SuperPoint [DeTone et al.(2018)DeTone, Malisiewicz, and Rabinovich].

##  5 Experimental Results

This section showcases our experimental results. We compare against Mesh-RCNN
[Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] and Pixel2Mesh [Wang et
al.(2018)Wang, Zhang, Li, Fu, Liu, and Jiang] (specifically the
reimplementation by [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] which
outperforms the original [Wang et al.(2018)Wang, Zhang, Li, Fu, Liu, and
Jiang]) as well as Mask2CAD [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] and
Patch2CAD [Kuo et al.(2021)Kuo, Angelova, Lin, and Dai]. Section 5.1 shows our
results on the Pix3D[Sun et al.(2018)Sun, Wu, Zhang, Zhang, Zhang, Xue,
Tenenbaum, and Freeman] S1 and S2 split when access to the correct CAD models
is provided at test time. In Section 5.2 we evaluate the proposed stretching
on modified versions of the original CAD models as well as when available CAD
models have to be adapted to match entirely different ones.

Figure 4: Qualitative comparison for predictions on the S2 split: The left
side shows results when access to the correct CAD model is given at test time,
but which were unseen at train time. The right side shows the case when no
access to correct CAD models is given and retrieved CAD models have to be
adapted dynamically. Overlaid numbers are the $\mathrm{F1}$ score. In general
the comparison shows that a geometric approach allows for very precise pose
estimation whereas the direct prediction method of Mask2CAD [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai] is less precise. Note that the
qualitative results are from our reimplementation of [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai] as neither the code nor the predictions are publicly
available. In comparison to CAD model retrieval direct mesh predictions
[Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] are very imprecise, often
failing to predict the correct topology and performing particularly poorly on
the backside of objects. Row 4 shows the sensitivity of the
$\mathrm{F1}^{0.3}$ score. Despite an appropriate object retrieval and very
good shape adaptation, slight imprecision in the alignments lead to a low F1
score. Finally row 5 shows a failure case of ours where poor segmentation
leads to a wrong shape retrieval and correspondingly false keypoint matches
resulting in a bad final pose and shape.

###  5.1 Geometry-Based Shape and Pose Predictions are very Precise

Table 1 shows the $\mathrm{AP}^{mesh}$ we obtain on the S1 and S2 splits of
Pix3D, originally proposed by Mesh R-CNN [Gkioxari et al.(2019)Gkioxari,
Malik, and Johnson]. For a fair comparison to Mask2CAD [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai], Patch2CAD [Kuo et al.(2021)Kuo, Angelova, Lin, and
Dai], Mesh R-CNN [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] and
Pixel2Mesh [Wang et al.(2018)Wang, Zhang, Li, Fu, Liu, and Jiang] we use the
ground truth $z$-coordinate for the final pose. While our approach does not
require the use of the ground truth $z$-coordinate (unlike [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai, Kuo et al.(2021)Kuo, Angelova, Lin, and
Dai]) it improves our performance as the low F1 score threshold is very
sensitive even to small displacements in the $z$ direction arising from slight
inaccuracies in the keypoint matches.

Seen Objects. Results on the S1 split show that particularly on seen objects,
CAD model retrieval is more precise than direct predictions. For the high
$\mathrm{AP}^{\mathrm{mesh}}$ regions (i.e. AP and AP75) retrieval-based
methods outperform direct prediction methods by a large margin. On the AP50
score Mesh-RCNN [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] performs
similar to the retrieval-based methods as the AP50 score only evaluates if at
least $50\%$ of a sampled object is predicted correctly and Mesh-RCNN
[Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] is generally able to
predict the object side in direct sight of the camera. In terms of the
category average we outperform the best competitor [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai] (37.8 vs 33.2). Mask2CAD [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai] performs well on large planar objects such as
bookcases or wardrobes, whereas we have very strong performance on high
fidelity objects, such as chairs, allowing for numerous keypoint matches. When
using ground truth masks compared to predicted masks we observe a very large
performance gain (37.8 vs 33.2). This observation is crucial as it shows the
potential of our approach with improved segmentation masks. While competing
approaches will also benefit from better segmentation, having accurate
instance masks for the pose prediction is not as an integral part in their
pipeline as it is for our keypoint matching and does therefore not benefit
them as much.

Unseen Objects. We show qualitative comparisons on the S2 split for [Kuo et
al.(2020)Kuo, Angelova, Lin, and Dai] and [Gkioxari et al.(2019)Gkioxari,
Malik, and Johnson] in Figure 4. Quantitatively, we outperform all competitors
on all metrics. We perform significantly better than [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai] (17.1 compared to 8.2 on the class average) because of
the geometric pose predictions. [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai]
struggles to predict poses of unseen objects whereas our pose prediction
relies on analytically computing poses from correspondences which generalises
a lot better to unseen shapes. Note that Mask2CAD [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai] performs well on sofas, not because it is able to
retrieve an unseen sofa but because for every sofa in the S2 split there is a
very good fitting sofa among the seen sofas, allowing to simply retrieve that
for a good performance (see the supplementary material).

Table 1: Quantitative results on the S1 and S2 split for seen and unseen
objects on the Pix3D dataset. Brackets indicate the segmentation masks that
were used. Bold numbers are the maximum values excluding experiments that were
run with ground truth (GT) masks. Figure 5: a) Retrieval accuracy for selected
CAD model splits. When considering the top 10 nearest neighbours the retrieval
network is able to return completely unseen CAD models in over 50% of cases.
Note that different renderings of the same CAD model are considered as
different nearest neighbours. b) Ablation experiments on the proposed object
stretching with ground truth masks. We plot the average
$\mathrm{AP}^{\mathrm{mesh}}$ score as a function of increasing shape
deformations of S2 CAD models. On the left no deformations were performed
while on the right objects were stretched along the $x$,$y$ and $z$ direction.
With increasing deformation simple object retrieval quickly becomes
inaccurate, while the proposed stretching is able to maintain a high accuracy.

###  5.2 Shape and Pose Estimation using Stretchable CAD Models

For realistic settings a given object will not have a perfect match among the
available CAD models and a retrieved CAD model will require adaptation. We
evaluate our adaptation approach on three different settings: stretched S1
models, stretched S2 models and predicting S2 test models with S2 train models
(see Figure 6). For the first two experiments the databases of S1 and S2
models are modified by applying random stretching in the $x$, $y$ and $z$
direction with planes passing through the center of the object. Stretch
factors $\tau_{i}$ in each direction are obtained by multiplying a random
number from a uniform distribution on the interval [-0.2, 0.3] by the side
length of the bounding box of the object in the relevant direction.
Experiments on stretched S1 models demonstrate that shape adaptation
substantially improves the shape predictions (15.5 vs. 4.4, see Figure 6). For
stretched S2 models we can observe significant improvements for certain
classes such as bookcases, chairs or tables. However, the overall accuracy
gain is smaller (9.3 vs. 8.5) and some classes, most notably sofas, show worse
performance. As explained above very accurate shape models already exist for
sofa objects in the non-stretched part of the database. Hence, stretching from
sometimes poor segmentation masks and corresponding keypoint matches can
deteriorate the estimated shapes. To further investigate the performance of
the system when enabling shape deformations, we perturb the models in the
database with progressively larger perturbations, and compare the predictions
generated when stretching is allowed and disallowed (see Figure 5 b). As CAD
model perturbations are increased, the proposed shape adaptation method can
maintain a high accuracy while estimating the poses and of retrieved objects
without using shape adaptation quickly becomes inaccurate. Finally, when
estimating shapes on S2 test images while only allowing to retrieve from the
S2 train model set (see Figure 6) we observe that similar to the experiments
on the stretched S2 models we show improvements only for some classes (e.g.
beds). This is due to the poor segmentation quality which prevents the
matching network from successfully establishing a sufficient number of
correspondences. When ground truth masks are used instead, significant
improvements for almost all categories are observed.

Figure 6: a) Quantitative results on Pix3D when no access to correct models is
provided at test time. For the first two main rows (containing two sub-rows
each) CAD models in the database are randomly stretched along all 3 principal
directions and our method has to recover the original shape. For the third
main row S2 test CAD models have to be estimated when the retrieval network
has no access to the correct models and differing S2 train CAD models have to
be adapted. b) Visual comparison of shape prediction with predicted (row one
and three) and ground truth (row two and four) segmentation masks.

##  6 Conclusion

In this work we propose to leverage geometric constraints for precisely
estimating 3D object shapes from retrieved CAD models. We demonstrate that our
approach is more accurate than direct pose prediction [Gkioxari et
al.(2019)Gkioxari, Malik, and Johnson, Wang et al.(2018)Wang, Zhang, Li, Fu,
Liu, and Jiang] and the image retrieval-based approaches [Kuo et al.(2020)Kuo,
Angelova, Lin, and Dai, Kuo et al.(2021)Kuo, Angelova, Lin, and Dai]. Further,
we show that by allowing object stretching we can modify retrieved CAD models
to better fit the observed shapes. We believe that adapting retrieved CAD
models is a promising avenue for future research as it combines the
reliability of object retrieval with the expressiveness of generative
approaches.

## Acknowledgments

This work was supported by the Engineering and Physical Sciences Research
Council
[RG105266].

## References

  * [Aubry et al.(2014)Aubry, Maturana, Efros, Russell, and Sivic] Mathieu Aubry, Daniel Maturana, Alexei Efros, Bryan Russell, and Josef Sivic.  Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Colombus, Ohio, US, June 2014.
  * [Chang et al.(2015)Chang, Funkhouser, Guibas, Hanrahan, Huang, Li, Savarese, Savva, Song, Su, Xiao, Yi, and Yu] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [Chechik et al.(2010)Chechik, Sharma, Shalit, and Bengio] Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio.  Large Scale Online Learning of Image Similarity Through Ranking.  _Journal of Machine Learning Research_ , 11(36):1109–1135, 2010.
  * [Chen et al.(2003)Chen, Tian, Shen, and Ouhyoung] Ding-Yun Chen, Xiao-Pei Tian, Yu-Te Shen, and Ming Ouhyoung.  On Visual Similarity Based 3D Model Retrieval.  _Computer Graphics Forum_ , 22(3):223–232, 2003\.
  * [Chen et al.(2020)Chen, Tagliasacchi, and Zhang] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang.  BSP-Net: Generating Compact Meshes via Binary Space Partitioning.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Virtual, June 2020.
  * [Choy et al.(2016)Choy, Xu, Gwak, Chen, and Savarese] Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese.  3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction.  In _Proc. 14th European Conference on Computer Vision_ , Amsterdam, NL, October 2016.
  * [Dalal and Triggs(2005)] N. Dalal and B. Triggs.  Histograms of oriented gradients for human detection.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , volume 1, pages 886–893 vol. 1, 2005.
  * [Deng et al.(2020)Deng, Genova, Yazdani, Bouaziz, Hinton, and Tagliasacchi] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.  CvxNet: Learnable Convex Decomposition.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Virtual, June 2020.
  * [DeTone et al.(2018)DeTone, Malisiewicz, and Rabinovich] Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich.  SuperPoint: Self-Supervised Interest Point Detection and Description.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Salt Lake City, Utah, USA, June 2018.
  * [Fan et al.(2017)Fan, Su, and Guibas] Haoqiang Fan, Hao Su, and Leonidas Guibas.  A Point Set Generation Network for 3D Object Reconstruction from a Single Image.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Honolulu, USA, July 2017.
  * [Georgakis et al.(2019)Georgakis, Karanam, Wu, and Kosecka] Georgios Georgakis, Srikrishna Karanam, Ziyan Wu, and Jana Kosecka.  Learning Local RGB-to-CAD Correspondences for Object Pose Estimation.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Seoul, Korea, October 2019.
  * [Gkioxari et al.(2019)Gkioxari, Malik, and Johnson] Georgia Gkioxari, Jitendra Malik, and Justin Johnson.  Mesh R-CNN.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Seoul, Korea, October 2019.
  * [Grabner et al.(2019)Grabner, Roth, and Lepetit] Alexander Grabner, Peter M. Roth, and Vincent Lepetit.  Location Field Descriptors: Single Image 3D Model Retrieval in the Wild.  In _2019 International Conference on 3D Vision (3DV)_ , pages 583–593, 2019.
  * [He et al.(2017)He, Gkioxari, Dollár, and Girshick] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Venice, Italy, October 2017.
  * [Izadinia et al.(2017)Izadinia, Shan, and Seitz] Hamid Izadinia, Qi Shan, and Steven M. Seitz.  IM2CAD.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Honolulu, USA, July 2017.
  * [Kneip et al.(2014)Kneip, Li, and Seo] Laurent Kneip, Hongdong Li, and Yongduek Seo.  UPnP: An Optimal O(n) Solution to the Absolute Pose Problem with Universal Applicability.  In _Proc. 13th European Conference on Computer Vision_ , Zurich, Switzerland, September 2014.
  * [Krizhevsky et al.(2017)Krizhevsky, Sutskever, and Hinton] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  _Commun. ACM_ , 60(6):84–90, May 2017.
  * [Kuo et al.(2020)Kuo, Angelova, Lin, and Dai] Weicheng Kuo, Anelia Angelova, Tsung-Yi Lin, and Angela Dai.  Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve.  In _Proc. 16th European Conference on Computer Vision_ , Glasgow, UK, August 2020.
  * [Kuo et al.(2021)Kuo, Angelova, Lin, and Dai] Weicheng Kuo, Anelia Angelova, Tsung-yi Lin, and Angela Dai.  Patch2CAD: Patchwise Embedding Learning for In-the-Wild Shape Retrieval from a Single Image.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Montreal (Virtual), October 2021.
  * [Li et al.(2015)Li, Su, Qi, Fish, Cohen-Or, and Guibas] Yangyan Li, Hao Su, Charles Ruizhongtai Qi, Noa Fish, Daniel Cohen-Or, and Leonidas J. Guibas.  Joint Embeddings of Shapes and Images via CNN Image Purification.  _ACM Trans. Graph._ , 34(6), October 2015.
  * [Lin et al.(2015)Lin, Maire, Belongie, Bourdev, Girshick, Hays, Perona, Ramanan, Zitnick, and Dollár] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár.  Microsoft COCO: Common Objects in Context, 2015.
  * [Liu and Nocedal(1989)] Dong C. Liu and Jorge Nocedal.  On the Limited Memory BFGS Method for Large Scale Optimization.  _Mathematical Programming_ , 45:503–528, 1989.
  * [Liu et al.(2021)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Montreal (Virtual), October 2021.
  * [Maninis et al.(2020)Maninis, Popov, Nießner, and Ferrari] Kevis-Kokitsi Maninis, Stefan Popov, Matthias Nießner, and Vittorio Ferrari.  Vid2cad: Cad model alignment using multi-view constraints from videos.  _arXiv_ , 2020.
  * [Mildenhall et al.(2020)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis.  In _Proc. 16th European Conference on Computer Vision_ , Glasgow, UK (Virtual), August 2020.
  * [Nie et al.(2020)Nie, Han, Guo, Zheng, Chang, and Zhang] Yinyu Nie, Xiaoguang Han, Shihui Guo, Yujian Zheng, Jian Chang, and Jian Jun Zhang.  Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Virtual, June 2020.
  * [Pan et al.(2019)Pan, Han, Chen, Tang, and Jia] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia.  Deep Mesh Reconstruction from Single RGB Images via Topology Modification Networks.  In _Proc. IEEE Int. Conf. on Computer Vision_ , Seoul, Korea, October 2019.
  * [Park et al.(2019)Park, Florence, Straub, Newcombe, and Lovegrove] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Long Beach, CA, USA, June 2019.
  * [Simonyan and Zisserman(2015)] Karen Simonyan and Andrew Zisserman.  Very Deep Convolutional Networks for Large-Scale Image Recognition.  In _3rd International Conference on Learning Representations_ , San Diego, CA, USA, May 2015.
  * [Sun et al.(2018)Sun, Wu, Zhang, Zhang, Zhang, Xue, Tenenbaum, and Freeman] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman.  Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Salt Lake City, Utah, USA, June 2018.
  * [Tatarchenko et al.(2019)Tatarchenko, Richter, Ranftl, Li, Koltun, and Brox] Maxim Tatarchenko, Stephan R. Richter, René Ranftl, Zhuwen Li, Vladlen Koltun, and Thomas Brox.  What Do Single-view 3D Reconstruction Networks Learn?  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Long Beach, CA, USA, June 2019.
  * [Tulsiani et al.(2018)Tulsiani, Su, Guibas, Efros, and Malik] Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A. Efros, and Jitendra Malik.  Learning Shape Abstractions by Assembling Volumetric Primitives.  In _Proc. IEEE Conf. on Computer Vision and Pattern Recognition_ , Honolulu, USA, July 2018.
  * [Wang et al.(2018)Wang, Zhang, Li, Fu, Liu, and Jiang] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang.  Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images.  In _Proc. 15th European Conference on Computer Vision_ , Munich, Germany, September 2018.
  * [Zeng et al.(2020)Zeng, Karaoglu, and Gevers] Wei Zeng, Sezer Karaoglu, and Theo Gevers.  Inferring Point Clouds from Single Monocular Images by Depth Intermediation.  _arXiv_ , 2020.