11institutetext: Inst. for Computer Graphics and Vision, Graz Univ. of
Technology, Graz, Austria 22institutetext: LIGM, École des Ponts, Univ Gustave
Eiffel, CNRS, Marne-la-Vallée, France 22email: {sinisa.stekovic,
stefan.ainetter<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>22email<EMAIL_ADDRESS>
# PyTorchGeoNodes: Enabling Differentiable Shape Programs for 3D Shape
Reconstruction

Sinisa Stekovic 11 Stefan Ainetter 11 Mattia D’Urso 11 Friedrich Fraundorfer
11 Vincent Lepetit 22

###### Abstract

We propose PyTorchGeoNodes, a differentiable module for reconstructing 3D
objects from images using interpretable shape programs. In comparison to
traditional CAD model retrieval methods, the use of shape programs for 3D
reconstruction allows for reasoning about the semantic properties of
reconstructed objects, editing, low memory footprint, etc. However, the
utilization of shape programs for 3D scene understanding has been largely
neglected in past works. As our main contribution, we enable gradient-based
optimization by introducing a module that translates shape programs designed
in Blender, for example, into efficient PyTorch code. We also provide a method
that relies on PyTorchGeoNodes and is inspired by Monte Carlo Tree Search
(MCTS) to jointly optimize discrete and continuous parameters of shape
programs and reconstruct 3D objects for input scenes. In our experiments, we
apply our algorithm to reconstruct 3D objects in the ScanNet dataset and
evaluate our results against CAD model retrieval-based reconstructions. Our
experiments indicate that our reconstructions match well the input scenes
while enabling semantic reasoning about reconstructed objects.

---
Shape parameter Differentiable graph Generated 3D shape
Our reconstructions
Figure 1: Our PyTorchGeoNodes is a framework that enables differentiable shape
programs in PyTorch. Given interpretable input shape parameters, a shape
program generates a mesh for a 3D object. Our differentiable pipeline makes
possible the flow of gradients from the generated shape to the continuous
parameters. We show a variety of sofas and armchairs generated by a single
shape program. The method we propose can retrieve these shapes using a
sequence of RGB-D scans as input, estimating both the continuous parameters
(Width, Depth, etc.) and the discrete parameters (Has Left Arm, Is L-Shaped,
etc.).

##  1 Introduction

In 3D scene understanding, many different representations of 3D objects that
can be recovered from images have already been considered: voxels [29, 20],
meshes [12, 38], SDFs [24], NeRFs [7], Gaussian Splatting [34], exemplars from
a shape database [3]. SDFs, NeRFs, or Gaussian Splatting can generalize to
shapes unseen during training but can still produce artifacts for example when
parts of the object are occluded in scenes captured “in the wild”. Moreover,
their memory footprint can become large. Using exemplars guarantees that the
retrieved shapes look good as they are usually created by human designers,
however the database may not contain shapes similar to the target object.

In this paper, we investigate the use of interpretable shape programs to
represent objects. Shape programs are a form of procedural modeling [11]
introduced in [35] for representing 3D objects. The top of Figure 1
illustrates a shape program generating sofa and armchair shapes.

Shape programs are a computational graph of parameterized geometric operations
that generate various 3D shapes from an object category given input shape
parameters, such as object dimensions as well as “semantic” properties like
’with armrests’. Thanks to this procedural modeling approach, they can capture
all the shape variation within an object category so that the generated shapes
remain appealing and can fit many different instances. By design, they also
allow the decomposition of generated shapes into smaller shape primitives.
This makes possible reasoning about semantic components, which is an advantage
over more traditional CAD model retrieval methods. Other nice properties
include being easily editable and having a small memory footprint. They became
a popular tool for building reconstruction [39, 23, 18, 36] and the interest
around shape programs for objects is growing [35, 15, 16, 26]. Blender [9]
already provides support for them. A research challenge targeting
reconstruction of walls, doors, and windows using such structured languages
was organized by Meta in 2024 111[https://eval.ai/web/challenges/challenge-
page/2115/overview (Accessed: March 2024)].

There have been a few attempts in the computer vision community to use
procedural shapes [30, 26]. Recovering the parameters of a shape program from
an image is however very challenging. Typically, a shape generated by a shape
program depends on a number of parameters. Some parameters such as the width
and height of the objects take continuous values but other parameters take
only discrete values, for example the number of shelves in a bookshelf.
Optimizing on these parameters turned out to be tricky: For example, changing
the number of shelves from 5 to 6 changes significantly the shape and
appearance of a bookshelf.

We introduce two contributions to make possible the estimation of shape
programs’ parameters from images with a render-and-compare algorithm. The
first contribution is a “compiler” that generates PyTorch [25] code for a
shape program, which can be defined in Blender for example. This code
generates 3D shapes that are differentiable with respect to the continuous
parameters and can therefore be used easily with 3D reconstruction algorithms
to optimize the continuous parameters.

While our compiler can be used to optimize over the continuous parameters,
there remains the challenge of estimating the discrete parameters. Our
experiments show that random sampling is highly inefficient. In a simple
experiment of identifying parameters of a sofa, random search needed $41$
minutes to find an acceptable solution. Our second contribution is to show how
to adapt recent methods based on Monte Carlo Tree Search to our problem of
jointly optimizing the continuous and the discrete parameters of a shape
program. For the same example, our method finds a better solution in $2.3$
minutes.

We evaluate our method on both synthetic and real scenes. In addition to the
usual shape metrics, we also evaluate how well we retrieve the shape program
discrete parameters. Because existing datasets are not annotated with the
value of these parameters, we manually annotated 276 objects. In summary, our
main contribution is a general tool to create PyTorch code from Blender shape
programs. In addition, we provide a method for estimating the parameters of
shape programs from views together with the poses of the objects. We will make
our code available on our project page,
https://vevenom.github.io/pytorchgeonodes/, with the hope that our compiler
will encourage the use of shape programs and that our method will inspire
other researchers to work on recovering shape programs from images. Beyond
shape retrieval, differentiable rendering of shape programs could open the
door to methods creating shape programs from images in a self-supervised way,
an exciting research direction.

##  2 Related Work

Since the field of retrieving 3D models for objects from images is very large,
we focus here only on methods that can provide a light CAD model designed for
specific categories: 3D model retrieval methods and Shape Programs.

###  2.1 3D Model Retrieval

3D model retrieval aims to retrieve a CAD model from a database of objects,
such as the ShapeNet dataset [8], that maximizes some scoring function which
is typically based on an input 3D representation or a single or multiple input
images.

Some methods approach CAD model retrieval as a classification problem [3, 22].
For a given input a neural network can be trained to predict a probability
distribution over all models in the database. However, these direct approaches
do not scale well to large object databases such as the ShapeNet database.
Another popular approach is computing a similarity metric between synthetic
objects and real inputs. There are several possible ways of computing such
metrics. It is possible to render an image from a synthetic object, and then
for a synthetic-real image pair, we can compute and compare image descriptors
[4, 13, 21] or encode images into embedding vector representations and
calculate the similarity in the embedding space [19].

While such approaches have displayed promising results, one clear limitation
is expressiveness of the database of objects. In fact, even with large
databases of objects, in practice we often encounter objects in real scenes
that are not represented in the database, or we might encounter modified or
articulated objects. For example, we can modify seat height, arm rest and back
support position of a work chair but the object in the database appears in the
default setting which will introduce errors during retrieval. While it is
possible to perform mesh refinement and deformation [5, 31, 14], these
approaches typically introduce artifacts and deformations in the
reconstruction and the final object shape might not be representative of the
actual object category after refinement.

###  2.2 Shape Programs

Procedural modelling is an attractive way of generating 3D shapes in computer
graphics, and as such it is an interesting feature of different 3D modelling
software. In general, procedural modelling is a family of techniques that
generate 3D content by abiding to a set of rules. In machine learning, [35]
introduced a procedural language that enables the generation of a variety of
shapes in form of so called shape programs. In [35], shape programs are
written in Domain Specific Language (DSL) for shapes with two types of
statements. “Draw” statements describes shape primitive, such as cuboid or a
cylinder, together with its geometric and semantic properties. “For”
statements represent loops of sub-programs including parameters that indicate
how the sub-program should be repeatedly executed. [35] also showed that it is
possible to learn to infer shape programs that describe input 3D shapes.
However, as it is based on basic geometric primitives, it becomes difficult to
model more complex geometries, and resulting shapes can be non-representative
of the target category. Other methods suffer from the same limitations [15,
16].

GeoCode [26] recently introduced hand-designed shape programs as a mapping
between 3D shapes of a specific object category and human-interpretable
parameter space. The language used for developing shape programs is based on
the geometry node feature of Blender, an open-source modelling software,
which, in contrast to [35], allows designing of more expressive shape programs
that can represent and interpolate between a variety of shapes by simply
modifying human-interpretable program’s parameters. Some nodes define a shape
primitive or pre-designed more complex shapes, _e.g_., chair seat or chair
leg. Other nodes represent geometric transformations on individual shapes, or
how two shapes should be connected, _e.g_., legs should always be connected to
the bottom of the seat. There are nodes representing mathematical operations
such as addition and multiplication, and there are nodes that are used to
replicate specific shapes, _e.g_., in case of chair legs we can add a
replicator node to create four instances of a chair leg of the same shape.
There are also nodes that serve as control statements, _e.g_., it is possible
to switch between a four-legged chair and a one-legged chair. A pre-defined
set of input program’s parameters allow non-expert user to easily control
shape-generation process. [26] also showed that it is possible to learn the
mapping from input point clouds of 3D shapes, or a sketch, to program’s
parameters.

##  3 PyTorchGeoNodes: Differentiable Computational Graphs for Shape Programs

In this section, we discuss shape programs and how they can be beneficial for
tasks in 3D scene understanding. We then describe our framework for enabling
efficient gradient-based optimization for shape programs and, in addition,
allowing to translate shape programs designed in Blender to PyTorch code to
enable differentiable optimization.

In Section 4, we explain how our framework can be applied for reconstructing
3D objects from images.

###  3.1 Shape Programs as Computational Graphs

We consider shape programs as defined in Blender, _i.e_., as a computational
graph that computes 3D shapes given a number of input parameters. Such graphs
contain nodes of different types; in our framework, we currently consider the
following node types:

  * •

An ’Input’ node contains input parameters of the shape programs and their
value ranges. We consider floating, integer, and boolean values parameters;

  * •

A ’Math’ node performs mathematical operations on given input such as
addition, subtraction, multiplication, division, and comparison;

  * •

A ’Switch’ node switches between two inputs conditioned on a boolean input;

  * •

A ’Combine’ node concatenates three input values into a 3D vector;

  * •

A ’Primitive’ node creates a geometric primitive, _e.g_. a cuboid or a
cylinder;

  * •

A ’Transform’ node applies scaling, rotation, and translation from input
vectors to the input geometry;

  * •

’Mesh Line’ node creates a tensor of 3D points on a line from input vectors
that define the start location and the offset of the line, and an integer that
defines the number of points to be generated;

  * •

A ’Points on instances’ node clones input geometry at given input 3D points;

  * •

A ’Join geometry’ node merges different geometries;

  * •

An ’Output geometry node’ generates the final mesh for the shape.

Note that for example, the Transform node depends on continuous parameters,
the Points on instances node on integer parameters, and the Switch node on
boolean parameters.

Figure 2: Gradient-based optimization of continuous parameters of a shape
program for the sofa category. From an initial estimate of the parameters of
the object, we can perform gradient descent on the parameters based on a 3D
geometric loss term. In contrast to methods that directly optimize the
reconstructed mesh, PyTorchGeoNodes allows optimization in the parameter space
which has several benefits. From the resulting shapes in this example, it is
observable that individual parameters can be scaled independently targeting
only specific parts of the shape geometry while preserving the compactness of
the 3D shape at the same time.

###  3.2 Blender Geometry Nodes to PyTorchGeoNodes

Figure 3: A shape program in Blender generating dividing boards of a cabinet.
The computational graph is designed and visualized in Blender using Geometry
Nodes feature. Underneath, we show how we abstract the different nodes using
PyTorch tensors and PyTorch3D meshes. The input node takes input parameters,
here {Width: 0.5, Dividing Board Thickness: 0.04, Height: 0.6, Number of
Dividing Boards: 5, Board Thickness: 0.04} and feeds them to a series of
operations. The blue nodes are arithmetic and concatenation nodes, which
transform input parameters and feed the results to geometry nodes, in green.
In this example, we generate a cuboid mesh and instantiate a line of points
which generates the final geometry for dividing boards. In practice, this
shape program is part of a larger shape program for modeling cabinets.

Our framework provides different computational nodes that reimplement the
functionalities of the individual geometry nodes in Blender. We visualize this
in Figure 3. More exactly, for every node type in Blender, we implement a
corresponding node type with the same functionalities using PyTorch, or
PyTorch3D[27] in case of geometric operations. We note that shape programs
from Blender are already saved as a graph structure which simplifies the
translation process. Therefore, our framework is designed to enable automatic
translation from shape programs designed in Blender into PyTorch code and we
can design shape programs directly using the Blender interface.

To make our implementation computationally efficient, we considered different
aspects. First, the nodes are ordered using a topological sort so that during
the forward pass of a node, all inputs are readily available, which avoids the
need for recursive calls. In addition, we cache outputs of nodes that do not
depend on input shape parameters such that they do not need to be recomputed
in subsequent forward calls. We provide additional implementation details in
the supplementary material. While computation time depends on the complexity
of shape programs, in our experiments inference time is always less than
$20$ms.

Figure 2 shows a simple example. Given a shape program created in Blender, our
parser generates a differentiable PyTorchGeoNodes code. We can then use this
code in PyTorch in an optimization procedure where we fit shape parameters to
generate a shape consistent with the input scene.

##  4 Search Algorithm

Figure 4: Reconstructing 3D objects with our PyTorchGeoNodes and our method.
At every iteration, our MCTS-based search algorithm first selects a set of
shape parameters and object pose. These are then passed through the
differentiable computational graph, parsed by our PyTorchGeoNodes framework
for a shape program, that generates a 3D mesh based on the selected shape
parameters. Then, we compute the loss that evaluates how well the shape fits
the input scene. We further optimize the continuous parameters of our shape
program using gradient descent and update the search tree based on this loss.

We provide in this section a way to retrieve the parameters of a shape program
for a given object from several RGB-D scans of this object. We do this by
exploiting our PyTorchGeoNodes framework. Following many current works, we use
a render-and-compare approach as it appears to provide the best results [2].
More exactly, we look for shape parameters ${\cal P}^{*}$ by minimizing the
following objective function:

$\displaystyle{\cal L}({\cal P})=$
$\displaystyle\sum_{i}\big{(}|D_{i}-M_{D}(D^{r}_{i}({\cal
P}),D_{i},S_{i})|+|N(D_{i})-N(M_{D}(D^{r}_{i}({\cal
P}),D_{i},S_{i}))|\big{)}\;+$
$\displaystyle\lambda_{\text{CD}}{\text{CD}}({\text{PC}},{\text{Sh}}({\cal
P}))\>.$ (1)

$D_{i}$ and $N_{i}$ denote the depth map and normal map respectively of the
$i$-th view, and $S_{i}$ is the estimated segmentation of object per input
view, obtained by running a segmentation network on the input color image of
the $i$-th view, as explained in Section 5.2. $D^{r}_{i}({\cal P})$ is the
depth map of the shape generated by the shape program given parameters ${\cal
P}$ corresponding to the $i$-th view. To deal with occlusions of the target
object in the input views, we merge the observed depth map $D_{i}$ with the
predicted depth map $D^{r}_{i}({\cal P})$ with operator $M_{D}$: For each
pixel where $S_{i}=1$, $M_{D}$ returns the values of $D^{r}_{i}({\cal P})$.
For each of the other pixels, $M_{D}$ returns the minimum depth between
$D_{i}$ and $D^{r}_{i}({\cal P})$ at this pixel. $N(D)$ returns the normal map
for depth map $D$.

We notice that including the Chamfer distance
${\text{CD}}({\text{PC}},{\text{Sh}}({\cal P}))$, between the scene and the
generated shape improves the speed of convergence. This was observed in [1] as
well. This can be explained by the fact that the Chamfer distance has a larger
radius of convergence than the render-and-compare terms and can be used to
guide the optimization. PC represents the point cloud reconstructed from the
depth maps, ${\text{Sh}}({\cal P})$ is the shape generated by the shape
program, and CD denotes the Chamfer distance of the point cloud to the
generated shape. It is weighted by $\lambda_{\text{CD}}=10$ based on our
empirical evaluations. Note that this objective function is not part of our
contribution and our framework could handle any other function that is
differentiable.

We consider object pose as additional continuous parameters in ${\cal P}$ to
enable the rendering of shape programs into scene views. In practice, we
assume an initialization of object translation is given and we are looking for
rotation around the y-axis only as we explain in Section 5.

We use our PyTorchGeoNodes framework to compute the $D^{r}_{i}({\cal P})$,
$N^{r}_{i}({\cal P})$, and ${\text{Sh}}({\cal P})$. Thanks to this framework,
PyTorch can thus compute their derivatives and the derivatives of ${\cal L}$
with respect to the continuous parameters in ${\cal P}$.

However, some parameters in ${\cal P}$ are discrete, _i.e_., integers or
booleans. Therefore, we cannot optimize ${\cal L}$ using a simple gradient
descent. Also, ${\cal L}$ has no special form that we can exploit to optimize
it. We therefore resort to an approach related to the one in [2], which
combines stochastic search, more specifically Monte Carlo Tree Search (MCTS),
and gradient descent which is enabled by our PyTorchGeoNodes framework.

###  4.1 MCTS for Shape Parameter Search

Monte Carlo Tree Search (MCTS) is a stochastic search algorithm that relies on
guided tree search and Monte Carlo simulations to maximize a given scoring
function. A major advantage of MCTS is that it can explore the most promising
parts of the tree simply based on the scoring function. It can also avoid
local minima and can shift the search focus on less explored sub-trees. It was
used in [33] to solve several scene understanding problems, and more details
can be found in [33] or in [6] which is a more general survey. We show here
how to adapt it to our problem. Further details about different hyper-
parameters for our method are given in the supplementary material.

Shape parameter tree. We organize the shape parameters in ${\cal P}$ into a
tree structure, with nodes at each level of the tree corresponding to a range
of values for a specific shape parameter. In the case of continuous values, we
first discretize them by linearly sampling values within a valid range.

In principle, we could only use gradient-based optimization to find the
optimal values for the continuous parameters. However, even for these
parameters, the objective function ${\cal L}$ is difficult to optimize when no
good initial guess is available and gradient descent can lead to poor local
minima. Instead, our search algorithm combines continuous and discrete
optimization to determine values of continuous parameters. Therefore, every
path in the tree instantiates parameters that can be fed to our shape program
to generate a 3D shape. We define the order of shape parameters tree based on
how much individual parameters influence the shape of generated objects which
is further discussed in supplementary material. Therefore, this guarantees
that objects in deeper parts of the tree become geometrically more consistent
with each other.

Modified selection criterion. At every iteration, MCTS typically selects nodes
${\cal N}$ that maximize the Upper Confidence Bound (UCB):

${\text{UCB}}({\cal N})={\text{score}}({\cal
N})+\lambda\sqrt{\frac{\ln{\\#{\text{visits}}}({\text{parent}}({\cal
N}))}{\\#{\text{visits}}({\cal N})}}\>,$ (2)

where ${\text{score}}({\cal N})$ is a score for node ${\cal N}$ which is
accumulated over evaluations of a negative of our objective function ${\cal
L}({\cal P})$ for the past iterations when node ${\cal N}$ was selected.
$\\#{\text{visits}}({\cal N})$ is the number of visits to node ${\cal N}$ in
previous iterations. We set $\lambda=0.2$ in all our experiments to weight the
exploration term.

In our case, each node ${\cal N}$ is associated to a pair of a parameter $p$
in ${\cal P}$ and a discretized value $v$ for this parameter. Within the tree,
many nodes will be associated to a given pair $(p,v)$. In the original
formulation of MCTS, these nodes remain independent. For our problem however,
it is interesting to make the nodes that correspond to the same pair $(p,v)$
share the same score. This is because assigning value $v$ to $p$ will have a
similar influence on the objective function ${\cal L}({\cal P})$ regardless of
the values assigned to the other parameters in ${\cal P}$.

We thus store the scores not for each node but for each pair $(p,v)$ and
modify the computation of UCB into:

${\text{UCB}}({\cal N})={\text{score}}({\text{pv}}({\cal
N}))+\lambda\sqrt{\frac{\ln{\\#{\text{iterations}}}}{\\#{\text{visits}}({\text{pv}}({\cal
N}))}}\>,$ (3)

where ${\text{pv}}({\cal N})$ is the pair parameter-value associated to node
${\cal N}$, $\\#{\text{visits}}({\text{pv}}({\cal N}))$ is the number of times
a node associated to this pair has been visited by MCTS so far, and
$\\#{\text{iterations}}$ is the current number of MCTS iterations. This
modification speeds up the convergence of MCTS.

Optimization step. Previous works already combined MCTS with refinement for
applications in tasks in 3D scene understanding [32, 33, 2]. Similarly, at the
end of MCTS iterations, we perform a gradient-based optimization to further
refine values of continuous shape parameters. This is enabled by our
PyTorchGeoNodes framework.

##  5 Validation

We validate our PyTorchGeoNodes framework and our optimization method for
reconstructing shape parameters on four common furniture categories: Cabinet,
Sofa, Chair, and Table, for which we created shape programs in Blender. Here
we give an overview, and provide more details in the supplementary material:

  * •

’Sofa’ has 18 parameters for creating and transforming legs, base, arms, arm
legs, and for the presence of an ’L-shaped’ extension and its dimensions;

  * •

’Chair’ has 20 parameters that control legs, leg support, backrest, and arms;

  * •

’Cabinet’ has 12 parameters for controlling legs, base, back, dividing boards,
and drawers;

  * •

’Table’ has 10 parameters that control the shape and scale of the top surface,
leg types, and middle board.

Our method jointly estimates the 6D pose and shape of the object. We do not
need initialization as we do not directly reconstruct global scale in contrast
to state-of-the-art approaches such as [2]: In our approach, the object scale
is determined by recovering the shape parameters instead, which we designed to
model scales of individual parts in meters. This makes our setting more
challenging. The orientation is defined as the rotation around the up-axis as
done in to [2], which is sufficient for indoor scenes.

Our experiments are divided into two parts: First, we evaluate our method for
shape reconstruction on synthetic images of objects. This allows us to
evaluate the capabilities of our method in an optimal setup, for example when
the object is never occluded. Second, we validate our method on object shape
reconstruction for real world scenes from the ScanNet dataset [10]. ScanNet is
a very challenging dataset of indoor environment RGB-D scans: Furniture is
often only partially visible or even highly occluded, and depth maps are often
incomplete due to missing sensor values—this happens for dark and reflective
materials, for example. We use SCANnotateDataset [2], which provides ground
truth 3D shapes for the objects in ScanNet, in the form of CAD models from
ShapeNet [8].

###  5.1 Evaluation Criteria

Evaluation of shape parameters. Each object category has individual semantic
properties which are an important aspect of the reconstruction. This is
neglected by current state-of-the-art methods. With our approach, we integrate
these properties as shape parameters into the reconstruction process. We
evaluate these properties by comparing the reconstructed and ground truth
parameters for the shapes of the target objects. For the experiments in the
synthetic domain, we extract the ground truth during the dataset generation
process. For the experiments with ScanNet, we manually annotated all relevant
objects with semantic properties. We decided to only annotate discrete
parameters, as it is extremely challenging and error-prone for humans to
manually annotate continuous 3D parameters from images. Instead, we adapt
shape similarity metrics.

Evaluation of shape similarity. Similarly to previous work [12, 38], we use
the Chamfer distance to quantify the similarity between sampled points of the
ground truth and the predicted object. We uniformly sample 10k points on the
surface of the predicted and ground truth meshes and use them to compute the
Chamfer distance. This metric is important for our experiments on real data
when ground truth for continuous shape parameters is difficult to obtain.

###  5.2 Implementation Details

Data pre-processing. We assume that for the target object in the RGB-D scan,
we have an initial estimate for the 3D center of the target object. Then, we
estimate 2D segmentation masks of the object for the objective function in
Equation (1).

For the synthetic experiment in Section 5.3, we extract the 3D object center
and 2D masks during scene generation. In our experiments on ScanNet, we
extract these from the annotations from SCANnotateDataset [2]. In general,
state-of-the-art 3D object recognition methods could be used instead [37, 28].

Search details. Based on empirical observations from our experiments on the
synthetic data, we set the number of MCTS iterations to $300$, with up to $30$
simulations per iteration. We use Adam [17] to perform gradient descent. To
deal with large computation overhead, we perform optimization only when the
leaf nodes of the tree are reached in the selection phase of MCTS, and skip
optimization during simulation. We provide details in the supplementary
material.

###  5.3 Validation on Synthetic Images

For this experiment, we generated $100$ scenes of rendered images of objects
generated by the shape programs per category, by randomly sampling shape
parameters. Then, we randomly sample $10000$ target points, and render $10$
target views with interpolated rotations around the up-axis. We fix the
translation to the origin. In this setting, we therefore evaluate our approach
on high-quality images and with precise ground truth annotations.

Table 1 shows quantitative results for cabinets. We observed similar behavior
for other categories in the supplementary. We consider three variants: Our
full method, MCTS without refinement (’No Refinement’), and MCTS without
refinement and the first term of Equation (3). This term encourages
exploitation and is typically required for difficult optimization problems.

The full method recovers both types of parameters very accurately, with a
somewhat lower number for the ’Has Legs’ parameter. We observe that fine
details such as this are very difficult to reconstruct using our current
objective function. When disabling refinement, we notice the accuracy of the
estimated continuous parameters significantly degrades. The accuracy of the
discrete parameters decreases slightly as well, which confirms that our joint
discrete and continuous optimization is critical to recovering these
parameters. When disabling the exploitation term of MCTS, the accuracy for the
discrete parameters additionally drops, which shows that random simulations
are not enough and the guided search in MCTS is important for recovering these
parameters.

| Parameter | Full Method | No Refinement |  | No Refinement
---
No Exploitation Term
Continuous Parameters |  | Mean Absolute Difference to Ground Truth (↓)
Width [cm] |  5.65 | 15.72 | 14.31
Height [cm] |  0.99 | 4.49 | 5.31
Depth [cm] |  2.42 | 14.16 | 13.70
Board Thickness [cm] |  2.01 | 2.14 | 2.14
Leg Width [cm] | 1.78 |  1.77 | 1.77
Leg Height [cm] |  1.61 | 1.69 | 1.73
Leg Depth [cm] | 1.70 | 1.71 | 1.66
Div. Board Thickness [cm] |  0.73 | 0.88 | 0.88
Rotation [∘] |  1.09 | 26.19 | 28.51
Discrete P. |  | Classification Accuracy (↑)
Has Back [%] |  98.07 | 90.38 | 88.00
Has Legs [%] |  82.00 | 75.00 | 62.50
Has Drawers [%] |  90.00 | 76.00 | 79.16
Number of Div. Boards [%] |  90.38 |  90.38 | 76.00

Table 1: Accuracy of the retrieved shape parameters for ’Cabinet’ on synthetic
scenes. For continuous parameters, we report the mean absolute difference to
the ground truth parameters. For discrete parameters, we report the
classification accuracy. We provide the results for the three variants
described in Section 5.3. MCTS does well to reconstruct the discrete
parameters, and gradient descent allows to precisely retrieve continuous
parameters. Small details such as legs do not contribute much to the objective
function, therefore in general we obtain lower accuracy in such situations.

###  5.4 Validation on the ScanNet Dataset

|
---|---
|
|
|
our model GT model | our model GT model

Figure 5: Qualitative comparisons between our reconstructions and the ground
truth from ScannotateDataset [2]. First row: Our approach correctly
reconstructs a one-armed one-seater, while the CAD model from
ScannotateDataset has two armrests. Second row: The width and length of the L
extension, and the back height, are too large in the ground truth. Third row:
The ground truth is more complex than the actual object. Fourth row: Our
approach accurately models the back and legs of the chair, while the
ScannotateDataset annotations provide a chair of unrealistic scale. Our method
correctly retrieves the height offset of leg support in this example.

Next, we evaluate our approach on challenging real-world scenarios on a subset
of validation scenes from ScanNet [10], which provides RGB-D scans of indoor
environments. We select a subset of object instances from ScannotateDataset
[2] for which we manually annotated 48 cabinets, 67 sofas, 61 chairs, and 100
tables. We consider only object instances that can be approximated by our
shape programs and ignore instances that are not yet included in our
framework, _e.g_. tables with built-in cabinets.

Table 2 evaluates quality for object shape reconstruction for sofa and chair
categories, we report the numbers for cabinet and table in the supplementary.
In addition to classification accuracy, we measure the quality of our
reconstructions by comparing them to the ground truth CAD models from
SCANnotateDataset. Similarly to synthetic experiments, we maintain solid
classification accuracy, except for shape parameters representing small
details. At the same time our method is comparable in quality to the ground
truth CAD models and we confirm the benefits of our refinement procedure. We
emphasize that our method generates good lightweight reconstructions with a
small number of vertices and faces: the average for our reconstructed meshes
is 64 and 100 respectively, compared to ground truth meshes with 2760 and
10918. Figure 5 shows some qualitative results.

|  |  | Classification accuracy [%] (↑) |  |  |  | Chamfer
---|---|---|---|---|---|---|---
| Legs | | Left Arm | | Right Arm | | Arm Legs | | Back | | L-shaped | | Flipped L | |  |  | Dist. [cm] (↓)
Sofa Random SP | 47.69 | 49.23 | 58.46 | 55.38 | 55.38 | 44.61 | 30.0 |  |  | 18.42
No Refinement | 43.07 | 76.92 | 66.15 | 64.61 | 100.0 | 95.38 | 100.0 |  |  | 7.2
Full (Ours) | 53.84 | 81.53 | 72.30 | 66.15 | 100.0 | 98.46 | 100.0 |  |  | 7.4
| Legs Type | | Has Leg Support | | Back | | Has Arms | |  |  |  |  |  |
Chair Random SP | 45.09 | 45.94 | 45.09 | 50.98 |  |  |  |  |  | 16.22
No Refinement | 68.62 | 89.18 | 98.03 | 66.66 |  |  |  |  |  | 12.48
Full (Ours) | 68.00 | 91.89 | 98.00 | 68.00 |  |  |  |  |  | 11.11

Table 2:  Results of shape reconstruction for ’Sofa’ and ’Chair’ categories on
the ScanNet scenes. We report classification accuracy for the discrete shape
parameters and the Chamfer distance to the CAD annotations SCANnotateDataset
[2]. Row-by-row: randomly selected shape parameters as a basic baseline, our
method without PyTorchGeoNodes-enabled refinement, and our full approach. For
’Sofa’, the refinement has a significant influence on classification accuracy,
except for the legs which are very small and difficult. The ground truth CAD
models do not always perfectly represent the scene, which is reflected by
similar Chamfer distance values. For ’Chair’, the classification accuracy is
not much affected when removing the refinement, which implies MCTS is an
important element. By contrast, the Chamfer distance significantly increases,
showing the importance of refinement for retrieving the continuous shape
parameters accurately.

##  6 Discussion and Future Work

|
---|---
|
retrieved model GT model | retrieved model GT model

Figure 6: Visualizations of current limitations. First row: Even though our
shape program considers legs, such details are usually very small, making the
reconstruction very challenging especially given the noisy observations from
the ScanNet dataset. The annotations from Scannotate have similar issues.
Second row: Our Table shape program does not yet consider cabinets built into
tables.

We introduced our PyTorchGeoNodes framework that generates efficient PyTorch
code for differentiable rendering of interpretable shape programs. In
addition, we provided a method for estimating the shape program’s parameters
for an object from multiple views.

As Figure 6 shows, our method does come with some limitations. While on
synthetic data, our method does well, noisy observations in real scenarios
make it difficult to retrieve small details such as the furniture legs.
Another limitation is in the expressiveness of hand-designed shape programs.
While better expressiveness can be achieved by improving the designs of shape
programs, this requires some expertise.

We will make our code publicly available and hope this will encourage other
researchers to develop the field. While shape programs are an attractive
representation of objects, there are indeed still many things to develop: They
could be extended to consider deformable and articulated objects and to
integrate textures. Moreover, as mentioned in the introduction, differentiable
rendering of shape programs, as allowed by PyTorchGeoNodes, makes it possible
to consider development of deep learning methods for creating shape programs
from images in a self-supervised way. In this context, as one of many possible
future directions, we propose to investigate whether deep networks can be
trained to predict shape parameters from images or 3D point clouds while using
our PyTorchGeoNodes to enable gradient propagation from various loss
functions.

## References

  * [1] Ainetter, S., Stekovic, S., Fraundorfer, F., Lepetit, V.: Automatically Annotating Indoor Images with CAD Models via RGB-D Scans. In: IEEE Winter Conference on Applications of Computer Vision (2023)
  * [2] Ainetter, S., Stekovic, S., Fraundorfer, F., Lepetit, V.: HOC-Search: Efficient CAD Model and Pose Retrieval from RGB-D Scans. International Conference on 3D Vision (2024)
  * [3] Aubry, M., Maturana, D., Efros, A.A., Russell, B.C., Sivic, J.: Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models. In: Conference on Computer Vision and Pattern Recognition (2014)
  * [4] Aubry, M., Russell, B.C.: Understanding Deep Features with Computer-Generated Imagery. In: International Conference on Computer Vision (2015)
  * [5] Botsch, M., Kobbelt, L.: An Intuitive Framework for Real-Time Freeform Modeling. ACM Trans. on Graphics (TOG) (2004)
  * [6] Browne, C., Powley, E., Whitehouse, D., Lucas, S., Cowling, P., Rohlfshagen, P., Tavener, S., Liebana, D.P., Samothrakis, S., Colton, S.: A Survey of Monte Carlo Tree Search Methods. IEEE Transactions on Computational Intelligence and AI in Games (2012)
  * [7] Cao, A.Q., de Charette, R.: SceneRF: Self-Supervised Monocular 3D Scene Reconstruction with Radiance Fields. In: International Conference on Computer Vision (2023)
  * [8] Chang, A.X., Funkhouser, T., Guibas, L., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., Yu, F.: ShapeNet: An Information-Rich 3D Model Repository. arXiv Preprint (2015)
  * [9] Community, B.O.: Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam (2018), http://www.blender.org
  * [10] Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes. In: Conference on Computer Vision and Pattern Recognition (2017)
  * [11] Ebert, D.S., Musgrave, F.K., Peachey, D., Perlin, K., Worley, S.: Texturing & Modeling: a Procedural Approach. Morgan Kaufmann (2003)
  * [12] Gkioxari, G., Malik, J., Johnson, J.: Mesh R-CNN. In: International Conference on Computer Vision (2019)
  * [13] Izadinia, H., Shan, Q., Seitz, S.M.: IM2CAD. In: Conference on Computer Vision and Pattern Recognition (2017)
  * [14] Jacobson, A., Tosun, E., Sorkine, O., Zorin, D.: Mixed Finite Elements for Variational Surface Modeling. In: Computer Graphics Forum (2010)
  * [15] Jones, R.K., Barton, T., Xu, X., Wang, K., Jiang, E., Guerrero, P., Mitra, N.J., Ritchie, D.: ShapeAssembly: Learning to Generate Programs for 3D Shape Structure Synthesis. ACM Transactions on Graphics (TOG) (2020)
  * [16] Jones, R.K., Charatan, D., Guerrero, P., Mitra, N.J., Ritchie, D.: ShapeMOD: Macro Operation Discovery for 3D Shape Programs. ACM Transactions on Graphics (TOG) (2021)
  * [17] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. In: International Conference for Learning Representations (2015)
  * [18] Koehl, M., Roussel, F.: Procedural Modelling for Reconstruction of Historic Monuments. Annals of the Photogrammetry Remote Sensing and Spatial Information Sciences (2015)
  * [19] Kuo, W., Angelova, A., Lin, T.Y., Dai, A.: Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve. In: European Conference on Computer Vision (2020)
  * [20] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou, Z., Liu, L., Ma, Y., Zhang, S.H., Habermann, M., Theobalt, C., et al.: Wonder3D: Single Image to 3D using Cross-Domain Diffusion. arXiv Preprint (2023)
  * [21] Massa, F., Russell, B.C., Aubry, M.: Deep Exemplar 2D-3D Detection by Adapting from Real to Rendered Views. In: Conference on Computer Vision and Pattern Recognition (2016)
  * [22] Mottaghi, R., Xiang, Y., Savarese, S.: A Coarse-to-Fine Model for 3D Pose Estimation and Sub-Category Recognition. In: Conference on Computer Vision and Pattern Recognition (2015)
  * [23] Müller, P., Wonka, P., Haegler, S., Ulmer, A., Van Gool, L.: Procedural Modeling of Buildings. In: ACM SIGGRAPH 2006 Papers (2006)
  * [24] Park, J.J., Florence, P., Straub, J., Newcombe, R.A., Lovegrove, S.J.: DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In: Conference on Computer Vision and Pattern Recognition (2019)
  * [25] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems (2019)
  * [26] Pearl, O., Lang, I., Hu, Y., Yeh, R.A., Hanocka, R.: GeoCode: Interpretable Shape Programs. arXiv Preprint (2022)
  * [27] Ravi, N., Reizenstein, J., Novotny, D., Gordon, T., Lo, W.Y., Johnson, J., Gkioxari, G.: Accelerating 3d deep learning with pytorch3d. arXiv Preprint (2020)
  * [28] Schult, J., Engelmann, F., Hermans, A., Litany, O., Tang, S., Leibe, B.: Mask3D: Mask Transformer for 3D Semantic Instance Segmentation. In: International Conference on Robotics and Automation (2023)
  * [29] Shin, D., Fowlkes, C.C., Hoiem, D.: Pixels, Voxels, and Views: A Study of Shape Representations for Single View 3D Object Shape Prediction. In: Conference on Computer Vision and Pattern Recognition (2018)
  * [30] Simon, L., Teboul, O., Koutsourakis, P., Paragios, N.: Random Exploration of the Procedural Space for Single-View 3D Modeling of Buildings. International Journal of Computer Vision (2011)
  * [31] Sorkine, O., Alexa, M.: As-Rigid-as-Possible Surface Modeling. In: Symposium on Geometry Processing (2007)
  * [32] Stekovic, S., Rad, M., Fraundorfer, F., Lepetit, V.: MonteFloor: Extending MCTS for Reconstructing Accurate Large-Scale Floor Plans. In: International Conference on Computer Vision (2021)
  * [33] Stekovic, S., Rad, M., Moradi, A., Fraundorfer, F., Lepetit, V.: MCTS with Refinement for Proposals Selection Games in Scene Understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022)
  * [34] Szymanowicz, S., Rupprecht, C., Vedaldi, A.: Splatter Image: Ultra-Fast Single-View 3D Reconstruction. arXiv Preprint (2023)
  * [35] Tian, Y., Luo, A., Sun, X., Ellis, K., Freeman, W.T., Tenenbaum, J.B., Wu, J.: Learning to Infer and Execute 3D Shape Programs. International Conference for Learning Representations (2019)
  * [36] Vaienti, B., Petitpierre, R., di Lenardo, I., Kaplan, F.: Machine-Learning-Enhanced Procedural Modeling for 4D Historical Cities Reconstruction. Remote Sensing 15(13) (2023)
  * [37] Vu, T., Kim, K., Luu, T.M., Nguyen, T., Yoo, C.D.: SoftGroup for 3D Instance Segmentation on Point Clouds. In: Conference on Computer Vision and Pattern Recognition (2022)
  * [38] Wang, N., Zhang, Y., Li, Z., Fu, Y., Liu, W., Jiang, Y.G.: Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images. In: European Conference on Computer Vision (2018)
  * [39] Wonka, P., Wimmer, M., Sillion, F., Ribarsky, W.: Instant Architecture. ACM Transactions on Graphics (TOG) (2003)