# 3D Neural Sculpting (3DNS): Editing Neural Signed Distance Functions

Petros Tzathas  School of Electrical & Computer Engineering, National
Technical University of Athens, Greece  Petros Maragos  School of Electrical &
Computer Engineering, National Technical University of Athens, Greece
Anastasios Roussos  Institute of Computer Science (ICS), Foundation for
Research & Technology - Hellas (FORTH), Greece  College of Engineering,
Mathematics and Physical Sciences, University of Exeter, UK

###### Abstract

In recent years, implicit surface representations through neural networks that
encode the signed distance have gained popularity and have achieved state-of-
the-art results in various tasks (_e.g_. shape representation, shape
reconstruction, and learning shape priors). However, in contrast to
conventional shape representations such as polygon meshes, the implicit
representations cannot be easily edited and existing works that attempt to
address this problem are extremely limited. In this work, we propose the first
method for efficient interactive editing of signed distance functions
expressed through neural networks, allowing free-form editing. Inspired by 3D
sculpting software for meshes, we use a brush-based framework that is
intuitive and can in the future be used by sculptors and digital artists. In
order to localize the desired surface deformations, we regulate the network by
using a copy of it to sample the previously expressed surface. We introduce a
novel framework for simulating sculpting-style surface edits, in conjunction
with interactive surface sampling and efficient adaptation of network weights.
We qualitatively and quantitatively evaluate our method in various different
3D objects and under many different edits. The reported results clearly show
that our method yields high accuracy, in terms of achieving the desired edits,
while at the same time preserving the geometry outside the interaction areas.

|  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |
Figure 1: Examples of shape editing capabilities offered by our 3DNS, acting
directly on the Neural SDF representation. First column: original shapes.
Following columns: results of multiple edits using various brush settings,
visualizing intermediate stages of the process. All edits and renderings are
done on the implicit neural representation, avoiding completely the use of
triangular meshes. Images were taken from our interactive editor, which uses
Sphere Tracing [1] to render the zero-level set of the Neural SDFs. Please
zoom in for details and refer to the paper’s webpage
(https://pettza.github.io/3DNS/) for a demo video.

##  1 Introduction

Representing and manipulating surfaces and 3D shapes is a problem of paramount
importance in many diverse applications, ranging from mechanical and
architectural design to computer animation, augmented/virtual reality, and
physical simulations. It thus comes as no surprise that the representations
devised over the years are as many and diverse as the applications, each with
their respective advantages and disadvantages. Bézier patches, B-splines and
subdivision surfaces are only some of the choices, with the most ubiquitous
being the polygon meshes [2].

Although polygon meshes offer a useful and efficient representation it is hard
to model diverse topologies, as that would require the vertices or their
connectivity to change. To surpass these limitations researchers have tried
incorporating different geometrical representations, such as voxel grids,
octrees, and implicit functions. Due to the grid (or grid-like) structure of
the former two, they have been used with convolutional networks [3, 4, 5].
Nevertheless, voxel grids cannot achieve high resolution and, even though
octrees address this, they, too, result in jagged models.

In the last years, given the ever-rising popularity of artificial neural
networks, a new class of surface representations has been proposed, namely the
Implicit Neural Representations (INRs). In this approach, the surface, which
is frequently required to be closed, is represented implicitly as a level set
of a neural network with one output. Several papers have presented very
interesting and promising results using such representations [6, 7]. In most
of the papers, the network tries to learn either the signed distance function
or the occupancy function. Also, the network can learn only one surface or a
class of surfaces by taking a class code along with the spatial coordinates as
input. In contrast to 3D meshes, voxel grids, and other common
representations, INRs are not coupled to spatial resolution and can be sampled
at arbitrary spatial resolutions, since they are continuous functions. In this
way, the memory required to accurately represent a 3D shape does not depend on
the spatial resolution but on its geometric complexity.

Despite the particularly promising results of implicit representations, there
are still limitations to their usage. One of the most important limitations is
that the shapes cannot be easily edited. This is due to the fact that in these
representations the geometric structures are not represented in a local
fashion. Each weight of the corresponding network affects the geometry over an
unbounded region of the output space. This means that, in order to perform a
localized modification on the 3D shape, generally all weights of the network
need to be modified.

This editability problem of INRs is an open challenge that has attracted very
limited attention in the literature. Existing works allow for some form of
interactive editing, by either optimizing the shape code fed to the network
[8, 9] or by training the networks for articulated objects [10, 11] and
changing the joints’ parameters (which are also fed to the network). In either
case, the editing is limited inside a learned shape space and, so, these
methods do not support arbitrary modifications of the shape’s 3D geometry.

To overcome the aforementioned limitations, this work introduces the first
method that allows interactive editing of INRs, specifically neural Signed
Distance Functions (SDFs). We approach the problem from a 3D sculpting
perspective, aiming at equipping INRs with functionalities that 3D modeling
software have for the standard mesh representations. Our method, which we call
3D Neural Sculpting (3DNS), edits the surface modeled by the zero-level set of
a neural network in a brush-based manner. As mentioned above, using a
feedforward neural network to represent an SDF creates a problem of locality.
For this, we propose using samples of the surface represented by the network
to regulate the learning of the desired deformation. The source code is
available at https://github.com/pettza/3DNS.

To recap, INRs are in their infancy. They have shown impressive results, but
have not yet found many applications. We believe that the editability
capabilities that this paper introduces will pave the way to a plethora of
applications for INRs, ranging from computer graphics, where machine learning
becomes more and more popular, to robotics, where some preliminary works have
given evidence about the great benefits of these representations [12, 13].

##  2 Related Work

###  2.1 Implicit Surface Representations

The idea to represent surfaces implicitly is by no means a new one. In fact,
there have been continual attempts to use implicit representations in computer
graphics and machine learning. In the shader art community, analytic implicit
representations have been used to render from simple primitives to complex
scenes and fractal objects [14]. On the other hand, in the machine learning
community earlier approaches have relied upon radial basis functions (RBFs)
[15] and octrees [16] to express SDFs. The authors of [17] use points sampled
on an implicit surface to control its shape.

Recently, the use of a neural network as the function expressing the surface
was proposed in by three concurrent works [18, 19, 6], which ignited interest
in these implicit neural representations. DeepSDF [6] uses a network to
represent the SDF for a shape (or a shape class, using additionally a shape
code as input to the network), while the other two [18, 19] express the
occupancy function. Since then many more works on INRs have ensued.

The works referenced above use a regression-type loss function for training.
For SDFs, this requires the computation of ground truth distances at points in
space which can be difficult. Various attempts have been made to reformulate
the loss function for training a neural SDF. SAL [20] neural SDFs are trained
using a loss that, nevertheless, disregards the sign of the distance, which
requires careful initialization. SAL++ [21] extends this method to utilize
information about the normal vectors of the surface. The authors of [22]
incorporate samples of the level sets of the network function to the loss.
Further progress was made by IGR [23], which uses the fact that SDFs satisfy
the eikonal equation to train the network as a differential equation solving
network [24], thus requiring only uniform samples inside a bounding box of the
surface and samples that lie on the surface, without computation of ground
truth distances. Our loss is derived from this.

Other works have experimented with the architecture of the networks. SIREN
[25], which uses sines instead of the usual RELUs, presented promising results
in a variety of tasks including surface reconstruction. Convolutional networks
are used in [26, 27] by discretizing the input point cloud. State-of-the-art
results have been attained by coupling neural networks with data structures
that retain localized spatial information. Octrees, which store learnable
weights are used in [28, 29] and a method called hash encoding is used by
Instant-NGP [30]. Besides high detailed representations, NGLoD [29] and
Instant-NGP [30] achieve interactive framerates, as well.

Besides training using raw geometric data like point clouds and meshes, there
have been efforts to train neural SDFs directly from images [31, 32, 33, 34].
The authors of [35] propose an SDF variant that takes direction into account
(Signed Directional Distance Function). In contrast to a neural SDF which
expresses the distance approximately, they prove that their network structure
ensures it expresses the SDDF of some object.

It is worth mentioning that, while we focus on neural representations of
shapes, implicit representations have found success in expressing quantities
other than distance functions. For example, NeRF [36] produces highly
realistic images by expressing the radiance and density of a scene. A recent
survey [7] explores these representations, to which the authors refer as
Neural Fields, in depth.

###  2.2 Neural SDF Editability

The research on editability is quite limited. DualSDF [9] proposes a two-level
representation, one which comprises a collection of primitives (_e.g_.
spheres) and a neural SDF which share the shape space. The user is able to
manipulate the fine representation of the neural SDF by specifying changes to
primitives’ parameters which affect the shape code. A similar process is
possible with DIF [8], where instead of primitives a sparse set of points is
used. The authors of [10] and those of [11] both deal with articulated
objects. The joints’ parameters are given as input to the network and, thus,
can be used to manipulate the shape.

Since the above works deform the expressed shape by proxy of the network’s
inputs the space of possible shapes is limited to the one learned during
training. In contrast, our method allows the user to more freely change the
local 3D geometry in ways that do not necessarily lie within a learned shape
space, similar to the functionalities that until now were offered by 3D
software for meshes only.

##  3 Background

We begin by presenting some material upon which we rely to develop our method.
In Section 3.1 we give a definition of SDF, in Section 3.2, we describe SIREN
[25] which is the architecture that we build upon, in Section 3.3 we present
weight normalization [37] and finally in Section 3.4, we describe the
formulation of the adopted loss function.

###  3.1 Signed Distance Functions (SDFs)

Let $S$ be a surface in $\mathbb{R}^{3}$, then the (unsigned) distance of a
point $\bm{x}$ to the surface is:

$\mathit{UDF}(\bm{x},S)=\min_{\bm{y}\in S}\\{d(\bm{x},\bm{y})\\}$ (1)

where $d$ is a metric on $\mathbb{R}^{3}$, typically (in our case as well) the
Euclidean distance.

If $S$ is closed the signed distance function is defined as follows:

$\mathit{SDF}(\bm{x},S)=\begin{cases}\mathit{UDF}(\bm{x},S)&\text{if
}\bm{x}\text{ inside }S\\\
-\mathit{UDF}(\bm{x},S)&\text{otherwise}\end{cases}$ (2)

A Neural SDF is a neural network that takes the spatial coordinate $\bm{x}$ as
input and approximates the SDF of a surface. As a consequence, Neural SDFs use
a whole trained neural network to parametrize a single shape. They effectively
use a continuous function to represent a shape and in this way they are not
dependent on a specific spatial resolution.

###  3.2 SIREN Representation

We build upon SIREN [25], which is a multilayer perceptron (MLP) that uses
sines, instead of the usual ReLUs for its non-linearities. Evidently, sines
allow the network to more accurately represent finer details. The use of sines
is related to Fourier features [38], for which the aforementioned property has
been theoretically explained [39] using the NTK framework [40]. We chose SIREN
as our architecture due to its simplicity and effectiveness. We note, however,
that our method is model-independent and, therefore, could be applied to a
variety of networks. The only alteration we experiment with is adding weight
normalization to each layer which we discuss next.

Figure 2: Brush template profile. See equation 11.

###  3.3 Weight Normalization

We propose to extend SIREN by equipping it with weight Normalization, which is
a memory-efficient technique proposed by [37] for accelerating convergence. It
works by reparametrizing the weights of a linear layer so that they are
represented by a length parameter and a direction vector. This approximately
‘whitens’ the gradients during training and this makes the 3D shape
represented by the Neural SDF to update across iterations in a more effective
manner. As we show in Section 5.4 applying it to SIREN has a positive effect.

###  3.4 Loss Function for Neural SDF

We adopt the loss function introduced by SIREN [25] and briefly present it
here for the sake of completeness. In more detail, many of the works on neural
SDFs [28, 27, 30, 6, 29] use a regression type loss to train the neural
networks. The loss is evaluated for points sampled on the surface, near the
surface, and in a bounding box around it. The ground truth signed distance
needs to be computed for off-the-surface points which can be difficult,
especially for non-mesh data which interest us (see Section 4). The authors of
[23] propose a loss function that comprises a regression loss evaluated only
on points on the surface (for which the distance is 0), a term for the normal
vectors at the same points, and a term that enforces the network to have unit
gradients w.r.t. to the input (this is commonly referred to as the eikonal
term). The latter term is evaluated for points that are sampled uniformly
inside a bounding box. The authors of [25], besides some minor changes, expand
the above loss with a term that penalizes small values of the neural SDF at
off-surface points. In summary, the loss function $L(\theta)$ that we minimize
during training is defined as:

$\displaystyle L(\theta)=L_{S}(\theta)+L_{eik}(\theta)+L_{es}(\theta),\text{
where:}$ (3) $\displaystyle
L_{S}(\theta)=\mathbb{E}_{p_{S}}\\{{\lambda_{1}\,|f_{\theta}(\bm{x})|+\lambda_{2}\,g\left(\nabla_{x}f_{\theta}(\bm{x}),n_{\bm{x}}\right)}\\}$
(4) $\displaystyle
L_{eik}(\theta)=\lambda_{3}\,\mathbb{E}_{q}\big{\\{}\big{|}\|\nabla_{\bm{x}}f_{\theta}(\bm{x})\|-1\big{|}\big{\\}}$
(5) $\displaystyle
L_{es}(\theta)=\lambda_{4}\,\mathbb{E}_{q}\\{e^{-\alpha|f_{\theta}(\bm{x})|}\\}$
(6)

where $\lambda_{1},\lambda_{2}$, $\lambda_{3}$ and $\lambda_{4}$ are balancing
weights (set to $1.5\cdot 10^{3}$, $5$, $2.5$, $5$ respectively), $S$ is the
target surface, $p_{S}$ is a distribution on the surface, $q$ is uniform
distribution in a bounding box, $\theta$ is the parameter vector, $f_{\theta}$
is the network function, $g$ is the cosine distance, $n_{\bm{x}}$ is the
normal vector at $\bm{x}$ and $\alpha$ a large positive number (set to $100$).
$L_{S}$ encompasses the regression and normal terms, $L_{eik}$ is the eikonal
term, and $L_{es}$ is the term described last.

##  4 Proposed 3D Neural Sculpting

Our goal is to deform a surface (represented as a neural SDF) around a
selected point on it in a manner similar to what is possible for meshes with
3D sculpting software. We refer to the selected point as the interaction
point, to the area around it as the interaction area and to the process in
general as an interaction or edit. The main problem in our case is how to
enforce locality. Generally, a change to the parameters of a neural network is
expected to affect its output for an unbounded region of the input space.
Consequently, naively trying to train the network only where the surface needs
to change will ostensibly distort the rest of the surface as well, which is
what we confirm through experimentation in Section 5.6. In order to ameliorate
this adverse effect as much as possible, we produce the point cloud that is to
be used to evaluate the loss of equation 4 by including both samples from the
surface that the network already represents (we call these model samples) and
samples from the desired deformation. After discarding samples from the former
set that are close to the interaction point we use the union with the latter
set for training. In Section 4.1 we describe our surface sampling, then we
present our formulation of sculpting brushes in Section 4.2 and then, in
Section 4.3 we describe how we sample the interaction area.

(a) Bunny

(b) Frog

(c) Bust

(d) Pumpkin

(e) Sphere

(f) Torus

Figure 3: The dataset of 3D shapes that we use in our experiments. Please zoom
in for details.

###  4.1 Surface Sampling

Two prior works [22, 27] have proposed similar algorithms that sample the
zero-level set of a neural network function. They work by sampling points
uniformly inside a bounding box and then projecting them on the level set with
generalized Newton-Raphson iterations. For a true SDF, this would move a point
to the surface point closest to it.

A naive way to produce samples for each training iteration would be to use
this algorithm. However, this approach has two major drawbacks. Firstly, it is
time inefficient. Secondly, the distribution of the resulting samples can be
quite non-uniform. Inspired by [27] where the sample set is extended by
perturbing existing samples with gaussian noise we produce the samples for the
next iterations using the ones we already have. To that purpose, we add to
each point a vector sampled uniformly from the tangent disk and then reproject
them on the surface using the aforementioned procedure. We opt for the
tangential disks, instead of gaussians, so that we explore the surface as much
as possible without moving too far from it. The radius of the tangent disks is
a hyperparameter we set to 0.04.

Figure 4: The brush application is demonstrated above. $S$ is the surface,
$\bm{x}_{0}$ is the interaction point, $\bm{n}$ is the normal vector at
$\bm{x}_{0}$, $p$ is the tangent plane and $B_{r,s}$ is the brush function
whose graph over $p$ is the dark green curve.

This way of sampling forms a Markov Chain [41]. Naturally, the stationary pdf
distribution is of interest. It is relatively easy to see that the
requirements of Theorem 1 of [42] (regarding the existence of the stationary
distribution of a continuous space Markov Chain) are satisfied and, hence, the
stationary distribution exists and has support over the surface. It is hard to
theoretically reason about the shape of the distribution. However, we provide
experimental results in Section 5.5, which demonstrate that our sampling
process produces more uniformly distributed samples. Uniformness guarantees
that every surface region is included equally during training.

wc1pt wc14pt *5wc & 1-5Brush Radius
0.05 0.10 0.15 0.20 0.25
3-1Brush Intensity 1-10.03
1-10.05
1-10.07

Figure 5: Example of the effect of a supported bumping brush (causing an
outward local deformation) on the same interaction point on a sphere using
different values for the radius and intensity. A similar effect is supported
for a denting brush (causing an inward local deformation).

###  4.2 Brushes

We define a brush template as a $C^{1}$ (or higher) positive 2D function
defined over the unit disk centered at the origin which reaches a maximum
value of 1 and vanishes at the unit circle (ideally its gradient and its
higher derivatives vanish as well). Suppose $b_{T}(\bm{x})$ is a brush
template, then the properties above are summarized as follows:

$\displaystyle b_{T}:\\{\bm{x}\in\mathbb{R}^{2}\,|\,\|\bm{x}\|\leq
1\\}\rightarrow\mathbb{R}$ (7) $\displaystyle\max\\{b_{T}(\bm{x})\\}=1$ (8)
$\displaystyle\|\bm{x}\|=1\Rightarrow
b_{T}(\bm{x})=0\left(\wedge\>\nabla_{x}b_{T}(\bm{x})=\bm{0}\right)$ (9)

We can, then, define a brush family $B_{r,s}$ parametrized over radius $r$ and
intensity $s$:

$\displaystyle
B_{r,s}(\bm{x})=s\,b_{T}\left(\frac{\bm{x}}{r}\right),\,r\in\mathbb{R^{+}},s\in\mathbb{R}$
(10)

We show the behaviour of different radii and intensities in Section 5.3.
Notice that a positive value for the intensity creates a bump on the surface,
while a negative value creates a dent. In our experiments, we use the
following brush template whose profile is shown in Figure 2. For other types,
please refer to the supplementary material.

$\displaystyle b_{T}(\bm{x})=\begin{cases}P(1-\|\bm{x}\|)&\text{if
}\|\bm{x}\|<1\\\ 0&otherwise\end{cases}$ (11) $\displaystyle
P(x)=6x^{5}-15x^{4}+10x^{3}$ (12)

In order to apply the brush at a point on the surface, we consider it defined
on the tangent plane at that point. Due to the implicit function theorem [43]
we can express the zero-level set of the network function, in a region of that
point, as the graph of a 2D function over the same plane. We can, thus, apply
the brush by simply adding the brush function to the latter. As we will see,
even though this is not important for computing the samples’ positions, we use
it to compute the correct deformed normals.

Figure 6: On the top row the estimated pdf is visualized with face colors. On
the bottom row, the respective histograms of the estimated pdf values are
shown. The dashed vertical red line indicates the value of the uniform pdf.
For the two columns on the left, our proposed algorithm was used, while for
the ones on the right, the naive approach.

###  4.3 Interaction Sampling

We will now describe how we produce samples on the area of the surface that is
affected by the brush. We begin by placing uniform samples on a disk tangent
to the interaction point whose radius is the same as the brush’s radius. We
can, then, project these samples on the unaltered surface and then move them
perpendicular to the tangent plane (at the interaction point) a distance equal
to the brush function’s value. Theoretically, for what we discussed in the
previous section to be applicable this needs to be a parallel projection,
however, we use the same procedure that we described in Section 4.1 so that we
don’t affect the surface in a greater area than intended. The above process is
shown in Figure 4 Next we compute the normals at the sampled position.

Suppose that $f$ is a 2D function defined over a 3D plane (its gradient
$\nabla f$ then lies on the plane), and $\bm{n}$ is the normal vector to that
plane (analogous to the z axis), then the following is an unnormalized vector,
perpendicular to the graph of the function:

$-\nabla f+\bm{n}$ (13)

Shape | Chamfer Distance $\times 10^{3}$ ($\downarrow$)
---|---
Without WN | With WN
Bunny | 9.021 | 8.995
Frog | 8.095 | 7.921
Bust | 7.167 | 7.139
Pumpkin | 8.646 | 8.506
Sphere | 7.087 | 6.861
Torus | 7.198 | 6.882
Average | 7.869 | 7.717
Table 1: Chamfer distances computed with 100000 points for weight
normalization ablation.

Accordingly, what we need in order to compute the normals of the samples is
the gradient of the 2D function whose graph is the deformed surface. As we
explained in the previous section, this function is the sum of the brush
function and the function defined implicitly by the network. We can directly
calculate the gradient of the brush function. By the implicit function
theorem, the gradient of the implicit function is given by:

$-\frac{\left(\nabla_{\bm{x}}f_{\theta}(\bm{x})\right)_{\parallel}}{\left(\nabla_{\bm{x}}f_{\theta}(\bm{x})\right)_{\perp}}$
(14)

where $\parallel$ denotes the component parallel to the tangent plane and
$\perp$ the component perpendicular to it.

We, firstly, sample the model and discard the samples that lie inside a sphere
centered at the interaction point with a radius equal to the brush’s radius.
Then, we sample the interaction. The number of samples we take is the number
of discarded samples multiplied by an integer factor bigger than 1. We utilize
this factor in order to balance the contributions of the model and interaction
samples. We want the influence of the interaction samples to be larger since
it is where the surface must change, while adapting to the size of the
affected area. We use a factor of 10 for our experiments. Finally, the set of
samples for the training is the union of the non-discarded model samples and
the interaction samples.

Shape | Mean Chamfer Distance $\times 10^{3}$ ($\downarrow$)
---|---
Over whole surface | Inside interaction area
Ours | Naive | Simple Mesh | Ours | Naive | Simple Mesh
Bunny | 9.407 | 14.106 | 11.127 | 5.527 | 12.919 | 17.707
Frog | 8.172 | 12.865 | 8.756 | 4.750 | 9.805 | 17.051
Bust | 7.279 | 11.486 | 7.901 | 3.818 | 8.779 | 14.926
Pumpkin | 8.774 | 13.558 | 11.489 | 4.315 | 5.910 | 20.693
Sphere | 7.209 | 12.550 | 7.399 | 3.555 | 6.117 | 12.982
Torus | 7.142 | 13.516 | 7.415 | 3.574 | 5.980 | 13.402
Average | 7.997 | 13.014 | 9.015 | 4.257 | 8.252 | 16.127
Table 2: Comparison of our editing method with and without model samples (Ours
and Naive, respectively) and direct mesh editing on a mesh with equivalent
size (Simple Mesh). Chamfer distances are computed with 100000 points. The
mean for each shape is taken over 10 independent edits.

##  5 Experiments

In this section, we qualitatively and quantitatively evaluate our proposed
3DNS in various 3D objects and under several different surface edits. For
additional results and visualizations, please refer to the Supplementary
Material section below. Also, the source code and a demo video can be found at
the paper’s webpage (https://pettza.github.io/3DNS/).

###  5.1 Dataset

For our experiments we have created a small dataset of six 3D shapes, see Fig.
3. For four of them, we start from a mesh representation: The Stanford Bunny,
which comes from [44] and the frog, pumpkin, and bust, which come from
TurboSquid 111https://www.turbosquid.com. The other two shapes are a sphere
and a torus which are usually provided by 3D modeling software as starting
shapes. We start from an analytical representation of these shapes.
Specifically, we use a sphere with a radius of 0.6 and a torus with a major
radius of 0.45 and a minor radius of 0.25. For our implementation, we use
PyTorch [45]. As a pre-processing step, we normalize the coordinates of the
four meshes of our dataset by translating and scaling them uniformly so that
they lie inside $[-(1-b),1-b]^{3}$, where $b$ is a positive number. The latter
parameter is used so that there is space around the models where we can edit
them. We set $b$ to 0.15. We train networks to represent the shapes of the
dataset by sampling them uniformly. The architecture of the networks is SIREN
with 2 hidden layers and 128 neurons each, and weight normalization (except in
Section 5.4). We use 120000 samples for the loss of equation 4 and another
120000 for the losses of equations 5 and 6 and train for $10^{6}$ iterations.

###  5.2 Performance Metric

For quantitative comparisons, we use the Chamfer distance, which is a common
metric used for geometric tasks. If $A$, $B$ are two point clouds, their
Chamfer distance is defined as follows:

$\mathit{CD}(A,B)=\frac{1}{|A|}\sum_{a\in A}\min_{b\in
B}d(a,b)+\frac{1}{|B|}\sum_{b\in B}\min_{a\in A}d(a,b)$ (15)

###  5.3 Brush Parameters

We demonstrate how the brush parameters allow the user to control the edit in
Figure 5. Specifically, we use different radii and intensities on the same
interaction point on the sphere. For each row, the intensity is the same and
is shown on the left. Similarly, for each column, the radius is the same and
is shown on the top.

###  5.4 Weight Normalization Ablation Study

Figure 7: Example of an edit using different methods. From left to right,
ground truth (ideal edit result), ours, naive and simplified mesh.

In order to prove the improvements provided by weight normalization, we train
six models for the shapes we presented above in the first case with weight
normalization and in the second case without. The Chamfer distance between a
point cloud sampled from the ground truth models and one sampled from the
trained network by our sampling algorithm is given in Table 1. It can be seen
that, even though not by a large margin, the models trained with weight
normalization perform better in every case.

###  5.5 PDF Estimation

We want to study the uniformness of the stationary distribution of our
sampling algorithm. We do this by estimating the pdf over the surface.
Firstly, we create a triangle mesh of the surface using the Marching Cubes
algorithm [50]. We, then, generate samples with our algorithm for $N$
iterations, with $M$ samples per iteration. In effect, we are simulating $M$
Markov chains. For each triangle of the mesh, we count the number of samples
$c$ that are closest to it. The estimated mean value of the pdf over the
triangle is then:

$pdf=\frac{c}{N\cdot M\cdot A}$ (16)

where $A$ is the area of the triangle.

We visualize the estimated pdf for the bunny and the sphere with the face
colors in Figure 6. In the same figure, we show histograms of the pdf
estimations, as well. The value of the uniform pdf, which is equal to the
inverse of the surface area, is shown in the histograms with a dashed vertical
red line. We can see that the histograms are centered tightly around this
value, indicating that the stationary distributions of our sampling process
are quite uniform. For comparison, we give the corresponding results for the
naive sampling outlined in Section 4.1, in the same Figure. Here, we notice
the bright areas which are sampled more frequently than the rest of the
surface, as well as the longer tails of the histograms and the fact that they
are off-centered.

###  5.6 Mesh Editing Comparison

We compare our proposed method with the editing of meshes which is by far the
most popular representation for 3D modeling and sculpting and, also, the naive
approach of using only interaction samples. We edit a mesh by changing the
positions of the vertices that lie inside the interaction area (the sphere
that was used to discard samples). We follow the process described in Section
4.3, the only difference being that, instead of projecting samples from the
tangent plane onto the surface, we compute a vertex’s corresponding position
on that plane as the intersection of a ray starting from the vertex with
direction along its normal. In order to have a fair comparison, we use a mesh
with approximately the same size as the network. We begin with a high-
resolution mesh as the ground truth (this is the mesh the network was trained
on for the four meshes and one constructed via Marching Cubes [50] for the
sphere and torus) and, following [29], use quadratic decimation [47] to get
the smallest mesh with size larger or equal to the network’s size. Afterward,
we perform the same ten edits on these three representations. Each edit is
performed on the unedited models. We compute the mean Chamfer distance of the
network and the simplified mesh to the ground truth mesh, over the whole
surface, as well as only inside the interaction areas. We set the brush’s
radius to 0.08 and its intensity to 0.06 for all the edits. The results are
summarized in Table 2, where it is shown that our method outperforms the other
approaches. We, also, provide an example in Figure 7.

##  6 Limitations and Future Work

Despite the successes of neural SDFs and the results we present above, there
are also drawbacks. One problem with this way of editing is that the shape
cannot easily be edited outside the bounding box where the eikonal loss has
been applied because there the network does not approximate an SDF. Also,
since the neural network function is smooth it is difficult to model hard
edges and corners using a neural SDF. However, there is research on neural
SDFs that use auxiliary data structures to represent the surface locally [28,
30, 29] that can be potentially utilized to address these shortcomings, which
is a direction we aim to pursue. Furthermore, our framework could in the
future be extended in editing neural scene representations such as NeRFs [36].

##  7 Conclusion

We have presented 3DNS, a method for editing neural SDF representations in an
interactive fashion inspired by existing 3D software. We believe that research
towards the editability of these representations can help render them viable
for more applications, either scientific or artistic in nature. Nevertheless,
in order to compete with the existing tools and capabilities of meshes, more
research is required.

## References

  * [1] John C. Hart “Sphere tracing: a geometric method for the antialiased ray tracing of implicit surfaces” In _The Visual Computer_ 12.10, 1996, pp. 527–545 DOI: 10.1007/s003710050084
  * [2] Donald Hearn, M Pauline Baker and M Pauline Baker “Computer graphics with OpenGL” Pearson Prentice Hall Upper Saddle River, NJ:, 2004
  * [3] Christopher B. Choy et al. “3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction”, 2016 arXiv:1604.00449 [cs.CV]
  * [4] Christian Häne, Shubham Tulsiani and Jitendra Malik “Hierarchical Surface Prediction for 3D Object Reconstruction”, 2017 arXiv:1704.00710 [cs.CV]
  * [5] Maxim Tatarchenko, Alexey Dosovitskiy and Thomas Brox “Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs”, 2017 arXiv:1703.09438 [cs.CV]
  * [6] Jeong Joon Park et al. “DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation”, 2019 arXiv:1901.05103 [cs.CV]
  * [7] Yiheng Xie et al. “Neural Fields in Visual Computing and Beyond”, 2021 URL: https://neuralfields.cs.brown.edu/
  * [8] Yu Deng, Jiaolong Yang and Xin Tong “Deformed Implicit Field: Modeling 3D Shapes with Learned Dense Correspondence” In _IEEE Computer Vision and Pattern Recognition_ , 2021
  * [9] Zekun Hao, Hadar Averbuch-Elor, Noah Snavely and Serge Belongie “DualSDF: Semantic Shape Manipulation using a Two-Level Representation” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 7631–7641
  * [10] Boyang Deng et al. “Neural Articulated Shape Approximation” In _The European Conference on Computer Vision (ECCV)_ , 2020 Springer
  * [11] Jiteng Mu et al. “A-SDF: Learning Disentangled Signed Distance Functions for Articulated Shape Representation” arXiv, 2021 DOI: 10.48550/ARXIV.2104.07645
  * [12] Korrawe Karunratanakul et al. “Grasping Field: Learning Implicit Representations for Human Grasps”, 2020 arXiv:2008.04451 [cs.CV]
  * [13] Anthony Simeonov et al. “Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation” In _ICRA_ , 2022
  * [14] Inigo Quilez “Inigo Quilez’s site” https://iquilezles.org, 2022
  * [15] J.. Carr et al. “Reconstruction and Representation of 3D Objects with Radial Basis Functions” In _Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques_ , SIGGRAPH ’01 New York, NY, USA: Association for Computing Machinery, 2001, pp. 67–76 DOI: 10.1145/383259.383266
  * [16] Sarah Frisken, Ronald Perry, Alyn Rockwood and Thouis Jones “Adaptively Sampled Distance Fields: A General Representation of Shape for Computer Graphics” In _ACM SIGGRAPH_ , 2000 DOI: 10.1145/344779.344899
  * [17] Andrew P. Witkin and Paul S. Heckbert “Using particles to sample and control implicit surfaces. Computer graphics and interactive techniques”, 1994
  * [18] Zhiqin Chen and Hao Zhang “Learning Implicit Fields for Generative Shape Modeling”, 2019 arXiv:1812.02822 [cs.GR]
  * [19] Lars Mescheder et al. “Occupancy Networks: Learning 3D Reconstruction in Function Space”, 2019 arXiv:1812.03828 [cs.CV]
  * [20] Matan Atzmon and Yaron Lipman “SAL: Sign Agnostic Learning of Shapes from Raw Data”, 2020 arXiv:1911.10414 [cs.CV]
  * [21] Matan Atzmon and Yaron Lipman “SALD: Sign Agnostic Learning with Derivatives”, 2020 arXiv:2006.05400 [cs.CV]
  * [22] Matan Atzmon et al. “Controlling Neural Level Sets”, 2019 arXiv:1905.11911 [cs.LG]
  * [23] Amos Gropp et al. “Implicit Geometric Regularization for Learning Shapes”, 2020 arXiv:2002.10099 [cs.LG]
  * [24] Justin Sirignano and Konstantinos Spiliopoulos “DGM: A deep learning algorithm for solving partial differential equations” In _Journal of Computational Physics_ 375 Elsevier BV, 2018, pp. 1339–1364 DOI: 10.1016/j.jcp.2018.08.029
  * [25] Vincent Sitzmann et al. “Implicit Neural Representations with Periodic Activation Functions”, 2020 arXiv:2006.09661 [cs.CV]
  * [26] Julian Chibane, Thiemo Alldieck and Gerard Pons-Moll “Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion”, 2020 arXiv:2003.01456 [cs.CV]
  * [27] Julian Chibane, Aymen Mir and Gerard Pons-Moll “Neural Unsigned Distance Fields for Implicit Function Learning”, 2020 arXiv:2010.13938 [cs.CV]
  * [28] Rohan Chabra et al. “Deep Local Shapes: Learning Local SDF Priors for Detailed 3D Reconstruction”, 2020 arXiv:2003.10983 [cs.CV]
  * [29] Towaki Takikawa et al. “Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes” In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021
  * [30] Thomas Müller, Alex Evans, Christoph Schied and Alexander Keller “Instant Neural Graphics Primitives with a Multiresolution Hash Encoding” In _ACM Trans. Graph._ 41.4 New York, NY, USA: ACM, 2022, pp. 102:1–102:15 DOI: 10.1145/3528223.3530127
  * [31] Sai Praveen Bangaru et al. “Differentiable Rendering of Neural SDFs through Reparameterization” arXiv, 2022 DOI: 10.48550/ARXIV.2206.05344
  * [32] Michael Niemeyer, Lars Mescheder, Michael Oechsle and Andreas Geiger “Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision” arXiv, 2019 DOI: 10.48550/ARXIV.1912.07372
  * [33] Michael Oechsle, Songyou Peng and Andreas Geiger “UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction” In _International Conference on Computer Vision (ICCV)_ , 2021
  * [34] Lior Yariv et al. “Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance” In _Advances in Neural Information Processing Systems_ 33, 2020
  * [35] Ehsan Zobeidi and Nikolay Atanasov “A Deep Signed Directional Distance Function for Object Shape Representation”, 2021 arXiv:2107.11024 [cs.CV]
  * [36] Ben Mildenhall et al. “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis” In _ECCV_ , 2020
  * [37] Tim Salimans and Diederik P. Kingma “Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks”, 2016 arXiv:1602.07868 [cs.LG]
  * [38] Nuri Benbarka, Timon Hofer, Hamd Riaz and Andreas Zell “Seeing Implicit Neural Representations as Fourier Series” In _ArXiv Pre-print_ , 2021 URL: http://arxiv.org/abs/2109.00249v1
  * [39] Matthew Tancik et al. “Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains”, 2020 arXiv:2006.10739 [cs.CV]
  * [40] Arthur Jacot, Franck Gabriel and Clément Hongler “Neural Tangent Kernel: Convergence and Generalization in Neural Networks”, 2020 arXiv:1806.07572 [cs.LG]
  * [41] Sheldon Ross “Introduction to Probability Models” Elsevier, 2017
  * [42] Persi Diaconis and David Freedman “On Markov chains with continuous state space” In _Technical Report no. 501_ , 1997
  * [43] Jerrold E. Marsden and Anthony Tromba “Vector Calculus” W. H. Freeman, 2011
  * [44] Greg Turk and Marc Levoy “Zippered Polygon Meshes from Range Images”, SIGGRAPH ’94 New York, NY, USA: Association for Computing Machinery, 1994, pp. 311–318 DOI: 10.1145/192161.192241
  * [45] Adam Paszke et al. “PyTorch: An Imperative Style, High-Performance Deep Learning Library” In _CoRR_ abs/1912.01703, 2019 arXiv: http://arxiv.org/abs/1912.01703
  * [46] William E. Lorensen and Harvey E. Cline “Marching cubes: A high resolution 3D surface construction algorithm” In _COMPUTER GRAPHICS_ 21.4, 1987, pp. 163–169
  * [47] Michael Garland and Paul S. Heckbert “Surface simplification using quadric error metrics” In _Proceedings of the 24th annual conference on Computer graphics and interactive techniques_ , 1997

## References

  * [48] Inigo Quilez “Smoothstep functions” https://iquilezles.org/articles/smoothsteps/, 2022
  * [49] Angel X. Chang et al. “ShapeNet: An Information-Rich 3D Model Repository” https://shapenet.org, 2015
  * [50] William E. Lorensen and Harvey E. Cline “Marching cubes: A high resolution 3D surface construction algorithm” In _COMPUTER GRAPHICS_ 21.4, 1987, pp. 163–169

##  Appendix A Other Brush Types

In the main paper (Section 4.2) we used a single brush template, which was
based on a quintic polynomial. Here, we describe a more general approach.

###  A.1 Smoothstep Functions

In computer graphics, the need to transition smoothly from one real number to
another arises very frequently. For this purpose, various functions are used,
which take the value $0$ for $x<0$, the value $1$ for $x>1$, and go from $0$
to $1$ in the interval $\left[0,1\right]$ in an continuously differentiable
increasing manner, with vanishing derivatives at $0$ and $1$. In graphics,
such a function is usually called _smoothstep_. Refer to Inigo Quilez’s site
[48] for a presentation of some smoothstep functions.

###  A.2 Radially Symmetric Brushes

If $f$ is a smoothstep function, then we can define a brush template that is
radially symmetric, as follows:

$b_{T}(\bm{x})=f\left(1-\|\bm{x}\|\right)$ (17)

Any function from [48] can be used. The brush we use for our experiments is of
this type. Besides smoothsteps, any function $f$ defined over
$\left[0,1\right]$ that has value $0$ at $x=0$ and a maximum value of $1$ can
be used in the equation above.

###  A.3 Vector Brushes

A possible extension we can make to our brush formulation is to allow the
brush template function to take vector values instead of just scalars. Such an
extension would allow to create brushes that twists the surface locally around
the interaction point.

##  Appendix B Pre-training

During the initial training of the networks so that they represent the
unedited shapes we encountered a problem. Sometimes the training fails. Figure
8 shows the sampled zero-level set of a network that was (unsuccessfully)
trained to represent a sphere. A sphere can actually be discerned.
Nevertheless, there also exists an outer shell. This situation manifested for
different shapes as well. We hypothesize that this is due to the nature of the
loss function, the network’s size, and poor initialization of the network
weights.

Figure 8: Sampled zero-level set of a neural network that was unsuccessfully
trained to represent a sphere.

Figure 9: The results of PDF estimation for all the shapes in the dataset. For
the top two rows our proposed algorithm was used, while for the bottom two the
naive approach. For an explanation of the figures refer to the main paper
(Section 5.5). Please zoom in for details.

(a) Dining chair 4dd46b9657c0e998b4d5420f7c27d2df

(b) Chair 02e76cb4f1039c482eb499cc8fbcd

(c) Armchair c5d880efc887f6f4f9111ef49c078dbe

(d) Sofa bcff6c5cb4127aa15e0ae65e074d3ee1

(e) Vase 13375f8fce3142e6597d391ab6fcc1

Figure 10: ShapeNet shapes used for surface editing comparisons. Beneath each
caption the ShapeNet model ID of the corresponding shape is written in
italics.

The loss function (equations 3-6) does not specify the value the network
should take anywhere other the zero-level set. Depending on initialization,
the network can take negative values outside the surface. The fact that we
want the network function to be positive outside the surface and negative
inside is expressed in the loss function only through the term for the normals
and, hence, in a local fashion. Early in training, this normal term will lead
to the network taking positive values outside the surface but only close to
the boundary. Far from the surface the network function continue to be
negative. This creates the outer shell we see in the figure. Once the shell is
created, the eikonal term $L_{eik}$ (equation 5) and the term that penalizes
small values of the network function $L_{es}$ (equation 6) lead to a local
minimum.

Shape | Mean Chamfer Distance $\times 10^{3}$ ($\downarrow$)
---|---
Over whole surface | Inside interaction area
Ours | Naive | Simple Mesh | Ours | Naive | Simple Mesh
Dining chair | 7.256 | 20.545 | 8.260 | 8.843 | 11.288 | 10.703
Chair | 8.498 | 30.620 | 8.650 | 8.741 | 19.072 | 15.344
Armchair | 14.152 | 19.433 | 14.311 | 5.085 | 10.942 | 23.654
Sofa | 11.160 | 18.268 | 11.899 | 4.477 | 8.623 | 23.940
Vase | 9.063 | 15.565 | 10.345 | 10.477 | 15.713 | 16.725
Average | 10.026 | 20.8862 | 10.693 | 7.525 | 13.128 | 18.073
Table 3: Comparison of our editing method with and without model samples (Ours
and Naive, respectively) and direct mesh editing on a mesh with equivalent
size (Simple Mesh). Chamfer distances are computed with 100000 points. The
mean for each shape is taken over 10 independent edits.

The training that proceeds can be regarded through a variational point of
view. $L_{eik}$ acts as a constraint on the network function forcing it to be
an SDF, which means that the outer shell created will change as a surface in a
continuous manner during gradient descent. The minimization of $L_{es}$
requires that the area of the outer shell decreases and so the shell shrinks.
When the outer shell is close enough to the surface of the shape which the
network is trained to represent the shell cannot shrink further because of the
network’s limited capacity. Between the shell’s surface and the shape’s
surface there exists a discontinuity in the SDF’s gradient. This
discontinuity, of course, exists even when the shell is far from the shape,
however when it gets close the network’s gradient is required to change too
abruptly which is not possible for its size. Thus, the training is stuck at a
local minimum.

We notice that larger networks sizes and/or more complex shapes help to avoid
the issue. The solution we opted for is pre-training the network. That is,
before starting the actual training we execute 100 iterations using the
following loss function:

$L_{pre}=\mathbb{E}_{q}\big{\\{}\big{|}f_{\theta}(\bm{x})-\|\bm{x}\|\big{|}\big{\\}}$
(18)

where $\theta$ is the parameter vector, $f_{\theta}$ is the network function,
and $q$ is the uniform distribution in a bounding box. This leads to the
network function being positive which avoids the creation of the shell.

##  Appendix C Additional PDF Estimation Results

In the main paper, we provided PDF Estimation results only for the bunny and
the sphere in the main paper (Section 5.5). In Figure 9, we provide them for
all the shapes in the dataset. Here, as well, it is clear the our proposed
algorithm produces a more uniform distribution than the naive approach.

##  Appendix D Additional Surface Editing Comparisons

We also run the surface editing comparison experiment for five shapes from the
ShapeNet dataset [49] shown in Figure 10. Since the ShapeNet meshes do not
have adequately high resolution to capture the desired edits accurately and in
order to have a fair comparison between the approaches, we consider as ground
truth a high resolution mesh constructed by Marching Cubes [50]. This is
similar to the process that we adopted for the Sphere and Torus shapes in the
experimental comparisons of the main paper. We follow the same protocol of
experimental comparisons as in Section 5.6 of the main paper and the results
are reported in Table 3. We observe that, once again, our method outperforms
the compared approaches consistently for all shapes and with respect to both
metrics used (i.e. Mean Chamfer Distance over the whole surface and inside the
interaction area).


#  TabSketchFM: Sketch-based Tabular Representation Learning
for Data Discovery over Data Lakes

(2023)

Table Representation Learning, Data Discovery, Data Lakes, Foundation Models

††copyright: acmcopyright††journalyear: 2023††doi: XXXXXXX.XXXXXXX††ccs:
Information systems††ccs: Information systems Data management systems††ccs:
Information systems Information retrieval