# Removing Objects From Neural Radiance Fields

Silvan Weder1,2 Guillermo Garcia-Hernando1 Áron Monszpart1 Marc Pollefeys2
Gabriel Brostow1,3 Michael Firman1 Sara Vicente1
1Niantic 2ETH Zurich 3University College London
nianticlabs.github.io/nerf-object-removal

###### Abstract

Neural Radiance Fields (NeRFs) are emerging as a ubiquitous scene
representation that allows for novel view synthesis. Increasingly, NeRFs will
be shareable with other people. Before sharing a NeRF, though, it might be
desirable to remove personal information or unsightly objects. Such removal is
not easily achieved with the current NeRF editing frameworks. We propose a
framework to remove objects from a NeRF representation created from an RGB-D
sequence. Our NeRF inpainting method leverages recent work in 2D image
inpainting and is guided by a user-provided mask. Our algorithm is underpinned
by a confidence based view selection procedure. It chooses which of the
individual 2D inpainted images to use in the creation of the NeRF, so that the
resulting inpainted NeRF is 3D consistent. We show that our method for NeRF
editing is effective for synthesizing plausible inpaintings in a multi-view
coherent manner. We validate our approach using a new and still-challenging
dataset for the task of NeRF inpainting.

##  1 Introduction

Since the initial publication of Neural Radiance Fields (NeRFs) [41], there
has been an explosion of extensions to the original framework, e.g., [3, 4, 8,
12, 25, 34, 38, 41]. NeRFs are being used beyond the initial task of novel
view synthesis. It is already appealing to get them into the hands of non-
expert users for novel applications, e.g., for NeRF editing [78] or live
capture and training [45]. More casual use-cases are driving interesting new
technical issues.

Figure 1: Removal of unsightly objects. Our method allows for objects to be
plausibly removed from NeRF reconstructions, inpainting missing regions whilst
preserving multi-view coherence.

One of those issues is how to seamlessly remove parts of the rendered scene.
Removing parts of the scene can be desirable for a variety of reasons. For
example, a house scan being shared on a property selling website may need
unappealing or personally identifiable objects to be removed [66]. Similarly,
objects could be removed so they can be replaced in an augmented reality
application, e.g., removing a chair from a scan to see how a new chair fits in
the environment [49]. Removing objects might also be desirable when a NeRF is
part of a traditional computer vision pipeline, e.g., removing parked cars
from scans that are going to be used for relocalization [42].

Some editing of NeRFs has already been explored. For example, object-centric
representations disentangle labeled objects from the background, which allows
editing of the trained scene with user-guided transformations [75, 72], while
semantic decomposition allows selective editing and transparency for certain
semantic parts of the scene [26]. However, these previous approaches only
augment information from the input scan, limiting their generative
capabilities, i.e., the hallucination of elements that haven’t been observed
from any view.

With this work, we tackle the problem of removing objects from scenes, while
realistically filling the resulting holes, as shown in Fig. 1. Solving this
problem requires: a) exploiting multi-view information when parts of the scene
are observed in some frames but occluded in others and, b) leveraging a
generative process to fill areas that are never observed. To this end, we pair
the multi-view consistency of NeRFs with the generative power of 2D inpainting
models [67] that are trained on large scale 2D image datasets. Such 2D
inpaintings are not multi-view consistent by construction, and may contain
severe artefacts. This causes corrupted reconstructions, so we design a new
confidence-based view-selection scheme that iteratively removes inconsistent
inpaintings from the optimization. We validate our approach on a new dataset
and show that we outperform existing approaches on standard metrics of image
quality, as well as producing multi-view consistent results.

In summary, we make the following contributions: 1) We propose the first
approach focusing on inpainting NeRFs by leveraging the power of single image
inpainting. 2) A novel view-selection mechanism that automatically removes
inconsistent views from the optimization. 3) We present a new dataset for
evaluating object removal and inpainting in indoor and outdoor scenes.

##  2 Related work

#### Image inpainting

Image inpainting tackles the problem of plausibly filling in missing regions
in a single image. A typical approach is to use an image-to-image network with
an adversarial loss, e.g., [51, 77, 80, 22], or with a diffusion model [39].
Different ways have been proposed to encode the input image, e.g., using
masked [32] or Fourier convolutions [67]. Image inpainting was extended to
also inpaint depth images by [14]. However, these methods do not give temporal
consistency between video frames, nor the ability to synthesize novel views.

#### Removing moving objects from videos

While video inpainting is a well studied problem in computer vision [73, 79],
most works focus on removing moving objects. This is typically achieved with
guidance from nearby frames, e.g., via estimating flow [15, 73, 21], sometimes
using depth [31, 6]. Perhaps counter-intuitively, moving objects make the task
easier, since their movement disoccludes the background, making most parts of
the scene visible in at least one of the frames. In contrast, to inpaint
videos of a static scene a generative approach is required, since large parts
of the scene have never been observed.

#### Removing static objects from videos

Where occluded pixels are visible in other frames in the sequence, these can
be used to inpaint regions [28, 43, 44]. For example, [36, 37] remove static
foreground distractors from videos, e.g., fences and raindrops. However, there
are typically still pixels which cannot be seen in other views, for which some
other method is required to fill them in. For example, [18] propagates patches
from visible pixels into the region to inpaint, and [43, 68] inpaint missing
pixels via PatchMatch. Kim et al. [24] rely on a pre-computed mesh of each
scene for object removal. Our key difference to these methods is that our
inpaintings can be extrapolated to novel viewpoints.

#### Inpainting in novel view synthesis

Inpainting is often used as a _component_ of novel view synthesis, to estimate
textures for regions unobserved in the inputs [64, 70], e.g., for panorama
generation [27, 20]. Philip et al.[52] enable object removal from image-based-
rendering, but with an assumption that background regions are locally planar.

###  2.1 Novel view synthesis and NeRFs

NeRF [41] is a highly popular image-based rendering method which uses a
differentiable volume-rendering formulation to represent a scene; a multi-
layer perceptron (MLP) is trained to regress the color and opacity given a 3D
coordinate and ray viewing direction. This combined works on implicit 3D scene
representations [33, 50, 40, 56, 57, 58, 9], with light-field rendering [11]
and novel view synthesis [17, 35, 74]. Extensions include work that reduces
aliasing artifacts [3], can cope with unbounded scenes [4], reconstructs a
scene from only sparse views [8, 47, 38, 25] or makes NeRFs more efficient,
e.g., by using octrees [69, 76] or other sparse structures [59].

#### Depth-aware Neural Radiance fields

To overcome some of the limitations of NeRFs, particularly the requirement for
dense views and the limits in the quality of the reconstructed geometry,
depths can be used in training [12, 55]. These can be sparse depths from
structure-from-motion [61, 62], or depth from sensors [53, 65].

#### Object-centric and semantic NeRFs for editing

One direction of progress in NeRFs is the decomposition of the scene into its
constituent objects [48, 75, 72]. This is done based on motion for dynamic
scenes [48], or instance segmentation for static scenes [75]. Both lines of
work also model the background of the scene as a separate model. However,
similar to video inpainting, dynamic scenes allow a better modelling of the
background since more of it is visible. In contrast, visual artefacts can be
seen in the background representation of [75, 72], which model static scenes.
Methods that decompose the scene based on semantics [82, 26] can also be used
to remove objects. However, they do not try to complete the scene when a
semantic part is removed and, for example, [26] discusses how “the background
behind the deleted objects can be noisy or have a hole because it lacks
observation”.

#### Generative models for novel view synthesis

3D aware generative models can be used to synthesize views of an object or
scene from different viewpoints, in a 3D consistent manner [46, 63, 13, 54].
In contrast with NeRF models, which only have a test time component and
“overfit” to a specific scene, generative models can be used to hallucinate
views of novel objects by sampling in the latent variable space. There has
also been some interest in 3D generative models that work for full indoor
scenes [13, 54, 29]. However, their capacity to fit the source views (or
memorization) can be limited, as shown in the qualitative results of [13]. To
train the generative model, [13, 29, 54] require a large dataset of indoor
scenes with RGB and camera poses and in some cases depth [13, 29]. In
contrast, our use of a 2D pretrained inpainting network, that can be trained
on any image, is less dependent on the existence of training data and less
constrained to indoor scenarios.

Bad Inpaintings | Good Inpaintings
---|---
|  |  |
|  |  |
Figure 2: Per-frame inpainting can give plausible results for each frame, but
they are not consistent between viewpoints and sometimes contain severe
artefacts corrupting the optimization.
Figure 3: An overview of our method We take a sequence of posed RGB-D images
together with corresponding 2D masks as input. The 2D frames are inpainted
using [67] and then used to optimize a neural radiance field. During
optimization, our confidence-based view selection automatically removes
inconsistent views from the optimization preventing unwanted artefacts in the
final result. Finally, novel views can be rendered from the scene, where the
object has been removed. Figure 4:

#### Our architecture

Our NeRF formulation contains two color heads $F_{\Theta}^{c}$ and
$F_{\Theta}^{MV}$, where $F_{\Theta}^{MV}$ does not take the view direction as
input. This is important to encourage multi-view consistency (see text for
details). It also includes uncertainty variables $u_{n}$ that are used to
jointly model the scene and the uncertainty of the 2D inpaintings for our
automated view selection.

##  3 Method

We assume we are given an RGB-D sequence with camera poses and intrinsics.
Depths and poses could be acquired, for example, using a dense structure-from-
motion pipeline [62, 61]. For most of our experiments we capture posed RGB-D
sequences directly using Apple’s ARKit framework [2], but we also show that we
can relax this requirement through use of a multi-view stereo method to
estimate depth for an RGB sequence. Along the way, we also assume access to a
per-frame mask of the object to be removed. The goal is to learn a NeRF model
from this input, which can be used to synthesize consistent novel views, where
the per-frame masked region should be plausibly inpainted. An overview of our
method is shown in Fig. 3.

###  3.1 RGB and depth inpainting network

Our method relies on a 2D single image inpainting method to inpaint each RGB
image individually. Furthermore, we also require a depth inpainting network.
We use both networks as black-boxes and our approach is agnostic to the method
chosen. Future improvements in single image inpainting can be directly
translated to improvements to our method. Given an image $I_{n}$ and
corresponding mask $M_{n}$, the per-image inpainting algorithm produces a new
image $\tilde{I}_{n}$. Similarly, the depth inpainting algorithm produces an
inpainted depth map $\tilde{D}_{n}$. We show some results for the 2D
inpainting network in Figure 2.

###  3.2 Background on NeRFs

Following the original NeRF paper [41], we represent the scene as an MLP
$F_{\Theta}$ that predicts color $\textbf{c}=[r,g,b]$ and density $\sigma$,
for a 5 dimensional input containing $x,y,z$ position and two viewing
directions. The predicted color for pixel $\mathbf{r}$,
$\hat{I}_{n}(\mathbf{r})$, is obtained by volume rendering along its
associated ray, so

$\displaystyle{\hat{I}_{n}(\textbf{r})=\sum_{i=1}^{K}\underbrace{T_{i}(1-\exp{(-\sigma_{i}\delta_{i})})}_{w_{i}}\textbf{c}_{i},~{}T_{i}=\exp\left(-\sum_{j=1}^{i-1}\sigma_{j}\delta_{j}\right)}$,

(1)

where $K$ is the number of samples along the ray, $t_{i}$ is a sample
location, $\delta_{i}=t_{i+1}-t_{i}$ is the distance between adjacent samples,
and $w_{i}$ is the alpha compositing weight which, by construction, sum to
less than or equal to 1.

The NeRF loss operates on training images as

$\mathcal{L}_{\text{RGB}}=\sum_{n=1}^{N}\sum_{\mathbf{r}\in\Omega_{n}}\left\|I_{n}(\mathbf{r})-\hat{I}_{n}(\mathbf{r})\right\|^{2},$
(2)

where $I_{n}(\mathbf{r})$ is the input RGB value for pixel $\mathbf{r}$, and
$\hat{I}_{n}(\mathbf{r})$ is its predicted color. $\Omega_{n}$ indicates the
2D domain of image $n$. The parameters of the MLP, $\Theta$, are optimized to
minimize this loss. Similarly to [55], if input depth is available, then an
additional loss can be added,

$\leavevmode\resizebox{216.81pt}{}{$\displaystyle{\mathcal{L}_{\text{depth}}=\sum_{n=1}^{N}\sum_{\mathbf{r}\in\Omega_{n}}\left|D_{n}(\mathbf{r})-\hat{D}_{n}(\mathbf{r})\right|,~{}\text{with~{}}\hat{D}_{n}(\mathbf{r})=\sum_{i=1}^{K}w_{i}t_{i}}$},$
(3)

where $D_{n}(\mathbf{r})$ is the input depth for pixel $\mathbf{r}$, and
$\hat{D}_{n}(\mathbf{r})$ is the corresponding predicted depth.

Finally, a distortion regularizer loss was introduced in [4] to better
constrain the NeRF optimization and remove “floaters”. It encourages the non-
zero compositing weights $w_{i}$ to be concentrated in a small region along
the ray, so for each pixel $\mathbf{r}$,

$l_{\text{dist}}(\mathbf{r})=\sum_{i,j}{w_{i}w_{j}\left|\frac{t_{i}+t_{i+1}}{2}-\frac{t_{j}+t_{j+1}}{2}\right|}+\frac{1}{3}\sum_{i}{w_{i}^{2}\delta_{i}}.$
(4)

###  3.3 Confidence-based view selection

Despite most of the individual inpainted RGB images $\tilde{I}_{n}$ looking
realistic, they still suffer from two issues: 1) some of the inpaintings are
incorrect, and 2) despite individual plausibility, they are not multi-view
consistent, i.e., the same area observed in multiple views is not necessarily
completed in a consistent way (Figure 2). For this reason, we propose a
confidence-based view selection scheme, that automatically chooses which views
are used in the NeRF optimization. We associate to each image $n$ a non-
negative uncertainty measure $u_{n}$. The corresponding per-image confidence,
$e^{-u_{n}}$, is used to re-weight the NeRF losses. This confidence value can
be seen as a loss attenuation term, similar to the aleatoric uncertainty
prediction term in [23].

The RGB loss for our model is then set out as

$\displaystyle{\mathcal{L}_{\text{RGB}}^{\mathcal{P}}(\hat{I})=\sum_{n=1}^{N}\sum_{\mathbf{r}\in\Omega_{n}\setminus
M_{n}}\left\|{I}_{n}(\mathbf{r})-\hat{I}_{n}(\mathbf{r})\right\|^{2}+\sum_{n\in\mathcal{P}}e^{-u_{n}}\sum_{\mathbf{r}\in
M_{n}}\left\|\tilde{I}_{n}(\mathbf{r})-\hat{I}_{n}(\mathbf{r})\right\|^{2}}$

(5)

where the color for pixels $\mathbf{r}$ is supervised by the inpainted image
for pixels inside the mask, and by the input RGB image for pixels outside the
mask. Note that the second term of this loss is only computed over a
restricted set of images $\mathcal{P}$, where
$\mathcal{P}\subseteq\\{1,..,N\\}$. This is denoted by the superscript
$\mathcal{P}$ in $\mathcal{L}_{\text{RGB}}^{\mathcal{P}}$. In practice, that
means that only some inpainted regions are used in the NeRF optimization. We
discuss below how we choose the set $\mathcal{P}$.

We use a similar split into pixels inside and outside the mask for the depth
loss, so

$\leavevmode\resizebox{216.81pt}{}{$\displaystyle{\mathcal{L}_{\text{depth}}^{\mathcal{P}}=\sum_{n=1}^{N}\sum_{\mathbf{r}\in\Omega_{n}\setminus
M_{n}}\left|{D}_{n}(\mathbf{r})-\hat{D}_{n}(\mathbf{r})\right|+\sum_{n\in\mathcal{P}}e^{-u_{n}}\sum_{\mathbf{r}\in
M_{n}}\left|\tilde{D}_{n}(\mathbf{r})-\hat{D}_{n}(\mathbf{r})\right|}$}.$ (6)

Finally, we include two regularizers. One is on the uncertainty weights
$\mathcal{L}_{\text{reg}}^{\mathcal{P}}=\sum_{n\in\mathcal{P}}u_{n}$, to
prevent a trivial solution where $e^{-u_{n}}$ is $0$. The other is a
distortion regularizer, based on [3], around the loss detailed in Equation 4,
so

$\displaystyle\mathcal{L}_{\text{dist}}^{\mathcal{P}}$
$\displaystyle=\sum_{n=1}^{N}\sum_{\mathbf{r}\in\Omega_{n}\setminus
M_{n}}l_{\text{dist}}(\mathbf{r})+\sum_{n\in\mathcal{P}}\sum_{\mathbf{r}\in
M_{n}}l_{\text{dist}}(\mathbf{r}).$ (7)

#### View direction and multi-view consistency

When optimizing the NeRF, we made three observations: a) the multi-view
inconsistencies in the inpaintings are modelled by the network using the
viewing direction; b) we can enforce multi-view consistency by removing the
viewing direction from the input; c) the inconsistencies introduce cloud-like
artefacts in the density when not using the viewing direction as input. To
prevent a) and c) and correctly optimize the variables $u_{n}$ that capture
the uncertainty about the inpaintings $\tilde{I}_{n}$, we propose: 1) adding
an auxiliary network head, $F_{\Theta}^{MV}$, to the NeRF that does not take
the viewing direction as input and, 2) stopping the gradient from the color
inpainting and $F_{\Theta}^{MV}$ to the density, leaving the uncertainty
variable $u_{n}$ as the only view-dependent input. This design forces the
model to encode the inconsistencies between inpaintings into the uncertainty
prediction while keeping the model consistent across views. $F_{\Theta}^{MV}$
has a loss term based on Eq. 5:
$\mathcal{L}_{\text{RGB}}^{\mathcal{P}}(\hat{I}^{MV})$. See Figure 4 for an
illustration of our architecture.

Our final loss is then
$\mathcal{L}^{\mathcal{P}}=\lambda_{\text{RGB}}\mathcal{L}_{\text{RGB}}^{\mathcal{P}}(\hat{I})+\lambda_{\text{RGB}}\mathcal{L}_{\text{RGB}}^{\mathcal{P}}(\hat{I}^{MV})+\lambda_{\text{depth}}\mathcal{L}_{\text{depth}}^{\mathcal{P}}+\lambda_{\text{reg}}\mathcal{L}_{\text{reg}}^{\mathcal{P}}+\lambda_{\text{dist}}\mathcal{L}_{\text{dist}}^{\mathcal{P}}$,
which is optimized over the MLP parameters
$\Theta=\left\\{{\Theta}^{\sigma},{\Theta}^{c},{\Theta}^{MV}\right\\}$ and the
uncertainty weights $\mathbf{U}^{\mathcal{P}}=\\{u_{n},n\in\mathcal{P}\\}$.
The confidence of all images is initialized to $1$, i.e., $u_{n}:=0$.

#### Iterative refinement

We use the predicted per-image uncertainty, $u_{n}$, in an iterative scheme
that progressively removes non-confident images from the NeRF optimization,
i.e., we iteratively update the set $\mathcal{P}$ of images that contribute to
the loss in masked regions. After $K_{\text{grad}}$ steps of optimizing
$\mathcal{L}^{\mathcal{P}}$, we find the median estimated confidence value
$m$. We then remove from the training set all 2D _inpainted regions_ which
have associated confidence scores less than $m$. We then retrain the NeRF with
the updated training set, and repeat these steps $K_{\text{outer}}$ times.
Note that images excluded from $\mathcal{P}$ still participate in the
optimization, but only for rays in the unmasked regions as they contain
valuable information about the scene. This is summarized in Algorithm 1.

Data: Input images $I_{n}$, Inpainted images $\tilde{I}_{n}$, Depth maps
$D_{n}$, Inpainted depth maps $\tilde{D}_{n}$, Masks $M_{n}$

Result: Trained NeRF model $F_{\Theta}$ with object removed

/* Set of images used for training NeRF is initialized with all images. */

$\mathcal{P}\leftarrow\\{1,...,N\\}$ and $u_{n}\leftarrow
0,~{}~{}n\in\mathcal{P}$

for  _$i\leftarrow 0$ to $K_{\text{outer}}$_ do

$\Theta\leftarrow$ randomly initialized

for  _$j\leftarrow 0$ to $K_{\text{grad}}$_ do

/* Gradient iterations of NeRF training. */

$\Theta\leftarrow\Theta-\nabla_{\Theta}\mathcal{L}^{\mathcal{P}}$

$\mathbf{U}^{\mathcal{P}}\leftarrow\mathbf{U}^{\mathcal{P}}-\nabla_{\mathbf{U}^{\mathcal{P}}}\mathcal{L}^{\mathcal{P}}$

/* Calculate median of confidence values. */

$m=\text{median}(\\{e^{-u_{n}},n\in\mathcal{P}\\})$

/* Update $\mathcal{P}$ by removing images with small confidence. */

for  _$n\in\mathcal{P}$_ do

if  _$e^{-u_{n}} <m$_ then

$\mathcal{P}\leftarrow\mathcal{P}\setminus n$

Algorithm 1 Iterative refinement using confidence based view selection.

###  3.4 Implementation details

#### Masking the object to be removed

Similarly to other inpainting methods, our method requires per-frame masks as
input. Manually annotating each frame with a 2D mask, as done in other
inpainting methods [67, 73, 30], is time consuming. Instead, we propose to
manually annotate a 3D box that contains the object, which only has to be done
once per scene. We reconstruct a 3D point cloud of the scene using the camera
poses and the input depth maps. This point cloud can then be imported into a
3D visualisation and editing software, such as MeshLab [10], where we specify
a 3D bounding box enclosing the object to be removed. Alternatively, we could
have relied on 2D object segmentation methods, e.g., [16], or 3D object
bounding box detectors, e.g., one of the baselines in [1], to mask the object.

#### Mask refinement

In practice, we observe that masks obtained from the annotated 3D bounding
boxes can be quite coarse and include large parts of the background. Since
large masks have a negative effect on the inpainting quality, we propose a
mask refinement step to obtain masks which are tighter around the object. This
step is not required for tight input masks. Intuitively, this mask refinement
step removes parts of the 3D bounding box that are empty space. We start by
taking all points in the reconstructed 3D point cloud that are inside the 3D
bounding box. The refined mask is then obtained by rendering these points into
each image and performing a simple comparison with the depth map to check
occlusions in the current image. The resulting mask is cleaned up by dilating
any pixel leaks caused by sensor noise using binary dilation and erosion. The
effect of our mask refinement step can be seen in Figure 5.

Raw masks | Refined masks
---|---
|  |  |
Figure 5: Mask refinement Our mask refinement leads to smaller masks and
therefore higher quality inpainting.

| Synthetic objects - Masked | Real objects - Masked
---|---|---
|  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  Depth $L_{1}\downarrow$ |  Depth $L_{2}\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  Depth $L_{1}\downarrow$ |  Depth $L_{2}\downarrow$
No novel view synthesis |  |  |  |  |  |  |  |  |  |
LaMa [67]† | 27.999 | 0.898 | 0.060 | 0.070 | 0.075 | - | - | - | - | -
E2FGVI [30]†‡ | 24.568 | 0.874 | 0.102 | - | - | - | - | - | - | -
NeRF based ablations |  |  |  |  |  |  |  |  |  |
Masked NeRF | 26.126 | 0.882 | 0.093 | 0.084 | 0.105 | 21.644 | 0.815 | 0.142 | 0.096 | 0.054
Inpainted NeRF | 28.760 | 0.905 | 0.086 | 0.278 | 0.400 | 23.705 | 0.848 | 0.134 | 0.145 | 0.121
Inpainted NeRF + inpainted Depth | 27.568 | 0.898 | 0.094 | 0.231 | 0.318 | 23.652 | 0.844 | 0.136 | 0.313 | 0.387
3D scene completion |  |  |  |  |  |  |  |  |  |
PixelSynth [54]‡ | 25.481 | 0.887 | 0.116 | - | - | 25.438 | 0.851 | 0.152 | - | -
CompNVS [29] | 17.389 | 0.823 | 0.171 | 1.697 | 3.641 | - | - | - | - | -
Object compositional NeRF [75]∗ | - | - | - | - | - | 21.757 | 0.836 | 0.134 | 0.312 | 0.341
Our method |  |  |  |  |  |  |  |  |  |
Ours | 29.437 | 0.916 | 0.078 | 0.069 | 0.096 | 25.271 | 0.859 | 0.125 | 0.071 | 0.044
Ours – depth predicted using [60] | 26.540 | 0.895 | 0.087 | 0.112 | 0.118 | 25.010 | 0.856 | 0.128 | 0.142 | 0.140

Table 1: Comparison with baselines and state of the art methods. Our method is
either best or second-best compared to other novel-view synthesis baselines in
inpainting the missing regions of the scene, by propagating multi-view
information and leveraging 2D inpainting information. Notes: †These methods
can’t be evaluated on the proposed real dataset as they do not synthesize
novel views. ‡These methods do not produce depth maps. ∗[75] requires the
actual object therefore it can’t be evaluated on the proposed synthetic
dataset.

#### Inpainting network

We used [67] for inpainting both RGB and depth. The inpainting of RGB images
and depth maps is done independently and we use the reference network provided
by the authors of [67] for both. Our depth maps are preprocessed by clipping
to $5\text{\,}\mathrm{m}$ and linearly mapping depths in
$\left[$0\text{\,}\mathrm{m}$,$5\text{\,}\mathrm{m}$\right]$ to pixel values
of $\left[0,255\right]$. This matrix is duplicated to make a $H\times W\times
3$ tensor for input to [67]. We observed empirically that this approach
provided good results, but an inpainting method specific for depth maps could
improve over this baseline.

#### NeRF estimation

The implementation of our method is built upon [47] and [3]. The shared MLP
$F_{\Theta}^{\sigma}$ consists of eight 256-wide layers while the branches
$F_{\Theta}^{c}$ and $F_{\Theta}^{MV}$ consist of four 128-wide layers each.
The final output is activated using _softplus_ for density, ReLU for the
uncertainty, and sigmoid for the color channel. We weight the terms in the
loss function with
$\lambda_{\text{RGB}}=\lambda_{\text{depth}}=\lambda_{\text{dist}}=1$, and
$\lambda_{\text{reg}}=0.005$. We optimize using Adam with an initial learning
rate of $l_{r}=0.0005$, and use the scheduler from [47]. We do a filtering
step, where we remove low confidence images every $K_{\text{grad}}=50,000$
steps, resulting in $K_{\text{outer}}=4$ filtering steps. More details
concerning implementation are in the supplementary material.

##  4 Experiments

###  4.1 Datasets

While previous approaches have tackled static object removal from videos, no
standard dataset/metrics to evaluate these systems has been proposed, to our
knowledge. This work introduces an RGB-D dataset of real scenes, designed to
evaluate the quality of object removal. Our dataset will be made public and
has two variants, which are used differently when benchmarking:

#### Ours — Real objects

This dataset comprises 17 scenes, which are a mixture of indoors/outdoors with
landscape/portrait orientation, and focus on a small area with at least one
object, with one of them being the labeled object of interest. They vary in
difficulty in terms of background texture, size of the object, and complexity
of scene geometry. Please see supplementary material for a visualization of
the full dataset. For each scene, we collected two sequences, one with and the
other without the object that we want to remove. The sequences are collected
using ARKit [2] on an iPhone 12 Pro with Lidar, and contain RGB-D images and
poses. The RGB images are resized to $192\times 256$, the same resolution as
the depth maps. Both sequences for the same scene use the same camera
coordinate system. The sequences vary in length, from 194 to 693 frames.

As previously described, the masks are obtained by annotating a 3D bounding
box of the object of interest and are refined for all of the scenes. For each
scene, we use the sequence with the object and corresponding masks for
training the NeRF model, and the sequence without the object for testing. The
use of real objects makes it easier to evaluate how the systems deal with real
shadows and reflections, as well as novel view synthesis.

#### Ours — Synthetic objects

Most video and image inpainting methods, e.g., [67, 30], do not perform novel
view synthesis, meaning such methods cannot be fairly evaluated on our ‘Real
objects’ dataset. We therefore introduce a separate synthetically-augmented
variant of our dataset. This uses the same scenes as the real objects dataset,
but we only use the sequence that _doesn’t_ contain the object. We then
manually position a 3D object mesh from ShapeNet [7] in each scene. The object
is placed so that it has a plausible location and size, e.g., a laptop on a
table. The masks are obtained by projecting the mesh into the input images,
which is the only use we make of the 3D object mesh. Please see supplementary
material for a visualisation of the full synthetic dataset. For this synthetic
dataset, following e.g., [4], we use every 8th frame for testing and the rest
of the frames for training the NeRF model.

#### ARKitScenes

We further validate our approach qualitatively on ARKitScenes [5]. This is an
RGB-D dataset of 1,661 scenes, where depth was captured via iPhone Lidar.

###  4.2 Metrics

To evaluate the object removal and inpainting quality, we compare our system’s
output image against the ground truth image, for each test image in the
dataset. All metrics in the paper are only computed inside the masked region.
Metrics for the full image are provided in the supplementary material. We use
the three standard metrics for NeRF evaluation [41]: PSNR [19], SSIM [71] and
LPIPS [81]. To evaluate the geometric completion, we compute the $L_{1}$ and
$L_{2}$ error between the rendered and the ground-truth depth maps inside
masked regions. The metrics are averaged over all frames of a sequence, and
then averaged over all sequences.

xxxxxxxxxInput |  |  |
---|---|---|---
xxxxxxRendering |  |  |
Figure 6: Results on ARKitScenes [5]. We can successfully remove objects from
casually captured sequences in indoor scenes.

###  4.3 Comparison with baselines

In Table 1, we compare our approach with alternative methods for object
removal, with a focus on methods that use an underlying NeRF representation.

#### Image and video inpainting baselines

We compare with two state-of-the-art methods for image LaMa [67] and video
inpainting E2FGVI [30]. In both cases, we use their reference implementation
and provided trained network. Neither of these methods allows novel view
synthesis, so they are only evaluated on the synthetic objects dataset.

#### NeRF-based ablations

We compare different ways of training a baseline NeRF model. The Masked NeRF
baseline corresponds to training a NeRF using the full input RGB-D data, but
pixels and depths in the masked regions are ignored in the NeRF losses.
Inpainted Images is a NeRF trained with all inpainted images, but not using
the inpainted depth maps, while Inpainted Images + Inpainted Depth uses all
the inpainted images and inpainted depth maps. This baseline corresponds to
All views in Table 2.

#### 3D scene completion

We compare with several published works for 3D scene completion. Note that
none of these baselines specifically targets inpainting, so results should be
viewed with this in mind. For all of them, we use their publicly available
implementation. PixelSynth [54] and CompNVS [29] were both proposed for scene
_outpainting_. Given one or a few frames of a scene their goal is to complete
the scene to enable novel view synthesis. Both of these methods rely on a
generative model of indoor scenes and neither requires test-time optimization.
Both methods are adapted to use our masks as input. We show qualitative
results for CompNVS in the supplementary materials. Object compositional NeRF
[75] enables objects in NeRFs to be edited via user-editing of pose
transformations. We adapt their code for object _removal_ by setting a
transformation that moves the object outside the camera’s field of view.

#### Our method

We compare two different versions of our method. Ours is our proposed
approach, which uses the method described in Section 3. Ours – depth from [60]
is our method, where the depth maps are obtained using a state-of-the art
multi-view depth prediction method [60] to show that we do not necessarily
rely on sensor depth.

As shown in Table 1, our method is superior to other novel-view synthesis
baselines across most appearance and depth metrics. Moreover, as opposed to
the single image inpainting LaMa [67], our method is close to multi-view
consistent, significantly reducing inter-frame flickering. To scrutinize the
flickering, we refer to the supplementary video.

#### Qualitative comparison

In Figure 7, we show that the proposed method successfully removes the
selected object compared to the baselines. While Masked NeRF fails to complete
large holes and Inpainted NeRF suffers from bad inpaintings in the training
set, our method can leverage the 2D inpaintings, while avoiding integrating
artefacts by removing those input frames. In contrast to Object compositional
NeRF [75], leveraging the inpaintings also helps to mitigate the appearance of
artefacts below the object’s surface. Compared to [75] and [54], our method is
better able to generate plausible scene completions. We also show results from
the ARKitScenes dataset in Figure 6, and please also see our supplementary
video.

Input |  Masked NeRF$\dagger$ |  Inpainted NeRF$\dagger$ |  Obj. comp. [75] |  PixelSynth [54] | Ours | Ground-truth
---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
Input |  Masked NeRF$\dagger$ |  Inpainted NeRF$\dagger$ |  Obj. comp. [75] |  PixelSynth [54] | Ours | Ground-truth
Figure 7: Qualitative comparisons with baseline. Our method significantly
improves over 3D scene completion baselines for the task of removing objects
from a scene using 3D inpainting. Further, it mitigates artefacts and
stabilizes convergence compared to the inpainting baseline without automatic
view selection. $\dagger$: We also compare with two NeRF based baselines:
Masked NeRF and Inpainted NeRF.

Method | # of views | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | $L_{1}\downarrow$ | $L_{2}\downarrow$
---|---|---|---|---|---|---
All views | 82 - 382 | 27.568 | 0.898 | 0.094 | 0.231 | 0.318
1/10th | 8 - 38 | 27.098 | 0.900 | 0.079 | 0.202 | 0.291
1/50th | 1 - 7 | 26.718 | 0.893 | 0.087 | 0.229 | 0.309
Single view | 1 | 26.232 | 0.892 | 0.079 | 0.133 | 0.198
Ours | 10 - 185 | 29.437 | 0.916 | 0.078 | 0.069 | 0.096

Table 2: Ablation on view selection methods. We validate our view selection
formulation by comparing to alternative approaches. Ours consistently produces
better performing models.

###  4.4 View selection ablation

Here we validate that our view selection procedure from Section 3.3
contributes to improved results. We compare our method with different view
selection strategies in Table 2 on our synthetic object dataset. All views
uses all the inpainting views when training the NeRF model. The other
baselines use a subset of views to train the NeRF model, spaced at regular
intervals: every 10th frame for 1/10th; every 50th frame for 1/50th and a
single middle frame for Single view. The number of views used for each
sequence varies depending on the length of the sequence. We outperform these
baselines, suggesting that our proposed strategy for view selection is
effective in choosing a good set of views to include in the NeRF optimization.

###  4.5 Limitations

Our method is upper bounded by the performance of the 2D inpainting method.
When the masks are too large along the entire trajectory, the 2D inpainting
fails and no realistic views can be selected. Our renderings sometimes suffer
from blurring, caused by flickering of high-frequency textures in the 2D
inpaintings. Furthermore, cast shadows or reflections of the object are not
handled well. We leave tackling these challenges for future work.

Input | Rendering | Input | Rendering
---|---|---|---
|  |  |
Figure 8: Failure cases and limitations. Our method can not recover when the
2D inpainting method fails all the frames, for example when the mask and
covers a large part of the image. Further, our method keeps the shadows of the
removed object, if they are not included in the object mask.

##  5 Conclusion

We have presented a framework to train neural radiance fields, where objects
are plausibly removed from the output renderings. Our method draws upon
existing work in 2D inpainting, and introduces an automatic confidence-based
view selection scheme to select single-view inpaintings with multi-view
consistency. We experimentally validated that our proposed method improves
novel-view synthesis from 3D inpainted scenes compared to existing work. We
have also introduced a dataset for evaluating this work, which sets a
benchmark for other researchers in the field.  Acknowledgments. We thank Jamie
Watson for fruitful discussions about the project, Paul-Edouard Sarlin for the
constructive feedback on an early draft, and Zuoyue Li for kindly running
CompNVS on our dataset.

## References

  * [1] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, and Matthias Grundmann.  Objectron: A large scale dataset of object-centric videos in the wild with pose annotations.  In CVPR, 2021.
  * [2] Apple.  ARKit.  Accessed: 14 October 2022.
  * [3] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan.  Mip-NeRF: A multiscale representation for anti-aliasing neural radiance fields.  In ICCV, 2021.
  * [4] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman.  Mip-NeRF 360: Unbounded anti-aliased neural radiance fields.  In CVPR, 2022.
  * [5] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman.  ARKitscenes - a diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data.  In NeurIPS, 2021.
  * [6] Borna Bešić and Abhinav Valada.  Dynamic object removal and spatio-temporal RGB-D inpainting via geometry-aware adversarial learning.  IEEE Transactions on Intelligent Vehicles, 7(2), 2022.
  * [7] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu.  ShapeNet: An Information-Rich 3D Model Repository.  Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  * [8] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su.  MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo.  In ICCV, 2021.
  * [9] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.  Implicit functions in feature space for 3D shape reconstruction and completion.  In CVPR, 2020.
  * [10] Paolo Cignoni, Massimiliano Corsini, and Guido Ranzuglia.  MeshLab: an open-source 3D mesh processing system.  ERCIM News, (73), 2008.
  * [11] Abe Davis, Marc Levoy, and Fredo Durand.  Unstructured light fields.  In Computer Graphics Forum, volume 31, pages 305–314. Wiley Online Library, 2012.
  * [12] Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan.  Depth-supervised NeRF: Fewer views and faster training for free.  In CVPR, June 2022.
  * [13] Terrance DeVries, Miguel Angel Bautista, Nitish Srivastava, Graham W Taylor, and Joshua M Susskind.  Unconstrained scene generation with locally conditioned radiance fields.  In ICCV, 2021.
  * [14] Ryo Fujii, Ryo Hachiuma, and Hideo Saito.  RGB-D image inpainting using generative adversarial network with a late fusion approach.  In International Conference on Augmented Reality, Virtual Reality and Computer Graphics, pages 440–451. Springer, 2020.
  * [15] Chen Gao, Ayush Saraf, Jia-Bin Huang, and Johannes Kopf.  Flow-edge guided video completion.  In ECCV, 2020.
  * [16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask R-CNN.  In ICCV, 2017.
  * [17] Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow.  Deep blending for free-viewpoint image-based rendering.  ACM Transactions on Graphics (TOG), 37(6):1–15, 2018.
  * [18] Jan Herling and Wolfgang Broll.  Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments.  In ISMAR, 2010.
  * [19] Alain Hore and Djemel Ziou.  Image quality metrics: PSNR vs. SSIM.  In ICPR. IEEE, 2010.
  * [20] Ching-Yu Hsu, Cheng Sun, and Hwann-Tzong Chen.  Moving in a 360 world: Synthesizing panoramic parallaxes from a single panorama.  arXiv preprint arXiv:2106.10859, 2021.
  * [21] Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf.  Temporally coherent completion of dynamic video.  ACM Transactions on Graphics (TOG), 35(6), 2016.
  * [22] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 2017.
  * [23] Alex Kendall and Yarin Gal.  What uncertainties do we need in bayesian deep learning for computer vision?  NeurIPS, 30, 2017.
  * [24] Joohyung Kim, Janghun Hyeon, and Nakju Doh.  Generative multiview inpainting for object removal in large indoor spaces.  International Journal of Advanced Robotic Systems, 2021.
  * [25] Mijeong Kim, Seonguk Seo, and Bohyung Han.  Infonerf: Ray entropy minimization for few-shot neural volume rendering.  In CVPR, 2022.
  * [26] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.  Decomposing NeRF for editing via feature field distillation.  In NeurIPS, 2022.
  * [27] Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson.  Pathdreamer: A world model for indoor navigation.  In ICCV, 2021.
  * [28] Vincent Lepetit, Marie-Odile Berger, and LORIA-INRIA Lorraine.  An intuitive tool for outlining objects in video sequences: Applications to augmented and diminished reality.  In ISMAR, pages 159–160, 2001.
  * [29] Zuoyue Li, Tianxing Fang, Zhenqiang Li, Zhaopeng Cui, Yoichi Sato, Marc Pollefeys, and Martin R. Oswald.  CompNVS: Novel view synthesis with scene completion.  In ECCV, 2022.
  * [30] Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng.  Towards an end-to-end framework for flow-guided video inpainting.  In CVPR, 2022.
  * [31] Miao Liao, Feixiang Lu, Dingfu Zhou, Sibo Zhang, Wei Li, and Ruigang Yang.  DVI: Depth guided video inpainting for autonomous driving.  In ECCV, 2020.
  * [32] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In ECCV, 2018.
  * [33] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li.  Learning to infer implicit surfaces without 3D supervision.  arXiv:1911.00767, 2019.
  * [34] Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell.  Editing conditional radiance fields.  In ICCV, 2021.
  * [35] Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao.  Liquid warping GAN: A unified framework for human motion imitation, appearance transfer and novel view synthesis.  In ICCV, 2019.
  * [36] Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang.  Learning to see through obstructions.  In CVPR, 2020.
  * [37] Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang, Yung-Yu Chuang, and Jia-Bin Huang.  Learning to see through obstructions with layered decomposition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 2021.
  * [38] Xiaoxiao Long, Cheng Lin, Peng Wang, Taku Komura, and Wenping Wang.  SparseNeuS: Fast generalizable neural surface reconstruction from sparse views.  In ECCV, 2022.
  * [39] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In CVPR, 2022.
  * [40] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger.  Occupancy networks: Learning 3D reconstruction in function space.  In CVPR, 2019.
  * [41] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  NeRF: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [42] Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciulescu, and Arnaud de La Fortelle.  LENS: Localization enhanced by NeRF synthesis.  In Conference on Robot Learning, 2022.
  * [43] Shohei Mori, Okan Erat, Wolfgang Broll, Hideo Saito, Dieter Schmalstieg, and Denis Kalkofen.  InpaintFusion: incremental RGB-D inpainting for 3D scenes.  IEEE TVCG, 2020.
  * [44] Shohei Mori, Dieter Schmalstieg, and Denis Kalkofen.  Good keyframes to inpaint.  IEEE Transactions on Visualization and Computer Graphics, 2022.
  * [45] Thomas Müller, Alex Evans, Christoph Schied, Marco Foco, András Bódis-Szomorú, Isaac Deutsch, Michael Shelley, and Alexander Keller.  Instant neural radiance fields.  In ACM SIGGRAPH 2022 Real-Time Live! 2022\.
  * [46] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang.  Hologan: Unsupervised learning of 3D representations from natural images.  In CVPR, 2019.
  * [47] Michael Niemeyer, Jonathan T. Barron, Ben Mildenhall, Mehdi S. M. Sajjadi, Andreas Geiger, and Noha Radwan.  RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs.  In CVPR, 2022.
  * [48] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide.  Neural scene graphs for dynamic scenes.  In CVPR, 2021.
  * [49] Selcen Ozturkcan.  Service innovation: Using augmented reality in the IKEA Place app.  Journal of Information Technology Teaching Cases, 11(1), 2021.
  * [50] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove.  DeepSDF: Learning continuous signed distance functions for shape representation.  In CVPR, 2019.
  * [51] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros.  Context encoders: Feature learning by inpainting.  In CVPR, 2016.
  * [52] Julien Philip and George Drettakis.  Plane-based multi-view inpainting for image-based rendering in large scenes.  In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pages 1–11, 2018.
  * [53] Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari.  Urban radiance fields.  In CVPR, pages 12932–12942, 2022.
  * [54] Chris Rockwell, David F Fouhey, and Justin Johnson.  PixelSynth: Generating a 3D-consistent experience from a single image.  In ICCV, 2021.
  * [55] Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul P Srinivasan, and Matthias Nießner.  Dense depth priors for neural radiance fields from sparse input views.  In CVPR, 2022.
  * [56] Martin Runz, Kejie Li, Meng Tang, Lingni Ma, Chen Kong, Tanner Schmidt, Ian Reid, Lourdes Agapito, Julian Straub, Steven Lovegrove, et al.  FroDO: From detections to 3D objects.  In CVPR, 2020.
  * [57] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In ICCV, 2019.
  * [58] Shunsuke Saito, Tomas Simon, Jason Saragih, and Hanbyul Joo.  PIFuHD: Multi-level pixel-aligned implicit function for high-resolution 3D human digitization.  In CVPR, 2020.
  * [59] Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In CVPR, 2021.
  * [60] Mohamed Sayed, John Gibson, Jamie Watson, Victor Prisacariu, Michael Firman, and Clément Godard.  SimpleRecon: 3D reconstruction without 3D convolutions.  In ECCV, 2022.
  * [61] Johannes L Schonberger and Jan-Michael Frahm.  Structure-from-motion revisited.  In CVPR, 2016.
  * [62] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm.  Pixelwise view selection for unstructured multi-view stereo.  In ECCV, 2016.
  * [63] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger.  Graf: Generative radiance fields for 3D-aware image synthesis.  NeurIPS, 2020.
  * [64] Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang.  3D photography using context-aware layered depth inpainting.  In CVPR, 2020.
  * [65] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison.  iMAP: Implicit mapping and positioning in real-time.  In ICCV, 2021.
  * [66] Mohamad Zaidi Sulaiman, Mohd Nasiruddin Abdul Aziz, Mohd Haidar Abu Bakar, Nur Akma Halili, and Muhammad Asri Azuddin.  Matterport: virtual tour as a new marketing approach in real estate business during pandemic COVID-19.  In International Conference of Innovation in Media and Visual Design (IMDES), 2020.
  * [67] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions.  In WACV, 2022.
  * [68] Theo Thonat, Eli Shechtman, Sylvain Paris, and George Drettakis.  Multi-view inpainting for image-based scene editing and rendering.  In 3DV, 2016.
  * [69] Jonathan Tremblay, Moustafa Meshry, Alex Evans, Jan Kautz, Alexander Keller, Sameh Khamis, Charles Loop, Nathan Morrical, Koki Nagano, Towaki Takikawa, and Stan Birchfield.  RTMV: A ray-traced multi-view synthetic dataset for novel view synthesis.  ECCV Workshop (Learn3DG ECCVW), 2022.
  * [70] Qianqian Wang, Zhengqi Li, David Salesin, Noah Snavely, Brian Curless, and Janne Kontkanen.  3D moments from near-duplicate photos.  In CVPR, 2022.
  * [71] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4), 2004.
  * [72] Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng.  Object-compositional neural implicit surfaces.  In ECCV, 2022.
  * [73] Rui Xu, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy.  Deep flow-guided video inpainting.  In CVPR, 2019.
  * [74] Xiaogang Xu, Ying-Cong Chen, and Jiaya Jia.  View independent generative adversarial network for novel view synthesis.  In ICCV, 2019.
  * [75] Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui.  Learning object-compositional neural radiance field for editable scene rendering.  In ICCV, October 2021.
  * [76] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa.  PlenOctrees for real-time rendering of neural radiance fields.  In ICCV, 2021.
  * [77] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In CVPR, 2018.
  * [78] Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao.  NeRF-editing: geometry editing of neural radiance fields.  In CVPR, 2022.
  * [79] Yanhong Zeng, Jianlong Fu, and Hongyang Chao.  Learning joint spatial-temporal transformations for video inpainting.  In ECCV, 2020.
  * [80] Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo.  Learning pyramid-context encoder network for high-quality image inpainting.  In CVPR, 2019.
  * [81] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [82] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison.  In-place scene labelling and understanding with implicit scene representation.  In ICCV, 2021.


# Design Space Exploration and Explanation via Conditional Variational
Autoencoders in Meta-model-based Conceptual Design of Pedestrian Bridges
††thanks: *Corresponding author. Stefano-Franscini-Platz 5, 8093 Zürich,
Switzerland, E-Mail adress<EMAIL_ADDRESS>
1 both authors contributed equally to the presented manuscript.
The authors would like to thankfully acknowledge the facilities of Design++ at
ETH Zürich and the funding through ETH Foundation grant No. 2020-HS-388
(provided by Kollbrunner/Rodio) as well as the SDSC Project ”Domain-Aware AI-
augmented Design of Bridges (DAAAD Bridges)”.

Vera M. Balmera,1, Sophia V. Kuhna,1, Rafael Bischofb, Luis Salamancab, Walter
Kaufmanna,
Fernando Perez-Cruzb, Michael A. Krausa,∗ aETH Zurich, Chair of Concrete
Structures and Bridge Design and Design++, Stefano-Franscini-Platz 5, 8093
Zürich, Switzerland
b Swiss Data Science Center (SDSC), Turnerstr. 5, 8006 Zürich, Switzerland

###### Abstract

For conceptual design, engineers rely on conventional iterative (often manual)
techniques. Emerging parametric models facilitate design space exploration
based on quantifiable performance metrics, yet remain time-consuming and
computationally expensive. Pure optimisation methods, however, ignore
qualitative aspects (e.g. aesthetics or construction methods). This paper
provides a performance-driven design exploration framework to augment the
human designer through a Conditional Variational Autoencoder (CVAE), which
serves as forward performance predictor for given design features as well as
an inverse design feature predictor conditioned on a set of performance
requests. The CVAE is trained on 18’000 synthetically generated instances of a
pedestrian bridge in Switzerland. Sensitivity analysis is employed for
explainability and informing designers about (i) relations of the model
between features and/or performances and (ii) structural improvements under
user-defined objectives. A case study proved our framework’s potential to
serve as a future co-pilot for conceptual design studies of pedestrian bridges
and beyond.

###### Index Terms:

Conceptual Design, Machine Learning, Architecture, Engineering and
Construction, Conditional Variational Autoencoder

##  I Introduction

The conceptual design phase in current architectural, engineering and
construction (AEC) practice is generally disconnected from later phases of the
building process, leading to an insufficient consideration of ultimate
functionality, performance, material impacts, manufacturing and construction
processes of the built structures. Yet, this phase is where the most
influential design decisions for a building project are made, which greatly
influence the material consumption and environmental impacts. In light of
increasing urbanization and associated environmental concerns, these topics
can no longer be ignored. In addition, digital workflows for design and
planning as well as systematic creation of design databases are not common
practice in AEC industry. Today, the initial design is mainly based on the
prior knowledge of (i) the design team, and (ii) the manual analysis of a few
similar reference projects. The practice of post-processing a relatively
fixated design from the conceptual phase requires significant manual - and
hence lengthy - labour with numerous translations, detailing, and re-
iterations of the design models between early-stage designers, engineers and
contractors involved in later stages [1].

Recent breakthroughs in artificial intelligence (AI) and especially machine
(ML) and deep learning (DL) have already had a transformative impact on many
fields, including medicine, physics and finance. However, the application of
these technologies in AEC is still in its infancy, due to the aforementioned
domain-specific siloed thinking, non-digitized workflows and resulting
processes and tools. This is also why traditional ML and DL approaches are
insufficient for direct application to most AEC design problems due to lack of
data and/or inconsistencies and biases on it.

The computational design paradigm offers designers systematic and digital
means of exploring vast parametrically generated design spaces. High-
dimensional design spaces with a large number of design parameters are
appealing, as these can potentially contain high-performing, yet unimagined or
unexpected design solutions. However, they come with the curse of
dimensionality and are hard to process for human designers, hence underlining
the need for tools to assist in their exploration and account for human
intuition. We propose to employ DL algorithms, specifically a variant of
Conditional Variational Autoencoders (CVAE), to form the basis for an AI-based
design co-pilot augmenting human creativity with the computational power of
modern algorithms to support early design decision-making processes, turning
them more informed, integrated, productive, and flexible.

The objective of this research is to (i) develop a synthetic dataset
generation pipeline for quantifyable (structural) design performances, (ii) to
implement, train and evaluate a suitable design meta-model for the forward and
inverse design setting, and (iii) to create a demonstrator application of an
explainable and intuitive co-pilot for conceptual design. We demonstrate the
applicability of this approach for the generic example of a bridge design
project in St. Gallen, Switzerland, cf. Fig. 1. By integrating our ML
algorithms into established Building-Information-Modelling (BIM) software
(here Autodesk Revit 2022 [2]), the proposed framework can easily be
transferred to future bridge project scenarios (and beyond, e.g. to housing or
office buildings) as soon as the parametric bridge model together with
requested boundary conditions and performance criteria are defined within
Revit. Therefore, our solution integrates seamlessly into the current design
paradigm known by AEC domain experts. In combination with implemented
explainable AI methods such as the Sensitivity Analysis (SA), this
ubiquitousness ensures acceptance amongst engineers or project managers with
limited expertise in ML or AI and fosters wide-spread use of such tools in
practice. Further information can be found on the project homepage.

Figure 1: Project site, M 1:750, from Woodtli et al. [3]

##  II Related work

In recent years, computer vision has been the avant-garde when it comes to the
development of deep generative models, such as Variational Autoencoders (VAE)
[4] or Generative Adversarial Networks (GAN) [5]. In engineering applications,
especially in civil engineering, data-driven design generation is a highly
under-explored research area, mainly due to the lack of systematised, freely
accessible datasets. However, generative models have recently been found to be
powerful tools in generative modelling of automobiles [6], mechanical
components [7], performance-based shading device design for office buildings
[8] or floorplans [9, 10]. In order for such tools to prevail in practice,
they must provide an interface for including human experts into the design
loop. Jang et al. [11] provide a definition of generative design as a
computational design method for automated conduction of design exploration
under consideration of user-defined constraints. Salamanca et al. [12]
developed a conditional Autoencoder for finding optimal designs of a vertical
garden under the objectives of rain occlusion or exposure to sunlight.
Danhaive et al. [13] proposed to utilise the latent space of a performance-
conditioned VAE to provide an intuitive tool for structural design-space
exploration of a spatial steel truss for the objective of structural mass.

Yet, the cited approaches stayed far below AEC practice complexity when it
comes to software integration as well as dimensionality of the design features
and objectives in addition to neglecting the consideration of state-of-the-art
structural design analysis requirements.

Figure 2: Overview of design features and performances for the bridge design
task for synthetic data generation

##  III Methods

This section presents the methodology, workflow and software used for
developing the design space learning and browsing framework for the pedestrian
bridge design example.

###  III-A Bridge Project Description

The specific use-case used as demonstrator project for the proposed framework
is a pedestrian bridge in St. Gallen, Switzerland, which is currently in the
conceptual design phase. The pedestrian bridge shall provide a pedestrian
crossing from an exit next to a parking garage (start point / SP in Fig. 1),
over a busy cantonal street and then through the St. Mangen park, to terminate
next to a church (end point / EP in Fig. 1). The currently existing
surroundings of the bridge project as well as the corresponding coordinate
system is given as a scanned point cloud within a BIM model in Autodesk Revit
[2]. A rendered walk-over the bridge supports imagination.

The city of St. Gallen defined fixed boundary conditions for the project, cf.
Fig. 1 and [3]: (i) SP and EP of the bridge are fixed to specific locations
together with the request of straight alignments, (ii) a prescribed clearance
height over the cantonal street and the path in the park need to be respected,
(iii) to minimise cut down of trees due to cultural heritage protection, and
(iv) a fixed bridge width of $b=2.50~{}m$.

###  III-B Design {Representation, Features, and Performances}

The bridge design model is represented by a $d$-dimensional (Euclidean) vector
of design feature variables $\mathbf{x}\in\mathcal{D}\subset\mathbb{R}^{d}$,
which together with predefined feature intervals
$\times_{j}[x_{j,min},x_{j,max}],(j=1,...,d)$ span the domain of the design
space $\mathcal{D}$. Evaluating a performance metric $\mathcal{P}(\cdot)$ for
a design may be computationally expensive and is a priori unknown except when
computed for specific instances $\mathbf{y}=\mathcal{P}(\mathbf{x})$. For the
design process, two situations have to be distinguished:

  1. 1.

Forward Design: for the full set of design features $\mathbf{x}\in\mathcal{D}$
(e.g. cross-section parameters, alignment geometry, etc.) we request the
associated performance metrics $\mathcal{P}(\mathbf{x})$ (e.g. costs,
structural utilisation, etc.).

  2. 2.

Inverse Design: for given performance metrics $y=\mathcal{P}(\mathbf{x})$
(i.e. costs, structural utilisation, etc.) and potentially a subset of design
features $\mathbf{\tilde{x}}\in\mathbb{R}^{d_{1}}$ we request feasible
(remaining) design features $\mathbf{x}\in\mathbb{R}^{d_{2}}$ with
$d_{1}+d_{2}=d$.

This study develops AI-based meta models for both design situations together
with respective interaction interfaces, cf. Sec. III-C and Sec. IV.

In order to choose the dimensionality of the bridge design features, we adopt
the propositions of SIA 112 [14], which specifies a level of development (LOD,
[15]) 300 for the task at hand. This implies certain parameters of the bridge
project to be fixated and hence not subject to further design variation such
as the bridge use (pedestrian bridge), the bridge typology (girder bridge) and
the boundary conditions stated in Sec. III-A. The design features $x_{j}$ are
provided on the left hand of Fig. 2 and define the girder height $h_{girder}$
and thickness $t_{girder}$, the amount $n_{p}$ and dimensions $h_{p}$ of the
piers as well as parameters $w,i$ of the NURBS curve for defining the bridge
alignment.

TABLE I: Design Performance Objectives $\mathcal{P}(\mathbf{x})$ Performance | Considered | Simulator
---|---|---
Structural Utilisation |  |
\- Ultimate Limit State | yes | Sofistik (FEA)
\- Serviceability Limit State | yes | Sofistik (FEA)
Structural Dynamics | yes | Sofistik (FEA)
Costs | yes | Revit Dynamo
Aesthetics | no | -
Sustainability | no | -

The set of performance metrics $y=\mathcal{P}(\mathbf{x})$ for the design of
the pedestrian bridge is provided on the right hand of Fig. 2, where mostly
structural safety and serviceability (resistances, utilisations, internal
actions and displacements) criteria are employed next to structural dynamics
aspects (eigenfrequencies), costs (computed via the weight and volume) and the
boundary conditions for SP, EP and the trees as defined in Sec. III-A. The
structural performance objectives together with the load definitions for the
pedestrian bridge (such as loads due to dead weight, pedestrians, wind, snow
and temperature) are derived from currently applicable design standards in
Switzerland by SIA 261 and 262 [14].

###  III-C Synthetic Data Generation Pipeline and Dataset

Training deep latent generative models requires a significant amount of data
which, for this study, is collected in a two-stage approach. At first, a
central Latin Hypercube Sampling (LHS, [16]) of the design space $\mathcal{D}$
is launched within the specifications as provided in Tab. II.

TABLE II: Input definition for LHS Input | Value
---|---
Design Features $\mathbf{x}$ | [$h_{girder},t_{girder},n_{p},h_{p},i,w$]
Minimum bounds | [0.25, 0.1, 2, 0.5, 0, 0.01]
Maximum bounds | [2.5, 0.23, 8, 1.5, 4, 7.00]
Amount of samples | 18’000

In the second stage, the sampled design features $\mathbf{x}$ are handed to
performance simulators to obtain the performance metrics
$\mathcal{P}(\mathbf{x})$ as defined on the right hand side of Fig. 2. For
obtaining design performances, either analytical expressions (for costs,
weight, volume as well as compliance with boundary conditions) or the Finite-
Element-Analysis (FEA) software ”Sofistik” (remaining performances) is
utilised. A parametric template for the FEA-based structural analysis was
developed and connected to Revit via zero-touch nodes inside Dynamo for the
FEA performance simulator to be able to evaluate the vast amounts of parameter
samples in a standardised way. The parametric FEA template contains load
definitions, a parametrized geometry description and evaluations for the
structural and dynamic performances. The results are saved to text files and
subsequently merged with the design features to form a data frame as basis for
ML modelling.

###  III-D Machine Learning Model

The ML model used in this study is a variation of Conditional Variational
Autoencoders (CVAE) [17]. In light of having to solve both a forward as well
as an inverse problem, we forgo feeding the conditional $\mathbf{y}$ to the
encoder and instead let it predict the performance metrics together with a
latent vector in two separate heads (cf. Fig 3).

Figure 3: Architecture of our CVAE network acting both as surrogate as well as
generative model.

More specifically, the encoder receives the design features $\mathbf{x}$ as
input and produces the outputs $\mathbf{\hat{y}}$ and $\mathbf{z}$, denoting
the performance metrics and an arbitrary latent vector, respectively. Since
the performance metrics may include categorical features with discrete values,
we replace the predicted $\mathbf{\hat{y}}$ with the ground truth $\mathbf{y}$
before passing it to the decoder to avoid skewing the model towards continuous
values which it is not going to encounter at inference time. Finally,
$\mathbf{y}$ and $\mathbf{z}$ are passed through the decoder to produce
$\mathbf{\hat{x}}$. Both the encoder and the decoder consist of an arbitrary
number $l$ of MLP-Blocks, where each block is comprised of a fully-connected
layer with leaky-ReLU activation and batch normalisation.

We chose this architecture over a GAN because it is more robust to train and
easier to thoroughly evaluate, as there is no straight-forward way of
quantitatively estimating the ”goodness” of a GAN. Furthermore, our framing
allows to train a surrogate model (encoder) as well as a conditional
generative model (decoder) simultaneously and is therefore tailored to the
setting where both a forward and an inverse problem have to be solved. Once
trained, the decoder generates new samples $\mathbf{\hat{x}}$ conditioned on
some specified performance metrics $\mathbf{y}$. On the other hand, the
encoder can be used to compute the mapping from $\mathbf{\hat{x}}$ to an
approximate performance metric $\mathbf{\hat{y}}$, thus acting as a
reliability estimator for designs generated by allowing to examine the
reconstruction error between requested performance $\mathbf{{y}}$ and the
predicted performance of the generated designs $\mathbf{\hat{y}}$.

###  III-E CVAE Objective Function

In order to deter the encoder from including information about $\mathbf{y}$
into the latent vector $\mathbf{z}$, we decorrelate the two vectors by
regularising for the covariance between $\mathbf{y}$ and $\mathbf{z}$ over the
batch dimension [18]:

$\displaystyle\operatorname{Cov}[\mathbf{y},\mathbf{z}]=\mathrm{E}\left[(\mathbf{y}-\mathrm{E}[\mathbf{y}])^{\top}\mathbf{z}\right]$
(1)
$\displaystyle\mathcal{L}_{\textit{cov}}(\mathbf{y},\mathbf{z})=\frac{1}{d_{y}d_{z}}\sum_{i,j}\operatorname{Cov}[\mathbf{y},\mathbf{z}]_{i,j}^{2}$
(2)

where $\mathbf{y}\in\mathbb{R}^{d_{y}}$ and $\mathbf{z}\in\mathbb{R}^{d_{z}}$.
This term enforces the independence
$p(\mathbf{y},\mathbf{z})=p(\mathbf{y})p(\mathbf{z})$ and avoids having to
estimate the conditional $p(\mathbf{z|y})$ at generation time. Note that we
can abstain from deducting $\mathrm{E}[\mathbf{z}]$ from $\mathbf{z}$ as its
mean is already driven towards zero by the KL-divergence.

The total objective function of the CVAE is therefore comprised of the
reconstruction loss $\mathcal{L}_{\textit{des}}$, the accuracy of the
predicted performance metrics $\mathcal{L}_{\textit{perf}}$, the KL divergence
and the decorrelation between $\mathbf{y}$ and $\mathbf{z}$:

$\displaystyle\mathcal{L}=\lambda_{1}\mathcal{L}_{\textit{des}}(\mathbf{x},\mathbf{\hat{x}})+\lambda_{2}\mathcal{L}_{\textit{perf}}(\mathbf{y},\mathbf{\hat{y}})$
(3)
$\displaystyle+\lambda_{3}\mathcal{L}_{\textit{KL}}(\mathbf{z})+\lambda_{4}\mathcal{L}_{\textit{cov}}(\mathbf{y},\mathbf{z})$

where $\lambda_{i}$ are hyperparameters for selecting the contribution of each
term towards the total loss. Both $\mathcal{L}_{\textit{des}}$ and
$\mathcal{L}_{\textit{perf}}$ consist of the mean squared error (MSE) loss for
continuous features, or the crossentropy (CE) loss for discrete and
categorical features, respectively.

###  III-F Explainability through Design Sensitivity Analysis

Sensitivity analysis is a well-known method for performing topology
optimisation by taking the derivative of the performance metrics in the output
w.r.t. the design variables in the input of the model [19]. However, computing
these derivatives in established solvers, e.g. leveraging the FEM, is
expensive as they have to be estimated numerically. With neural networks being
fully differentiable, this feature is readily implemented in our CVAE and can
be computed very efficiently through Automatic Differentiation (AD).

When inspecting the sensitivity of a performance metric
$\frac{\partial\mathbf{\hat{y}_{i}}}{\partial\mathbf{x}}$ for a certain design
$\mathbf{x}$, the designer receives information in which direction the design
variables should be changed in order to improve the particular performance
attribute. Furthermore, the distribution of sensitivities over a large set of
designs yields information about the network’s decision-making. An expert
designer with prior knowledge in the design, analysis and construction of
bridges can therefore, to a certain extent, estimate the model’s reliability
based on the relations it found between the features.

##  IV Results & Discussion

We sampled 18’000 instances of the pedestrian bridge together with their
performances within the generative design as described in Sec. III to form the
dataset for subsequent CVAE training. While LHS of the design spaces
$\mathcal{D}$ was conducted in a few seconds, obtaining performances of a
batch of 600 design instances via FEA took on average around 55 minutes.

Figure 4: Model performance on four different performance metrics predicted by
the encoder. Optimally, all points should lie on the diagonal.

Figure 5: Sensitivities of the costs w.r.t. the input features (design
variables) for: (left) a single user-chosen bridge, (right) a batch of 100
randomly generated bridges around the user-chosen bridge.

For training the model and assessing its ability to generalise, the dataset
was split into training, validation and test sets, where the training set
accounted for 70%, the validation set for 10% and the test set for 20% of the
total number of samples. All continuous and ordinal features, both in
$\mathbf{x}$ as well as in $\mathbf{y}$, were standardised to have mean
$\mu=0$ and a standard deviation $\sigma=1$, while categorical features were
one-hot encoded. Training was then performed using the Adam optimiser [20]
with an initial learning rate $\eta=0.001$ and a decrease of $\eta$ by a
factor of 0.1 after every six epochs without improvement on the validation
set. Finally, an early stopping was employed after 12 epochs without
improvement.

The best-performing widths per MLP-Block were empirically found to be [128,
256, 512, 256, 128] in both the encoder and decoder, which accumulates to a
sizeable amount of parameters and indicates that the complexity of this
problem is non-negligible. The dimensionality of the latent space spanning
$\mathbf{z}$ is set to 2 to allow for human interpretation. Finally, we set
the weights of the loss terms to $\lambda_{1}=1$, $\lambda_{2}=10$,
$\lambda_{3}=0.1$ and $\lambda_{4}=0.01$. Note that the effects of
$\lambda_{3}$ for the KL-divergence and $\lambda_{4}$ for the decorrelation
can be assessed by visually inspecting the latent space $\mathbf{z}$ and
coloring the points according to their value of $\mathbf{y}$ (cf. Fig. 6). If
$\mathbf{z}$ is not distributed normally, the hyperparameter $\lambda_{3}$
should be increased, whereas $\lambda_{4}$ should be increased if the scatter
plot reveals correlations between $\mathbf{z}$ and $\mathbf{y}$.

Figure 6: Visualisation of the latent space with the points colored according
to their respective cost without decorrelation (left) and with decorrelation
(right). Figure 7: User interface for interaction with the meta-model for the
inverse bridge design setting (Screenshot Dynamo)

Given the brevity of this paper, we show only selected results. First, in Fig.
4 we show the performance on the test set of the surrogate model, i.e. the
forward mapping from design features $\mathbf{x}$ to performances
$\mathbf{y}=\mathcal{P}(\mathbf{x})$, on a selection of four features. It can
be observed that the main bulk of predictions lies accurately on the diagonal
and hence proves that the CVAE predicted every target well on average while
the prediction’s standard deviation is dependent on the respective target
quantity. In addition, the prediction intervals are reasonable for a realistic
pedestrian bridge design.

Fig. 5 (right) depicts the sensitivities of the predicted costs w.r.t. the
design variables for 100 generated bridges in a swarm plot. In addition to the
horizontal placement of the bridges based on the calculated sensitivities, we
also coloured the points according to their cost. This can give further
insight, as cheaper bridges may be influenced by different design variables
compared to more costly structures. Our trained model found a clear, positive
relation between the height and thickness of the girder and the final costs of
the bridge. This is obviously expected, and helps validating the sensitivity
analysis, as the costs in this synthetic dataset were calculated based on the
amount of material necessary for building the bridge. Both the height and the
thickness of the girder have a direct influence on the amount of concrete used
in the structure. The alignment parameters define the curvature and thus the
length of the bridge and have therefore an effect on the costs as well, though
to a lesser extent.

Fig. 5 (left) shows the derivative of the costs w.r.t. the design variables
for a single generated bridge. This plot is useful to designers for making
informed decisions on how to improve the bridge according to the user-chosen
objectives.

Figure 7 shows the prototype of a user interface for the inverse design
situation developed within Revit Dynamo. It provides the user with sliders and
check-boxes, allowing to set desired performance metrics such as ranges for
costs or the maximum structural utilisation in the ultimate as well as
serviceability limit state. More fine grained requests for different
objectives, as well as an additional visualisation of the latent space and the
mapping of the objectives, is also possible, yet has been omitted for the sake
of clearness. The right hand side of the user interface displays the
sensitivity plots as well as a scatter plot for finding the Pareto front.
Finally, a rendering of the generated bridge is shown, which can be inspected
either on the screen in 2D, or in 3D through virtual reality on smartphones or
dedicated devices.

We demonstrated our developments to a selected group of researchers and
practitioners (15 persons) within a hands-on session. The collected feedback
towards ergonomics, efficiency and quality prove our framework to be
intuitive, efficient, reliable and to bear a great potential for applications
in practice.

##  V Conclusions and Future Research

Subspace learning establishes a new paradigm for performance-conditioned
exploration of design spaces, which is neither an optimisation setting nor a
random process. Rather, it provides an intuitive and efficient cartography of
the vastness of these design spaces. Instead of replacing human intuition with
predefined, deterministic, quantitative rules, the AI acts as a design
collaborator/co-pilot that augments the human designer’s intuition on the
problem at hand.

This research provides a variation of CVAEs tailored to forward and inverse
design situations. We showed the potential of our CVAE in meta-modelling (i)
the forward problem by providing a surrogate to estimate more efficiently and
quickly design performances given design features, (ii) compression of complex
design spaces into continuous, smooth, low-dimensional design subspaces. With
a forward pass through our CVAE being extremely efficient, it can provide
performance conditioned designs in quasi real-time and thus augment human
designers by providing instant feedback and proposals during the iterative
prototyping phase. Furthermore, with analytical derivatives inherently
provided in neural networks, we demonstrated that the sensitivity analysis
serves as powerful tool for both design optimisation as well as model
interpretability. The latter is crucial for building trust and achieving wide
acceptance of this kind of design augmentation tools in the AEC domain. The
collected user responses prove our framework possesses the potential to find
wide application in industry and research as a co-pilot for conceptual design
studies in the AEC domain beyond pedestrian bridges.

Future research will be concerned with multiple open points identified within
this research. First, a more efficient performance-based sampling of the
design space to create a meaningful synthetic dataset under multiple design
objectives has to be investigated. Finally, more effort is necessary for
improving the human-machine interfaces. The reported AI model can be enhanced
with a recommender system that allows users to directly interact with the
results and thus capture less quantifiable metrics like aesthetics. In
addition, displaying results to the human designer and collecting feedback
could be shifted from a computer screen to immersive environments employing
knowledge from education in civil engineering with extended reality [21, 22].

## References

  * [1] S. V. Kuhn, R. Bischof, G. Klonaris, W. Kaufmann, and M. A. Kraus, “$ntab_{0}$: Design priors for AI-augmented generative design of network tied-arch-bridges.” Forum Bauinformatik 2022, 2022.
  * [2] Autodesk, “Revit,” 2022. [Online]. Available: https://www.autodesk.de/products/revit
  * [3] H. Woodtli and C. Svec, “Projektwettbewerb Brücke über den Graben (Passerelle St. Gallen, Unterer Graben 25), Projektwettbewerb für Senn Resources AG,” Mar 2021.
  * [4] D. P. Kingma and M. Welling, “Auto-Encoding Variational Bayes,” _arXiv e-prints_ , p. arXiv:1312.6114, Dec. 2013.
  * [5] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Networks,” _arXiv e-prints_ , p. arXiv:1406.2661, Jun. 2014.
  * [6] A. Burnap, Y. Liu, Y. Pan, H. Lee, R. Gonzalez, and P. Y. Papalambros, “Estimating and exploring the product form design space using deep generative models,” in _International Design Engineering Technical Conferences and Computers and Information in Engineering Conference_ , vol. 50107\. American Society of Mechanical Engineers, 2016, p. V02AT03A013.
  * [7] S. Oh, Y. Jung, S. Kim, I. Lee, and N. Kang, “Deep Generative Design: Integration of Topology Optimization and Generative Models,” _arXiv e-prints_ , p. arXiv:1903.01548, Mar. 2019.
  * [8] B. Ercan and S. T. Elias-Ozkan, “Performance-based parametric design explorations: A method for generating appropriate building components,” _Design Studies_ , vol. 38, pp. 33–53, 2015.
  * [9] N. Nauata, K.-H. Chang, C.-Y. Cheng, G. Mori, and Y. Furukawa, _House-GAN: Relational Generative Adversarial Networks for Graph-Constrained House Layout Generation_ , 11 2020, pp. 162–177.
  * [10] S. Chaillou, “Archigan: Artificial intelligence x architecture,” in _Architectural intelligence_. Springer, 2020, pp. 117–127.
  * [11] S. Jang, S. Yoo, and N. Kang, “Generative design by reinforcement learning: Enhancing the diversity of topology optimization designs,” _Computer-Aided Design_ , vol. 146, p. 103225, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0010448522000239
  * [12] L. Salamanca, A. A. Apolinarska, M. Kohler, and F. Perez-Cruz, “Augmented Intelligence for Architectural Design with Conditional Autoencoders: Semiramis case study.” Design Modelling Symposium Berlin, 2022.
  * [13] R. Danhaive and C. T. Mueller, “Design subspace learning: Structural design space exploration using performance-conditioned generative modeling,” _Automation in Construction_ , vol. 127, p. 103664, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0926580521001151
  * [14] SIA, “SIA: Schweizer Ingenieure- und Architektenverein, Normenreihe:
SIA 112 (2014): Model project planning (Terminology standards)
SIA 260 (2013): Basics of project planning for structures
SIA 261 (2014): Actions on Structures
SIA 262 (2013): Structural concrete,” 2013.

  * [15] J. Abualdenien, A. Borrmann, and M. Koenig, “Ausarbeitungsgrade von bim-modellen,” _Building Information Modeling: Technologische Grundlagen und industrielle Praxis, Springer Verlag_ , p. 165–191, 2021.
  * [16] M. D. McKay, R. J. Beckman, and W. J. Conover, “Comparison of three methods for selecting values of input variables in the analysis of output from a computer code,” _Technometrics_ , vol. 21, no. 2, p. 239–245, May 1979\.
  * [17] K. Sohn, H. Lee, and X. Yan, “Learning structured output representation using deep conditional generative models,” _Advances in neural information processing systems_ , vol. 28, 2015.
  * [18] J. Zbontar, L. Jing, I. Misra, Y. LeCun, and S. Deny, “Barlow Twins: Self-Supervised Learning via Redundancy Reduction,” _arXiv e-prints_ , p. arXiv:2103.03230, Mar. 2021.
  * [19] S. Cho and H.-S. Jung, “Design sensitivity analysis and topology optimization of displacement–loaded non-linear structures,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 192, no. 22, pp. 2539–2553, 2003. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0045782503002743
  * [20] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” in _3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings_ , 2015. [Online]. Available: https://arxiv.org/abs/1412.6980
  * [21] M. Kraus, I. Custovic, and W. Kaufmann, “Struct-mrt: Immersive learning and teaching of design and verification in structural civil engineering using mixed reality,” _arXiv preprint arXiv:2109.09489_ , 2021.
  * [22] M. Kraus, I. Čustović, and W. Kaufmann, “Mixed reality applications for teaching structural design,” in _Structures Congress 2022_ , pp. 283–295.