# ED-NeRF: Efficient Text-Guided Editing of 3D Scene using Latent Space NeRF

Jangho Park2∗, Gihyun Kwon3∗, Jong Chul Ye1,2,3
Kim Jaechul Graduate School of AI1, Robotics Program2,
Department of Bio and Brain Engineering3, KAIST
<EMAIL_ADDRESS>

###### Abstract

Recently, there has been a significant advancement in text-to-image diffusion
models, leading to groundbreaking performance in 2D image generation. These
advancements have been extended to 3D models, enabling the generation of novel
3D objects from textual descriptions. This has evolved into NeRF editing
methods, which allow the manipulation of existing 3D objects through textual
conditioning. However, existing NeRF editing techniques have faced limitations
in their performance due to slow training speeds and the use of loss functions
that do not adequately consider editing. To address this, here we present a
novel 3D NeRF editing approach dubbed ED-NeRF by successfully embedding real-
world scenes into the latent space of the latent diffusion model (LDM) through
a unique refinement layer. This approach enables us to obtain a NeRF backbone
that is not only faster but also more amenable to editing compared to
traditional image space NeRF editing. Furthermore, we propose an improved loss
function tailored for editing by migrating the delta denoising score (DDS)
distillation loss, originally used in 2D image editing to the three-
dimensional domain. This novel loss function surpasses the well-known score
distillation sampling (SDS) loss in terms of suitability for editing purposes.
Our experimental results demonstrate that ED-NeRF achieves faster editing
speed while producing improved output quality compared to state-of-the-art 3D
editing models. ††* equally contributed

##  1 Introduction

In recent years, the development of neural implicit representation for
embedding three-dimensional images in neural networks has seen remarkable
progress. This advancement has made it possible to render images from all
angles using only a limited set of training viewpoints. Starting with the
seminar work known as the Neural Radiance Field (NeRF) (Mildenhall et al.,
2021), which trained radiance fields using a simple MLP network, various
improved techniques (Barron et al., 2021; Reiser et al., 2021; Müller et al.,
2022) based on advanced network architectures or modified encoding have been
proposed. Alternatively, several methods (Sun et al., 2022; Fridovich-Keil et
al., 2022; Karnewar et al., 2022; Chen et al., 2022) proposed to directly
optimize voxel points serving as sources for rendering, bypassing the
traditional approach of encapsulating all information within implicit
networks. These methods have gained prominence for their ability to train
radiance fields in a remarkably short time. In addition to representing
existing 2D image data in the 3D space, recent research has explored expanded
approaches for generating entirely novel 3D objects. With the emergence of
text-to-image embedding models like CLIP (Radford et al., 2021), various
methods have been proposed to train implicit networks that can generate new
objects solely from text prompts (Jain et al., 2022). This trend has been
accelerated with the advent of text-to-image diffusion generation models such
as Stable Diffusion (Rombach et al., 2022), particularly through the score
distillation sampling (SDS) (Poole et al., 2022) which conveys the
representation of the text-to-image model to NeRF model.

Figure 1: Qualitative results of our method. ED-NeRF successfully edited 3D
scenes with given target text prompts while preserving the original object
structure and background regions.

However, the challenge of editing pre-trained 3D implicit networks according
to specific conditions still remains as an open problem due to the constraints
of tasks: maintaining the integrity of the original 3D images while making
desired modifications. As an initial work, several approaches (Wang et al.,
2022; 2023a) tried to edit the pre-trained NeRF models based on text
conditions, utilizing the pre-trained CLIP model to fine-tune the parameters
of NeRF models. Nevertheless, these methods exhibit notable weaknesses,
including the performance limitations of the CLIP model itself and the need
for rendering high-resolution images during training, which results in
significant time consumption.

Recently, several editing methods proposed to leverage the enhanced
expressiveness of text-to-image diffusion models such as Stable Diffusion.
Some methods (Sella et al., 2023) proposed to directly employ the score
distillation sampling method, with additional regularizations. However, these
methods suffer from significant time consumption and instability in generation
performance due to the requirement of full-resolution rendering in the
training stage and limitations of the score distillation loss itself. Other
alternative approaches (Haque et al., 2023) proposed to directly manipulate
the training images of NeRF using text-guided image translation models. This
method aims to enable the generation of 3D images corresponding to text
conditions. However, it suffers from a significant drawback in terms of
training time, as it requires periodic translation of training images during
the training process.

To address these challenges, we are interested in developing a novel NeRF
editing method to efficiently and effectively edit 3D scenes using only text
prompts. To achieve this, we enable NeRF to operate directly in the NeRF
latent space, similar to Latent-NeRF (Metzer et al., 2023), which helps reduce
time and computational costs. However, naively rendering the latent feature of
real-world scenes directly with NeRF may lead to a significant drop in view
synthesis performance due to the lack of geometric consistency in the latent
space. To tackle this issue, we conduct an analysis of the latent generation
process and propose a novel refinement layer to enhance performance based on
the analysis. Furthermore, to solve the drawback of the existing SDS-based
method in editing, we propose a new sampling strategy by extending Delta
Denoising Score (DDS) (Hertz et al., 2023), a 2D image editing technique based
on score distillation sampling, into the 3D domain. This extension allows us
to achieve high-performance editing capabilities while keeping computational
costs affordable, even with large Diffusion Models such as Stable Diffusion.
Given the superior editing proficiency of our approach, we’ve named it ED-NeRF
(EDiting NeRF).

##  2 Related Work

Starting from the Neural Radiance Field (NeRF) (Mildenhall et al., 2021),
there have been approaches to represent three-dimensional scenes in neural
fields. However, due to the slow training speed, several approaches tried to
improve the performance by modifying the network architecture or training
strategy (Barron et al., 2021; Müller et al., 2022; Reiser et al., 2021).
Several methods without relying on neural networks showed great performance in
accelerating. These include a method for optimizing the voxel fields (Sun et
al., 2022; Fridovich-Keil et al., 2022; Chen et al., 2022; Karnewar et al.,
2022), or decomposing the components of field representation. Based on the
success of these techniques, methods for generating ‘novel’ 3D scenes have
been proposed. Especially with the emergence of the text-to-image embedding
model of CLIP (Radford et al., 2021), DreamField (Jain et al., 2022) leveraged
CLIP to train the NeRF model for novel 3D object synthesis. Recently, the
performance of the text-to-image diffusion model enabled remarkable
improvement in 3D generation. Starting from DreamFusion (Poole et al., 2022),
several methods (Metzer et al., 2023; Liu et al., 2023b; Xu et al., 2023)
showed impactful results using the diffusion-based prior. However, these
methods are limited to generating ‘novel’ 3D objects and, therefore cannot be
applied to our case of NeRF-editing which tries to modify the existing 3D
scenes according to the given conditions.

Compared to the novel object generation, NeRF editing is still not an explored
field, due to the complexity of the task. As a basic work, several methods
focused on color or geometric editing (Yuan et al., 2022; Liu et al., 2021;
Kuang et al., 2023). Other works tried style transfer or appearance transfer
on 3D neural fields (Zhang et al., 2022; Liu et al., 2023a; Bao et al., 2023)
and showed promising results. With incorporating the CLIP model, several
approaches (Wang et al., 2022; 2023a; Kim et al., 2023) tried to modify the
pre-trained NeRF towards the given text conditions. Although the results show
pleasing results, the method still has limitations in detailed expression due
to the limitation of CLIP model itself.

Similar to the novel scene generation case, the development of text-to-image
diffusion models brought significant improvement in the editing field.
Starting from Score Distillation Sampling method proposed in DreamFusion,
Vox-e tried to edit the pre-trained voxel fields with regularization (Sella et
al., 2023). As an alternative method, InstructNerf2Nerf (Haque et al., 2023)
proposed to directly leverage 2D image translation models for changing the
attribute of 2D images for NeRF training. However, these methods still have
limitations due to excessive training time or unstable editing from loss
functions. To address the above problems, we propose an efficient method of
editing with novel latent space NeRF training and improved edit-friendly loss
functions.

##  3 Methods

Figure 2 provides an overview of training ED-NeRF. First, we optimize NeRF in
the latent space of Stable Diffusion. To do this, we encode all images using a
pre-trained Variational Autoencoder (VAE) to obtain the feature vectors and
guide NeRF to predict these feature vectors directly. Also, we introduce an
additional refinement layer, which enhances the novel view synthesis
performance of NeRF (Fig. 2(a)). At the inference stage, we can render a
natural image by latent NeRF via decoding rendered latent map (Fig. 2(b)). At
the editing phase, by utilizing DDS, we adjust the parameters of both NeRF and
the refinement process to align the 3D scene with the provided target text
(Figure 3). The detailed pipeline for this approach is outlined in the
following sections.

###  3.1 ED-NeRF for 3D Scene Editing

Figure 2: Overall pipeline of training and inference stage. (a) We optimize
ED-NeRF in the latent space, supervised by source latent. Naively matching
NeRF to a latent feature map during optimization can degrade view synthesis
quality. (b) Inspired by the embedding process of Stable Diffusion, we
integrated additional ResNet blocks and self-attention layers as a refinement
layer. (c) All 3D scenes are decoded from the Decoder when ED-NeRF renders a
novel view feature map.

NeRF (Mildenhall et al., 2021) uses MLPs to predict density $\sigma$ and color
$\mathbf{c}$ for a given 3D point coordinate $\mathbf{x}=(x,y,z)$ and view
direction $\mathbf{d}$. Through positional encoding $\gamma(\cdot)$,
$\mathbf{x}$ and $\mathbf{d}$ are mapped into high-frequency vectors, and then
fed into the neural network of NeRF, resulting in two outputs: density
$\sigma\in\mathbb{R}$ and color $\mathbf{c}\in\mathbb{R}^{3}$.

$(\mathbf{c},\sigma)=F_{\theta}^{c}(\mathbf{\gamma}(\mathbf{x}),\mathbf{\gamma}(\mathbf{d}))$
(1)

Through volume rendering Eq. (2), NeRF predicts the pixel color along the
camera ray $\mathbf{r}(t)=\mathbf{o}$ \+ $t\mathbf{d}$, with $t$ representing
the depth within the range $[t_{near},t_{far}]$ , $\mathbf{o}$ stands for the
camera position, and $\mathbf{d}$ represents the view direction:

$\hat{C}(r)=\int_{t_{n}}^{t_{f}}T(t)\mathbf{\sigma}(\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),d)dt,\hskip
2.84544pt\textrm{where}\hskip
2.84544ptT(t)=\textrm{exp}\left(-\int_{t_{n}}^{t}\sigma(\mathbf{r}(s))ds\right).$
(2)

Optimizing NeRF to render the latent feature value of the latent diffusion
model offers several advantages in text-guided 3D generation. These advantages
include a reduced training burden due to the decreased dimensionality of the
space, and enhanced editability for the NeRF model, as the rendered outputs
can be directly employed as input for the latent diffusion models. The concept
of migrating NeRF to the latent space is first proposed by Latent-NeRF (Metzer
et al., 2023), in which the NeRF is directly trained with the latent feature
rather than RGB color. Therefore it can render a 3D scene without the encoding
process during optimization when using the latent diffusion model as semantic
knowledge prior. However, this work exclusively focuses on generating
‘virtual’ 3D assets without supervision, making it unsuitable for real-world
scenes.

Thus, ED-NeRF is realized based on a novel latent NeRF training pipeline for
synthesizing real-world scenes in the latent space. As depicted in Figure 2,
when a real-world image dataset $I$ contains multi-view images
$I=\\{I^{i}\\}_{i=1}^{N}$, we can encode all images to the latent space of
Stable Diffusion via encoder to obtain the feature:
${z^{i}}=\mathcal{E}({{I^{i}}})\in\mathbb{R}^{64\times 64\times 4}$. After
embedding all images, we can use the latent feature maps
${z}:=\\{{z}^{i}\\}_{i=1}^{N}$ as label data set for ED-NeRF training using
the loss function:

$\mathcal{L}_{rec}=\sum_{\mathbf{r}\in\mathcal{R}}\left\|Z^{i}(\mathbf{r})-\hat{Z}^{i}(\mathbf{r})\right\|^{2}$
(3)

where $Z^{i}$ denotes the pixel latent value of the latent $z^{i}$ and
$\hat{Z^{i}}(\mathbf{r})$ is rendered by the volume rendering equation:

$\hat{Z}^{i}(\mathbf{r})=\int_{t_{n}}^{t_{f}}T(t)\mathbf{\sigma}(\gamma(t))\mathbf{f}_{z}(\mathbf{r}(t),d)dt,\hskip
2.84544pt\textrm{where}\hskip
2.84544ptT(t)=\textrm{exp}\left(\int_{t_{n}}^{t}\sigma(\mathbf{r}(s))ds\right).$
(4)

where $\mathbf{f}_{z}\in\mathbb{R}^{4}$ denotes the predicted feature value by
the neural network, taking $\gamma(\mathbf{x})$ and $\gamma(\mathbf{d})$ as
input:

$(\mathbf{f}_{z},\sigma)=F_{\theta}(\mathbf{\gamma}(\mathbf{x}),\mathbf{\gamma}(\mathbf{d}))$
(5)

By minimizing the loss Eq. (3) to update the parameters of the neural network
$F_{\theta}$, we obtain a novel ED-NeRF model optimized in the latent space of
the Stable Diffusion.

###  3.2 Refinement Layer based on Latent Feature Analysis

When naively matching the latent generated by Eq. (3), we observed that the
reconstruction performance significantly deteriorated. In addressing this
issue, we analyzed the Encoder $\mathcal{E}$ and Decoder $\mathcal{D}$ of
Stable Diffusion and discovered the following insight in the process:

1) The encoder and decoder consist of ResNet blocks and self-attention layers.
Therefore during the process of mapping the image to the latent space and
forming a feature map, pixel values exhibit interference between each other,
primarily due to ResNet and self-attention layers. Thus the latent and image
pixels are not directly aligned.

2) When NeRF renders a single pixel value from the latent feature map, each
ray independently passes through an MLP to determine the pixel value of the
feature map. Therefore, the feature value rendered by NeRF for a single pixel
is determined without interactions with other pixels.

Based on this analysis, we find that the reason for the deformed
reconstruction performance of the latent NeRF lies in the inconsideration of
the interactions mentioned above. Therefore, we aim to incorporate the
interactions among pixels introduced by the ResNet and self-attention layers
into the ED-NeRF rendering stage. Fortunately, in the Encoder and Decoder of
Stable Diffusion, the embedded feature maps pass through self-attention layers
at the same dimension, allowing us to concatenate two attention layers
straightly. Taking advantage of this, we can design a refinement layer
${F_{\phi}(\cdot)}$ as shown in Figure 2, without dimension change of input
and output vector. Let $\tilde{{Z}}^{i}$ as the pixel latent vector of the
refined feature map $\tilde{z}^{i}$, where formed from
$\tilde{z}^{i}={F}_{\phi}(\hat{z}^{i}).$ Therefore, we can design a refined
reconstruction loss function as follows :

$\mathcal{L}_{ref}=\sum_{\mathbf{r}\in\mathcal{R}}\left\|Z^{i}(\mathbf{r})-\tilde{Z}^{i}(\mathbf{r})\right\|^{2}\hskip
2.84544pt,\textrm{where}\hskip 2.84544pt\tilde{z}^{i}={F}_{\phi}(\hat{z}^{i})$
(6)

Ultimately, we can formulate total training loss as the sum of the refinement
loss $\mathcal{L}_{ref}$ and reconstruction loss $\mathcal{L}_{rec}$, as
follows.

$\mathcal{L}_{rtot}=\lambda_{rec}\mathcal{L}_{rec}+\lambda_{ref}\mathcal{L}_{ref}$
(7)

We update NeRF and refinement layer concurrently denoted as $F_{\theta}$ and
$F_{\phi}$ by minimizing total loss $\mathcal{L}_{rtot}$ to reconstruct latent
vectors in various views. To ensure stable learning, training with
$\lambda_{rec}$ set to 1.0 and $\lambda_{ref}$ set to 0.1 during the initial
stages of training. Beyond a specific iteration threshold, we set it to 0 to
encourage the refinement layer to focus more on matching the latent
representations.

Figure 3: Expanding DDS into 3D for ED-NeRF editing. Pretrained ED-NeRF
renders the target latent feature map, and a scheduler of the denoising model
perturbs it to the sampled time step. Concurrently, the scheduler adds noise
to the source latent using the same time step. Each of them is fed into the
denoising model, and the DDS is determined by subtracting two different SDS
scores. In combination with a binary mask, masked DDS guides NeRF in the
intended direction of the target prompt without causing unintended
deformations.

###  3.3 Editing ED-NeRF via Delta Denoising Score

After optimizing ED-NeRF in the latent space, it is possible to directly
employ the latent diffusion model to update ED-NeRF parameter via rendered
latent map $z$ in the direction of the target text prompt $y_{trg}$. The most
well-known method for text-guided NeRF update is Score Distillation Sampling
(SDS), which directly transfers the score estimation output as a gradient of
NeRF training:

$\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\mathbf{z},y_{trg},\epsilon,t)=\omega(t)(\epsilon_{\psi}\left(\mathbf{z}_{\mathbf{t}},y_{trg},t\right)-\epsilon)\frac{\partial\mathbf{z}_{\mathbf{t}}}{\partial\theta}$
(8)

However, in our NeRF editing case, the updating rule for SDS often shows
several problems including color saturation and mode-seeking (Wang et al.,
2023b). We conjecture that the problem originated from the properties of score
estimation itself. Since the target noise $\epsilon$ is pure Gaussian, the
score difference is not aware of any prior knowledge of source images.
Therefore the generated outputs are just the replacement of hallucinated
objects without consideration of source NeRF.

To solve the problem of SDS, we focus on the recently proposed 2D editing
method of Delta Denoising Score (DDS) (Hertz et al., 2023). The major
difference between SDS and DDS is that the distilled score is the difference
between the denoising scores from target and source. As shown in Eq. (9), DDS
can be formed as a difference between two SDS scores conditioned on two
different text prompts:

$\nabla_{\theta}\mathcal{L}_{\mathrm{DDS}}=\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\mathbf{z},y_{src})-\nabla_{\theta}\mathcal{L}_{\mathrm{SDS}}(\hat{\mathbf{z}},y_{trg}),$
(9)

where $\mathbf{z}$ is source latent, $\hat{\mathbf{z}}$ is rendered target
latent, $y_{trg}$ represents the target text embedding, $y_{src}$ represents
the reference text embedding. DDS guides the optimized latent towards the
target prompt from the source prompt without the influence of the pure noise
component, therefore it can easily edit 2D images.

We aim to extend this manipulation capability of DDS to 3D space as shown in
Fig. 3. As we already have embedded source latent $z^{i}$ for the $i$-th
camera pose, we can directly use them as source components of DDS. To fine-
tune the model, we render the edited output $\tilde{z}^{i}$ which is also
rendered from the $i$-th camera pose. With the paired latents, we add the same
sampled noise $\epsilon_{t}$ with the noise scale of timestep $t$ to the both
source and edited latents so that we obtain noisy latent
$\tilde{z}^{i}_{t},{z}^{i}_{t}$. Then we apply the diffusion model to obtain
estimated score outputs from noisy latents using different text conditions for
source and edited images. As in Eq. (9), we can use the difference between the
two outputs as a gradient for updating the NeRF parameters. In this step, we
simultaneously train the NeRF parameters $\theta$ with refinement parameters
$\phi$ as it showed better editing quality. Therefore with the random $i$-th
camera pose, our 3D DDS is formulated as:

$\nabla_{\theta,\phi}\mathcal{L}_{\mathrm{DDS}}=\nabla_{\theta,\phi}\mathcal{L}_{\mathrm{SDS}}(\mathbf{z}^{i},y_{src})-\nabla_{\theta,\phi}\mathcal{L}_{\mathrm{SDS}}(\tilde{\mathbf{z}}^{i},y_{trg}).$
(10)

Although the DDS formulation improves the performance, using vanilla DDS leads
to excessive changes in unwanted areas and inconsistency between two different
scenes. Therefore, we propose an additional binary mask for utilizing DDS in
3D images. The objective function that combines the binary mask $\mathcal{M}$
and DDS is as follows:

$\nabla_{\theta,\phi}\mathcal{L}_{\mathrm{MDDS}}=\mathcal{M}\cdot(\nabla_{\theta,\phi}\mathcal{L}_{\mathrm{DDS}}),$
(11)

where $\cdot$ denotes the pixel-wise multiplication and $\mathcal{M}$ is the
conditional binary mask of the specific region of the target prompt to change.
This mask is generated by utilizing off-the-shelf text prompt segmentation
models such as CLIPSeg (Lüddecke & Ecker, 2022) and SAM (Kirillov et al.,
2023) to segment the target region by a text prompt.

Despite the use of a binary mask, masked DDS loss $\nabla\mathcal{L}_{MDDS}$
update all parameters of NeRF potentially affecting even undesired areas. As a
result, solely depending on the masked DDS loss may inadvertently result in
alterations beyond the mask boundaries. Hence, we introduce an additional
reconstruction loss as follows to mitigate undesired deformations beyond the
mask.

$\mathcal{L}_{\mathrm{Mrec}}=\lambda_{im}\cdot\mathcal{M}\cdot\mathcal{L}_{\mathrm{rtot}}+\lambda_{om}\cdot(1-\mathcal{M})\cdot\mathcal{L}_{\mathrm{rtot}}.$
(12)

Finally, the total editing loss is as follows:

$\mathcal{L}_{\mathrm{tot}}=\mathcal{L}_{\mathrm{MDDS}}+\mathcal{L}_{\mathrm{Mrec}}$
(13)

By suppressing undesired alterations through the use of the masked
reconstruction loss $\mathcal{L}_{Mrec}$, our total editing objective function
updates NeRF and refinement layer $F_{\theta}$ and $F_{\phi}$, ensuring NeRF
renders novel views in accordance with the desired text conditions.

##  4 Experimental Results

Figure 4: Comparison with baseline models. ED-NeRF demonstrates outstanding
performance in effectively altering specific objects compared to other models.
Baseline methods often failed to maintain the region beyond the target objects
and failed to guide the model towards the target text.

###  4.1 Baseline methods

To comprehensively evaluate the performance of our method, we perform
comparative experiments comparing it to state-of-the-art methods. As CLIP-
based text guidance editing methods, we used CLIP-NeRF (Wang et al., 2022) and
NeRF-ART (Wang et al., 2023a). CLIP-NeRF encodes the images rendered by NeRF
to the CLIP embedding space, allowing it to transform the images according to
the text condition. As an improved method, NeRF-ART trains NeRF with various
regularization functions to ensure that CLIP-edited NeRF can preserve the
structure of the original NeRF. For fair experiments, we re-implemented the
methods to TensoRF backbone, referencing the official source codes. For the
diffusion-based editing, we chose Masked SDS (Poole et al., 2022) and
InstructNeRF2NeRF (Haque et al., 2023) as the methods that target local
editing. In the masked SDS setting, we fine-tuned the pre-trained NeRF with
applying basic SDS loss only to the masked regions so that the NeRF model is
locally edited. InstructNeRF2NeRF (Haque et al., 2023) leverages the powerful
generation capabilities of diffusion models to sequentially modify the entire
dataset to align with text conditions and use the modified dataset as a new
source for NeRF training. We utilized a database comprising real-world images,
including LLFF (Mildenhall et al., 2019) and IBRNet (Wang et al., 2021)
datasets, as well as the human face dataset employed in Instruction-NeRF2NeRF
(Haque et al., 2023).

###  4.2 Qualitative Results

Text-guided editing of 3D scenes. As shown in Figure 1, our method shows its
capability to edit various image types with different textual contexts.
Specifically, when altering 3D scenes, it is possible to achieve the effective
transformation of specific objects without affecting other parts. Our baseline
method InstructNeRF2NeRF (Haque et al., 2023) shows decent results with high
consistency between images and text conditions, as well as view consistency
across scenes. However, it faces limitations in accurately transforming the
specific objects to match text conditions and may introduce undesired image
alterations beyond the specific objects. In Masked SDS, the output accurately
edits the local objects towards target text conditions, but the edited output
fails to reflect the structure of the original NeRF structure and shows
unwanted artifacts. In the case of NeRF-ART, the entire image is embedded into
the CLIP space, and it does not inherently recognize and modify only specific
objects. Therefore, it exhibits limitations in recognizing and altering
specific objects. CLIP-NeRF also encodes the images rendered by NeRF to the
CLIP embedding space, allowing it to transform the images according to the
text condition. However, its performance falls short when it comes to altering
specific parts or objects in a similar manner. On the other hand, our ED-NeRF
exhibited powerful abilities in editing 3D scenes by specifying certain parts
through text, surpassing other models. It not only excelled in changing
objects but also demonstrated the capability to faithfully follow and modify
areas that are not objects, such as the ground, in accordance with the text
condition.

Figure 5: Ablation studies. (a) If we only use DDS loss, the model fails to
maintain the attribute of untargeted regions and often fails to reflect text
conditions. (b) If we do not use masked reconstruction regularization, again
the regions beyond the target objects are excessively changed. (c) If we
remove the mask from DDS, unwanted artifacts occur in untargeted regions. (d)
With removing the proposed refinement layer, the results become blurry as the
backbone NeRF cannot fully embed real-world scenes. Our proposed setting can
modify a specific region in a 3D scene and follow the target word without
causing unwanted deformations.

###  4.3 Quantitative Results

CLIP Directional Score. In order to quantitatively measure the editing
performance, we show the comparison results using CLIP Directional scores(Gal
et al., 2021). The CLIP Directional score quantifies the alignment between
textual caption modifications and corresponding image alterations. We rendered
multiple view images from NeRF and measured the average score over images.
When compared to baseline methods, our model obtained the best similarity
scores. The result indicates that our edited NeRF accurately reflects the
target text conditions.

User Study. In order to further measure the perceptual preference of human
subjects, we conducted an additional user study. For the study, we rendered
images from edited NeRF using 5 different scenes from LLFF and IBRnet. We
gathered feedback from 20 subjects aged between their 20s and 40s. Each
participant was presented with randomly selected multi-view renderings from
our model and baselines and provided feedback through a preference scoring
survey. We set the minimum score as 1 and the maximum score is 5, and users
can choose the score among 5 options: 1-very low, 2-low, 3-middle, 4-high,
5-very high. To measure the performance of editing, we asked two questions for
each sample: 1) Does the image reflect the target text condition?(Text score)
2) Does the model accurately edit the target object?(Preservation). 3) Does
the 3D scenes preserve view consistency?(view consistency). In Table 1, we
show the user study results. Compared with baseline methods, our method showed
the best score in text score and preservation, and second best in view
consistency. Overall, ours outperformed the baseline models in perceptual
quality.

Metrics | CLIP-NeRF | NeRF-Art | Instruct N2N | Mask SDS | Ours
---|---|---|---|---|---
CLIP Direction Score $\uparrow$ | 0.1648 | 0.1947 | 0.2053 | 0.1409 | 0.2265
Text score$\uparrow$ | 2.56 | 3.20 | 3.29 | 3.14 | 3.88
Preservation $\uparrow$ | 2.30 | 2.97 | 3.08 | 2.76 | 4.09
view consistency $\uparrow$ | 3.21 | 3.79 | 3.28 | 3.56 | 3.64
Table 1: Quantitative Comparison. We compared the text-image similarity
between the target text and rendered output from edited NeRF (CLIP Directional
Score). Also, we show the user study results on three categories of text-
guidance score, source preservation score , and view consistency. The results
show that ours shows improved perceptual score among baseline models. Figure
6: Ablation study on refinement layer. Novel view synthesis results of our ED-
NeRF method. An additional refinement layer enhances the performance of
rendering quality compared to naive training cases.

###  4.4 Ablation Studies

To evaluate our proposed components, we conducted an ablations study in Figure
5. (a) If we only use DDS, the method fails to maintain the untargeted regions
with artifacts, even failing in training (e.g. fossil). (b) If we do not use
regularization $\mathcal{L}_{Mrec}$, the edited results show the target text
attribute, but again the regions beyond the target objects are severely
degraded. (c) When we remove mask guidance on DDS, (w/o $\mathcal{L}_{MDDS}$),
unwanted minor deformations occur due to the gradient of DDS affecting the
regions outside the mask. (d) When we remove our refinement layer, the results
show blurry outputs, which indicate that latent NeRF is not accurately
trained. When we utilize all the components we proposed, we can reliably
transform the 3D scene into the desired target object while preserving the
original structure source NeRF. In order to further show the effect of our
proposed refinement layer, we compare the reconstruction quality in Figure 6.
We can observe that the reconstruction output from our refinement layer shows
fine details when compared to the basic setting. As already shown in our
previous ablation study in Figure 5, the refinement layer improves the latent
space NeRF training which is crucial in the final edited output.

##  5 Conclusion

In this paper, we introduced a novel ED-NeRF method optimized in the latent
space. By enabling NeRF to directly predict latent features, it efficiently
harnesses the text-guided score function of latent diffusion models without
the need for an encoder. By doing so, our approach is able to effectively
reduce computation costs and address the burden of previous models that
required rendering at full resolution to utilize the diffusion model. We
extended the strong 2D image editing performance of DDS to the 3D scene and
also introduced a new loss function based on the mask. As a result, it showed
high performance in object-specific editing, a task that previous models
struggled with. We experimented with our proposed approach across various
datasets, and as a result, it demonstrated strong adherence to text prompts in
diverse scenes without undesired deformation.

##  6 Ethics and Reproducibility Statements

Ethics statement. ED-NeRF enables efficient and accurate text-guided NeRF
editing, which can be applied to various applications. However, our ED-NeRF
can be used for creating obscene objects which may cause the users to feel
offended. In order to prevent the possible side effects, we can use a filtered
diffusion model that does not contain malicious text conditions.

Reproducibility statement. We detailed our experimental process and parameter
settings in our Appendix. We will upload our source code to an anonymous
repository for reproduction.

## References

  * Bao et al. (2023) Chong Bao, Yinda Zhang, Bangbang Yang, Tianxing Fan, Zesong Yang, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui.  Sine: Semantic-driven image-based nerf editing with prior-guided editing field.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20919–20929, 2023.
  * Barron et al. (2021) Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 5855–5864, 2021.
  * Chen et al. (2022) Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su.  Tensorf: Tensorial radiance fields.  In _European Conference on Computer Vision_ , pp. 333–350. Springer, 2022.
  * Fridovich-Keil et al. (2022) Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa.  Plenoxels: Radiance fields without neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5501–5510, 2022.
  * Gal et al. (2021) Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  _arXiv preprint arXiv:2108.00946_ , 2021.
  * Haque et al. (2023) Ayaan Haque, Matthew Tancik, Alexei A Efros, Aleksander Holynski, and Angjoo Kanazawa.  Instruct-nerf2nerf: Editing 3d scenes with instructions.  _arXiv preprint arXiv:2303.12789_ , 2023.
  * Hertz et al. (2023) Amir Hertz, Kfir Aberman, and Daniel Cohen-Or.  Delta denoising score.  _arXiv preprint arXiv:2304.07090_ , 2023.
  * Jain et al. (2022) Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 867–876, 2022.
  * Karnewar et al. (2022) Animesh Karnewar, Tobias Ritschel, Oliver Wang, and Niloy Mitra.  Relu fields: The little non-linearity that could.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pp. 1–9, 2022.
  * Kim et al. (2023) Hyunsu Kim, Gayoung Lee, Yunjey Choi, Jin-Hwa Kim, and Jun-Yan Zhu.  3d-aware blending with generative nerfs.  _arXiv preprint arXiv:2302.06608_ , 2023.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kuang et al. (2023) Zhengfei Kuang, Fujun Luan, Sai Bi, Zhixin Shu, Gordon Wetzstein, and Kalyan Sunkavalli.  Palettenerf: Palette-based appearance editing of neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20691–20700, 2023.
  * Liu et al. (2023a) Kunhao Liu, Fangneng Zhan, Yiwen Chen, Jiahui Zhang, Yingchen Yu, Abdulmotaleb El Saddik, Shijian Lu, and Eric P Xing.  Stylerf: Zero-shot 3d style transfer of neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8338–8348, 2023a.
  * Liu et al. (2023b) Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  _arXiv preprint arXiv:2303.11328_ , 2023b.
  * Liu et al. (2021) Steven Liu, Xiuming Zhang, Zhoutong Zhang, Richard Zhang, Jun-Yan Zhu, and Bryan Russell.  Editing conditional radiance fields.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 5773–5783, 2021.
  * Lüddecke & Ecker (2022) Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7086–7096, 2022.
  * Metzer et al. (2023) Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12663–12673, 2023.
  * Mildenhall et al. (2019) Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar.  Local light field fusion: Practical view synthesis with prescriptive sampling guidelines.  _ACM Transactions on Graphics (TOG)_ , 2019.
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Müller et al. (2022) Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  _ACM Transactions on Graphics (ToG)_ , 41(4):1–15, 2022.
  * Poole et al. (2022) Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Reiser et al. (2021) Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger.  Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 14335–14345, 2021.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Sella et al. (2023) Etai Sella, Gal Fiebelman, Peter Hedman, and Hadar Averbuch-Elor.  Vox-e: Text-guided voxel editing of 3d objects.  _arXiv preprint arXiv:2303.12048_ , 2023.
  * Sun et al. (2022) Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5459–5469, 2022.
  * Wang et al. (2022) Can Wang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  Clip-nerf: Text-and-image driven manipulation of neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3835–3844, 2022.
  * Wang et al. (2023a) Can Wang, Ruixiang Jiang, Menglei Chai, Mingming He, Dongdong Chen, and Jing Liao.  Nerf-art: Text-driven neural radiance fields stylization.  _IEEE Transactions on Visualization and Computer Graphics_ , 2023a.
  * Wang et al. (2021) Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, and Thomas Funkhouser.  Ibrnet: Learning multi-view image-based rendering.  In _CVPR_ , 2021.
  * Wang et al. (2023b) Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  _arXiv preprint arXiv:2305.16213_ , 2023b.
  * Xu et al. (2023) Jiale Xu, Xintao Wang, Weihao Cheng, Yan-Pei Cao, Ying Shan, Xiaohu Qie, and Shenghua Gao.  Dream3d: Zero-shot text-to-3d synthesis using 3d shape prior and text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20908–20918, 2023.
  * Yuan et al. (2022) Yu-Jie Yuan, Yang-Tian Sun, Yu-Kun Lai, Yuewen Ma, Rongfei Jia, and Lin Gao.  Nerf-editing: geometry editing of neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18353–18364, 2022.
  * Zhang et al. (2022) Kai Zhang, Nick Kolkin, Sai Bi, Fujun Luan, Zexiang Xu, Eli Shechtman, and Noah Snavely.  Arf: Artistic radiance fields.  In _European Conference on Computer Vision_ , pp. 717–733. Springer, 2022.

##  Appendix A Experimental Details

Training ED-NeRF details. For optimizing NeRF in Latent space, we use TensoRF
as backbone for fast and efficient rendering with up to 64 resolution by
supervision. As stated in the original TensoRF, we use Adam optimizer. For RGB
NeRF, the color values to predict are constrained within the range of 0 to 1.
In the case of NeRF trained in the latent space, the range of latent values is
at least between -3 and 3, which leads to faster convergence than density.
Therefore, we set the learning rate for trainable density voxels, which form
the sigma in the original TensoRF, to 0.04, while keeping the learning rates
for other components the same as TensoRF at 0.02 during training. During the
training process, ED-NeRF needed to form a feature map of size 64 resolution,
therefore we configured the batch size to be 4096-pixel rays. We train ED-NeRF
in latent space with source latent feature map supervision for 100k steps on
the single NVIDIA RTX 3090. When fine-tuning ED-NeRF for editing purposes, we
train ED-NeRF for 5k steps. We set $\lambda_{om}$ as 100 and $\lambda_{im}$ as
0.01 to strongly preserve the regions outside the mask while editing and
reflecting the objects within the mask.

Generate source image and latent feature map. Datasets such as LLFF and IBRNet
typically contain categories with a limited quantity of images. As mentioned
earlier, due to the lack of guaranteed geometric consistency within the latent
space, synthesizing views using NeRF with a limited dataset is an
exceptionally challenging task. To address this issue from an editing
perspective, we leveraged a pre-trained NeRF model trained on RGB images. We
employed publicly available NeRF models for novel view synthesis, and when
required, we trained our own RGB NeRF. Using this pre-trained NeRF, we
generated novel view images by capturing over 100 views along a spiral camera
path. Subsequently, these generated images were utilized as part of the source
image dataset, denoted as $I$. The required number of iterations for
convergence to the target scene can vary depending on the dataset, influenced
by the provided images and text prompts.

##  Appendix B Time Comparison

To evaluate the effectiveness of our proposed ED-NeRF, we compared the overall
fine-tuning time of text-guided editing. In the baseline methods, as the
models apply editing on full-resolution NeRF backbones, they require a much
longer time to obtain reasonable edited outputs. Especially for the Instruct-
NeRF2NeRF, the overall editing time is over 30 minutes, as there should be
periodic replacement of edited 2D images. Therefore the training time is much
longer than other baselines. In the case of CLIP-NeRF, the editing time is
relatively short (about 6 min) compared to other baselines, but the overall
quality is degraded as shown in our main script. Although NeRF art uses a
similar training scheme to CLIP-NeRF, the model takes longer time over 15
minutes as the method uses additional regularizations for editing. In Masked
SDS the average editing time is about 8 minutes, but again the output quality
is not better than other methods. In our case, when it comes to easy cases
such as color change the fine-tuning time takes less than 3 minutes. In more
complex cases, the training time is up to 14 minutes. In the case of CLIP-Nerf
and mask-SDS, the longer training could not get improvement. Compared with
other state-of-the-art baselines such as Instruct NeRF2NeRF and Nerf-Art, our
method shows better efficiency in training. Furthermore, we compare the time
it takes for the existing RGB NeRF model and ED-NeRF to generate 512x512 RGB
images. Based on the LLFF dataset, TensoRF takes an average of 2.7143 seconds
to render a single 512-resolution image. In contrast, our ED-NeRF, which
renders the latent feature map and decodes it into an RGB image, takes an
average of 1.1240 seconds per image. As a result of training in the latent
space and reducing the resolution, ED-NeRF shortens the inference time by 2.4
times.

Figure 7: Failure case. The diffusion model is a 2D-based image generation
model, so it may not consistently perform well in 3D image editing tasks. In
this example, we aimed to change ‘statue’ to ‘black hole under a statue’ and
‘a hat’ to ‘a burning hat.’

##  Appendix C Limitations

Since we leverage a pre-trained Stable Diffusion model, there should be
limitations in the generative performance of the diffusion model itself. For
example, as shown in Figure 7if we try to apply text conditions that are very
far from the source object,(e.g., ‘a statue’ to ‘black hole under a statue’)
sometimes the output quality is degraded. This may be solved when we use an
improved version of the latent diffusion model.

##  Appendix D Additional Results

To further show the edited results, we show multi-view rendered outputs from
our edited NeRF in Figure.8. The top scenes are altered from a reference to
‘statue’ to ‘gold statue’ and the scene just below it is changed from ‘ground
under the statue’ to ‘water pond ground under the statue.’ In the case of the
car images, ‘SUV’ is modified to ‘gold SUV,’ followed by ‘shallow water pond
under the SUV.’ Lastly, in the picture of the man, it changes from ‘a man in
the gray sweater’ to ‘a man in the batman suit,’ and subsequently to ‘Einstein
in the gray sweater.’ Through our unique method, we were able to achieve a
very high-quality edited 3D scene. In particular, our model could even depict
objects in the original scene being reflected in the water when we changed the
target region to a ‘water pond.

Figure 8: Edited scene in various views. We conduct an experiment where we
manipulate different objects with text prompts in various views. It
demonstrates the ability to edit various elements within 3D scenes and verify
their accurate transformation in response to the provided text prompts. ED-
NeRF has a depth map for the training scene in the latent space, with a
resolution of 64. For visualization purposes, we resize it to 504x378
resolution in the above image.

In Figure 9, we present the rendering results of the ED-NeRF trained in the
latent space. These results represent the state before changing the image with
the target text, demonstrating that ED-NeRF can effectively represent real-
world 3D scenes with a 64-resolution latent feature map and depth map. As
depicted in Figure 2 (c), the RGB image is obtained by rendering the
64-resolution feature map with ED-NeRF, followed by decoding to generate a
512-resolution image, which is then resized to $504\times 378$. The depth map,
on the other hand, is directly resized from 64 resolution to $504\times 378$.

Figure 9: Visualized latent feature and depth map. As illustrated in the
figure, the ED-NeRF trained in the latent space is capable of generating a
high-quality depth map. We show visualizations of the features rendered by ED-
NeRF from various scenes and angles before the editing phase. RGB images are
the result of decoding latent feature maps rendered by our NeRF using Stable
Diffusion, and they are resized from $512\times 512$ resolution to $504\times
378$ resolution for visualization purposes. Similarly, the depth maps are
formed alongside the latent feature map at a resolution of $64\times 64$, and
it has also been resized to $504\times 378$ resolution for visualization.


# The Problem of Fragmented Occlusion in Object Detection

Julian Pegoraro1, Roman Pflugfelder1,2
1 AIT Austrian Institute of Technology, 1,2 TU Wien
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Object detection in natural environments is still a very challenging task,
even though deep learning has brought a tremendous improvement in performance
over the last years. A fundamental problem of object detection based on deep
learning is that neither the training data nor the suggested models are
intended for the challenge of fragmented occlusion. Fragmented occlusion is
much more challenging than ordinary partial occlusion and occurs frequently in
natural environments such as forests. A motivating example of fragmented
occlusion is object detection through foliage which is an essential
requirement in green border surveillance. This paper presents an analysis of
state-of-the-art detectors with imagery of green borders and proposes to train
Mask R-CNN on new training data which captures explicitly the problem of
fragmented occlusion. The results show clear improvements of Mask R-CNN with
this new training strategy (also against other detectors) for data showing
slight fragmented occlusion.

##  1 Introduction

Automated surveillance at green borders has become a hot topic for European
border guards. Border guards today face several challenges in protecting EU
borders. One well known occasion in public is illegal migration which had its
peak in 2015.

Border surveillance today limited to 2D imaging sensors consists of color and
thermal cameras, mounted on poles or used as handheld cameras by the border
guards. Innovating these technical systems by adding further capabilities of
automatic inference such as the automatic detection of persons, vehicles,
animals and suspicious objects in general will need to apply object detectors
to such imagery.

However, video of green borders especially at EU borders show significant
differences to typical imagery of video surveillance such as indoor video or
video taken in man-made outdoor scenes. For example, green borders are scenes
showing dense forest, hills, harsh weather and climate conditions. Such scenes
draw challenges to automated surveillance and raise several interesting
research questions.

Figure 1: The problem of fragmented occlusion in object detection. Top Left:
no occlusion (level $L_{0}$). Top Right: slight occ. ($L_{1}$). Bottom Left:
moderate occ. ($L_{2}$). Bottom Right: heavy occ. ($L_{3}$) occlusion.

This paper considers a challenge for state-of-the-art object detection in
green border surveillance which is the problem of through foliage detection.
To the best of our knowledge, none of the current approaches for object
detection allow the detection of objects through foliage. This problem raises
an interesting scientific question, namely how to detect objects with
fragmented occlusion? This problem is also different to the problem of partial
occlusion in object detection. Fragmented occlusion occurs by viewing objects
behind tree ans bush leaves. Contrary to partial occlusion, fragmented
occlusion gives no clear view on minimal recognisable parts of the object [10]
which is used to detect the object [7].

We show in this work that the state-of-the-art in object detection fails on
fragmented occlusion even for the moderate case. For this, we created a new
dataset (Figure 1) capturing people behind trees. We labelled nearly 40,000
images in three representative videos. This data raises new challenges on the
labelling and evaluation which we only partially answer in this paper. For
example, bounding boxes are the standard in current evaluation of detectors
but such labels are hard to find in data that contains fragmented occlusion.
As the state-of-the-art detectors deliver bounding boxes, fragmented occlusion
poses new questions on the evaluation methodology.

Furthermore, we augmented Microsoft COCO111http://cocodataset.org training
data by occluding the ground truth masks similarly as leaves occlude people
behind bushes and trees. We then show results on training Mask R-CNN [4] on
this new data showing improvement of Mask R-CNN trained on the original data
with slight fragmented occlusion.

##  2 Related Work

State-of-the-art object detection is based on deep learning. Two-stage
detectors work by finding as an intermediate step bounding box proposals [3,
2] on the feature maps of the backbone CNN. A region proposal network further
improves efficiency [9, 4]. One-stage detectors regress the bounding boxes
directly [8, 6] which is computationally efficient on GPUs but this approach
is inherently less accurate as it assumes a coarsely discretised search space.
Although these methods show usually excellent performance for fully visible
objects, they break down in the case of fragmented occlusion. Fragmented
occlusion has not been considered for object detection so far, however there
is literature about this topic in the field of motion analysis [1].

##  3 Methodology

We created a dataset recorded in a forest consisting of three videos with a
total of 18,360 frames and 33,933 bounding boxes which were manually defined
by human annotators. These bounding boxes are divided into four different
occlusion levels including the unoccluded case (Figure 1).

Figure 2: A training image from Microsoft COCO
(http://images.cocodataset.org/train2017/000000001700.jpg). Top Left: the
image. Top Right: Segmentation mask of the image. Bottom Left: image overlaid
with artificial trees. Bottom Right: Mask of the overlaid image.

Then, we extended the Microsoft COCO dataset by adding artificial trees as
foreground to the images of objects (Figure 2). We chose this dataset, because
it contains pixel-wise segmentation masks in the ground truth as well as a
large number of different categories including the human person.

The underlying basic idea of our approach is to add artificial fragmented
occlusion to Microsoft COCO and train Mask R-CNN on this new data. By this we
can adapt the original distribution of data to the case of fragmentally
occluded objects. Since we are only interested in humans, we apply this
augmentation only to images containing humans and use only these images for
training. The trees used for the augmentation are generated from real images
we have obtained from the test data. The method generates whole artificial
trees by randomly adding branches to previously manually segmented tree
trunks. In total 14 such trunks are extracted from the test dataset. The
branches attached to these trunks are also randomly generated by also adding a
few manually segmented leaves.

The trees are placed in front of objects by randomly selecting the
x-coordinate on which they will be placed and an angle at which the tree will
be rotated. The calculated foreground is applied to the image and its negative
mask is multiplied by the segmentation mask of the objects in the image. The
Mask R-CNN model is then trained with the augmented images. The selected
backbone model is the Inception v2 [5] network. This network is selected for
its faster computation.

##  4 Evaluation

To evaluate whether training with the augmented dataset is useful, the model
trained on the augmented data must be compared with the model not trained on
this data. However, the intersection over Union (IoU) measure is not
meaningful in this case.

Standard evaluation metrics such as the mean average precision (mAP) define an
IoU threshold (e.g. 0.5) and check whether a ground truth object and a
detected object have an IoU value above this value. If this is the case, the
detected object is defined as a True Positive (TP). If an object is detected
but there is no respective ground truth with an IoU above this specific
threshold, the detected object is defined as a False Positive (FP). If there
is ground truth but no detected object with an IoU above the threshold, the
object is defined as a False Negative (FN).

These evaluation methods cannot be easily applied to ground truth showing
fragmented occlusion, because of the following two observations:

IoU too small: Since the data is based on fragmented detections, a detector
can only detect parts of the person. An image where this problem occurs is
shown in Figure 3. The bounding box is clearly a TP, based on the fact, that
fragmented objects should be detected, but due to the occlusion by the
branches of the tree, the whole body cannot be recognized. This leads to an
IoU of only $\approx 0.2$.

Figure 3: Ground truth (green) and the detection (blue) vary substantially due
to the occlusion effects.

Multiple detections: Another major problem with the standard evaluation
metrics is that exactly one detected bounding box and one ground truth
bounding box match. However, when handling fragmented objects, human heads
and/or other body parts should be detected separately if body parts are
covered. This creates the problem that parts of the body (like a head) is
detected as well as the whole body. Figure 4 shows some examples.

Figure 4: The problem of multiple detections. Ground truth is shown in green.
Left: state-of-the-art yields two bounding boxes of the same, single person.
Middle: two persons are visible. Detection yields two bounding boxes which are
diffucult to associate. Right: an even harder case with three persons.

To tackle these two problems, this paper proposes a different evaluation
metric. For each bounding box in the evaluation data set, we calculate the
maximum region in the image where there is no overlap with another ground
truth bounding box. This region is then extracted and fed into the model. If
the model detects an object, we define it as TP, otherwise as FN. To assess
FPs, we create an additional dataset that represents the maximum region in an
image without overlap with any ground truth bounding box. We extracted in
total 45,340 such regions with different aspect ratios, different parts of the
image and at different time instants. In addition to FPs, we can also
calculate the TNs using this evaluation metrics.

Figure 5 shows these results as recall vs. precision curve (ROC). There is no
significant difference between Mask R-CNN trained on Microsoft COCO and on the
augmented dataset for $L_{0}$ occlusion. However, clear improvement has been
achieved for $L_{1}$ and $L_{2}$ occlusion which proves the applicability of
the idea to model fragmented occlusion by the masks. Nevertheless, all
approaches basically do not reach the expected robustness and accuracy for
moderate $L_{2}$ and heavy $L_{3}$ occlusion. One reason for this is that our
current technique is not accurate enough to model fragmented occlusion.
Furthermore, clear limits exist as heavy fragmented occlusion removes local
spatial and structural information necessary for current approaches in object
detection.

We further recognise that bounding box labelling is not the appropriate
approach for labelling data showing fragmented occlusion. Especially for
$L_{3}$ and $L_{4}$ occlusion, it is frequently impossible to manually define
the bounding box. Such occlusion levels allow an approximate localisation of
the object in the image but make the observation of the object’s extent
impossible. While the recall in Figure 5 is still meaningful, the precision is
basically undefined. This observation has severe consequences on the
labelling, but also on the evaluation and on the detector which we leave open
for future research.

Figure 5: This ROC plot shows results of Faster R-CNN (green), YOLO (blue),
Mask R-CNN (red) and our method (purple) for all occlusion levels.

##  5 Conclusion

This paper formulates a new scientific question on object detection with
fragmented occlusion which is different to partial occlusion. We show by a
study that current object detectors fail in this case. We generated and
labelled a new dataset showing people behind trees in a forestry environment.
Such scenes frequently occur in border surveillance which has become very
important in EU security policies. We try to tackle the occlusion challenge by
augmenting Microsoft COCO including the pixel-wise segmentation masks to
capture the occlusion problem. We show that Mask R-CNN trained on this data
improves on fragmented occlusion, however, we also observe severe loss of
spatial, structural information and that the bounding box itself is not the
appropriate description to cope with fragmented occlusion. This has severe
implications on the detection approach itself, but also on dataset labelling
and evaluation. A potential solution is left open for future work.

## Acknowledgments

This research was supported by the European Union H2020 programme under grant
agreement FOLDOUT-787021. We thank all our students on internship to label the
new dataset.

## References

  * [1] M. Black and P. Anandan.  The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields.  Computer Vision and Image Understanding, 63:75–104, 01 1996.
  * [2] R. Girshick.  Fast r-cnn.  In CVPR, pages 1440–1448, 2015.
  * [3] R. Girshick, J. Donahue, T. Darrell, and J. Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In CVPR, pages 580–587, 2014.
  * [4] K. He, G. Gkioxari, P. Dollár, and R. Girshick.  Mask r-cnn.  In CVPR, pages 2961–2969, 2017.
  * [5] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [6] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg.  Ssd: Single shot multibox detector.  In ECCV, pages 21–37. Springer, 2016.
  * [7] G. Nebehay and R. Pflugfelder.  Clustering of static-adaptive correspondences for deformable object tracking.  In CVPR, June 2015.
  * [8] J. Redmon and A. Farhadi.  Yolov3: An incremental improvement.  arXiv, 2018.
  * [9] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In NIPS, pages 91–99, 2015.
  * [10] S. Ullman, L. Assif, E. Fetaya, and D. Harari.  Atoms of recognition in human and computer vision.  PNAS, 113(10):2744–2749, 2016.