11institutetext: Visual Geometry Group, University of Oxford
11email: {minghao, iro<EMAIL_ADDRESS>
silent-chen.github.io/DGE

(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

# DGE: Direct Gaussian 3D Editing by Consistent Multi-view Editing

Minghao Chen  Iro Laina  Andrea Vedaldi

###### Abstract

We consider the problem of editing 3D objects and scenes based on open-ended
language instructions. The established paradigm to solve this problem is to
use a 2D image generator or editor to guide the 3D editing process. However,
this is often slow as it requires do update a computationally expensive 3D
representations such as a neural radiance field, and to do so by using
contradictory guidance from a 2D model which is inherently not multi-view
consistent. We thus introduce the Direct Gaussian Editor (DGE), a method that
addresses these issues in two ways. First, we modify a given high-quality
image editor like InstructPix2Pix to be multi-view consistent. We do so by
utilizing a training-free approach which integrates cues from the underlying
3D geometry of the scene. Second, given a multi-view consistent edited
sequence of images of the object, we _directly_ and efficiently optimize the
3D object representation, which is based on 3D Gaussian Splatting. Because it
does not require to apply edits incrementally and iteratively, DGE is
significantly more efficient than existing approaches, and comes with other
perks such as allowing selective editing of parts of the scene.

##  1 Introduction

AI-based content creation and editing have advanced substantially due to
recent breakthroughs in 2D and 3D generation [82, 33, 78]. The benefits are
particularly significant for 3D content, processing which normally requries
highly-specialized talent and software. AI-powered tools are unlocking new
creative possibilities for artists and non-professional users alike.

In this work, we consider the problem of editing 3D models based on textual
instructions. Part of the recent progress in this area is due to the
introduction of new radiance field representations [69, 43] for objects and
scenes, which are both high-quality and flexible. Because they are
differentiable, radiance fields combine well with recent advances in image
generation. A notable example is InstructPix2Pix (IP2P) [4], a denoising
diffusion model [89, 83] that can edit the content of an image based on
textual instructions [4]. This and similar models can be used to update
radiance fields, thus resulting in open-ended editing of 3D contents. So far,
this has been done primarily in two ways. The first was pioneered by
InstructNeRF2NeRF [28], which utilizes IP2P to alternate between editing one
rendered view of the 3D model and updating the latter. The second approach
[123, 40, 85] is to update the model in a distillation-like fashion [78].
Either way, edits are slowly incorporated by alternating between updating the
3D model and generating and processing different views of it, so that a single
edit often requires several minutes, and sometimes even hours, to complete.

In this paper, we introduce a new approach to 3D editing, the _Direct Gaussian
Editor_ (DGE), which addresses this and other shortcomings of these earlier
attempts. We design our method with three goals in mind: (i) high fidelity,
(ii) high efficiency, and (iii) selective editing (_i.e_., the ability to only
edit part of an object or a scene).

To achieve these goals, we propose to change both the representation of the 3D
model and the way it is updated. As a representation, we use Gaussian
Splatting (GS) [43]. This is an alternative representation of a radiance field
with the main obvious advantage of being orders of magnitude faster than NeRF
and NeRF-adjacent models [11, 91, 9], for both rendering and gradient
computation. There is another, less obvious but also important advantage of
GS: because it is an explicit 3D representation made of local 3D primitives,
the Gaussians, it supports easily and efficiently local edits, as long as one
knows which Gaussian(s) to modify. In practice, we can easily identify the
Gaussians by fusing the output of a 2D instance segmenter [45] from several
rendered views of the scene.

While GS can contribute to the speed of the 3D editor, it does not remove the
bottleneck of iterating several times between rendering, reconstruction and
evaluation of the underlying image-based diffusion model, meaning that edits
would still require a considerable amount of time to complete. Hence, our
second contribution is to develop a faster alternative to this slow iterative
process.

The reason why distillation-like methods are used in prior works is the lack
of a set of view-consistent edits to update the 3D model coherently. Because
2D editors like IP2P are _monocular_ and provide a distribution over all
possible edits, the probability of drawing independently two or more
consistent edits is nearly zero. Distillation is a way to incrementally reach
a “multi-view” consensus w.r.t the application of the monocular editor to the
different views.

Here, we suggest to use instead a method that can sample _multi-view
consistent_ edits such that the 3D model can be updated by _directly_ fitting
it to the edited views. Our approach to multi-view consistent editing is
inspired by recent progress in video generation and editing and, in
particular, methods that extend image generators to video without additional
training [23, 106, 44, 8, 80]. The key insight is that multiple views of the
3D model can be interpreted as a video of a static scene generated by a moving
camera. This then means that achieving multi-view consistency is analogous to
temporal consistency in video editing. To achieve multi-view consistency we
adopt the spatio-temporal attention mechanism used in the video editing
literature and extend it with additional epipolar constraints, which we can
enforce due to the 3D nature of our problem.

Through qualitative and quantitative comparisons to prior works, we
demonstrate two key advantages of DGE, even when compared to very recent works
like the GaussianEditor [15], which also use Gaussian Splatting. The first
advantage is that, as a direct editing approach, DGE is more efficient and
results in a noticeable speed boost compared to prior approaches
(approximately 4mins for a single edit). Second, ensuring multi-view
consistent editing in the image space significantly simplifies the process of
integrating edits from various views into the 3D model. This is then reflected
in both the number of updates that are required for convergence as well as the
higher fidelity of the final result.

##  2 Related Work

#### Image Editing.

Most methods for 3D image editing are based on 2D image editing due to the
lack of large scale training data to learn 3D editors directly, so we discuss
those first. GLIDE [74] controls 2D image generation and editing using CLIP
features [81]. Methods like [21, 49] and DreamBooth [84] consider the problem
of personalizing an image; GLIGEN [57] and others [10, 13, 20] perform layout
control and [118] considers other forms of control. DragDiffusion [86] and
Drag-a-Video [93] edit images and videos by dragging. Others [65, 72, 2, 96,
76, 42, 31] cast image editing as image-to-image translation. InstructPix2Pix
(IP2P) [5] fine-tunes Stable Diffusion [82] large synthetic dataset of example
language-driven edits. The works of [117, 120] further improve InstructPix2Pix
via manual labelling and the work of [70] aims at localizing the edits better.

#### Ad-hoc 3D Editing.

Several authors have explored various types of inputs and controls to edit 3D
objects. EditNeRF [60] updates shape and color in a radiance field based on
user-provided scribbles, 3Designer [54] and SINE [1] based on a single edited
view, SKED [67] based on 2D sketches, Editable-NeRF [121] based on keypoints
and CoNeRF [41] based on attributes. The work of [61] also considers sketch-
based editing, but for category-level 3D generators. NeRF-Editing [113],
NeuMesh [110] and NeRFShop [39] modify a radiance field based on meshes and
the work of [109] based on cages. NeuralEditor [12] does so based on point
clouds. N3F [95], DFF [46], SPIn-NeRF [71], NeRF-in [59] and the work of [105]
consider segmenting and removing objects from a radiance field. Control-NeRF
[50] allows both removing and moving objects. Component-NeRF [58] and the
works of [115, 111] consider compositional editing. Palette-NeRF [48],
RecolorNeRF [25], ICE-NeRF [51] address recoloring and ARF [116], DeSRF [108],
StylizedNeRF [37], SNeRF [73] and the work of [18, 36] style transfer.
NeRFEditor [92] integrates GAN-based stylization in NeRF. Seal-3D [104]
consider pixel-level editing of 3D scenes learning interactive tools like
brushes, deformation and recoloring and SealD-NeRF [38] extend it to dynamic
NeRF models. SceNeRFlow [94] allows for dynamic edits as well using DensePose
for correspondence estimation [27].

#### Open-ended 3D Editing.

Like us, several authors have considered open-ended and language-based 3D
editing. Some authors start from CLIP or other vision-language model [62, 53,
55] to drive editing/stylisation of a radiance field as a whole, like CLIP-
NeRF [101], NeRF-Art [102], Text2Mesh [66] and Tango [52], or in part [87, 47,
103, 26]. Some like AvatarCLIP [35] consider animations as well. Most recent
works start instead from diffusion-based image generators like StableDiffusion
[82] or editors like Instruct-Pix2Pix [4]. Most of these methods are
iterative. Some, like Instruct-NeRF2NeRF [29], InstructP2P [107], ProteusNeRF
[100], Edit-DiffNeRF [112], Vica-NeRF [19] edit views of the object and update
the 3D model accordingly. Others, like Instruct3Dto3D [40], DreamEditor [123],
Vox-E [85], ED-NeRF [75], FocalDreamer [56], Progressive3D [17] and [114,
122], do the same, but using score distillation sampling. TextDeformer [22]
focuses on manipulating only the shape of objects. Blending-NeRF [88] blends
an edited radiance field with the original one. SHAP-EDITOR [14] learns to
quickly edit 3D shapes encoded in a latent space, but require retraining for
each new set of instructions.

Our method allows direct editing, or substantially reduces the number of
iterations required. Furthermore, methods like Vox-E [85] and FocalDreamer
[56] consider local 3D editing, but struggle to handle both local and global
ones; in contrast, our DGE allows precise editing of small or large regions of
the 3D scene.

##  3 Preliminaries

#### Gaussian Radiance Fields.

A _radiance field_ is a pair of functions
$\sigma:\mathbb{R}^{3}\rightarrow\mathbb{R}_{+}$ and
$c:\mathbb{R}^{3}\times\mathbb{S}^{2}\rightarrow\mathbb{R}^{3}$ mapping 3D
points $\boldsymbol{x}\in\mathbb{R}^{3}$ to opacities $\sigma(\boldsymbol{x})$
and directional colors $c(\boldsymbol{x},\boldsymbol{\nu})$, where
$\boldsymbol{\nu}\in\mathbb{S}^{2}$ is a unit vector expressing the viewing
direction. to obtain an image $I$ of the radiance field, one uses the
_emission absorption_ equation:

$I(\boldsymbol{u})=\int_{0}^{\infty}c(\boldsymbol{x}_{t},\boldsymbol{\nu})\sigma(\boldsymbol{x}_{t})e^{-\int_{0}^{t}\sigma(\boldsymbol{x}_{\tau})d\tau}dt,$
(1)

where the 3D point $\boldsymbol{x}_{t}=\boldsymbol{x}_{0}-t\boldsymbol{\nu}$
sweeps the ray that connects the camera center $\boldsymbol{x}_{0}$ to the
pixel $\boldsymbol{u}$ along direction $-\boldsymbol{\nu}$ (where the 3D point
is expressed in the reference frame of the world, not the camera).

Prior works have explored several representations for these functions,
including MLPs [69], voxel grids [91] and low-rank factorizations of the
latter [11, 9]. _Gaussian Splatting_ (GS) [43] proposes yet another but very
efficient representation. GS is given by a mixture of Gaussians
$\mathcal{G}=\\{(\sigma_{i},\boldsymbol{\mu}_{i},\Sigma_{i},c_{i})\\}_{i=1}^{G}$,
where $\sigma_{i}\geq 0$ is the opacity,
$\boldsymbol{\mu}_{i}\in\mathbb{R}^{3}$ is the mean,
$\Sigma_{i}\in\mathbb{R}^{3\times 3}$ is the covariance matrix, and
$c_{i}:\mathbb{S}\rightarrow\mathbb{R}^{3}$ the directional color of each
Gaussian. The Gaussian functions are given by:

$g_{i}(\boldsymbol{x})=\exp\left(-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu}_{i})^{\top}\Sigma_{i}^{-1}(\boldsymbol{x}-\boldsymbol{\mu}_{i})\right).$
(2)

The directional colors are given by functions
$[c_{i}(\boldsymbol{\nu})]_{j}=\sum_{l=0}^{L}\sum_{m=-l}^{l}c_{ijlm}Y_{lm}(\boldsymbol{\nu}),$
where $Y_{lm}$ are spherical harmonics and $c_{ijlm}\in\mathbb{R}$ are
corresponding coefficients. The Gaussian mixture then defines the opacity and
color functions as:

$\sigma(\boldsymbol{x})=\sum_{i=1}^{G}\sigma_{i}g_{i}(\boldsymbol{x}),~{}~{}~{}c(\boldsymbol{x},\boldsymbol{\nu})=\frac{\sum_{i=1}^{G}c_{i}(\boldsymbol{\nu})\sigma_{i}g_{i}(\boldsymbol{x})}{\sum_{i=1}^{G}\sigma_{i}g_{i}(\boldsymbol{x})}.$
(3)

Importantly, the integral in Eq. 1 under the model in Eq. 3 can be
approximated very efficiently and in a differentiable manner, as noted in
[43].

#### Diffusion-based Generators and Editors.

A _diffusion-based image generator_ is a model that allows one to sample image
$I\in\mathbb{R}^{3\times H\times W}$ from a conditional distribution $p(I\mid
y)$. Here the condition $y$ is often a textual prompt, but it can also be an
image or a combination of both. Let
$I_{k}=\sqrt{1-\sigma_{k}^{2}}I+\sigma_{k}\epsilon$ be a noised version of the
image $I$ one wishes to sample, where $\sigma$ is a sequence monotonically
increasing from 0 to 1 and $\epsilon\sim\mathcal{N}$ is normal i.i.d. noise.
The diffusion model consists of a denoising neural network $\Phi$ that, given
the noisy image $I_{k}$, the noise level $k$ and the conditioning $y$,
estimates the noise $\hat{\epsilon}=\Phi(I,y,k).$ The image $I$ is sampled by
starting from pure noise ($\sigma_{T}=1$) followed by iterative denoising of
the signal until $I=I_{0}$ is obtained [34, 89, 98].

Several authors have suggested to use the denoising network of an off-the-
shelf image generator such as Stable Diffusion (SD) [83, 77] to modify an
existing image according to a prompt $y$ instead of generating it from
scratch, thus editing it. However, this process is somewhat suboptimal;
instead, we make use of editors specifically trained to apply such
modifications. An example is IP2P [6], which also uses diffusion but directly
implements conditional distribution $p(I^{\prime}\mid I,y)$ where $I^{\prime}$
is the edited image, $I$ is the original image, and $y$ is a prompt describing
how $I^{\prime}$ should be obtained from $I$. IP2P is in itself derived from
the SD model by finetuning it to training triplets $(I^{\prime},I,y)$ which
are automatically generated using Prompt-to-Prompt [32], a slow but training-
free editing method.

##  4 Method

In this section, we describe our method for text-guided 3D scene editing — the
Direct Gaussian Editor (DGE). Our main objective is 3D editing that is (i)
faithful to the text instructions (high fidelity), (ii) fast and efficient,
and (iii) partial to specific scene elements when desired. Next, we introduce
our approach and discuss how these characteristics are met.

###  4.1 Direct Gaussian Editor

DGE is motivated by the following key observation: if one can obtain _multi-
view consistent edits_ in image space, then the 3D model can be updated
_directly_ and efficiently by fitting it to the edited images. This approach
is alternative to slow iterative techniques like SDS [79] and iterative
dataset updates (IDU) [28] that work around inconsistent image-level edits.
Thus, the main challenge to allow direct 3D editing is to improve image-based
editors to produce several _consistent_ views of the edited objects or scenes.

In more detail, consider an initial reconstruction of a 3D scene in terms of a
mixture of Gaussians $\mathcal{G}$ (Sec. 3) and consider a camera trajectory
$(\pi_{t})_{t=1}^{T}$ consisting of $T$ viewpoints. We first render the
corresponding views $\mathcal{I}=(I_{t})_{t=1}^{t}$ where
$I_{t}=\operatorname{Rend}(\mathcal{G},\pi_{t})$ is obtained by utilizing Eq.
1. Let $y$ be the textual prompt that specifies the desired edits and let
$\mathcal{I}^{\prime}=(I^{\prime}_{t})_{t=1}^{T}$ be the images obtained by
editing images $\mathcal{I}$ according to $y$. We can then update the 3D model
$\mathcal{G}$ by fitting it to $\mathcal{I}^{\prime}$ according to

$\mathcal{G}^{\prime}=\operatorname*{arg\,min}_{\mathcal{G}}\sum_{t=1}^{T}\|I^{\prime}_{t}-\operatorname{Rend}(\mathcal{G},\pi_{t})\|$
(4)

This rendering loss is often used to reconstruct a 3D scene from multiple
views, and here it is used to update it. In practice, GS makes this
optimization efficient.

The key question is how to obtain the edited views $\mathcal{I}^{\prime}$. A
standard approach is to apply a diffusion-based image editor $\mathcal{E}$,
such as IP2P [107] to the individual views, obtaining edited views
$\mathcal{I}^{\prime}$. However, each of these views would then be drawn
_independently_ from the distribution $I^{\prime}_{t}\sim p(I^{\prime}_{t}\mid
I_{t},y).$ However, even though the initial views $I_{t}$ are consistent by
construction, the edited views would not be as this method disregards their
statistical dependency. As a result, updating the 3D model $\mathcal{G}$
according to Eq. 4 would fuse inconsistent views, yielding a blurry outcome.
We thus seek to modify the image editor to account for the mutual dependency
of the views and approximately draw samples from the joint distribution
$p(I^{\prime}_{1},\dots,I^{\prime}_{t}\mid I_{1},\dots,I_{T},y)$.

In summary, we divide the 3D editing process into two parts: (a) multi-view
consistent editing with epipolar constraints, and (b) 3D reconstruction from
edited images, which we detail next.

###  4.2 Consistent Multi-view Editing with Epipolar Constraint

In order to achieve multi-view consistent edits, we consider the views
$\mathcal{I}$ rendered from $\mathcal{G}$ as if they were arising from a
relatively smooth camera trajectory and thus form a _video_. In this way, we
reduce consistent multi-view editing to video editing.

While there is no off-the-shelf fast and video editor, we can take advantage
of recent methods that extend image generators to video generators without
additional training [24, 106, 44, 8, 80]. By applying them to image _editors_
instead, we can obtain corresponding video editors.

Let $\mathcal{E}$ be a diffusion-based image editor that utilizes self-
attention (_e.g_., SD-based). Based on the above insight, we propose to extend
$\mathcal{E}$ with a modified, spatio-temporal version of self-attention [99,
24], which jointly considers all frames in $\mathcal{I}$, effectively turning
the single-image editor into a multi-image editor
$\mathcal{E_{\operatorname{mv}}}$. Specifically, given queries
$\\{Q_{t}\\}_{t=1}^{T}$, keys $\\{K_{t}\\}_{t=1}^{T}$, and values
$\\{V_{t}\\}_{t=1}^{T}$ for each viewpoint $t$, the spatio-temporal attention
block ensures that each frame attends to every other frame and is given by:

$\operatorname{STAttn}(Q,K,t)=\operatorname{Softmax}\left(\frac{Q_{t}[\,K_{1},\dots,K_{T}]\
}{\sqrt{d}}\right),$ (5)

where $d$ is the embedding dimension of keys and queries. The corresponding
output features for frame $t$ are then computed as
$\Phi_{t}=\operatorname{STAttn}(Q,K,t)\cdot[\,V_{1},\dots,V_{T}]\ $ (at each
layer).

As the underlying image editor $\mathcal{E}$, we utilize IP2P [4], which is a
common choice in the literature. Following the framework of [23], each
denoising iteration of the editing process consists of two steps.

The first step is keyframe editing. Because attention can be computationally
expensive for longer sequences, $\mathcal{E_{\operatorname{mv}}}$ is only
applied to a number of keyframes. In practice, we first sort the cameras given
their pose matrix to have smooth views. Then, we randomly subsample the video
(viewpoint) sequence at each denoising iteration. We detail the sorting
process in the supplementary. This step results in roughly consistent edits
across all keyframes.

Figure 1: Overview. Our method is divided into two main parts: multi-view
consistent editing and direct 3D fitting. In the multi-view editing stage,
keyframes are randomly selected and fed to the editing diffusion network to
extract features. In the editing process of other frames, the features of
keyframes are injected into the diffusion network with the proposed epipolar
constraint; only features in the red boxes are considered for correspondence
matching to have a consistent appearance.

The second step is epipolar correspondence matching. The goal of this step is
to propagate the edited _keyframe_ features to all _other_ frames by finding
inter-frame correspondences. To find the correspondences, we leverage both
geometric and appearance cues as follows.

First, we extract visual features $\Psi_{t}$ for all frames, using
intermediate outputs (_i.e_., the inputs to each self-attention block) across
all layers in $\mathcal{E_{\operatorname{mv}}}$. Then, point correspondences
across frames can be established by simply comparing their respective features
at all spatial locations, and the keyframe features can be injected (based on
these correspondences) into all remaining frames, avoiding expensive self-
attention blocks across all frames.

However, we can also leverage the fact that we can directly use 3D information
to constrain the correspondence problem. In fact, we have multiple views of
the scene available, and the pose and calibration matrix of cameras are known.
This is without loss of generality since these assumptions are required to
apply GS for scene reconstruction in the first place. Therefore, we can
estimate the fundamental matrix $F$ between two views and constrain the
correspondence problem to points along an epipolar line.

Formally, given a feature map $\Psi_{t^{\prime}}$ corresponding to frame
$I_{t^{\prime}}$, where $t^{\prime}\notin\mathcal{K}$, and the features
$\Psi_{k^{*}}$ of a keyframe ${I_{k^{*}}}$, we compute a correspondence map
$M_{t^{\prime}}$ as

$\displaystyle M_{t^{\prime}}[u]$
$\displaystyle=\operatorname*{arg\,min}_{v,\,v^{\top}Fu=0}D\left(\Psi_{t^{\prime}}[u],\,\Psi_{k^{*}}[v]\right),\quad\forall\,t^{\prime}\in\mathcal{T}\backslash\mathcal{K}$
(6)

where $D$ is the cosine distance, $u$ and $v$ index the feature maps
spatially, $k^{*}$ is the index of the keyframe that is the closest to frame
$t^{\prime}$ (in terms of camera viewpoint), and $F$ is the fundamental matrix
corresponding to the two views $t^{\prime}$ and $k^{*}$. $Fu$ is the epipolar
line in view $t^{\prime}$ along which the corresponding point of $v$ in view
$k^{*}$ must lie [30]. Intuitively, using the epipolar constraint
significantly reduces the search space of correspondences from a plane to
spatial locations along a single line. Then the correspondence is decided
based on the cosine distance of the features.

In practice, we compute the correspondence maps between a frame $t^{\prime}$
and its two nearest neighbors. We then linearly combine the _edited_ features
of the corresponding keyframes to obtain
$\Phi_{t^{\prime}},\forall\,t^{\prime}\in\mathcal{T}\backslash\mathcal{K}$.

This approach is similar in spirit to TokenFlow [24] but contributes two
noteworthy improvements that are tailored to the 3D editing task in
particular. First, using IP2P instead of the 2D editor [97] used in TokenFlow
avoids the time-consuming DDIM [90] inversion process. Second, the epipolar
constraint in Eq. 6 allows us to leverage the 3D information that is available
from the original model $\mathcal{G}$ when propagating edits. This, in turn,
can be particularly useful in cases where appearance is insufficient to
compute correspondences (_e.g_., mostly uniform appearance, as shown in Fig.
4).

###  4.3 Direct Reconstruction and Iterative Refinement

After obtaining a set of edited images $\mathcal{I}$ with roughly consistent
content, we can directly fit $\mathcal{G}$ to the resulting images using Eq.
4, without the need for distillation. One of the notable advantages of 3D GS
over NeRF is that it enables the rendering of images at full resolution and
the application of robust image-level losses. Specifically, we optimize
$\mathcal{G}$ using LPIPS [119] as an objective function between the edited
$I^{\prime}_{t}$ and the rendered $\hat{I}_{t}$ views in Eq. 4. The LPIPS loss
is insensitive to small inconsistencies in the edited images, and helps to
obtain a more consistent 3D output.

Finally, we observe that even though $\mathcal{E_{\operatorname{mv}}}$ results
in generally consistent edits, some inconsistencies or artifacts may remain,
especially around fine and detailed textures, which in turn lead to blurry
results after updating $\mathcal{G}$. We conjecture that this is due to the
limited resolution of the feature space. To address this issue, after we
obtain $\mathcal{G}^{\prime}$, we can again render images to be edited,
repeating the process described in the previous section and re-updating the 3D
model, similar to SDEdit [65] or IM-3D [64].

###  4.4 Partial Editing

Because 3D GS is an _explicit_ 3D representation, if desired, a user can
selectively edit a 3D scene by allowing only certain Gaussians to change, thus
focusing on a specific region of interest while ensuring that the rest of the
scene remains unchanged. This is a significant advantage compared to implicit
radiance field representation, where there is no direct correspondence between
specific parameters and regions of space, so any update to the model tends to
be global.

We thus propose to first obtain a mask for the Gaussians that should undergo
editing. This could be done following GaussianEditor [15] or SAGA [7].
Specifically, SAGA [7] optimize a feature field, alongside the radiance field,
given user-provided seed point or mask for the object of interest.

We then follow the same steps outlined in Secs. 4.2 and 4.3, but only optimize
the reconstruction objective on regions defined by the rendered segmentation
masks. This training procedure leads to the partial change of a 3D model when
a specific prompt is provided.

##  5 Experiments

In this section, we first provide the implementation details of our method
followed by qualitative and quantitative comparison with other methods. We
then provide an ablation study of our method.

#### Implementation Details.

As our image editor, we use InstructPix2Pix [5], which is a fine-tuned image-
to-image translation model based on Stable Diffusion [82]. We use scenes from
IN2N [28] and other real-world datasets, including LLFF and Mip-NeRF360 [68,
3] to demonstrate the ability of our method on editing 3D models. We use 3D GS
[43] as the 3D representation and adopt the segmentation pipeline from SAGA
[7] and GaussianEditor [16] when partial editing is desired. For editing, we
use 20-30 views, which are edited according to the approach outlined in Sec.
4.2 and then used to train $\mathcal{G}^{\prime}$ with 500-1000 iterations
depending on the complexity of the scenes. We use LIPSP and L1 loss to train
the 3D GS as suggested in IN2N [28]. We also use classifier-free guidance to
control the effect strength of the editing. Most of the edits use the default
setting, 7.5 for textual conditions and 1.5 for the image condition. We
perform the interactive refinement every 500 iterations. We provide more
detailed hyperparameters in the supplementary material.

Method | 3D | CLIP | CLIP Directional | Avg. Editing
---|---|---|---|---
Model | Similarity | Similarity | Time
Instruct-N2N [28] | NeRF | 0.215 | 0.64 |  $\sim$ 51min
GaussianEditor [15] | GS | 0.201 | 0.60 |  $\sim$ 7min
IP2P [107] \+ SDS [79] | GS | 0.206 | 0.61 |  $\sim$ 6min
Ours | GS | 0.222 | 0.66 |  $\sim$ 4min
Table 1: Comparison with other editing methods. Methods based on Gaussian
Splatting are much faster than NeRF-based Instruct-N2N. Besides, our DGE
achieves the best performance while only using around half of the time
compared to GE .

#### Evaluation.

We provide both qualitative and quantitative evaluations of our method. For
quantitative evaluation, we followed the common practice [28, 16]. We evaluate
the alignment of edited 3D models and target text prompts with CLIP similarity
score, which is the cosine distance between the text and image embeddings
encoded by CLIP. To be specific, we randomly sample 20 camera poses from the
training dataset of the 3D models and measure the CLIP similarity scores
between the rendered images and the target text prompt. Additionally, we use
the CLIP directional similarity score, which is the cosine similarity distance
between the image and text editing directions (target embeddings minus source
embeddings), to measure the editing effect. We evaluate all the methods on a
total of 3 scenes and 10 different prompts. The detailed scene and prompt
pairs are provided in the supplementary.

###  5.1 Comparisons with Prior Work

In our experiments, we aim to demonstrate the advantages of our direct-and-
consistent editing approach vs. commonly used techniques, _i.e_., score
distillation sampling (SDS) and iterative dataset updates (IDU). We compare
our method with representative text-guided 3D editing methods, IN2N [28] and
GaussianEditor [15]. They both employ the IDU approach, while the former
builds on a NeRF representation and the latter on Gaussian Splatting. To edit
a 3D model, they both iterate between the 3D model training and image editing.
Every 10 steps, they use the same 2D editor (IP2P) to edit the rendered image,
replace the edited image in the 3D model training dataset, and continue the
model training. Finally, we also include a baseline using the same GS
framework as in our approach, but applying edits via SDS with InstructPix2Pix
(IP2P + SDS). This baseline enables a direct and fair comparison between our
consistent multi-view editing approach and the popular SDS-based approach.

Figure 2: Comparison with other methods. Our method can provide fast and detailed editing effects, such as the textures on the Venetian mask and mosaic sculpture. Other methods, such as InstructN2N and IP2P $+$ SD, fail to get the mosaic effects because they average over inconsistent editing. Multi-view | Epipolar | CLIP | CLIP Directional
---|---|---|---
Editing | Constraints | Score | Score
|  | 0.203 | 0.59
| ✓ | 0.219 | 0.64
✓ | ✓ | 0.222 | 0.66
Table 2: Ablation study on different components of our methods. Without multi-
view consistency editing, both the CLIP and CLIP directional scores drop
significantly. Although the difference between with and without epipolar
constraints is small, we provide a visual comparison in Fig. 4 showing that it
helps to improve the detailed texture especially when the appearance is
relatively similar in different places.  Figure 3: The comparison between with
and without multi-view consistency. With the proposed multi-view consistent
editing, the edited 3D GS is clear and clean while without multi-view
consistent editing, it either fails to converge or leads to blurry results.

Figure 2 shows visual comparisons with the above methods. Our proposed method
achieves realistic and detailed 3D editing. One notable advantage of our
approach is that ensuring the consistency of the edits in image space results
in 3D edits that are overall more faithful to the editing instruction (higher
fidelity). In contrast, IDU and SDS work by progressively aligning the edited
views of the 3D object, effectively averaging the inconsistencies from the
edited images and leading to blurrier or lower-fidelity reconstructions and
artifacts. For example, for the edit instruction “Make him wear a Venetian
mask”, the texture of the mask generated by our method is detailed and vivid,
while other methods fail to properly generate the texture. For example,
GaussianEditor hardly changes the skin color, while IN2N changes the clothes
and background of the 3D models as well. In the example of “Make the man look
like a mosaic sculpture”, our DGE generates a consistent texture of the
sculpture However, IN2N and IP2P $+$ SDS fail to converge to a reasonable
editing outcome, mainly because of the inconsistency of edits during each
iteration. GaussianEditor is able to generate some texture, but it is
inconsistent from different views. It is worth noting that IN2N usually gets
smoother results than GaussianEditor and IP2P + SDS; this is mainly due to the
fact the NeRF is continuous, while GS is not.

Table 1 shows quantitative comparisons between our method and others. We
achieve higher CLIP similarly score and CLIP directional similarly score,
indicating that our editing 3D models align better than other methods and also
follow the editing instructions.

###  5.2 Ablation Study

Next, we ablate the effectiveness of two main components in our editing
pipeline: the multi-view consistent editing and the epipolar correspondence
matching. Finally, we demonstrate the convergence speed (in terms of editing)
of our approach through a qualitative example.

Figure 4: The comparison between edited 2D images with and without epipolar
constraints. The one with epipolar constraints successfully matches the
correspondences while the one without epipolar constraints fail to match the
correspondences thus result in inconsistent multi-view editings. Figure 5:
Comparison between our DGE and GaussianEditor in terms of interaction numbers.
Our method achieves realistic editing result with much fewer iterations. With
iteration increases, our method also gradually refines the details. Figure 6:
Partial editing results on the Horns scene. We achieve realistic editing
results only on the required object while keeping the rest unchanged.

#### Editing without Multi-View Consistent Editing.

To validate the effectiveness of the multi-view consistent editing, we replace
the multi-view consistent editing with _independently_ editing the same number
of views, and using those for fitting $\mathcal{G}^{\prime}$. Table 2
illustrates that editing with multi-view consistency is important to achieve a
high CLIP score and CLIP directional score. To discuss this further, in Fig.
3, we show some intermediate 2D editing results with and without multi-view
consistent editing. We observe that with multi-view consistent editing,
different views get similar appearances, while per-frame editing can lead to
dramatically different edits. This is especially true in the first example
(“turn the man into a robot”), where the prompt allows for greater variation
and occasionally even results in unsuccessful editing (third frame). As
expected, fitting $\mathcal{G}^{\prime}$ to these edited views produces a low-
quality 3D edited model. Our multi-view consistent approach clearly alleviates
these issues. We show more detailed comparisons in supplementary materials.

#### Effect of the Epipolar Constraints.

In Table 2, we show that using the epipolar constraints during feature
propagation (_i.e_. from keyframes to intermediate frames), both the CLIP
score and CLIP directional score increase. Quantitatively, the difference
between using or not using epipolar constraints is not large. However, we
conjecture that this is mainly because the CLIP score measures the image on a
high level while the epipolar constraints help to improve the detailed
editing, such as textures. We better illustrate the difference in Figure 4.
Without using the epipolar constraint, the resulting edits are inconsistent
between the two view — red points (just below the bear’s ear). The reason is
that it wrongly matches the point with another point in black which leads to
failed consistent editing. The red line in the images indicates the epipolar
line of the point in the red point on the keyframe. As it can only receive
attention from the epipolar line, it generates the correct color.

#### Effect of the Number of Iterations

We show here a comparison between our method and GaussianEditor in Figure 5.
The figure demonstrates that our method can rapidly edit the 3D model using
only 400 iterations, while GaussianEditor requires much longer to converge. It
is worth noting that the results in the column of 400 iterations are without
refinement and generated within two minutes. Besides, as the interaction
increases, the details of the mask rendered by the 3D model are gradually
refined as we update the multi-view editing results every 500 iterations.

###  5.3 Partial Editing

Finally, we showcase some local editing results on the Horns scene from LLFF
[68] dataset. Firstly, we segment the Gaussians and get the mask of the
Gaussians. Then, we render images from different views and edit those views
with our proposed DGE. The results are realistic and only applied to the
segmented skeleton, demonstrating our method’s ability of partial editing.

##  6 Conclusion

We have presented DGE, a robust framework for _directly_ editing of a 3D model
by simply fitting it to a small set of views, which are edited by a text-based
image editor. The key insight of our approach is that one must obtain multi-
view _consistent_ edits in image space, to be able to update the 3D model
directly. Thus we propose an editing mechanism that jointly considers multiple
frames based on both appearance cues (image features) and 3D cues (epipolar
constraints based on scene geometry). This approach enables 3D editing that is
both more efficient and more faithful to the text instruction in comparison to
SDS-like alternatives.

## Ethics

For further details on ethics, data protection, and copyright please see
https://www.robots.ox.ac.uk/~vedaldi/research/union/ethics.html.

## Acknowledgements

This research is supported by ERC-CoG UNION 101001212. I. L. is also partially
supported by the VisualAI EPSRC grant (EP/T028572/1).

## References

  * [1] Bao, C., Zhang, Y., Yang, B., Fan, T., Yang, Z., Bao, H., Zhang, G., Cui, Z.: SINE: semantic-driven image-based NeRF editing with prior-guided editing field. In: Proc. CVPR (2023)
  * [2] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing. In: ECCV (2022)
  * [3] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-NeRF 360: Unbounded anti-aliased neural radiance fields. In: Proc. CVPR (2022)
  * [4] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. arXiv.cs abs/2211.09800 (2022)
  * [5] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR. pp. 18392–18402 (2023)
  * [6] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR (2023)
  * [7] Cen, J., Fang, J., Yang, C., Xie, L., Zhang, X., Shen, W., Tian, Q.: Segment any 3d gaussians. arXiv preprint arXiv:2312.00860 (2023)
  * [8] Ceylan, D., Huang, C.H.P., Mitra, N.J.: Pix2video: Video editing using image diffusion. In: ICCV. pp. 23206–23217 (2023)
  * [9] Chan, E.R., Lin, C.Z., Chan, M.A., Nagano, K., Pan, B., Mello, S.D., Gallo, O., Guibas, L.J., Tremblay, J., Khamis, S., Karras, T., Wetzstein, G.: Efficient geometry-aware 3D generative adversarial networks. In: Proc. CVPR (2022)
  * [10] Chefer, H., Alaluf, Y., Vinker, Y., Wolf, L., Cohen-Or, D.: Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. In: SIGGRAPH (2023)
  * [11] Chen, A., Xu, Z., Geiger, A., Yu, J., Su, H.: TensoRF: Tensorial radiance fields. In: arXiv (2022)
  * [12] Chen, J., Lyu, J., Wang, Y.: NeuralEditor: Editing neural radiance fields via manipulating point clouds. In: Proc. CVPR (2023)
  * [13] Chen, M., Laina, I., Vedaldi, A.: Training-free layout control with cross-attention guidance. In: WACV (2023)
  * [14] Chen, M., Xie, J., Laina, I., Vedaldi, A.: SHAP-EDITOR: Instruction-guided latent 3D editing in seconds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2024)
  * [15] Chen, Y., Chen, Z., Zhang, C., Wang, F., Yang, X., Wang, Y., Cai, Z., Yang, L., Liu, H., Lin, G.: Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. arXiv.cs abs/2311.14521 (2023)
  * [16] Chen, Y., Chen, Z., Zhang, C., Wang, F., Yang, X., Wang, Y., Cai, Z., Yang, L., Liu, H., Lin, G.: Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. CVPR (2024)
  * [17] Cheng, X., Yang, T., Wang, J., Li, Y., Zhang, L., Zhang, J., Yuan, L.: Progressive3d: Progressively local editing for text-to-3d content creation with complex semantic prompts. arXiv preprint arXiv:2310.11784 (2023)
  * [18] Chiang, P.Z., Tsai, M.S., Tseng, H.Y., sheng Lai, W., Chiu, W.C.: Stylizing 3D scene via implicit representation and hypernetwork. arXiv 2105.13016 (2022)
  * [19] Dong, J., Wang, Y.: ViCA-NeRF: View-consistency-aware 3d editing of neural radiance fields. In: Proc. NeurIPS (2024)
  * [20] Epstein, D., Jabri, A., Poole, B., Efros, A.A., Holynski, A.: Diffusion self-guidance for controllable image generation. In: NeurIPS (2023)
  * [21] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. In: ICLR (2023)
  * [22] Gao, W., Aigerman, N., Groueix, T., Kim, V., Hanocka, R.: TextDeformer: Geometry manipulation using text guidance. In: Proc. SIGGRAPH (2023)
  * [23] Geyer, M., Bar-Tal, O., Bagon, S., Dekel, T.: TokenFlow: Consistent diffusion features for consistent video editing. arXiv.cs abs/2307.10373 (2023)
  * [24] Geyer, M., Bar-Tal, O., Bagon, S., Dekel, T.: Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arxiv:2307.10373 (2023)
  * [25] Gong, B., Wang, Y., Han, X., Dou, Q.: Recolornerf: Layer decomposed radiance field for efficient color editing of 3d scenes. arXiv preprint arXiv:2301.07958 (2023)
  * [26] Gordon, O., Avrahami, O., Lischinski, D.: Blended-nerf: Zero-shot object generation and blending in existing neural radiance fields. ICCV (2023)
  * [27] Güler, R.A., Neverova, N., Kokkinos, I.: DensePose: Dense human pose estimation in the wild. In: Proc. CVPR (2018)
  * [28] Haque, A., Tancik, M., Efros, A., Holynski, A., Kanazawa, A.: Instruct-nerf2nerf: Editing 3d scenes with instructions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
  * [29] Haque, A., Tancik, M., Efros, A.A., Holynski, A., Kanazawa, A.: Instruct-NeRF2NeRF: Editing 3D scenes with instructions. In: Proc. ICCV (2023)
  * [30] Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press (2000)
  * [31] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-or, D.: Prompt-to-prompt image editing with cross-attention control. In: ICLR (2023)
  * [32] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross-attention control. In: Proc. ICLR (2023)
  * [33] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. NeurIPS 33, 6840–6851 (2020)
  * [34] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Proc. NeurIPS (2020)
  * [35] Hong, F., Zhang, M., Pan, L., Cai, Z., Yang, L., Liu, Z.: AvatarCLIP: zero-shot text-driven generation and animation of 3D avatars. Proc. SIGGRAPH 41(4) (2022)
  * [36] Huang, H., Tseng, H., Saini, S., Singh, M., Yang, M.: Learning to stylize novel views. In: Proc. ICCV (2021)
  * [37] Huang, Y., He, Y., Yuan, Y., Lai, Y., Gao, L.: StylizedNeRF: Consistent 3D scene stylization as stylized NeRF via 2D-3D mutual learning. In: Proc. CVPR (2022)
  * [38] Huang, Z., Shi, Y., Bruce, N., Gong, M.: SealD-NeRF: Interactive pixel-level editing for dynamic scenes by neural radiance fields. arXiv (2402.13510) (2024)
  * [39] Jambon, C., Kerbl, B., Kopanas, G., Diolatzis, S., Leimkühler, T., Drettakis, G.: NeRFshop: interactive editing of neural radiance fields. Proc. ACM Comput. Graph. Interact. Tech. 6(1) (2023)
  * [40] Kamata, H., Sakuma, Y., Hayakawa, A., Ishii, M., Narihira, T.: Instruct 3d-to-3d: Text instruction guided 3d-to-3d conversion. arXiv preprint arXiv:2303.15780 (2023)
  * [41] Kania, K., Yi, K.M., Kowalski, M., Trzciński, T., Tagliasacchi, A.: Conerf: Controllable neural radiance fields. In: CVPR. pp. 18623–18632 (2022)
  * [42] Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., Irani, M.: Imagic: Text-based real image editing with diffusion models. In: CVPR (2023)
  * [43] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3D Gaussian Splatting for Real-Time Radiance Field Rendering. Proc. SIGGRAPH 42(4) (2023)
  * [44] Khachatryan, L., Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., Shi, H.: Text2video-zero: Text-to-image diffusion models are zero-shot video generators. arXiv preprint arXiv:2303.13439 (2023)
  * [45] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. In: Proc. CVPR (2023)
  * [46] Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing NeRF for editing via feature field distillation. arXiv.cs (2022)
  * [47] Kobayashi, S., Matsumoto, E., Sitzmann, V.: Decomposing nerf for editing via feature field distillation. NeurIPS 35, 23311–23330 (2022)
  * [48] Kuang, Z., Luan, F., Bi, S., Shu, Z., Wetzstein, G., Sunkavalli, K.: PaletteNeRF: Palette-based appearance editing of neural radiance fields. In: Proc. CVPR (2023)
  * [49] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. In: CVPR (2023)
  * [50] Lazova, V., Guzov, V., Olszewski, K., Tulyakov, S., Pons-Moll, G.: Control-NeRF: Editable feature volumes for scene rendering and manipulation. In: Proc. WACV (2023)
  * [51] Lee, J.H., Kim, D.S.: Ice-nerf: Interactive color editing of nerfs via decomposition-aware weight optimization. In: ICCV. pp. 3491–3501 (2023)
  * [52] Lei, J., Zhang, Y., Jia, K., et al.: Tango: Text-driven photorealistic and robust 3d stylization via lighting decomposition. NeurIPS 35, 30923–30936 (2022)
  * [53] Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: ICLR (2022)
  * [54] Li, G., Zheng, H., Wang, C., Li, C., Zheng, C., Tao, D.: 3DDesigner: Towards photorealistic 3d object generation and editing with text-guided diffusion models. arXiv.cs abs/2211.14108 (2022)
  * [55] Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., et al.: Grounded language-image pre-training. In: CVPR. pp. 10965–10975 (2022)
  * [56] Li, Y., Dou, Y., Shi, Y., Lei, Y., Chen, X., Zhang, Y., Zhou, P., Ni, B.: Focaldreamer: Text-driven 3d editing via focal-fusion assembly. arXiv preprint arXiv:2308.10608 (2023)
  * [57] Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In: CVPR (2023)
  * [58] Lin, Y., Bai, H., Li, S., Lu, H., Lin, X., Xiong, H., Wang, L.: CompoNeRF: Text-guided multi-object compositional nerf with editable 3d scene layout. arXiv.cs abs/2303.13843 (2023)
  * [59] Liu, H., Shen, I., Chen, B.: NeRF-In: Free-form NeRF inpainting with RGB-D priors. arXiv.cs abs/2206.04901 (2022)
  * [60] Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J.Y., Russell, B.: Editing conditional radiance fields. In: ICCV. pp. 5773–5783 (2021)
  * [61] Liu, S., Zhang, X., Zhang, Z., Zhang, R., Zhu, J., Russell, B.: Editing conditional radiance fields. In: Proc. ICCV (2021)
  * [62] Lüddecke, T., Ecker, A.: Image segmentation using text and image prompts. In: CVPR. pp. 7086–7096 (2022)
  * [63] Mascaro, R., Teixeira, L., Chli, M.: Diffuser: Multi-view 2D-to-3D label diffusion for semantic scene segmentation. In: Proc. ICRA (2021)
  * [64] Melas-Kyriazi, L., Laina, I., Rupprecht, C., Neverova, N., Vedaldi, A., Gafni, O., Kokkinos, F.: IM-3D: Iterative multiview diffusion and reconstruction for high-quality 3D generation. arXiv preprint (abs/2402.08682) (2024)
  * [65] Meng, C., He, Y., Song, Y., Song, J., Wu, J., Zhu, J.Y., Ermon, S.: SDEdit: Guided image synthesis and editing with stochastic differential equations. In: ICLR (2022)
  * [66] Michel, O., Bar-On, R., Liu, R., Benaim, S., Hanocka, R.: Text2Mesh: text-driven neural stylization for meshes. In: Proc. CVPR (2022)
  * [67] Mikaeili, A., Perel, O., Safaee, M., Cohen-Or, D., Mahdavi-Amiri, A.: SKED: sketch-guided text-based 3D editing. In: Proc. ICCV (2023)
  * [68] Mildenhall, B., Srinivasan, P.P., Ortiz-Cayon, R., Kalantari, N.K., Ramamoorthi, R., Ng, R., Kar, A.: Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) (2019)
  * [69] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: Representing scenes as neural radiance fields for view synthesis. In: Proc. ECCV (2020)
  * [70] Mirzaei, A., Aumentado-Armstrong, T., Brubaker, M.A., Kelly, J., Levinshtein, A., Derpanis, K.G., Gilitschenski, I.: Watch your steps: Local image and scene editing by text instructions. arXiv.cs abs/2308.08947 (2023)
  * [71] Mirzaei, A., Aumentado-Armstrong, T., Derpanis, K.G., Kelly, J., Brubaker, M.A., Gilitschenski, I., Levinshtein, A.: Spin-nerf: Multiview segmentation and perceptual inpainting with neural radiance fields. In: CVPR. pp. 20669–20679 (2023)
  * [72] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. In: CVPR. pp. 6038–6047 (2023)
  * [73] Nguyen-Phuoc, T., Liu, F., Xiao, L.: SNeRF: stylized neural implicit representations for 3d scenes. Proc. SIGGRAPH 41(4) (2022)
  * [74] Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., Chen, M.: GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. arXiv.cs abs/2112.10741 (2021)
  * [75] Park, J., Kwon, G., Ye, J.C.: Ed-nerf: Efficient text-guided editing of 3d scene using latent space nerf. arXiv preprint arXiv:2310.02712 (2023)
  * [76] Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, J., Zhu, J.Y.: Zero-shot image-to-image translation. In: SIGGRAPH (2023)
  * [77] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: SDXL: improving latent diffusion models for high-resolution image synthesis. arXiv.cs abs/2307.01952 (2023)
  * [78] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. In: ICLR (2023)
  * [79] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: DreamFusion: Text-to-3D using 2D diffusion. In: Proc. ICLR (2023)
  * [80] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. ICCV (2023)
  * [81] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from natural language supervision. In: Proc. ICML. vol. 139, pp. 8748–8763 (2021)
  * [82] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684–10695 (2022)
  * [83] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proc. CVPR (2022)
  * [84] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: CVPR (2023)
  * [85] Sella, E., Fiebelman, G., Hedman, P., Averbuch-Elor, H.: Vox-e: Text-guided voxel editing of 3d objects. In: Proc. ICCV (2023)
  * [86] Shi, Y., Xue, C., Pan, J., Zhang, W., Tan, V.Y., Bai, S.: DragDiffusion: Harnessing diffusion models for interactive point-based image editing. arXiv (2306.14435) (2023)
  * [87] Song, H., Choi, S., Do, H., Lee, C., Kim, T.: Blending-nerf: Text-driven localized editing in neural radiance fields. In: ICCV. pp. 14383–14393 (2023)
  * [88] Song, H., Choi, S., Do, H., Lee, C., Kim, T.: Blending-NeRF: Text-driven localized editing in neural radiance fields. In: Proc. ICCV (2023)
  * [89] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: Proc. ICLR (2021)
  * [90] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: ICLR (2021)
  * [91] Sun, C., Sun, M., Chen, H.: Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In: Proc. CVPR (2022)
  * [92] Sun, C., Liu, Y., Han, J., Gould, S.: NeRFEditor: differentiable style decomposition for full 3D scene editing. In: Proc. WACV (2022)
  * [93] Teng, Y., Xie, E., Wu, Y., Han, H., Li, Z., Liu, X.: Drag-a-video: Non-rigid video editing with point-based interaction. arXiv.cs abs/2312.02936 (2023)
  * [94] Tretschk, E., Golyanik, V., Zollhöfer, M., Bozic, A., Lassner, C., Theobalt, C.: Scenerflow: Time-consistent reconstruction of general dynamic scenes. arXiv.cs abs/2308.08258 (2023)
  * [95] Tschernezki, V., Laina, I., Larlus, D., Vedaldi, A.: Neural Feature Fusion Fields: 3D distillation of self-supervised 2D image representation. In: Proceedings of the International Conference on 3D Vision (3DV) (2022)
  * [96] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-play diffusion features for text-driven image-to-image translation. In: CVPR (2023)
  * [97] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-play diffusion features for text-driven image-to-image translation. In: Proc. CVPR (2023)
  * [98] Vargas, F., Grathwohl, W.S., Doucet, A.: Denoising diffusion samplers. In: Proc. ICLR (2023)
  * [99] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NIPS (2017)
  * [100] Wang, B., Dutt, N.S., Mitra, N.J.: ProteusNeRF: Fast lightweight NeRF editing using 3D-aware image context. arXiv.cs abs/2310.09965 (2023)
  * [101] Wang, C., Chai, M., He, M., Chen, D., Liao, J.: CLIP-NeRF: Text-and-image driven manipulation of neural radiance fields. In: Proc. CVPR (2022)
  * [102] Wang, C., Jiang, R., Chai, M., He, M., Chen, D., Liao, J.: NeRF-Art: Text-driven neural radiance fields stylization. arXiv.cs abs/2212.08070 (2022)
  * [103] Wang, D., Zhang, T., Abboud, A., Süsstrunk, S.: Inpaintnerf360: Text-guided 3d inpainting on unbounded neural radiance fields. arXiv preprint arXiv:2305.15094 (2023)
  * [104] Wang, X., Zhu, J., Ye, Q., Huo, Y., Ran, Y., Zhong, Z., Chen, J.: Seal-3D: Interactive pixel-level editing for neural radiance fields. In: Proc. ICCV (2023)
  * [105] Weder, S., Garcia-Hernando, G., Monszpart, A., Pollefeys, M., Brostow, G.J., Firman, M., Vicente, S.: Removing objects from neural radiance fields. In: CVPR. pp. 16528–16538 (2023)
  * [106] Wu, J.Z., Ge, Y., Wang, X., Lei, S.W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7623–7633 (2023)
  * [107] Xu, J., Wang, X., Cao, Y., Cheng, W., Shan, Y., Gao, S.: InstructP2P: Learning to edit 3D point clouds with text instructions. arXiv.cs abs/2306.07154 (2023)
  * [108] Xu, S., Li, L., Shen, L., Lian, Z.: Desrf: Deformable stylized radiance field. In: CVPR. pp. 709–718 (2023)
  * [109] Xu, T., Harada, T.: Deforming radiance fields with cages. In: Proc. ECCV (2022)
  * [110] Yang, B., Bao, C., Zeng, J., Bao, H., Zhang, Y., Cui, Z., Zhang, G.: Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In: Proc. ECCV (2022)
  * [111] Yang, B., Zhang, Y., Xu, Y., Li, Y., Zhou, H., Bao, H., Zhang, G., Cui, Z.: Learning object-compositional neural radiance field for editable scene rendering. In: Proc. ICCV (2021)
  * [112] Yu, L., Xiang, W., Han, K.: Edit-diffnerf: Editing 3d neural radiance fields using 2d diffusion model. arXiv preprint arXiv:2306.09551 (2023)
  * [113] Yuan, Y., Sun, Y., Lai, Y., Ma, Y., Jia, R., Gao, L.: NeRF-editing: Geometry editing of neural radiance fields. In: Proc. CVPR (2022)
  * [114] Zhang, H., Feng, Y., Kulits, P., Wen, Y., Thies, J., Black, M.J.: Text-guided generation and editing of compositional 3d avatars. arXiv preprint arXiv:2309.07125 (2023)
  * [115] Zhang, J., Liu, X., Ye, X., Zhao, F., Zhang, Y., Wu, M., Zhang, Y., Xu, L., Yu, J.: Editable free-viewpoint video using a layered neural representation. ACM Trans. Graph. 40(4) (2021)
  * [116] Zhang, K., Kolkin, N., Bi, S., Luan, F., Xu, Z., Shechtman, E., Snavely, N.: ARF: Artistic radiance fields. In: Proc. ECCV (2022)
  * [117] Zhang, K., Mo, L., Chen, W., Sun, H., Su, Y.: Magicbrush: A manually annotated dataset for instruction-guided image editing. In: NeurIPS (2023)
  * [118] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models. In: ICCV (2023)
  * [119] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)
  * [120] Zhang, S., Yang, X., Feng, Y., Qin, C., Chen, C.C., Yu, N., Chen, Z., Wang, H., Savarese, S., Ermon, S., Xiong, C., Xu, R.: Hive: Harnessing human feedback for instructional visual editing. arXiv preprint arXiv:2303.09618 (2023)
  * [121] Zheng, C., Lin, W., Xu, F.: EditableNeRF: Editing topologically varying neural radiance fields by key points. In: Proc. CVPR (2023)
  * [122] Zhou, X., He, Y., Yu, F.R., Li, J., Li, Y.: Repaint-nerf: Nerf editting via semantic masks and diffusion models. arXiv preprint arXiv:2306.05668 (2023)
  * [123] Zhuang, J., Wang, C., Lin, L., Liu, L., Li, G.: DreamEditor: Text-driven 3D scene editing with neural fields. In: Proc. SIGGRAPH (2023)

## Appendix

##  Appendix 0.A Overview

This supplementary material contains the following parts:

  * •

Additional Editing Results (Appendix 0.B). We provide additional qualitative
examples of our method, including more editing prompts and partial editing
results on complex scenes with many objects.

  * •

Additional Comparisons (Appendix 0.C). We provide in-depth comparisons of our
approach and other update schemes and further comparisons with prior work,
GaussianEditor and ViCA-NeRF.

  * •

Additional Details (Appendix 0.D). We provide more details regarding the
experimental implementation and evaluation protocol.

  * •

Social Impact and Limitations (Appendix 0.E). We discuss the social impact and
limitations of our work.

##  Appendix 0.B Additional Editing Results

In this section, we provide additional qualitative editing results of DGE with
more prompts and additional partial editing results on the Mip-NeRF360 [3]
dataset.

####  Results with Additional Prompts.

In Figure 7, we provide further qualitative editing results from our method,
demonstrating the ability of DGE to generate complex and visually compelling
editing effects. Specifically, when tasked with transforming the appearance of
a human figure into mythical creatures, such as a “Vampire,” “Tolkien Elf,” or
Werewolf,” DGE produces highly realistic results. This capability is not
solely restricted to character modifications. We also show the ability of DGE
to handle style transfer, such as converting the 3D models’ style into
“Fauvism painting”. Finally, by directly and consistently editing multiple
images at once, we show that DGE produces intricate 3D edits with detailed
patterns and textures, _e.g_., changing the clothes of the human figure to a
“pineapple pattern” or “floral pattern”.

Figure 7: Additional editing examples. We provide more qualitative results,
including prompts to change the character or style. Our DGE produces realistic
editing results with detailed textures.

#### Results with Partial Editing.

We provide additional partial editing results using three different scenes
provided by Mip-NeRF360 [3] in Figure 8. Our DGE successfully edits the color
of the selected objects such as vase and dozer while keeping other stuff
unchanged. For example, only the vase in the first row is changed to either
green or red. Besides, we can still distinguish the color changes from the
transparent bowl before the vase. As shown in the last two columns of the
figure, without partial editing, the whole scene turns either red or green,
which clearly demonstrates the advantages of partial editing. The core issue
is the inability of the image editor (InstructPix2Pix) to isolate and modify
the color of just the selected object, affecting the entire image instead.

Figure 8: Additional partial editing results. Our DGE successfully changes
only the selected objects faithfully. We can even tell the color change
through the transparent bowl before the vase. Figure 9: Comparison of
different updating methods. This figure compares the 2D editing results (i)
with multi-view consistency, (ii) without multi-view consistency, and (iii)
with iterative dataset updates. Multi-view consistent editing yields a
consistent appearance across all different views.

Figure 10: CLIP and CLIP directional scores w.r.t. the number of iterations.
We compare the CLIP scores of DGE and GaussianEditor over training iterations
for the example depicted in Figure 9. Our method achieves high CLIP and CLIP
directional scores in under 400 iterations, which is before the additional
refinement step. We perform refining after 500 iterations.

##  Appendix 0.C Additional Comparisons

#### Multi-view Consistent Editing vs. Iterative Dataset Updates.

Next, we provide a qualitative comparison of our _direct_ approach with and
without multi-view consistency, as well as the iterative dataset update
scheme. In Figure 9, we show the 2D edited images that are later used to
update the corresponding 3D model. Note that all variants use the same base 3D
model. We observe that when employing multi-view consistency in 2D editing, we
achieve a uniform and consistent appearance across all images, which in turn
accelerates the reconstruction process. Conversely, without multi-view
consistency, the images are edited independently by InstructPix2Pix. As a
result, the appearance of the edited images tends to vary, despite the input
views being similar (notably, the 3rd to 5th images in the middle row).
Finally, the inconsistencies are exacerbated under the iterative dataset
update scheme, which involves introducing different amounts of noise,
resulting in considerable variability after editing. This variability
contributes to the slower convergence speeds of methods such as IN2N [28] and
GE [15].

#### Extended Comparisons with GaussianEditor.

GaussianEditor [15] is a recent work that attempts to edit Gaussian Splatting-
based models similarly to IN2N [28]. Based on the iterative dataset update
scheme, both GaussianEditor and IN2N iterate between 3D model updating and
image editing. As discussed above, iterative dataset updates require many
iterations to successfully apply the desired editing effect to the 3D model.
In contrast, our DGE edits rendered images _simultaneously_ and subsequently
updates the Gaussian Splatting model, achieving faster speed and better
quality. Figure 10 further demonstrates this by comparing the number of
iterations required for convergence between DGE and GaussianEditor. By
directly integrating consistent edits into the 3D model, we observe our
approach that converges in significantly fewer iterations compared to
GaussianEditor.

We provide an additional qualitative comparison with GaussianEditor to assess
the edited image details in Figure 11. As shown in the figure, our method
generates high-fidelity textures that align well with the textual description,
which gives the human a pineapple-like outfit. In contrast, GaussianEditor
falls short in delivering this complex pattern with adequate detail. This
could be attributed to inconsistent editing results of InstructPix2Pix for
different views during the process, as also previously noted by IN2N [28].

Figure 11: Details Comparison. DGE generates editing results with more detailed textures such as the pineapple or floral pattern. Figure 12: Qualitative Comparison with ViCA-NeRF. We use the official implementation from its GitHub repository. ViCA-NeRF results in blurry editing results when asked to turn the man into “mosaic sculpture” or “Iron Man”. It also fails when given prompts such as “Turn him into a Spider Man with a mask” or “Make him wear a Venetian mask”.  Method | 3D | CLIP | CLIP Directional | Avg. Editing
---|---|---|---|---
Model | Similarity | Similarity | Time
ViCA-NeRF [28] | NeRF | 0.204 | 0.44 |  $\sim$ 28min
Ours | GS | 0.222 | 0.66 |  $\sim$ 4min
Table 3: Comparison with ViCA-NeRf. Our DGE achieves better performance with
much less time.

#### Comparison with ViCA-NeRF.

We provide an additional comparison between our work and ViCA-NeRF [19], which
also focuses on editing different views consistently. We use the official
implementation provided in their GitHub repository. We highlight some key
differences as below. ViCA-NeRF uses 3D model depth information to project
features from key views into others using a blending module. Our approach,
however, takes inspiration from [24], which injects attention values from
keyframes into other frames. In addition, we incorporate further 3D priors to
constrain this process with epipolar lines, minimizing the corresponding
matching errors. This leads to clearer results compared to ViCA-NeRF’s, which
can sometimes produce blurry edits or fail. We have conducted both qualitative
and quantitative evaluations, showing the superiority of our method in
achieving more accurate and higher-quality edits in only a fraction of the
time. Examples and detailed comparisons are in Figure 12 and Table 3. As shown
in Figure 12, ViCA-NeRF results in blurry editing results and fails to produce
meaningful editing given prompts such as “Turn him into spider man with a
mask”. The quantitative results in Table 3 also suggest that our method
surpasses ViCA with less time and higher quality.

##  Appendix 0.D Additional Details

###  0.D.1 Implementation Details

#### Image Editor.

We use instructPix2Pix as the base image editor. We follow the same setting in
[28, 15], which uses DDIM [90] scheduler with 20 steps and the implementation
included in the Diffusers [63] library. Different from IN2N [28] or
GaussianEditor [15], which sample noise from $t\in[0.02,0.98]$, we directly
use $t=1$ as we do not require iterative dataset updates. For the refinement
step, we add noise with $t=0.3$ to capture more details of 3D editing. We
select one keyframe in every five frames.

#### Camera Sorting.

Our approach to multi-view is inspired by video editing. While in videos
camera trajectories are typically smooth, in our case, when rendering views
from the 3D model, we do not assume that a camera trajectory is known in
advance. However, obtaining such a trajectory is useful to ensure that the
selected keyframes will cover a variety of viewpoints. To this end, prior to
multi-view editing, we sort the rendered viewpoints as follows. (1) We take
the camera with the largest value along the x-axis in world coordinates as the
reference camera. (2) We sort the cameras using the relative angle between
their forward vectors and that of the reference camera. The forward vector is
defined as the normalized vector along the z-axis of the camera in world
coordinates.

In Figure 13, we visualize the images corresponding to sorted cameras and
unsorted cameras for two different scenes.

#### Computation Time.

The average editing time reported (for both DGE and other methods) is
evaluated on one Nvidia RTX A6000 GPU with 48G memory.

Figure 13: The effect of camera sorting. The first two rows for each scene are
the ground-truth images of sorted cameras and the bottom two rows are ground-
truth images of unsorted cameras. Note the smoother transitions from one frame
to the next after sorting.

###  0.D.2 Evaluation Prompts

We have created a set of 10 scene-prompt pairs for evaluation purposes, shown
in Table 4. These are used for computing the CLIP and CLIP directional scores.
We randomly sample 20 views from the training cameras and use the average
score as the score for each scene-prompt pair.

Scene |  Source Prompt |  Target Prompt |  Edit Instruction
---|---|---|---
Face |  “A man with curly hair in a grey jacket.” |  “A man with curly hair in a grey jacket with a Venetian mask.” |  “Give him a Venetian mask.”
Face |  “A man with curly hair in a grey jacket.” |  “A man with curly hair in a checkered cloth.” |  “Give him a checkered jacket.”
Face |  “A man with curly hair in a grey jacket.” |  “A spider man with a mask and curly hair.” |  “Turn him into spider man with a mask.”
Bear |  “A stone bear in a garden.” |  “A robotic bear in the garden.” |  “Make the bear look like a robot.”
Bear |  “A stone bear in a garden.” |  “A panda in the garden.” |  “Make the bear look like a panda.”
Bear |  “A stone bear in a garden.” |  “A bear with rainbow color” |  “Make the color of the bear look like rainbow.”
Person |  “A man standing next to a wall wearing a blue T-shirt and brown pants.” |  “A man looks like a mosaic sculpture standing next to a wall.” |  “Make the man look like a mosaic sculpture.”
Person |  “A man standing next to a wall wearing a blue T-shirt and brown pants.” |  “A man wearing a shirt with a pineapple pattern.” |  “Make the person wear a shirt with a pineapple pattern.”
Person |  “A man standing next to a wall wearing a blue T-shirt and brown pants.” |  “An Iron Man stands to a wall.” |  “Turn him into Iron Man.”
Person |  “A man standing next to a wall wearing a blue T-shirt and brown pants.” |  “A robot stands to a wall.” |  “Turn the man into a robot.”
Table 4: Detailed evaluation dataset. As shown above, we evaluate different
methods on the above 10 different scene-prompt pairs. The source and target
prompts are used for computing CLIP and CLIP directional scores. The edit
instructions are textual inputs of InstructPix2Pix [4].

##  Appendix 0.E Social Impact and Limitations

#### Social Impact.

In this work, we leverage existing datasets in previous studies [28, 3, 68].
Our examples may include human subjects for whom consent has been obtained, as
outlined in [28]. The potential outcomes of our editing results may lead to
negative societal impacts if utilized for harmful purposes. We urge users to
employ our methodology responsibly, adhering to ethical guidelines,
regulations, and relevant laws.

#### Limitations.

The editing performance of our approach is primarily constrained by the
capabilities of the underlying image editor, in this case, InstructPix2Pix
[4]. Consequently, it inherits limitations from InstructPix2Pix. As such, our
method’s capability to handle substantial geometric transformations is
limited, such as adjusting the position of a person’s arm or simulating
actions like jumping. We anticipate that incorporating advanced or alternative
2D diffusion priors could pave the way for overcoming these constraints in the
future.