DRAGGAUSSIAN: ENABLING DRAG-STYLE MANIPULATION ON 3D GAUSSIAN REPRESENTATION

SUBMITTED BY
Sitian Shen

Beijing Institute of Technology

The final year project work was carried out under the 3+1+1 Educational
Framework at the National University of Singapore (Chongqing) Research
Institute

May 2024

## Abstract

User-friendly 3D object editing is a challenging task that has attracted
significant attention recently. The limitations of direct 3D object editing
without 2D prior knowledge have prompted increased attention towards utilizing
2D generative models for 3D editing. While existing methods like Instruct
NeRF-to-NeRF offer a solution, they often lack user-friendliness, particularly
due to semantic guided editing. In the realm of 3D representation, 3D Gaussian
Splatting emerges as a promising approach for its efficiency and natural
explicit property, facilitating precise editing tasks. Building upon these
insights, we propose DragGaussian, a 3D object drag-editing framework based on
3D Gaussian Splatting, leveraging diffusion models for interactive image
editing with open-vocabulary input. This framework enables users to perform
drag-based editing on pre-trained 3D Gaussian object models, producing
modified 2D images through multi-view consistent editing. Our contributions
include the introduction of a new task, the development of DragGaussian for
interactive point-based 3D editing, and comprehensive validation of its
effectiveness through qualitative and quantitative experiments.

###### Contents

  1. Abstract
  2. 1 Introduction
  3. 2 Background
    1. 2.1 Preliminaries on 3D Gaussian Splatting
    2. 2.2 Preliminaries on Diffusion Models
    3. 2.3 Preliminaries on Multi-view Diffusion Model
  4. 3 Literature Review
    1. 3.1 2D Editing
    2. 3.2 3D Editing
    3. 3.3 Multi-view Diffusion Model
  5. 4 Methodology
    1. 4.1 Interactive 3D Point-based Manipulation
    2. 4.2 Multi-view Identity Preserving Fine-tuning
    3. 4.3 Multi-view Consistent Editing
      1. 4.3.1 Motion Supervision
      2. 4.3.2 Point Tracking
      3. 4.3.3 Multi-view Consistent Denoising
  6. 5 Experiments
    1. 5.1 Implementation Details
    2. 5.2 Editing Results
    3. 5.3 Ablation Study
  7. 6 Discussion
  8. 7 Conclusion

###### List of Figures

  1. 4.1 Pipeline of DragGaussian.
  2. 4.2 Overview of our UI.
  3. (a) Splat size configured to 0.
  4. (b) Spat size configured to 1.
  5. 4.3 Drawing mask using the brush.
  6. (a) Splat size configured to 0.
  7. (b) Spat size configured to 1.
  8. 4.4 Stages for Multi-view Consistent Editing.
  9. 5.1 Multi-view consistent editing on a chair.
  10. 5.2 Editing and masking on a Gaussian chair.
  11. 5.3 Drawing mask using the brush.
  12. 5.4 Editing results on multi-view images generated by MVDream. (a) column shows the initial images, (b) colunm shows the images after DDIM Inversion and DDIM Sampling process without editing, (c) column shows the images edited by our multi-view image consistent editing pipeline without LoRA finetune, and (d) column shows the editing results with LoRA finetune.

##  Chapter 1 Introduction

Due to the lack of large scale 3D dataset, editing directly on 3D object or
scene without 2D prior often lacks generalizability. 3D editing utilizing 2D
generative models [1] has gain extensive attention recently. A famous
framework of three editing was proposed in Instruct NeRF-to-NeRF [2], that is
to edited the multi-view projected 2D images from a 3D representation using 2D
editing methods, and then use the edited 2D images as new training data to
finetune the initial 3D representation. However, the 2D editing algorithm in
this work called Instruct Pix2Pix [3] is a semantic guided editing method,
which isn’t very user-friendly. Compared with semantic guidance, some dragging
can better represent the user’s editing goal.

To address this issue, we tried to leverage the widely adopted 2D drag editing
approach known as DragGAN [4], renowned for its ability to facilitate
interactive point-based image manipulation. Within this framework, users
initiate the editing process by selecting pairs of anchor and destination
points on an image. Subsequently, the model executes semantically coherent
modifications, seamlessly relocating the content from the anchor points to
their corresponding destinations. Moreover, users have the option to delineate
specific editable regions by drawing masks, ensuring targeted alterations
while preserving the remainder of the image. However, it’s worth noting that
due to its reliance on GAN [5] within its network architecture, this method
cannot deal with open-vocabulary inputs, posing a significant drawback on
generalizability for its practical application in real-world scenarios. In
this case, we turned to DragDiffusion, the first interactive point-based image
editing method with diffusion models. Empowered by large-scale pre-trained
diffusion models [6], DragDiffusion achieves accurate spatial control in image
editing with significantly better generalizability, while allowing open-
vocabulary input data.

When it comes to 3D representation, Neural radiance field (NeRF) methods [7,
8] have shown great power and synthesizing novel-view images. However, as
current 2D diffusion models have the limited ability to localize editing
regions, it is hard to generate delicate edited scenes as the unwanted regions
are usually changed with diffusion models. Recent 3D Gaussian Splatting
[9](3D-GS) has been a groundbreaking work in the field of radiance field,
which is the first to achieve a real sense of real-time rendering while
enjoying high rendering quality and training speed. Besides its efficiency, we
further notice its natural explicit property. 3D-GS has a great advantage for
editing tasks as each 3D Gaussian exists individually. It is easy to edit 3D
scenes by directly manipulating 3D Gaussians with desired constraints applied,
including choosing explicit editing mask and choosing the center of one of the
Gaussian points as the editing start points (handle points). Some works [10,
11] have proposed effective semantic guided editing on 3D Gaussian recently.

Taking into account the above considerations, we propose a 3D object drag-
editing framework based on 3D Gaussian Splatting representation, named
DragGaussian. In our pipeline, users input pre-trained 3D Gaussian object
models into the system through a user interface. Upon visualization, they can
designate initial and final points for drag editing or specify regions for
editing. These points are projected onto 2D images from various camera angles
via the projection module. Subsequently, employing Multi-view Consistent
Editing, we produce the modified 2D images. Prior to this, to enhance the
adaptability of the pre-trained 2D editing network to diverse inputs, we fine-
tune the editing network using an enhanced version of multi-view LoRA.
Finally, we refine the original 3D Gaussian model using the modified 2D images
to demonstrate the edited appearance of the 3D objects.

Our contributions are summarized as follows:

1) We proposed a new task: 3D object editing through drag operation on 3D
Gaussian.

2) We present a novel 3D object editing method DragGaussian, the first to
achieve interactive point-based 3D editing with diffusion models on 3D
Gaussian representation.

3) Comprehensive qualitative and experiments demonstrate the effectiveness of
our DragGaussian.

##  Chapter 2 Background

###  2.1 Preliminaries on 3D Gaussian Splatting

3D Gaussian splatting[9] is a modern and effective method for 3D
visualization. It uses point-based 3D Gaussians, labeled as
$G=\\{g_{1},g_{2},\dots g_{N}\\}$, where each $g_{i}$ consists of a set
$\\{\mu,\Sigma,c,\alpha\\}$ with $i$ ranging from 1 to $N$. Here, $\mu$
represents the 3D position of the Gaussian’s center, $\Sigma$ is the 3D
covariance matrix, $c$ is the RGB color value, and $\alpha$ denotes the
opacity. This method is notable for its streamlined Gaussian representation
and its efficient, differentiable rendering technique, allowing for high-
quality, real-time rendering. The rendering process of splatting is described
by the equation

$C=\sum_{i\in N}c_{i}\sigma_{i}\prod_{j=1}^{i-1}(1-\alpha_{j}),$ (2.1)

where $\sigma_{i}=\alpha_{i}e^{-\frac{1}{2}(x_{i})^{T}\Sigma^{-1}(x_{i})}$
indicates the Gaussian’s impact on a pixel, with $x_{i}$ being the pixel’s
distance from the $i$-th Gaussian’s center.

###  2.2 Preliminaries on Diffusion Models

Denoising diffusion probabilistic models (DDPM) [12] constitutes a family of
latent generative models. Concerning a data distribution $q(\mathbf{z})$, DDPM
approximates $q(\mathbf{z})$ as the marginal $p_{\theta}(\mathbf{z}_{0})$ of
the joint distribution between $\mathbf{Z}_{0}$ and a collection of latent
random variables $\mathbf{Z}_{1:T}$. Specifically,

$p_{\theta}(\mathbf{z}_{0})=\int
p_{\theta}(\mathbf{z}_{0}:\mathbf{z}_{1:T})\,d\mathbf{z}_{1:T},$ (2.2)

where $p_{\theta}(\mathbf{z}_{T})$ is a standard normal distribution and the
transition kernels $p_{\theta}(\mathbf{z}_{t}|\mathbf{z}_{t-1})$ of this
Markov chain are all Gaussian conditioned on $\mathbf{z}_{t}$. In our context,
$\mathbf{Z}_{0}$ corresponds to image samples given by users, and
$\mathbf{Z}_{t}$ corresponds to the latent after $t$ steps of the diffusion
process. [13] proposes Latent Diffusion Model (LDM), which maps data into a
lower-dimensional space via a Variational Auto-Encoder (VAE) [14] and models
the distribution of the latent embeddings instead. Based on the framework of
LDM, several powerful pretrained diffusion models have been released publicly,
including the Stable Diffusion (SD) model
(https://huggingface.co/stabilityai). In SD, the network responsible for
modeling $p_{\theta}(\mathbf{z}_{t}|\mathbf{z}_{t-1})$ is implemented as a
UNet [15] that comprises multiple self-attention and cross-attention modules
[16].

Based on DDPM method, Denoising Diffusion Implicit Models (DDIM) [17] has also
been proposed, who’s sampling process is shown in Equ. 2.3. DDIM perfectly
solves the two issues of DDPM. DDIM has a fast sampling speed, and its
sampling process is deterministic because $\sigma$ can be set to 0. Therefore,
no noise is introduced during the sampling process.

$x_{t-1}=\sqrt{\bar{\alpha_{t-1}}}\left(\frac{x_{t}-\sqrt{1-\sqrt{\bar{\alpha_{t}}}}\varepsilon_{t}}{\sqrt{\bar{\alpha_{t}}}}\right)+\sqrt{1-\bar{\alpha_{t-1}}-\sigma^{2}\varepsilon_{t}}+\sigma^{2}\varepsilon$
(2.3)

The above formula can be reversed to derive the calculation formula for
$x_{t}$ from $x_{t-1}$ as shown in Equ. 2.4, called DDIM Inversion. Given the
initial image $x_{0}$, all that is needed is to use a pre-trained U-net model
to predict noise, and then continuously update to obtain $x_{t}$.

$x_{t}=\frac{\sqrt{\bar{\alpha_{t}}}}{\sqrt{\bar{\alpha_{t-1}}}}(x_{t-1}-\sqrt{1-\bar{\alpha_{t-1}}}\varepsilon_{t})+\sqrt{1-\bar{\alpha_{t}}}\varepsilon_{t}.$
(2.4)

###  2.3 Preliminaries on Multi-view Diffusion Model

By leveraging insights from both 2D and 3D datasets, a multi-view diffusion
model can attain the generalizability of 2D diffusion models while maintaining
the consistency of 3D renderings. In an earlier study, MVDream [18] laid the
groundwork by effectively training a multi-view diffusion network. This
network generates four orthogonal and coherent multi-view images based on a
given text prompt and corresponding camera embedding. In our DragGaussian
approach, we utilize the U-net architecture pretrained in MVDream as a
foundational model.

In the training stage of MVDream, each block of the multi-view network
contains a densely connected 3D attention on the four view images, which
allows a strong interaction in learning the correspondence relationship
between different views. To train such a network, it adopts a joint training
with the rendered dataset from the Objaverse [19] and a larger scale text-to-
image (t2i) dataset, LAION5B [20], to maintain the generalizability of the
fine-tuned model. Formally, given text-image dataset
$\mathcal{X}=\\{\mathbf{x},y\\}$ and a multi-view dataset
$\mathcal{X}_{mv}=\\{\mathbf{x}_{mv},y,\mathbf{c}_{mv}\\}$, where $x$ is a
latent image embedding from VAE [14], $y$ is a text embedding from CLIP [21],
and $\mathbf{c}$ is their self-designed camera embedding, we may formulate the
multi-view (MV) diffusion loss as,

$\mathcal{L}_{MV}(\theta,\mathcal{X},\mathcal{X}_{mv})=\mathbb{E}_{\mathbf{x},y,\mathbf{c},t,\varepsilon}\left[\left\|\varepsilon-\varepsilon_{\theta}(x^{p};y,\mathbf{c}^{p},t)\right\|^{2}\right]$
(2.5)

where,
$(\mathbf{x}^{p},\mathbf{c}^{p})=\begin{cases}(\mathbf{x},\mathbf{0})&\text{with
probability }p,\\\ (\mathbf{x}_{mv},\mathbf{c}_{mv})&\text{with probability
}1-p.\end{cases}$

Here, $\mathbf{x}$ is the noisy latent image generated from a random noise
$\varepsilon$ and image latent, $\varepsilon_{\theta}$ is the multi-view
diffusion (MVDiffusion) model parameterized by $\theta$.

##  Chapter 3 Literature Review

###  3.1 2D Editing

In the domain of 2D image editing, guidance modalities encompass visual,
textual, and other forms such as point-based mouth tracking [22]. Visual
guidance leverages pixel-space properties to afford precise control, allowing
for the repurposing of image synthesis techniques across a range of editing
tasks by adjusting visual elements like semantic maps, as seen in [23, 24].
Textual guidance, in contrast, offers a more dynamic and flexible medium for
articulating visual concepts, exemplified by methods like Instruct Pix2Pix[3].

Recent advancements have spotlighted point-based editing as a method for
nuanced image content manipulation, illustrated by [25, 4, 26]. Specifically,
DragGAN [4] employs latent code optimization and point tracking for effective
dragging manipulation, though its utility is bounded by the intrinsic
limitations of GANs. FreeDrag [27] seeks to enhance DragGAN through a novel
point-tracking-free approach. Furthermore, advancements in diffusion models
[12] have led to developments such as DragDiffusion [28] and DragonDiffusion
[29], which adapt DragGAN’s framework to the diffusion model context,
significantly enhancing its versatility and generalization capabilities.

###  3.2 3D Editing

In the realm of NeRF-based 3D editing, the integration of text and image-
guided methodologies marks a significant advancement. Works such as Sine [30],
NeRF-Art [31], and TextDeformer [32] employ semantic-driven and geometric
manipulation techniques, while Clip-NeRF [33] innovatively combines text and
image inputs for enhanced NeRF manipulation. Building on these foundations,
InstructNeRF2NeRF [2] introduces a novel text-guided scene editing approach
using 2D diffusion models, albeit with the caveat of potential global scene
impacts due to its 2D image reliance. In parallel, methods like Ed-NeRF [34]
and DreamEditor [35] utilize static masking to define editing zones, and Watch
Your Steps [36] dynamically localizes edits via relevance mapping during NeRF
training, further refining the precision of NeRF-based editing.

Focusing on object-level modifications, FocalDreamer [37] offers meticulous
control over object attributes, ensuring detailed and consistent alterations,
while Image Sculpting [38] introduces an intuitive platform for geometric
adjustments, enabling users to artistically sculpt and modify objects within
3D spaces. APAP [1] offers a novel shape deformation method, enables
plausibility-aware mesh deformation and preservation of fine details of the
original mesh while offering an interface that alters geometry by directly
displacing a handle along a certain direction. These approaches exemplify the
nuanced control and artistic freedom now achievable in object editing.

Expanding the spectrum of 3D editing techniques, Gaussian-based methods, as
exemplified by GaussianEditor [11, 10], utilize text directives for the
refined manipulation of 3D Gaussians. This underscores the agility, precision,
and controllability inherent in Gaussian representations for 3D edits. In a
complementary vein, Drag3D merges the capabilities of DragGAN[4] and Get3D
[39] to pioneer an interactive 3D dragging edit demonstration. This mesh-based
manipulation technique empowers users with direct, intuitive interactions,
leveraging advanced generative and reconstructive technologies to facilitate
object transformations, thereby enriching the toolkit available for 3D object
editing. However, Drag3D’s mesh-based nature can result in less smooth and
realistic 3D representations, and its reliance on DragGAN restricts it to
predefined semantics, limiting open-ended drag editing.

###  3.3 Multi-view Diffusion Model

Traditional 2D diffusion models[40, 6] are tailored for single-view image
generation, lacking capabilities for 3D viewpoint adjustments. In response,
recent innovations such as SweetDreamer [41], Wonder3D [42], Zero123++ [43],
Viewset Diffusion [44], SyncDreamer [45], Mvdream [18], and Imagedream [46]
have emerged, refining multi-view diffusion models with 3D data to integrate
camera poses, thereby facilitating multi-view image generation of a singular
object from text prompts or single images. Concurrently, 3D editing techniques
like Efficient-NeRF2NeRF [47] have leveraged multi-view enhanced diffusion
models to heighten editing precision.

##  Chapter 4 Methodology

In this section, we formally present the proposed DragGaussian pipeline and
the whole system. The whole pipeline is shown in Fig. 4.4. Users upload pre-
trained 3D Gaussian object models into the system via a UI interface,
whereupon visualization, they can select starting and ending points for
dragging editing or editable regions (Sec. 4.1). Through the projection
module, we obtain the coordinates of the selected points on 2D images from
different camera perspectives. Subsequently, via Multi-view Consistent Editing
(Sec. 4.3), we generate the edited 2D images. Prior to this, to better adapt
the pre-trained 2D editing network to the open-vocabulary input, we fine-tune
the editing network using an improved version of multi-view LoRA (Sec. 4.2).
Finally, we further train the original 3D Gaussian model using the edited 2D
images to present the edited effect of the 3D objects.

Figure 4.1: Pipeline of DragGaussian.

###  4.1 Interactive 3D Point-based Manipulation

The user interface (UI) overview is illustrated in Fig. 4.2. Leveraging an
initial UI repository, we have developed our custom supersplat tool. Several
buttons, including ’start points,’ ’end points,’ and ’brush,’ have been
integrated into this interface. Users have the flexibility to select a precise
$n$ Gaussian points as start points for dragging
($\mathbf{S}={S_{1},S_{2},...,S_{n}}$), followed by the selection of
corresponding end points anywhere within the 3D space
($\mathbf{E}={E_{1},E_{2},...,E_{n}}$). Subsequently, the coordinates of these
points, along with four randomly chosen camera poses
$\mathbf{C}=\\{C_{1},C_{2},C_{3},C_{4}\\}$, are passed to a projection module
$P$. The projection module computes the coordinates of these points on 2D
splatted images of the 3D Gaussian from specific poses, denoted as
$\mathbf{s}=\\{s_{1},s_{2},...,s_{n}\\}$ and
$\mathbf{e}=\\{e_{1},e_{2},...,e_{n}\\}$, using the following functions:

$\mathbf{s}=P(\mathbf{S},\mathbf{C}),\mathbf{e}=P(\mathbf{E},\mathbf{C}).$
(4.1)

(a) Splat size configured to 0.

(b) Spat size configured to 1.

Figure 4.2: Overview of our UI.

Different from other 3D object representation methods, the 3D Gaussian is
explicit, allowing users to directly specify editable Gaussians in 3D space.
In our UI, users can click the ’brush’ button to apply a mask to Gaussian
points eligible for editing, as illustrated in Fig. 3. This set of Gaussian
points, $G_{mask}$, which is subject to updates, will be immediately fed into
the model. During the subsequent fine-tuning process of pre-trained Gaussians,
$G_{mask}$ serves as a control mechanism, ensuring that parameters of
Gaussians not belonging to $G_{mask}$ are not updated.

(a) Splat size configured to 0.

(b) Spat size configured to 1.

Figure 4.3: Drawing mask using the brush.

###  4.2 Multi-view Identity Preserving Fine-tuning

Before editing on multi-view 2d images, we conduct an extra multi-view
identity-preserving fine-tuning on the pre-trained multi-view diffusion U-Net.
This stage aims to ensure that the diffusion denoising U-Net encodes the
features of multi-view images more accurately (than in the absence of this
procedure), thus facilitating the consistency of the identity of the multi-
view images throughout the editing process. Following the fine-tuning method
in DragDiffusion, our fine-tuning process is also implemented with LoRA [48],
whose objective function is

$\mathcal{L}_{\text{ft}}(z,\Delta\theta)=\mathbb{E}_{\epsilon,t}\left[\left\|\epsilon-\epsilon_{\theta+\Delta\theta}(\alpha_{t}z+\sigma_{t}\epsilon)\right\|_{2}^{2}\right],$
(4.2)

where $\theta$ and $\Delta\theta$ represent the U-Net and LoRA parameters
respectively, $z$ is the real image, $\epsilon\sim\mathcal{N}(0,\mathbf{I})$
is the randomly sampled noise map, $\epsilon_{\theta+\Delta\theta}(\cdot)$ is
the noise map predicted by the LoRA-integrated U-Net, and $\alpha_{t}$ and
$\sigma_{t}$ are parameters of the diffusion noise schedule at diffusion step
$t$. The fine-tuning objective is optimized via gradient descent on
$\Delta\theta$.

During fine-tuning, images of $n$ views from the same 3D gaussian are combined
into a single batch, noted as $\mathbf{z}$, sharing a same sampled timestep
$t$ and independently sampled noise maps $\epsilon_{i}$. For each batch, our
loss function can be specified as:

$\mathcal{L}_{\text{batch}}(\mathbf{z},\Delta\theta)=\frac{1}{n}\sum_{i=1}^{n}\left[\left\|\epsilon_{i}-\epsilon_{\theta+\Delta\theta}(\alpha_{t}\mathbf{z}_{i}+\sigma_{t}\epsilon_{i})\right\|_{2}^{2}\right].$
(4.3)

Following the setting of MVDream, we set $n=4$ during fine-tuning.

###  4.3 Multi-view Consistent Editing

Figure 4.4: Stages for Multi-view Consistent Editing.

After multi-view identity preserving fune-tuning, we add noise to the
projected 2d images using DDIM inversion (Sec. 2.2), and then implement motion
supervision and point tracking methods to optimize the diffusion latent
according to the user directions to achieve editing on 2D images.

####  4.3.1 Motion Supervision

We denote the $n$ 2d handle points (start points) at the $k$-th motion
supervision iteration as $\\{h_{i}^{k}=(x_{i}^{k},y_{i}^{k}):i=1,\ldots,n\\}$
and their corresponding target points as
$\\{e_{i}^{k}=(x_{i}^{k},y_{i}^{k}):i=1,\ldots,n\\}$. Each of the input four
images is denoted as $z_{0}$; the $t$-th step latent (the result of $t$-th
DDIM inversion) is denoted as $z_{t}$. We pass $z_{t}$ through the U-Net
architecture extracted from the pre-trained MVDream model. Let $F(z_{t})$
represent the output feature maps of the U-Net used for motion supervision,
and $F_{h_{i}^{k}}(z_{t})$ denote the feature vector at pixel location
$h_{i}^{k}$. Also, we denote the square patch centered around $h_{i}^{k}$ as
$\Omega(h_{i}^{k};r_{1})=\\{(x,y):|x-x_{i}^{k}|\leq r_{1},|y-y_{i}^{k}|\leq
r_{1}\\}$. Then, the motion supervision loss at the $k$-th iteration is
defined as:

$\displaystyle L_{ms}(z_{t}^{k})$
$\displaystyle=\sum_{i=1}^{n}\sum_{q}\Omega\left(\left\|F_{q+d_{i}}(\hat{z}_{t}^{k})-sg(F_{q}(\hat{z}_{t}^{k}))\right\|_{1}\right.$
(4.4)
$\displaystyle\quad+\lambda\left\|\left(\hat{z}_{t}^{k}-sg(\hat{z}_{t-1}^{0})\right)\odot\left(1-M\right)\right\|_{1},$

where $\hat{z}_{t}^{k}$ is the $t$-th step latent after the $k$-th update,
$\text{sg}(\cdot)$ is the stop gradient operator (i.e., the gradient will not
be back-propagated for the term $\text{sg}(F_{q}(\hat{z}_{t}^{k})))$,
$d_{i}=\left(\frac{{g_{i}-\hat{h}_{k}^{i}}}{{\|g_{i}-\hat{h}_{k}^{i}\|_{2}}}\right)$,
$\|g_{i}-\hat{h}_{k}^{i}\|_{2}$ is the normalized vector pointing from
$\hat{h}_{k}^{i}$ to $g_{i}$, $M$ is the binary mask specified by the user,
$F_{q+d_{i}}(\hat{z}_{t}^{k})$ is obtained via bilinear interpolation as the
elements of $q+d_{i}$ may not be integers. In each iteration,
$\hat{z}_{t}^{k}$ is updated by taking one gradient descent step to minimize
$L_{ms}$:

$\hat{z}_{t}^{k+1}=\hat{z}_{t}^{k}-\eta\frac{{\partial
L_{ms}(\hat{z}_{t}^{k})}}{{\partial\hat{z}_{t}^{k}}},$ (4.5)

where $\eta$ is the learning rate for latent optimization.

####  4.3.2 Point Tracking

Since the motion supervision updates $\hat{z}_{t}^{k}$ on the four 2d images,
the positions of the handle points on them may also change. Therefore, we need
to perform point tracking on each of the four images to update the handle
points after each motion supervision step. To achieve this goal, we use Multi-
view U-Net feature maps $F(\hat{z}_{t}^{k+1})$ and $F(\hat{z}_{t})$ to track
the new start points. Specifically, we update each of the start points
$h_{i}^{k}$ with a nearest neighbor search within the square patch
$\Omega(h_{i}^{k};r2)=\\{(x,y):|x-x_{i}^{k}|\leq r2,|y-y_{i}^{k}|\leq r2\\}$
as follows:

$h_{i}^{k+1}=\text{arg
min}_{q\in\Omega(h_{i}^{k};r2)}\|F_{q}(\hat{z}_{t}^{k+1})-F_{h_{i}^{0}}(\hat{z}_{t})\|_{1}.$
(4.6)

####  4.3.3 Multi-view Consistent Denoising

After we get editted 2d images from several viewpoints (here we set it as 4),
we come to the stage to denoise them. Implementing the U-net pre-trained by
MVDream (Sec. 2.3) and finetuned by LoRA (Sec. 4.2) to predict noise,
following the equation of DDIM Sampling (Equ. 2.3), the four edited latent 2d
images can be denoised step by step. Same as the Sampling process in
DragDiffusion which uses U-net from Stable Diffusion [40], we replace the key
and value vectors generated from $\hat{z}_{t}$ with the ones generated from
$z_{t}$. With this simple replacement technique, the query vectors generated
from $\hat{z}_{t}$ will be directed to query the correlated contents and
texture of $z_{t}$.

##  Chapter 5 Experiments

###  5.1 Implementation Details

In our experimental setup, we configured the number of steps for DDIM
Inversion and Sampling to be 50, and set the guidance scale $\eta$ to 1.0.
Specifically, during the DDIM Sampling process, we shared the Key-Value (KV)
pairs from layer 8 out of 10, and from step 0. When editing real images, we do
not apply classifier-free guidance (CFG) [49] in both DDIM Inversion and DDIM
Denoising process. For the LoRA fine-tuning process, we employed a learning
rate of 0.0005 for 300 LoRA steps. During the Multi-view Consistent Editing
Module, we use the Adam optimizer with a learning rate of 0.01 to optimize the
latent. The maximum optimization step is set to be 80. The hyperparameter
$r_{1}$ in Eqn. 4.4 and $r_{2}$ in Equ. 4.6 are tuned to be 1 and 3,
respectively. $\lambda$ in Equ. 4.4 is set to 0.1 by default. Our experiments
were conducted utilizing 2 NVIDIA RTX 3090 GPUs.

###  5.2 Editing Results

Figure 5.1: Multi-view consistent editing on a chair.

Fig. 5.3 shows the editing results of a pretrained lego 3D Gaussian. Images on
the left column are rendered from initial well-trained Gaussian, images on the
middle column are 2d editing results of the model with start points (red) and
end points (blue), and images on the right column are rendered results of the
editted 3D Gaussian after finetuning 5000 iterations. In the editing results
of both 2D and 3D, we can clearly see the shrinkage deformation caused by
editing the LEGO tractor. However, constrained by the limitations of the 2D
drag editing model, the resolution of the 2D editing results is lower, further
leading to a blurred visual effect of the edited 3D Gaussian model.

We also compared our method with Drag3D, a 3d mesh editing methods based on
dragging operation. Because of the usage of diffusion model, our method is
much power than Drag3D, which extends the idea of DragGAN into GET3D. However,
the representation of 3D Gaussian provides a much clearer rendering result and
also provide users the possibility to choose editing areas explicitly.

###  5.3 Ablation Study

Figure 5.2: Editing and masking on a Gaussian chair.

To showcase the effectiveness of our editing approach for 2D images using the
pre-trained MVDream network, we conducted several preliminary experiments.
These experiments involved editing both multi-view 2D images derived from 3D
Gaussian datasets’ projections and multi-view images generated by MVDream
itself.

In Fig.5.1, an exemplary editing outcome is presented. The chair data
originates from the NeRF Synthetic dataset and was utilized for pre-training a
3D Gaussian model. Fig.5.2 demonstrates the user’s editing process and mask
selection, depicting the utilization of brush tools for mask creation. The
editing outcome on an open vocabulary dataset reveals conspicuous traces of
editing on the chair’s back, along with evident distortions in unmasked areas
such as the chair legs. Additionally, the overall size of the chair appears
diminished post-editing. The constraints imposed by the pre-trained MVDream
U-net’s generalizability become apparent when confronted with unseen data,
highlighting limitations in our 2D consistent editing pipeline.

Figure 5.3: Drawing mask using the brush.

Nonetheless, our reliable 2D editing pipeline adeptly handles data generated
by the U-net. When provided with a text prompt like ”A chair,” MVDream
generates chair images from four different viewpoints. Subsequently, we feed
these four chairs into DDIM Inversion, the 2D image editing process, and the
multi-view denoising stage within DragGaussian’s pipeline, as illustrated in
Fig.4.3. To enhance the editing process, we include four pairs of manually
selected start and end points. The resulting purely 2D edited images are
depicted in Fig.5.4.

Figure 5.4: Editing results on multi-view images generated by MVDream. (a)
column shows the initial images, (b) colunm shows the images after DDIM
Inversion and DDIM Sampling process without editing, (c) column shows the
images edited by our multi-view image consistent editing pipeline without LoRA
finetune, and (d) column shows the editing results with LoRA finetune.

To validate the effectiveness of LoRA finetuning, we contract the editing
result on multi-view 2d images with and without LoRA finetune. In Fig. 5.4,
(c) column shows the images edited by our multi-view image consistent editing
pipeline without LoRA finetune, and (d) column shows the editing results with
LoRA finetune. It’s evident that editing outcomes lacking LoRA fine-tuning
fail to maintain the original background color and inadvertently alter parts
of the chair that weren’t intended to be edited. Conversely, employing LoRA
fine-tuning on the U-net of MVDream yields significantly superior results,
preserving undisturbed regions more effectively.

##  Chapter 6 Discussion

Miss Point Tracking Issue The point tracking method used in our DragGaussian
bases on a traditional framework for point based image editing. However, there
may exist several issues in this tracking framework, which is discussed in a
recent work called FreeDrag [27].

One issue is miss tracking, where the process of dragging points faces
difficulty in accurately following the desired handle points. This problem is
especially prevalent in highly curved areas with a significant perceptual path
length, as observed within latent space [50]. In such instances, the optimized
image undergoes significant alterations, causing handle points in subsequent
iterations to be placed outside the intended search area. Furthermore, in
certain cases, miss tracking can result in handle points disappearing. It’s
noteworthy that during miss tracking, the cumulative error in the motion
supervision step progressively increases as iterations continue, due to the
misalignment of tracked features. Another issue is ambiguous tracking, where
tracked points are positioned within regions resembling the handle points.
This dilemma occurs when parts of the image share features similar to the
intended handle points, leading to uncertainty in the tracking process. This
problem presents a significant challenge as it can mislead the motion
supervision process in subsequent iterations, resulting in inaccurate or
misleading guidance.

##  Chapter 7 Conclusion

In this work, we introduced DragGaussian, an innovative 3D object drag-editing
framework built upon the principles of 3D Gaussian Splatting and leveraging
diffusion models for interactive image editing with open-vocabulary input.
This framework empowers users to engage in drag-based editing on pre-trained
3D Gaussian object models, resulting in the production of modified 2D images
through multi-view consistent editing. Our contributions encompass the
inception of a novel editing task, the development of DragGaussian for
interactive point-based 3D editing, and the thorough validation of its
efficacy through both qualitative and quantitative experiments.

While our experiments have demonstrated the effectiveness of our 3D editing
method, several limitations remain evident. Notably, the utilization of
diffusion models impedes real-time editing capabilities. Moreover, constraints
associated with the pre-trained MVDream network may hinder the attainment of
elegant editing results across certain datasets.

Moving forward, our focus will center on the development of real-time editing
frameworks directly tailored to 3D Gaussian representations. By addressing
these drawbacks and refining our approach, we aim to further enhance the
utility and versatility of DragGaussian, thereby advancing the landscape of 3D
object editing technologies.

## Bibliography

  * [1] Seungwoo Yoo, Kunho Kim, Vladimir G Kim and Minhyuk Sung  “As-Plausible-As-Possible: Plausibility-Aware Mesh Deformation Using 2D Diffusion Priors”  In _arXiv preprint arXiv:2311.16739_ , 2023
  * [2] Ayaan Haque et al.  “Instruct-nerf2nerf: Editing 3d scenes with instructions”  In _arXiv preprint arXiv:2303.12789_ , 2023
  * [3] Tim Brooks, Aleksander Holynski and Alexei A Efros  “Instructpix2pix: Learning to follow image editing instructions”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18392–18402
  * [4] Xingang Pan et al.  “Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold”  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023
  * [5] Ian Goodfellow et al.  “Generative adversarial networks”  In _Communications of the ACM_ 63.11 ACM New York, NY, USA, 2020, pp. 139–144
  * [6] Chitwan Saharia et al.  “Photorealistic text-to-image diffusion models with deep language understanding”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 36479–36494
  * [7] Jonathan T Barron et al.  “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5855–5864
  * [8] Jonathan T Barron et al.  “Mip-nerf 360: Unbounded anti-aliased neural radiance fields”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5470–5479
  * [9] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis  “3D Gaussian Splatting for Real-Time Radiance Field Rendering”  In _ACM Transactions on Graphics_ 42.4, 2023
  * [10] Yiwen Chen et al.  “Gaussianeditor: Swift and controllable 3d editing with gaussian splatting”  In _arXiv preprint arXiv:2311.14521_ , 2023
  * [11] Jiemin Fang et al.  “Gaussianeditor: Editing 3d gaussians delicately with text instructions”  In _arXiv preprint arXiv:2311.16037_ , 2023
  * [12] Jonathan Ho, Ajay Jain and Pieter Abbeel  “Denoising diffusion probabilistic models”  In _Advances in neural information processing systems_ 33, 2020, pp. 6840–6851
  * [13] Robin Rombach et al.  “High-resolution image synthesis with latent diffusion models”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10684–10695
  * [14] Diederik P Kingma and Max Welling  “Auto-encoding variational bayes”  In _arXiv preprint arXiv:1312.6114_ , 2013
  * [15] Olaf Ronneberger, Philipp Fischer and Thomas Brox  “U-net: Convolutional networks for biomedical image segmentation”  In _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_ , 2015, pp. 234–241  Springer
  * [16] Ashish Vaswani et al.  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [17] Jiaming Song, Chenlin Meng and Stefano Ermon  “Denoising diffusion implicit models”  In _arXiv preprint arXiv:2010.02502_ , 2020
  * [18] Yichun Shi et al.  “Mvdream: Multi-view diffusion for 3d generation”  In _arXiv preprint arXiv:2308.16512_ , 2023
  * [19] Matt Deitke et al.  “Objaverse: A universe of annotated 3d objects”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 13142–13153
  * [20] Christoph Schuhmann et al.  “Laion-5b: An open large-scale dataset for training next generation image-text models”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 25278–25294
  * [21] Alec Radford et al.  “Learning transferable visual models from natural language supervision”  In _International conference on machine learning_ , 2021, pp. 8748–8763  PMLR
  * [22] Fangneng Zhan et al.  “Multimodal image synthesis and editing: A survey and taxonomy”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ IEEE, 2023
  * [23] Fangneng Zhan et al.  “Bi-level feature alignment for versatile image translation and manipulation”  In _European Conference on Computer Vision_ , 2022, pp. 224–241  Springer
  * [24] Haitian Zheng et al.  “Semantic layout manipulation with high-resolution sparse attention”  In _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45.3 IEEE, 2022, pp. 3768–3782
  * [25] N Umetani, E Vouga and C Wojtan  “User-Controllable Latent Transformer for StyleGAN Image Layout Editing”, 2022
  * [26] Sheng-Yu Wang, David Bau and Jun-Yan Zhu  “Rewriting geometric rules of a gan”  In _ACM Transactions on Graphics (TOG)_ 41.4 ACM New York, NY, USA, 2022, pp. 1–16
  * [27] Pengyang Ling et al.  “Freedrag: Point tracking is not you need for interactive point-based image editing”  In _arXiv preprint arXiv:2307.04684_ , 2023
  * [28] Yujun Shi et al.  “DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing”  In _arXiv preprint arXiv:2306.14435_ , 2023
  * [29] Chong Mou et al.  “Dragondiffusion: Enabling drag-style manipulation on diffusion models”  In _arXiv preprint arXiv:2307.02421_ , 2023
  * [30] Chong Bao et al.  “Sine: Semantic-driven image-based nerf editing with prior-guided editing field”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 20919–20929
  * [31] Can Wang et al.  “Nerf-art: Text-driven neural radiance fields stylization”  In _IEEE Transactions on Visualization and Computer Graphics_ IEEE, 2023
  * [32] William Gao et al.  “Textdeformer: Geometry manipulation using text guidance”  In _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023, pp. 1–11
  * [33] Can Wang et al.  “Clip-nerf: Text-and-image driven manipulation of neural radiance fields”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3835–3844
  * [34] Jangho Park, Gihyun Kwon and Jong Chul Ye  “ED-NeRF: Efficient Text-Guided Editing of 3D Scene using Latent Space NeRF”  In _arXiv preprint arXiv:2310.02712_ , 2023
  * [35] Jingyu Zhuang et al.  “Dreameditor: Text-driven 3d scene editing with neural fields”  In _SIGGRAPH Asia 2023 Conference Papers_ , 2023, pp. 1–10
  * [36] Ashkan Mirzaei et al.  “Watch your steps: Local image and scene editing by text instructions”  In _arXiv preprint arXiv:2308.08947_ , 2023
  * [37] Yuhan Li et al.  “Focaldreamer: Text-driven 3d editing via focal-fusion assembly”  In _arXiv preprint arXiv:2308.10608_ , 2023
  * [38] Jiraphon Yenphraphai et al.  “Image Sculpting: Precise Object Editing with 3D Geometry Control”  In _arXiv preprint arXiv:2401.01702_ , 2024
  * [39] Jun Gao et al.  “GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images”  In _Advances In Neural Information Processing Systems_ , 2022
  * [40] Robin Rombach et al.  “High-resolution image synthesis with latent diffusion models”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10684–10695
  * [41] Weiyu Li, Rui Chen, Xuelin Chen and Ping Tan  “Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d”  In _arXiv preprint arXiv:2310.02596_ , 2023
  * [42] Xiaoxiao Long et al.  “Wonder3d: Single image to 3d using cross-domain diffusion”  In _arXiv preprint arXiv:2310.15008_ , 2023
  * [43] Ruoxi Shi et al.  “Zero123++: a single image to consistent multi-view diffusion base model”  In _arXiv preprint arXiv:2310.15110_ , 2023
  * [44] Stanislaw Szymanowicz, Christian Rupprecht and Andrea Vedaldi  “Viewset Diffusion:(0-) Image-Conditioned 3D Generative Models from 2D Data”  In _arXiv preprint arXiv:2306.07881_ , 2023
  * [45] Yuan Liu et al.  “Syncdreamer: Generating multiview-consistent images from a single-view image”  In _arXiv preprint arXiv:2309.03453_ , 2023
  * [46] Peng Wang and Yichun Shi  “ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation”  In _arXiv preprint arXiv:2312.02201_ , 2023
  * [47] Liangchen Song et al.  “Efficient-NeRF2NeRF: Streamlining Text-Driven 3D Editing with Multiview Correspondence-Enhanced Diffusion Models”  In _arXiv preprint arXiv:2312.08563_ , 2023
  * [48] Edward J Hu et al.  “Lora: Low-rank adaptation of large language models”  In _arXiv preprint arXiv:2106.09685_ , 2021
  * [49] Jonathan Ho and Tim Salimans  “Classifier-free diffusion guidance”  In _arXiv preprint arXiv:2207.12598_ , 2022
  * [50] Tero Karras, Samuli Laine and Timo Aila  “A style-based generator architecture for generative adversarial networks”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4401–4410