#  D-MiSo: Editing Dynamic 3D Scenes using Multi-Gaussians Soup

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

Over the past years, we have observed an abundance of approaches for modeling
dynamic 3D scenes using Gaussian Splatting (GS). Such solutions use GS to
represent the scene’s structure and the neural network to model dynamics. Such
approaches allow fast rendering and extracting each element of such a dynamic
scene. However, modifying such objects over time is challenging. SC-GS (Sparse
Controlled Gaussian Splatting) enhanced with Deformed Control Points partially
solves this issue. However, this approach necessitates selecting elements that
need to be kept fixed, as well as centroids that should be adjusted throughout
editing. Moreover, this task poses additional difficulties regarding the re-
productivity of such editing. To address this, we propose Dynamic Multi-
Gaussian Soup (D-MiSo), which allows us to model the mesh-inspired
representation of dynamic GS. Additionally, we propose a strategy of linking
parameterized Gaussian splats, forming a Triangle Soup with the estimated
mesh. Consequently, we can separately construct new trajectories for the 3D
objects composing the scene. Thus, we can make the scene’s dynamic editable
over time or while maintaining partial dynamics.

##  1 Introduction

Figure 1: D-MiSo model parameterized dynamic scenes by Triangle Soup (disjoint
triangles cloud), which allows modification of objects during time.

Recently introduced Gaussian Splatting (GS) kerbl20233d represents the 3D
scene structure through Gaussian components. We can combine GS with the neural
network (i.e., deform network) to model dynamic scenes kratimenos2024dynmf.
This approach involves the joint training of both the GS components and the
neural network. GS characterizes the 3D object’s shape and color, while the
neural network utilizes time embedding and Gaussian parameters to generate
updated initial positions to model dynamic scenes. Such an approach allows for
fast rendering and extracting each element of a dynamic scene.

Figure 2: Each object using the D-MiSo model is represented by Core-Gaussians
and Sub-Gaussians, which form Multi-Gaussians. Each Gaussian is related to a
triangle using parameterization proposed in GaMeS waczynska2024games.
Triangles define the Gaussian shape (i.e., location, scale, rotation), and
triangles clouds form Triangles Soups.

Most existing methods can effectively model dynamic scenes, but generating new
3D objects’ positions remains challenging. Consequently, we cannot edit
objects over time when using such approaches. To tackle this issue, SC-GS
(Sparse Controlled Gaussian Splatting) huang2023sc uses Deformed Control
Points to manage Gaussians. After the training phase, we can manually modify
the model at any point in time. However, this method requires identifying
elements to remain static and adjusting 3D objects’ centroids (nodes) during
editing when a relationship between the selected nodes is visible. For
example, by moving the humanoid 3D model’s hand, the part of the head or leg
is also changed.

To address this issue, we introduce Dynamic Multi-Gaussian Soup (D-MiSo),
which is easier to modify (see Fig. 1) and obtain renders comparable to SC-GS.
D-MiSo estimates the mesh as Triangle Soup, i.e. a set of disconnected
triangle faces 1290060; inproceedings, and uses a dynamic function to control
the vertices.

Figure 3: D-MiSo allows us to modify scenes in similar ways as classical mesh-
based models.

Thanks to using Triangle Soup, we control two types of Gaussian components.
D-MiSo employing Multi-Gaussians, defined as larger Core-Gaussians
encompassing smaller ones termed Sub-Gaussians (Fig. 2). Sub-Gaussians are
defined in the local coordinate system given by principal components of Core-
Gaussian. Therefore, by modifying Core-Gaussian, we change all Sub-Gaussians,
which allows scene modifications (Fig 3). Core-Gaussians are an alternative to
the control points discussed in huang2023sc, with the added advantage of
allowing individual modifications. Consequently, there is no necessity for
static and dynamic markers. In Fig. 4 we present the difference between
modification applied by SC-GS and D-MiSo. In D-MiSo, we can select and modify
one part of the object like a mesh. In contrast, we must select static and
dynamic points when using SC-GS, but editing only one part of the object is
difficult.

In D-MiSo, we use flat Gaussians from GaMeS waczynska2024games, which can be
parametrized by triangle faces. Consequently, we obtain two Triangle Soups as
shown in Fig. 2 where Core-Triangle Soup is marked by red color, and Sub-
Triangle Soup is denoted in blue. During training, the positions of Core-
Gaussians are managed by deformation MLP, while the Sub-Gaussians are
collectively manipulated through global transformation and small local
deformation. The former describes the general flow of objects in the scene,
while the local deformation is responsible for modeling small changes like
shadows and light reflections. After training, we can modify our model
directly by using the vertex of the Sub-Triangle Soup, or we can generate mesh
from the Core-Triangle Soup (Fig. 5).

Figure 4: Comparison of possible modifications in D-MiSo and the SC-GS. In the
latter, authors use nodes while D-MiSo apply Sub-Triangle Soup (see the second
column). We also must add static (pink) and dynamic (yellow) points in SC-GS
to obtain modification by editing dynamic points. In practice, we have to use
many static points to stop artifacts. Moreover, SC-GS is not an affine
invariant and produces space when we change the size of the objects. In the
case of D-MiSo, we marked points and applied modifications. Our model is
superior in handling object scaling.

The contributions of this paper are significant and are outlined as follows:

  * •

We introduce the Multi-Gaussian components, which consist of a single large
Gaussian response for global transformations and many small components
dedicated to rendering. Multi-Gaussian components allow for the modeling of
large 3D scene elements.

  * •

We propose D-MiSo a model that uses Multi-Gaussian components and two
deformation networks for modeling dynamic scenes.

  * •

Our D-MiSo allows an object to be edited at a selected moment in time. The
edited components are independent, and the editing does not affect other parts
of the object. In addition, it also allows for full or partial dynamics to be
maintained. Modifications also include scaling and rotation.

##  2 Related Works

Recent advancements in view synthesis, particularly driven by NeRFs
mildenhall2020nerf, have significantly contributed to the rapid development of
novel view synthesis techniques. However, majority of these approaches model
static scenes implicitly using multilayer perceptron (MLP). Moreover, several
works have extended classical NeRF to dynamic scenes through the use of
deformation fields park2021nerfies; park2021hypernerf; li2021neural and
pumarola2020dnerf. The alternative approaches, such as Gao-ICCV-DynNeRF and
li-Neural3DVideoSynthesis, represent scenes as 4D radiance fields. However,
NeRF-based solutions often suffer from long training and rendering times. To
address this, grid-plane-based methods Cao2023HexPlane; kplanes_2023;
shao2023tensor4d have been proposed. In addition, several NeRF-based
approaches have also been extended for scene editing purposes
yuan2022nerfediting; zheng2023editablenerf; kania2022conerf;
Wang2023RIPNeRFLR.

The recently introduced Gaussian Splatting technique kerbl20233d addresses
many limitations of other methods, offering multiple advantages due to their
explicit geometry representation, enabling easier dynamics modeling. The
efficient rendering of 3D GS also avoids densely sampling and querying neural
fields, making downstream applications such as free-viewpoint video
reconstruction more feasible. While the original GS was developed for static
scenes, several extensions for dynamic scenes were proposed. For example,
luiten2023dynamictracking utilizes a multiview dynamic dataset and a frame-by-
frame approach to model such scenes. However, this method lacks inter-frame
correlation and requires high storage overhead for long-term sequences. In
yang2023deformable3dgs; kratimenos2024dynmf MLP is introduced to model changes
in Gaussians over time, and in wu20234dgaussians MLP together with decomposed
neural voxel encoding algorithm are utilized for training and storage
efficiency. In liang2023gaufre, dynamic scenes are divided into dynamic and
static parts, optimized separately and rendered together to achieve
decoupling. Other works enhance dynamic scene reconstruction using external
priors. For example, the diffusion priors can be used as regularization terms
during optimization zhang2024bags.

Furthermore, GS was employed for mesh-based scene geometry editing. In
gao2024meshbased 3D Gaussians are defined over an explicit mesh and utilizes
mesh rendering to guide adaptive refinement. This approach depends on the
extracted mesh as a proxy and fails if the mesh cannot be extracted. In
contrast, in guedon2023sugar explicit meshes are extracted from 3D GS
representations by regularizing Gaussians over surfaces. However, this method
involves a costly optimization and refinement pipeline. Another example of
huang2023sc-gs employs sparse control points for 3D scene dynamics, but this
method struggles with intense edit movements and necessitates accurate static
node selection. Also, waczynska2024games combines GS with mesh extraction.
However, it only works for static scenes. In contrast to all these approaches,
we propose a D-MiSo, a mesh-based method specifically designed to handle
dynamic scenes. D-MiSo leverages a straightforward pipeline of GS techniques
to enable real-time editing of dynamic scenes.

Figure 5: One way to modify the object at the selected time $t_{i}$ is to take
Core-Gaussians and apply a meshing strategy to obtain the correct mesh instead
of Triangle Soup. Then, we can parametrize Sub-Gaussian in the coordinate
system given bay mesh faces instead of Core-Triangle Soup. Finally, we can
modify our mesh to obtain new modifications.  Figure 6: Multi-Gaussian,
consisting of one Core-Gaussian $\mathcal{G}_{V_{j}}$ and a Sub-Gaussian
$\mathcal{G}_{V_{j}}^{i}$. The Core Gaussian is parametrized by a
$V_{j}$-triangle, and the Sub-Gaussian by a $V_{j}^{i}$-triangle. The relative
distance of the center of the Sub-Gaussian from the Core-Gaussian is indicated
by $\boldsymbol{\alpha^{i}}=(\alpha^{i}_{1},\alpha^{i}_{2},\alpha^{i}_{3})$.

##  3 Dynamic Multi-Gaussian Soup

Here we present the main components of D-MiSo. We start with the classical GS
to provide the foundations for our model. Next, we introduce the concept of
Multi-Gaussians and present how to estimate the mesh for editing. Finally, we
show D-MiSo, which uses Multi-Gaussians in dynamic 3D scenes.

#### Gaussian Splatting

The Gaussian Splatting (GS) technique models 3D scenes using an array of 3D
Gaussians, each specified by its mean position, covariance matrix, opacity,
and color expressed using spherical harmonics (SH) fridovich2022plenoxels;
muller2022instant. The GS algorithm constructs the radiance field by
iteratively optimizing the parameters of all Gaussian components. Ultimately,
the GS efficiency mainly depends on its rendering method, which involves
projecting Gaussian components.

The GS framework employs a dense collection of 3D Gaussians:
$\mathcal{G}=\\{(\mathcal{N}({\bf
m}_{i},\Sigma_{i}),\sigma_{i},c_{i})\\}_{i=1}^{n},$ where ${\bf m}_{i}$
denotes the position, $\Sigma_{i}$ the covariance, $\sigma_{i}$ the opacity,
and $c_{i}$ the SH colors for the $i$-th Gaussian. The GS optimization process
involves a repetitive cycle of rendering and comparing the resultant images
with the training views. In our work, we will use Multi-Gaussian approaches
(Fig. 6).

#### Multi-Gaussians

Multi-Gaussians $\mathcal{G}_{multi}$ are dedicated to describing relatively
large parts of the 3D scene to allow modification of large blocks instead of
modifying each small Gaussian separately. The Multi-Gaussian model comprises a
primary large 3D Gaussian (referred to as the Core-Gaussian
$\mathcal{G}_{core}$), which encompasses numerous smaller Gaussians (termed
Sub-Gaussians $\mathcal{G}_{sub}$), all of which are parameterized by the main
Core-Gaussian. Multi-Gaussianare is similar to anchor Gaussians from
yang2024specgaussian, but we do not use a neural network to produce child
components. We parametrize Sub-Gaussians in a local coordinate system.

Similarly to classical GS, we parameterize the Core-Gaussian distribution by
center ${\bf m}$ and the covariance parameterized by factorization:
$\Sigma=RSSR^{T},$ where $R$ is the rotation matrix and $S$ the scaling
parameters. More precisely we consider $p$ Core-Gaussians uses flat Guassinas
as in waczynska2024games, and is defined by:

$\mathcal{G}_{core}=\\{(\mathcal{N}_{core}({\bf
m}_{i},R_{i},S_{i}),\sigma_{i},c_{i})\\}_{i=1}^{p},$ (1)

where $S=\mathrm{diag}(s_{1},s_{2},s_{3}),$ $s_{1}=\varepsilon$ and $R$ is
rotation matrix of Core-Gaussian defined as: $R=({\bf r}_{1},{\bf r}_{2},{\bf
r}_{3}),\mbox{ where }{\bf r}_{i}\in\mathbb{R}^{3}$ which can be interpreted
as a local coordinate system used by Sub-Gaussian.

Sub-Gaussian can be interpreted as a child of Core-Gausian. We define centers
of Sub-Gaussian $\mathcal{N}_{sub}({\bf m}^{i},R^{i},S^{i})$ in the local
coordinate system of Core-Gaussian $\mathcal{N}_{core}({\bf m},R,S)$ by:

${\bf m}^{i}={\bf m}+R\boldsymbol{\alpha^{i}}^{T}$ (2)

where ${\bf m}$, $R$ is Core-Gaussian position and rotation; and
$\boldsymbol{\alpha^{i}}=(\alpha^{i}_{1},\alpha^{i}_{2},\alpha^{i}_{3})$ are
trainable parameters used to define the positions of the Sub-Gaussian relative
to the Core-Gaussian (Fig. 6). Sub-Gaussians are used for rendering and can be
seen as a main component of our model:

$\mathcal{G}_{sub}=\left\\{\left(\mathcal{N}_{sub}\left({\bf
m}+R\boldsymbol{\alpha^{i}}^{T},R^{i},S^{i}\right),\sigma^{i},c^{i}\right)\right\\}_{i=1}^{k},$
(3)

where ${\bf m},R,S$ are parameters of Core-Gaussian and,
${\boldsymbol{\alpha}}^{i},S^{i},R^{i}$ marks parameters of $i$-th Sub-
Gaussians with opacity $\sigma^{i}$, and SH colors $c^{i}$.

Figure 7: D-MiSo model diagram. The input of the model consists of images at
different moments in time and information regarding the position of the
camera. Training distinguishes two main phases (i.e., stages). The first
includes the preparation of Core-Gaussians, describing the movement of the
object. The second focuses on the fitting of Sub-Gaussians responsible for the
render’s quality. The model has the ability to produce high-quality renders or
create an animation/modification of the object due to Sub-Gaussians (i.e.,
Sub-Triangles Soup) shape modification. Figure 8: Representation of change
over time acting on Core-Gaussians using a neural network responsible for
movement. In practice, $t_{i}$ is an abstract time. The network’s output
returns information about the change in location $\Delta v_{1}$, scale
($\Delta v_{2}$, $\Delta v_{3}$), and rotation $\Delta R$.

Core-Gaussians is generally dedicated mainly to transformations, hence, in
practice opacity or colors are not used during rendering. On the other hand,
Sub-Gaussian is devoted to rendering and modification. It has its own opacity
and colors.

One of our mode’s most important properties is its ability to model dynamic
scenes in each time step. To obtain such properties, we parameterize all
Gaussian using Triangle Soup proposed by GaMeS waczynska2024games. Thanks to a
few simple transformations, we can convert the mean ${\bf m}$ rotation $R$ and
scaling $S$ into triangle $V=({\bf v}_{1},{\bf v}_{2},{\bf v}_{3})$, which
parametrize Gaussian distribution. Such transformation is unambiguous and
reversible, for more details see Appedix A. GaMeS transformation between
Gaussian parameters and triangle face we denote by $\mathcal{T}$:

$\mathcal{N}(V)=\mathcal{N}(\mathcal{T}^{-1}(V))=\mathcal{N}(\hat{\bf
m}_{V},\hat{R}_{V},\hat{S}_{V})$

In our D-MiSo we parameterize the $p$ Core-Gaussians with $k$ Triangle soup:

$\mathcal{G}_{multi}=\left\\{\left(\mathcal{N}_{core}(V_{j}),\left\\{\left(\mathcal{N}_{sub}\left(\hat{\bf
m}_{V_{j}}+\hat{R}_{V_{j}}\boldsymbol{\alpha^{i}}^{T},\hat{R}^{i},\hat{S}^{i}\right),\sigma^{i},c^{i}\right)\right\\}_{i=1}^{k}\right)\right\\}_{j=1}^{p},$
(4)

where $V_{j}$, $\boldsymbol{\alpha^{i}}$, $R^{i}$,$\hat{S}^{i}$, $\sigma^{i}$,
$c^{i}$ are trainable parameters and $(\hat{\bf
m}_{V_{j}},\hat{R}_{V_{j}},\hat{S}_{V_{j}})=\mathcal{T}^{-1}(V_{j})$.
Alternatively, we can parameterize Core-Gaussian and Sub Gaussians by Triangle
Soup:

$\mathcal{G}_{multi}=\left\\{\left(\mathcal{N}_{core}(V_{j}),\left\\{\left(\mathcal{N}_{sub}\left(V_{j}^{i}\right),\sigma^{i},c^{i}\right)\right\\}_{i=1}^{k}\right)\right\\}_{j=1}^{p},$
(5)

where $(\hat{\bf
m}_{V_{j}},\hat{R}_{V_{j}},\hat{S}_{V_{j}})=\mathcal{T}^{-1}(V_{j})$,
$V^{i}=\mathcal{T}(\hat{\bf
m}_{V}+\hat{R}_{V}\boldsymbol{\alpha^{i}}^{T},\hat{R}^{i},\hat{S}^{i})$.

Figure 9: One way to animate is to assign the nearest triangle (face) to each
Sub-Gaussian from the estimated mesh. The mesh modification changes the
assigned Gaussian.

In D-MiSo, we use the collation of Multi-Gaussian distribution for rendering
and Sub-Triangle Soup for editing. The formal definition of our model uses
equation (4) since in training, we store Core-Gaussian as a triangle face
(Core-Gaussian does not have colours) and Sub-Gaussian as a collection from
classical GS components with colour and opacity. After training, we
parametrize our model to equation (5) for editing.

###  3.1 Dynamic Multi-Gaussian Soup (D-MiSo)

Figure 10: An example of Sub-Triangle Soup modification and the render
obtained by this change from a different viewpoint. It is possible not only to
change the position of the hand but also to raise the thumb.

Previously, we defined Multi-Gaussians and their parametrization using
Triangle Soup. Now, we have all the tools to present the D-MiSo model. The
overview of our method is illustrated in Fig. 7. The input to our model is a
set of images of a dynamic scene, together with the time label and the
corresponding camera poses. Our training is divided into two stages. In the
first stage, we initialise the Core-Gaussians. In the second, we add Sub-
Gaussian components.

#### Stage 1

First, we train only Core-Gaussians to obtain good initialization for Multi-
Gaussins. As Core-Gaussians are mainly employed to capture motion, the model
does not require much of them, see Fig. 7. In our approach, the Core-Triangle
Soup, constructed via the Core-Gaussians parameterization (as depicted in Fig.
2), is adjusted depending on the time t.

Practically, when random initialization of Gaussians is necessary, redundant
Gaussians must be pruned first to ensure that the remaining ones emphasize the
object’s shape. To reduce the number of Gaussians and obtain consistent Core-
Gaussians, we train GS on a batch containing a few views instead of one. In
practice, we render few views from different positions and use back-
propagation.

Figure 11: Reconstruction and three ways of modification of the output object.
The first involves modifying the estimated mesh, which does not have to be
accurate. The next two focus on Sub-Triangle Soup editing. The red box shows
the direct modification of the Triangle Soup in a logical way (e.g., raising a
hand). The yellow box shows a change in space, i.e., giving fluidity to an
object by creating an abstract modification.

In particular, we parameterize Gaussians $\mathcal{N}({\bf m},R,S)$ by face
$V=({\bf v}_{1},{\bf v}_{2},{\bf v}_{3})$ to obtain $\mathcal{N}({\bf
v}_{1},{\bf v}_{2},{\bf v}_{3})$. Our Deform Network takes as in input the
triangle vertixes $V$ assigned to the parameterized 3D Core-Gaussians and the
current time $t$ and returns updated $\psi(V,t)=(\Delta{\bf
v}_{1}(t),\Delta{\bf v}_{2}(t),\Delta{\bf v}_{3}(t),\Delta R_{V}(t)).$ Such
updates consist of translation and rotation (see Fig. 8):

$V(t)=V\odot\psi(V,t)=({\bf v}_{1}+\Delta{\bf v}_{1}(t),{\bf v}_{2}+\Delta{\bf
v}_{2}(t),{\bf v}_{3}+\Delta{\bf v}_{3}(t))\cdot\Delta R_{V}(t).$

D-MiSo parameterized Core-Gaussians in time $t$ by:
$\mathcal{N}_{core}(V(t),\sigma,c)=\mathcal{N}_{core}(V\odot\psi(V,t),\sigma,c)$
where $\psi$ is a deformable network that moves triangle $V$ according to time
$t$. The opacity and colour of the Core-Gaussian are used only in the first
stage.

#### Stage 2: Preparation

Figure 12: Limbs render and modification obtained with D-MiSo. It is worth
noting that it is also possible to close the hand.

To move to the second phase of the model, it is imperative to prepare the
Multi-Gaussians. This involves attaching $k$ Sub-Gaussians to each Core-
Gaussian generated in Stage 1, as shown in Fig. 6. Henceforth, Sub-Gaussians
assume responsibility for the resultant rendering. Initially, the Sub-Gaussian
adopts the same features as the Core-Gaussian, except the position.

#### Stage 2

The primary objective of the second phase is to parallelized Core-Gaussians
(Core-Triangles Soup) to enhance understanding of movement and increase
rendering quality through the precise training of Sub-Gaussians. Since the
centers of Sub-Gaussians are parameterized by the local coordinate system
given by the rotation matrix of Core-Gaussian, when the Deform Network $\psi$
changes the Core-Gaussian all Sub-Gaussians (attached to this Core-Gaussian)
are modified by global transformation $\psi(V,t)$. D-MiSo use an additional
deformation network Sub-Rot Network $\phi$ dedicated to each $i$-th Sub-
Gassian’s small changes. Sub-Rot Network takes the Sub-Gaussian rotation
matrix $R_{V}^{i}$ and the current time $t$ as input and produces an updated
rotation matrix $\Delta R_{V}^{i}(t)$.

Figure 13: Examples of object modifications. The first method allows for a
smooth modification (bending) and also removes (e.g. plate), scales and/or
adds (e.g. small blue balls) objects.

The position $\hat{\bf m}^{i}_{V}(t)=\hat{\bf
m}_{V(t)}+\hat{R}_{V(t)}\boldsymbol{\alpha^{i}}^{T}$ of the Sub-Gaussian in
time $t$ is determined by the position $\hat{\bf m}_{V(t)}$, and rotation
$\hat{R}_{V(t)}$ of the Core-Gaussian (parameterized by triangle $V$) and the
learning parameter $\boldsymbol{\alpha^{i}}$. It should be noted that scale
$S^{i}$, color $c_{i}$, and opacity $\sigma_{i}$ of Sub-Gaussian are trainable
and do not depend on time. Sub-Rot Network produce updated
$\phi(R^{i},t)=\Delta R^{i}(t)$ for rotation parameter of Sub-Gaussians.
Finally parameters of Sub-Gaussians in time $t$ depends on Deform Network
$\psi$, and Sub-Rot Network $\phi$, and the Corr-Gaussians parameter $V$, and
Sub-Gaussian parameters $R^{i}$, $S^{i}$, $c_{i}$ and $\sigma_{i}$:

$\mathcal{G}_{Sub}(t)=\\{(\mathcal{N}_{sub}(\hat{\bf
m}_{V\odot\psi(V,t)}+\hat{R}_{V\odot\psi(V,t)}\boldsymbol{\alpha^{i}}^{T},R^{i}+\phi(R^{i},t),S^{i}),c_{i},\sigma_{i})\\}_{i=1}^{k}.$

D-MiSo final model consists of two deformable networks and two levels of
Gaussian distributions. Such a solution allows effectively modeling time in
dynamic scenes and modifying objects in each time frame.

The result of the model’s inference depends on the camera’s view angle as well
as on the selected time. Only Sub-Gaussian features are used in the rendering
process. Hence, the implementation of generating an output image from
Gaussians is no different from a vanilla GS.

The modifications involve the appropriate transformation of Sub-Gaussians. We
base this on the fact that the transformed Gaussians have exactly the same
color properties as before editing. Only the shape of the Gaussians is
changed. In our work, we present three editing methods (Fig. 11).

##  4 Experiments

The experimental section is divided into two parts. First, we show that D-MiSo
can model dynamic scenes with high quality; in the second part, that our model
allows an easy editing procedure, which is our main contribution.

###  4.1 Reconstruction of dynamic scenes

Here, we outline the specifics of our implementation and provide a detailed
description of the used datasets. We demonstrated the core advantages of our
model by conducting experiments using three different datasets. The source
code is available in GitHub111The GiHub link will be available after review..
Our code is developed on top of the GS vanilla code, according to their
license. We used NVIDIA GeForce RTX 4090 and A100 GPUs. The experiments focus
on a benchmark tasks of reconstruction. PSNR metric is used to compare our
model with another ones. Furthermore, additional numerical comparisons, i.e.,
SSIM/LPIPS, are available in the supplementary material.

#### D-NeRF Datasets:

Contains seven dynamic objects with realistic materials described with a
single camera pumarola2020dnerf. This means the model had access to only one
view at a given moment. Tab. 1 shows we are very comparable to other methods,
and we achieve a higher PSNR on one object. The differences in metrics are
small, however, our method presents an easier way to edit (Fig. 4). Providing,
among other things, better scaling.

Table 1: Quantitative comparisons (PSNR) on a D-NeRF dataset showing that
D-MiSo gives comparable results with other models. PSNR $\uparrow$
---
| Hook | Jumpin. | Trex | Bounc. | Hell. | Mutant | Standup
D-NeRF pumarola2020dnerf | 29.25 | 32.80 | 31.75 | 38.93 | 25.02 | 31.29 | 32.79
TiNeuVox-B Fang_2022 | 31.45 | 34.23 | 32.70 | 40.73 | 28.17 | 33.61 | 35.43
Tensor4D shao2023tensor4d | 29.03 | 24.01 | 23.51 | 25.36 | 31.40 | 29.99 | 30.86
K-Planes kplanes_2023 | 28.59 | 32.27 | 31.41 | 40.61 | 25.27 | 33.79 | 34.31
FF-NVS Guo_2023_ICCV | 32.29 | 33.55 | 30.71 | 40.02 | 27.71 | 34.97 | 36.91
4D-GS wu20234dgaussians | 30.99 | 33.59 | 32.16 | 38.59 | 31.39 | 35.98 | 35.37
DynMF kratimenos2024dynmf | 33.94 | 38.04 | 35.82 | 41.92 | 37.51 | 41.68 | 41.16
Editable
SC-GS huang2023sc-gs | 39.87 | 41.13 | 41.24 | 44.91 | 42.93 | 45.19 | 47.89
D-MiSo (our) | 38.13 | 42.05 | 40.88 | 41.49 | 41.49 | 44.38 | 47.66

#### NeRF-DSyan2023nerfds:

This dataset contains again seven real-world scenarios containing a moving or
deforming specular object. Each scene was recorded using two cameras, with the
video captured by one of them being used as a training set, while the footage
from the second one was treated as a test set. The camera pose was estimated
using COLMAP for both cameras. Tab. 2 shows that our method for four objects
achieves the SOTA results.

#### PanopticSports Datasets:

The dataset comprises six dynamic scenes featuring significant object and
actor movements luiten2023dynamictracking. The scenes are categorized by the
activities performed in the video sequences, i.e., juggling, boxing, softball,
tennis, football, and basketball. Each scene was recorded using 31 cameras
over 150 timesteps. Following the official data split, we use footage from 27
cameras for training and the remaining 4 cameras for testing. Numerical
results are shown in Tab. 3. The results show that the model achieved for five
objects SOTA according to PSNR metrics, and six times according to LPIPS
metrics.

We show that D-MiSo is comparable to other methods. However, our main
contribution in comparison to other methods is the very easy editing of the
resulting object (Fig. 4 11).

###  4.2 Editing of dynamic scenes

As mentioned earlier, we proposed three new methods of output object
modification. The first one focuses on moving and preparing a formal mesh,
which is connected in contrast to Triangle Soup. We apply a simple meshing
strategy on Core-Gaussians. We used the basic Alpha Shape algorithm 1056714;
articleEdelsbrunner and showed that the mesh does not have to estimate the
surface perfectly to be effective. Then, we reparametrize the Sub-Gaussian by
finding the closest face from the mesh (Fig. 9). In practice, we represent
each Sub-Gaussian in a local coordinate system (analogically to Multi-
Gaussians). Therefore, each Sub-Gaussian is assigned to the nearest face (Fig.
9). Each face can have a different number of Sub-Gaussians prescribed in such
a configuration. This method allows us to maintain the consistency bestowed by
the mesh. The whole process is shown in Fig. 5. Thanks to such representation,
we can edit our connected mesh to produce the correct edition of the dynamic
scene (Fig. 2).

Table 2: PSNR comparisons on a NeRF-DS dataset showing that D-MiSo gives
comparable results with other models. PSNR $\uparrow$
---
| Bell | Sheet | Press | Basin | Cup | Sieve | Plate
HyperNeRF park2021hypernerf | 24.0 | 24.3 | 25.4 | 20.2 | 20.5 | 25.0 | 18.1
NeRF-DS yan2023nerfds | 23.3 | 25.7 | 26.4 | 20.3 | 24.5 | 26.1 | 20.8
TiNeuVox-B Fang_2022 | 23.1 | 21.1 | 24.1 | 20.7 | 20.5 | 20.1 | 20.6
Editable
SC-GS huang2023sc | 25.1 | 26.2 | 26.6 | 19.6 | 24.5 | 26.0 | 20.2
D-MiSo (our) | 25.3 | 25.8 | 25.6 | 19.8 | 24.5 | 26.5 | 20.8

The second editing method allows us to define the connections, i.e., editing
Sub-Triangles Soup directly e.g. moving a hand or bending horns (Fig. 10) or
rotating a human body (Fig. 3). Changes are possible because we can also
transform a group of Sub-Triangles instead of individual ones. This way allows
for even very subtle changes like raising the thumb (Fig. 10), turning the
hand over (Fig 4), opening or closing the hand (Fig. 12). These edits would be
difficult in other approaches based on adjusting the 3D objects’ centroids
(nodes) since the space of nodes is limited in details area (Fig. 4). Nodes
use is preeminent for defining movement, which was insignificant in these
places.

With this method, we can also change complicated objects like a $360^{\circ}$
scene without losing the dynamics-related model. For example, the rotation of
a person stacking boxes (Fig. 3).

Table 3: Comparison on PanopticSports dataset. Metrics | Method | Juggle | Boxes | Softball | Tennis | Football | Basketball | Mean
---|---|---|---|---|---|---|---|---
|  3DGS kerbl20233d | 28.19 | 28.74 | 28.77 | 28.03 | 28.49 | 27.02 | 28.21
PSNR $\uparrow$ |  Dyn3DG luiten2023dynamictracking | 29.48 | 29.46 | 28.43 | 28.11 | 28.49 | 28.22 | 28.7
| D-MiSo (our) | 29.79 | 29.39 | 28.6 | 29.02 | 28.99 | 28.49 | 29.04
|  3DGS kerbl20233d | 0.91 | 0.91 | 0.91 | 0.90 | 0.90 | 0.89 | 0.90
SSIM $\uparrow$ |  Dyn3DG luiten2023dynamictracking | 0.92 | 0.91 | 0.91 | 0.91 | 0.91 | 0.91 | 0.91
| D-MiSo (our) | 0.93 | 0.92 | 0.92 | 0.92 | 0.92 | 0.91 | 0.92
|  3DGS kerbl20233d | 0.15 | 0.15 | 0.14 | 0.16 | 0.16 | 0.18 | 0.16
LPIPS $\downarrow$ |  Dyn3DG luiten2023dynamictracking | 0.15 | 0.17 | 0.19 | 0.17 | 0.19 | 0.18 | 0.17
| D-MiSo (our) | 0.13 | 0.13 | 0.15 | 0.14 | 0.13 | 0.15 | 0.14

The third method focuses on transforming the space in which the object is
located. Similar to the second method, the third one also works directly on
the Sub-Triangle soup. We can achieve such an effect by applying a certain
function to the selected plane. In practice, the object, or a portion of it,
is modified, for instance, through the use of sinusoidal functions. It allows
us to obtain fluidity and more easily define the physical nature of the
movement (Fig. 1, 11, 13).

Our methods are also scalable, so we can easily remove or duplicate elements
from an image. Moreover, the duplicated elements can be given their own
dynamics. Examples of these effects are shown in Fig. 13 where we removed the
plate, and both duplicated and rescaled the blue balls multiple times.

##  5 Conclusion

D-MiSo is a novel method based on Gaussian Splatting parameterization, which
produces a cloud of triangles called Triangle Soup. The method allows easy
editing of objects created in inference with the possible transformations
including moving, scaling, and rotating. By defining Multi-Gaussians, the
obligatory separability of modified elements seen in other models is combated.
In addition, certain elements of objects can be duplicated and removed (Fig.
13). Furthermore, the D-MiSo method can facilitate giving different dynamics
to separate parts of an object (Fig. 1).

#### Limitation

The method allows for complex changes at a given moment in time. However, if
some area is not well represented in the training set, it is not possible to
edit them. For example, a person’s hand can be changed but not fingers (Fig.
4). This is due to the liminality of Triangle Soup relative to a well-fitted
mesh.

##

##  Appendix A GaMeS [waczynska2024games] parametrisation of Gaussian
component

Multi-Gaussians describe 3D scenes using solid blocks rather than tiny
Gaussian distributions. This method enhances our model’s suitability for
dynamic environments. Additionally, we desire the capability to modify 3D
objects at any moment. As a result, we employ a mesh-inspired to condition our
model. In order to do that, we incorporate the parametrization concept from
GaMeS [waczynska2024games].

As it was mentioned earlier, our model uses flat Gaussians. Therefore, we can
approximate Gaussian components using triangle face mesh by parameterizing
Gaussian components by the vertices of the mesh face. We denote such
transformation by $\mathcal{T}(\cdot)$. In practice as a consequence of
parameterizing each Gaussian, we obtain a cloud of triangles called Triangle
Soup [1290060]. Accordingly, we get: Sub-Triangle Soup, Core-Triangle Soup and
Multi-Triangle Soup, see Fig. 2.

Let us assume that we have a Gaussian component parameterized by mean ${\bf
m}$, rotation matrix $R=[{\bf r}_{1},{\bf r}_{2},{\bf r}_{3}]$ and scaling
$S=\mathrm{dig}(\varepsilon,s_{2},s_{3}).$ We define three vertex of triangle
(face):

$V=[{\bf v}_{1},{\bf v}_{2},{\bf v}_{3}],$

where ${\bf v}_{1}={\bf m}$, ${\bf v}_{2}={\bf m}+s_{2}{\bf r}_{2}$, ${\bf
v}_{3}={\bf m}+s_{3}{\bf r}_{3}$.

Now we froze the vertex of the face $V=[{\bf v}_{1},{\bf v}_{2},{\bf v}_{3}]$
and reparameterize the Gaussian component by defining $\hat{\bf m}$,
$\hat{R}=[\hat{\bf r}_{1},\hat{\bf r}_{2},\hat{\bf r}_{3}]$ and
$\hat{S}=\mathrm{dig}(\hat{s}_{1},\hat{s}_{2},\hat{s}_{3}).$ First, we put
$\hat{\bf m}={\bf v}_{1}$. The first vertex of $\hat{R}$ is given by a normal
vector:

$\hat{\bf r}_{1}=\frac{({\bf v}_{2}-{\bf v}_{1})\times({\bf v}_{3}-{\bf
v}_{1})}{\|({\bf v}_{2}-{\bf v}_{1})\times({\bf v}_{3}-{\bf v}_{1})\|},$

where $\times$ is the cross product. The second one is defined by $\hat{\bf
r}_{2}=\frac{({\bf v}_{2}-{\bf v}_{1})}{\|({\bf v}_{2}-{\bf v}_{1})\|}.$ The
third one is obtained as a single step in the Gram–Schmidt process
[bjorck1994numerics]:

$\hat{\bf r}_{3}=\mathrm{orth}({\bf v}_{3}-{\bf v}_{1};{\bf r}_{1},{\bf
r}_{2}).$

Scaling parameters can also be easily calculated as $s_{1}=\varepsilon$,
$\hat{s}_{2}=\|{\bf v}_{2}-{\bf v}_{1}\|$ and $\hat{s}_{3}=\langle{\bf
v}_{3}-{\bf v}_{1},\hat{\bf r}_{3}\rangle$.

Consequently, the covariance of Gaussian distribution positioned to face is
given by:

$\hat{\Sigma}_{V}=\hat{R}_{V}\hat{S}_{V}\hat{S}_{V}\hat{R}_{V}^{T},$

and correspond with the shape of a triangle $V$. For one face $({\bf
v}_{1},{\bf v}_{2},{\bf v}_{3})$, we define the corresponding Gaussian
component:

$\mathcal{N}(({\bf v}_{1},{\bf v}_{2},{\bf v}_{3}))=\mathcal{N}(\hat{\bf
m}_{V},\hat{R}_{V},\hat{S}_{V}).$

Finally, the Gaussian component is derived from the mesh face parameters. This
approach can be applied in a Multi-Gaussian framework. Therefor we will use
invertible transformation between Gaussian parameters and triangle face $T$
and notation

$\mathcal{N}(V):=\mathcal{N}(\mathcal{T}^{-1}(V))=\mathcal{N}(\hat{\bf
m}_{V},\hat{R}_{V},\hat{S}_{V}).$

##  Appendix B Extension of numerical results from main paper

This section contains a numerical comparison of the D-MiSo with others models,
regarding the experiments described in the main document. We used additional
LPIPS and SSIM metrics. We see that we get comparable competitive results. We
can see that our method, because of the LPIPS metric, achieves better results
for five objects from D-NeRF dataset (Tab. 4). Numerical comparison with
metrics for the NeRF-DS dataset show that we are comparable to other methods
in the reproduction task ( Tab. 5).

In our method, a batch of images is taken as an input to the model. Table 6
shows a numerical comparison using batch: 4, 8, 10 on D-NeRF. Training takes
80 thousand iterations and second stages started at the 5 thousandth
iteration. Each Core-Gaussian have attached 25 Sub-Gaussians. This is
comparable to the SC-GS implementation. Similar study have been done for NeRF-
DS dataset presented in 7. We can see that batch is playing a bigger role in
the dataset, which is considering a more pronounced move.

Methods | Hook | Jumpingjacks | Trex | BouncingBalls
---|---|---|---|---
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
D-NeRF | 29.25 | .968 | .1120 | 32.80 | .981 | .0381 | 31.75 | .974 | .0367 | 38.93 | .987 | .1074
TiNeuVox-B | 31.45 | .971 | .0569 | 34.23 | .986 | .0383 | 32.70 | .987 | .0340 | 40.73 | .991 | .0472
Tensor4D | 29.03 | .955 | .0499 | 24.01 | .919 | .0768 | 23.51 | .934 | .0640 | 25.36 | .961 | .0411
K-Planes | 28.59 | .953 | .0581 | 32.27 | .971 | .0389 | 31.41 | .980 | .0234 | 40.61 | .991 | .0297
FF-NVS | 32.29 | .980 | .0400 | 33.55 | .980 | .0300 | 30.71 | .960 | .0400 | 40.02 | .990 | .0400
4D-GS | 30.99 | .990 | .0248 | 33.59 | .990 | .0242 | 32.16 | .988 | .0216 | 38.59 | .993 | .0267
SC-GS | 39.87 | .997 | .0076 | 41.13 | .998 | .0067 | 41.24 | .998 | .0046 | 44.91 | .998 | .0166
D-MiSo (our) | 38.13 | .990 | .0086 | 42.05 | .995 | .0049 | 40.88 | .996 | .0029 | 41.49 | .993 | .0079
Methods | Hellwarrior | Mutant | Standup | Average
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
D-NeRF | 25.02 | .955 | .0633 | 31.29 | .978 | .0212 | 32.79 | .991 | .0241 | 31.69 | .975 | .0575
TiNeuVox-B | 28.17 | .978 | .0706 | 33.61 | .982 | .0388 | 35.43 | .991 | .0230 | 33.76 | .983 | .0441
Tensor4D | 31.40 | .925 | .0675 | 29.99 | .951 | .0422 | 30.86 | .964 | .0214 | 27.62 | .947 | .0471
K-Planes | 25.27 | .948 | .0775 | 33.79 | .982 | .0207 | 34.31 | .984 | .0194 | 32.32 | .973 | .0382
FF-NVS | 27.71 | .970 | .0500 | 34.97 | .980 | .0300 | 36.91 | .990 | .0200 | 33.73 | .979 | .0357
4D-GS | 31.39 | .974 | .0436 | 35.98 | .996 | .0120 | 35.37 | .994 | .0136 | 34.01 | .987 | .0316
SC-GS | 42.93 | .994 | .0155 | 45.19 | .999 | .0028 | 47.89 | .999 | .0023 | 43.31 | .997 | .0063
D-MiSo (our) | 41.49 | .986 | .0173 | 44.38 | .997 | .0026 | 47.66 | .998 | .0016 | 42.27 | .993 | .0065
Table 4: Quantitative comparison on a D-NeRF dataset Methods | Bell | Sheet | Press | Basin
---|---|---|---|---
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
HyperNeRF | 24.0 | .884 | .159 | 24.3 | .874 | .148 | 25.4 | .873 | .164 | 20.2 | .829 | .168
NeRF-DS | 23.3 | .872 | .134 | 25.7 | .918 | .115 | 26.4 | .911 | .123 | 20.3 | .868 | .127
TiNeuVox-B | 23.1 | .876 | .113 | 21.1 | .745 | .234 | 24.1 | .892 | .133 | 20.7 | .896 | .105
SC-GS | 25.1 | .918 | .117 | 26.2 | .898 | .142 | 26.6 | .901 | .135 | 19.6 | .846 | .154
D-MiSo (our) | 25.3 | .846 | .174 | 25.8 | .875 | .211 | 25.6 | .867 | .206 | 19.8 | .788 | .217
Methods | Cup | Sieve | Plant | Average
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
HyperNeRF | 20.5 | .705 | .318 | 25.0 | .909 | .129 | 18.1 | .714 | .359 | 22.5 | .827 | .206
NeRF-DS | 24.5 | .916 | .118 | 26.1 | .935 | .108 | 20.8 | .867 | .164 | 23.9 | .898 | .127
TiNeuVox-B | 20.5 | .806 | .182 | 20.1 | .822 | .205 | 20.6 | .863 | .161 | 21.5 | .843 | .162
SC-GS | 24.5 | .916 | .115 | 26.0 | .919 | .114 | 20.2 | .837 | .202 | 24.1 | .891 | .140
D-MiSo (our) | 24.5 | .874 | .185 | 26.5 | .881 | .159 | 20.8 | .815 | .234 | 24.0 | .849 | .198
Table 5: Quantitative comparison on a NeRF-DS dataset Batch size | Hook | Jumpingjacks | Trex | BouncingBalls
---|---|---|---|---
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
10 | 37.02 | .986 | .0108 | 42.05 | .995 | .0049 | 40.88 | .996 | .0029 | 39.19 | .988 | .0132
8 | 38.13 | .990 | .0086 | 41.52 | .995 | .0052 | 40.60 | .994 | .0043 | 41.49 | .993 | .0079
4 | 37.68 | .989 | .0105 | 40.44 | .994 | .0063 | 39.58 | .995 | .0047 | 39.76 | .993 | .0073
Batch size | Hellwarrior | Mutant | Standup | Average
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
10 | 40.02 | .980 | .0276 | 44.29 | .997 | .0027 | 47.66 | .998 | .0016 | 41.58 | .990 | .0091
8 | 41.32 | .986 | .0174 | 44.38 | .997 | .0026 | 47.17 | .998 | .0018 | 42.08 | .993 | .0068
4 | 41.49 | .986 | .0173 | 43.32 | .996 | .0034 | 45.91 | .997 | .0025 | 41.16 | .993 | .0074
Table 6: Batch study on a D-NeRF dataset Batch size | Bell | Sheet | Press | Basin
---|---|---|---|---
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
8 | 25.1 | .851 | .165 | 24.8 | .863 | .219 | 25.2 | .858 | .205 | 19.7 | .788 | .217
4 | 24.5 | .832 | .190 | 25.1 | .873 | .202 | 25.3 | .860 | .215 | 19.8 | .781 | .245
2 | 25.3 | .846 | .174 | 25.8 | .875 | .211 | 25.6 | .867 | .206 | 19.6 | .785 | .207
1 | 25.1 | .844 | .190 | 25.6 | .873 | .216 | 24.3 | .842 | .273 | 19.7 | .785 | .229
Batch size | Cup | Sieve | Plant | Average
|  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$ |  PSNR $\uparrow$ |  SSIM $\uparrow$ |  LPIPS $\downarrow$
8 | 24.0 | .884 | .163 | 26.5 | .881 | .158 | 20.2 | .812 | .226 | 23.6 | .848 | .193
4 | 24.2 | .886 | .163 | 25.8 | .871 | .167 | 20.8 | .815 | .234 | 23.6 | .845 | .202
2 | 24.4 | .883 | .169 | 26.0 | .875 | .169 | 20.6 | .817 | .232 | 23.9 | .849 | .195
1 | 24.5 | .874 | .185 | 25.8 | .864 | .195 | 20.5 | .808 | .245 | 23.6 | .841 | .219
Table 7: Batch study on a NeRF-DS dataset

##  Appendix C Extension of examples modification

Below is an example of scene modification from the PanopticSports dataset
(Fig. 14). The dynamics of the object has been stopped, so as to show the
possibility of changing the position of the ball at a given time instant $t$.

Figure 14: Example of a ball position modification on a $360^{\circ}$ scene
from PanopticSports Datasets

Due to the characteristics of Multi-Gaussians and the ease of their control –
we show that we can easily duplicate and/or scale selected objects (Fig. 15).

Third method of modification allows to give new dynamics and fluidity to the
object (Fig. 16). We can see that even difficult edits like bending the back,
changing the shape of the face, bending the hand are possible and the result
is natural from the point of view of graphics, this is made possible by
editing the scale and rotation of Gaussians in an explicitly defined way.

Figure 15: Example of animation by duplicating elements or changing their
scale Figure 16: Example of third way of modification – Transformation of
space, allowing to give new dynamics.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: The abstract and introduction clearly outline the claims made in the paper, detailing the contributions, key assumptions, and scope. The claims are consistent with experimental results, indicating the extent to which the findings can be generalized to other settings.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes]

  9. Justification: The paper clearly presents both technical as well as ethical limitations in dedicated paragraph.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [N/A]

  14. Justification: The paper does not include theoretical results.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: The paper provides comprehensive technical details necessary for reproducing the code, along with datasets and experimental settings. The authors will also release the code and resulting checkpoints to ensure complete reproducibility.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: The code, along with the checkpoints used to report results, will be publicly available. All experiments were conducted using publicly available datasets.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: All results were obtained using official data splits which is stated in the paper. The hyperparameters of the model and their influence are also thoroughly discussed.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [No]

  34. Justification: Statistical significance is not provided due to the large computational resources required. No related work provides such statistics, making it unfeasible to recompute all results for comparison with our method.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Justification: Compute resources used for experiments are listed in the Implementation Details section.

  39. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  40. 9.

Code Of Ethics

  41. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  42. Answer: [Yes]

  43. Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

  44. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  45. 10.

Broader Impacts

  46. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  47. Answer: [Yes]

  48. Justification: We discuss the broader impact of our work in the Limitations section.

  49. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  50. 11.

Safeguards

  51. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  52. Answer: [N/A]

  53. Justification: The paper poses no such risks.

  54. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  55. 12.

Licenses for existing assets

  56. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  57. Answer: [Yes]

  58. Justification: We cited all utilized works and resources according to the citing guidelines.

  59. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  60. 13.

New Assets

  61. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the [N/A]

  62. Answer: [Yes]

  63. Justification: The paper does not relate new assets.

  64. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  65. 14.

Crowdsourcing and Research with Human Subjects

  66. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  67. Answer: [N/A]

  68. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  69. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  70. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  71. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  72. Answer: [N/A]

  73. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  74. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.