(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: University of Oxford 22institutetext: Mohamed bin Zayed
University of Artificial Intelligence

# GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editing

Jing Wu* 11 Jia-Wang Bian 11 Xinghui Li 11 Guangrun Wang 11 Ian Reid 22 Philip
Torr 11 Victor Adrian Prisacariu 11

###### Abstract

We propose GaussCtrl, a text-driven method to edit a 3D scene reconstructed by
the 3D Gaussian Splatting (3DGS). Our method first renders a collection of
images by using the 3DGS and edits them by using a pre-trained 2D diffusion
model (ControlNet) based on the input prompt, which is then used to optimise
the 3D model. Our key contribution is multi-view consistent editing, which
enables editing all images together instead of iteratively editing one image
while updating the 3D model as in previous works. It leads to faster editing
as well as higher visual quality. This is achieved by the two terms: (a)
depth-conditioned editing that enforces geometric consistency across multi-
view images by leveraging naturally consistent depth maps. (b) attention-based
latent code alignment that unifies the appearance of edited images by
conditioning their editing to several reference views through self and cross-
view attention between images’ latent representations. Experiments demonstrate
that our method achieves faster editing and better visual results than
previous state-of-the-art methods. Project website:
https://gaussctrl.active.vision/

###### Keywords:

3D Editing Diffusion Models Gaussian Splatting Neural Radiance Fields

††* indicates equal contribution. Figure 1: GaussCtrl. Our method edits a 3D
Gaussian Splatting (3DGS) scene by modifying its descriptive prompt (Upper
Left). This is achieved by editing the rendered images of 3DGS and re-training
the 3D model (Upper Right). Our contribution is a depth-conditioned multi-view
consistent editing framework, which substantially improves the blurry or
unreasonable 3D results caused by inconsistent editing in previous work
(Bottom).

##  1 Introduction

Neural representations such as Neural Radiance Field (NeRF) [26] and 3D
Gaussian Splatting (3DGS) [16] have demonstrated an ability to create a 3D
reconstruction that can guarantee remarkably high-quality novel view
rendering. However, the ability to edit these resulting 3D representations for
the creation of 3D assets remains a crucial yet underexplored aspect. The
first pipeline was recently proposed by Instruct NeRF2NeRF [9] (IN2N) to edit
a NeRF scene using text-based instructions. It leverages an image-conditioned
diffusion model (Instruct Pix2Pix [2]) to iteratively edit the images rendered
from the NeRF while updating the underlying NeRF-based reconstruction. Follow-
ups [5, 43] follow the same idea of editing 3D from 2D and show encouraging
results.

While IN2N transforms the 3D editing problem into a more manageable 2D editing
task, consistency across multiple views is not encouraged. Such consistency is
critical to the final 3D editing to prevent visual artefacts such as blurring
and inconsistent appearance at different viewpoints. However, this is
challenging for modern 2D diffusion models to achieve since their input is a
single image and they do not enforce geometric consistency across different
views. IN2N edits one image at a time while optimising the 3D NeRF, generating
an averaged result. This process leads to some geometric consistency, however,
the convergence is slow, and it requires many full NeRF optimisations.
Although its follow-ups yield better visual quality with their own
innovations, inconsistencies across rendered views remain—See Fig. 1 (Bottom).

We propose GaussCtrl to address the challenge of inconsistency. Our method
edits 3D scenes with text instructions, as illustrated in Fig. 1 (Upper Left &
Right). Distinguishing from IN2N, we propose a depth-controlled editing
framework and a latent code alignment module to explicitly enforce the multi-
view consistency, enabling editing all images together and updating the 3D
model only once. Specifically, first, as depth maps of the scene are naturally
geometry-consistent, we condition image editing on them by employing
ControlNet [49]. Images are inverted to their respective latent codes together
with their depth maps via DDIM inversion [41] and then are edited by the
denoising process with edited prompts. By doing so, the edited images inherit
the consistency from depth maps, avoiding the abrupt and incoherent geometric
changes that may present in other editing methods [27, 11]. Second, we propose
an attention-based latent code alignment module to encourage appearance
consistency during the editing process. Specifically, we choose several
reference views and align all other views’ latent codes to these reference
views during the denoising process, through the use of both self and cross-
view attention. This greatly improves high-level semantic consistency over the
entire dataset.

We evaluate our approach on a variety of scenes with different text prompts,
ranging from forward-facing scenes to challenging 360-degree object-centred
scenes. We also perform an ablation study on different components of our
method to validate their effectiveness. Experiments demonstrate that our
method significantly improves the visual quality of editing and greatly
reduces processing time. We summarise our contribution as follows:

  1. 1.

We propose GaussCtrl, to enable efficient editing of 3DGS scenes with text
instructions.

  2. 2.

We employ depth guidance and the attention-based latent code alignment module
to encourage multi-view consistent editing.

  3. 3.

The proposed method demonstrates more realistic editing and achieves higher
visual quality than previous work on a variety of 3D editing scenes.

##  2 Related Works

###  2.1 2D Editing with Diffusion Models

Diffusion models [41, 13, 30, 25, 40, 14] have gained popularity in image
generation due to their ability to produce highly realistic images. By
training on billions of image-text pairs, these models not only offer the
flexibility to customise generation through textual prompts [36, 4, 12, 38,
35] but also enable various forms of editing. Most editing methods leverage
pre-trained Stable Diffusion [36]. Starting with inverting the latent
representation of the to-be-edited image to its corresponding noise by DDIM
inversion, the editing is achieved through the denoising process. One form of
editing [32, 29, 28, 6] allows users to label several anchor points and drag
them to target positions. DragDiffusion [39] optimises the latent code of an
image by minimising feature differences between initial and target positions.
SDE-Drag [31] replaces the optimisation with copy-and-paste of features by
switching to DDPM scheduler [13]. Another form is to edit images through
textual prompts, which is more relevant to our work. Delta denoising score
(DDS) [10] directly optimises the image latent representation by forcing the
similarity between noises predicted using the original and edited texts.
Prompt-to-Prompt (P2P) [11] achieves editing by manipulating the cross
attention between image and text. Null-text inversion [27] tackles artefacts
at DDIM inversion when using classifier-free guidance [12] and integrates
their method to P2P. While significant progress has been made in 2D image
editing, none of them considers multi-view consistency, which may lead to
artefacts in editing. Our method, tailored for 3D editing, ensures such
consistency through initial latent code control and newly proposed attention-
based latent code alignment.

###  2.2 3D Editing in NeRF and Gaussian Splatting

NeRF and 3DGS are two of the most popular 3D models for neural novel view
synthesis. NeRF implicitly encodes the geometry and colours of a scene in a
Multi-Layer Perceptron (MLP), whereas 3DGS explicitly expresses the scene as
Gaussian ellipse point clouds. Although they exhibit promising results in 3D
reconstruction, their editing remains challenging. Current attempts can be
largely categorised into two main categories: 3D Style Transfer, and Text-
driven Editing.

3D Style Transfer:  Similar to 2D style transfer [8], 3D style transfer aligns
the style of a 3D scene to the style of a provided reference image. Notable
examples include StyleRF [22], StylizedNeRF [15], ARF [48], and PaletteNeRF
[18]. However, this line of work fails to edit the local details of the scene,
and the reference image is not always available.

Text-driven Editing:  Instruct NeRF2NeRF (IN2N) [9] is the first NeRF editing
work that edits 3D models with text instructions. This method effectively
transforms the 3D editing challenge into a 2D image editing task. By rendering
images from the 3D scene and editing them using Instruct Pix2Pix (IPix2Pix)
[2], IN2N iteratively updates the 3D scene until convergence. As there is no
guarantee of consistent editing of multi-view images, this method suffers from
instability, slow processing speeds, and notable artefacts, particularly
evident in 360-degree scenes. ViCA-NeRF [5], following a similar idea to IN2N,
selects several reference images from the dataset of the scene, edits them by
IPix2Pix, and edits the rest of the dataset as blended results of the
projection of reference images to alleviate the inconsistency. However, the
blending does not fully address the consistency issue and suffers from blurry
editing. DreamEditor [50] converts NeRF to mesh and directly optimises the
mesh with SDS loss [34] and DreamBooth [37]. Our method shares similarities
with IN2N and ViCA-NeRF but aims to address their limitations and offer
superior consistency and visual quality in the editing results.

Figure 2: GaussCtrl pipeline. Given a 3DGS scene and text instructions, our
method renders images using the 3DGS and edits the rendered images with text
instructions, which are then used to optimise the original 3DGS. Our key
contribution is multi-view consistent editing. Towards this, we propose (1)
depth-conditioned editing based on ControlNet for geometry consistency; and
(2) attention-based latent code alignment for improving consistency during
editing.

##  3 Method

We propose GaussCtrl, a novel approach to edit a 3D Gaussian Splatting (3DGS)
model using textual prompts. Given a collection of images and their
reconstructed 3D model, our method first re-renders each dataset image to the
required resolution and renders their respective depth maps. Then, we employ
ControlNet [49] to conduct depth-conditioned editing for all images
supplemented by attention-based latent code alignment to encourage geometry
and appearance consistency. Finally, we optimise the original 3D model using
the edited images to obtain the new edited 3D model. Optionally, a mask
generated by Language-based Segment Anything (Lang SAM) [17] is applied to
filter the background for better quality when editing local objects. The
comprehensive pipeline is illustrated in Fig. 2. In the following, we commence
by reviewing the background of 3DGS and ControlNet in Sec. 3.1, followed by
the introduction of our proposed methods, including depth-conditioned image
editing in Sec. 3.2, and attention-based latent code alignment in Sec. 3.3.

###  3.1 Background

3D Gaussian Splatting:  Gaussian Splatting [16] is an explicit 3D
representation based on point clouds. A set of 3D Gaussians is modelled to
represent the scene. Each Gaussian ellipse has a colour and an opacity and is
defined by its centred position $x$ (mean), and a full covariance matrix
$\Sigma$: $G(x)=e^{-\frac{1}{2}x^{T}\Sigma^{-1}x}$. When projecting 3D
Gaussians to 2D for rendering, a method of splatting [51] is used to position
the Gaussians on 2D planes, which involves a new covariance matrix
$\Sigma^{\prime}$ in camera coordinates defined as $\Sigma^{\prime}=JW\Sigma
W^{T}J^{T}$, where $W$ denotes a given viewing transformation matrix and $J$
is the Jacobian of the affine approximation of the projective transformation.
To enable differentiable optimisation, $\Sigma$ is further decomposed into s
scaling matrix $S$ and a rotation matrix $R$: $\Sigma=RSS^{T}R^{T}$.

ControlNet:  ControlNet [49] is an end-to-end spatial conditional generation
model built on top of Stable Diffusion (SD) [36]. It implants additional U-Net
encoders to SD, which enables image generation controlled by various kinds of
extra information, _e.g._ , depth, normals, edges, or hand-drawn priors. We
employ ControlNet for its ability of depth-controlled generation.

###  3.2 Depth-conditioned Image Editing

Previous work [9] employs Instruct-Pix2Pix [2] for image editing, resulting in
visually appealing individual images. However, ensuring multi-view consistency
between these images remains a challenge, often resulting in visual artefacts
and unstable editing outcomes. To this end, we conduct depth-conditioned image
editing by employing ControlNet $\mathcal{F}$, comprising a U-Net block
$\mathcal{F}_{U}$ and a ControlNet block $\mathcal{F}_{C}$. As the depths
$\mathcal{D}$ are extracted from the 3D model, they are naturally consistent
across multiple views. By conditioning image editing on these consistent depth
maps, our method effectively promotes consistency in 3D geometry across all
edited images.

Given a to-be-edited image $\mathcal{I}$ and its corresponding description
prompt $\hat{p}$, we begin by computing its latent code $z^{0}$, using the VAE
encoder of the ControlNet. We then iteratively invert it to its corresponding
Gaussian noise $z^{T}$ via DDIM inversion. Mathematically, the inversion can
be described as follows:

$\displaystyle\epsilon^{t}$
$\displaystyle=\mathcal{F}_{U}(z^{t},t,\hat{p},\mathcal{F}_{C}(z^{t},t,\hat{p},\mathcal{D}))$
(1) $\displaystyle z^{t+1}$
$\displaystyle=\sqrt{\alpha_{t+1}}\frac{z^{t}-\sqrt{1-\alpha_{t}}\cdot\epsilon^{t}}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t+1}}\epsilon^{t}$
(2)

where $t$ is the time step of the diffusion process and $\alpha_{t}$ is the
scheduling coefficient in DDIM scheduler. After reaching $z^{T}$, we replace
the original prompt $\hat{p}$ with the edited prompt $\hat{p}_{e}$ containing
changed content, and obtain the edited latent code $z^{0}_{e}$ through the
denoising process:

$\displaystyle\epsilon_{p}^{t}$
$\displaystyle=\mathcal{F}_{U}(z_{e}^{t},t,\hat{p}_{e},\mathcal{F}_{C}(z_{e}^{t},t,\hat{p}_{e},\mathcal{D}))$
(3) $\displaystyle\epsilon_{\emptyset}^{t}$
$\displaystyle=\mathcal{F}_{U}(z_{e}^{t},t,\emptyset,\mathcal{F}_{C}(z_{e}^{t},t,\emptyset,\mathcal{D}))$
(4) $\displaystyle\epsilon^{t}$
$\displaystyle=\epsilon_{\emptyset}^{t}+\omega\cdot(\epsilon_{p}^{t}-\epsilon_{\emptyset}^{t})$
(5) $\displaystyle z_{e}^{t-1}$
$\displaystyle=\sqrt{\alpha_{t-1}}\frac{z_{e}^{t}-\sqrt{1-\alpha_{t}}\cdot\epsilon^{t}}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t-1}}\epsilon^{t}$
(6)

where $z_{e}^{t}$ denotes the latent code of the edited images
($z_{e}^{T}=z^{T}$), $\emptyset$ is an empty prompt and Eq. 5 is the
classifier-free guidance [12] to improve the fidelity of the editing to the
edited prompt $\hat{p}_{e}$. We obtain the final image $\mathcal{I}_{e}$ by
decoding $z_{e}^{0}$ using the VAE decoder of the ControlNet.

Discussion on DDIM inversion: The original ControlNet operates as a generative
model, typically accepting randomly initialised noise $z^{T}$ as input, thus
yielding diverse results. However, for editing tasks, we adopt a different
approach by reversing the to-be-edited images into noise and utilising them as
input for ControlNet. By doing so, the output is conditioned on the original
image, and more importantly, multi-view consistency is improved during
editing. This is because the original images have naturally consistent colour
and geometry, where we use DDIM inversion to obtain consistent initial noises
for all to-be-edited images towards consistent editing.

###  3.3 Attention-based Latent Code Alignment

While our depth-conditioned editing approach enhances geometric consistency,
individual images are still edited independently, posing challenges for
appearance consistency. Despite sharing the same editing prompt, edited images
may exhibit discrepancies in colours or produce peculiar results, particularly
at challenging viewpoints. Previous studies [11, 3] have identified a
relationship between the appearance of images generated by diffusion models
and the key-value pairs in the image self-attention mechanism of the U-Net.
Inspired by this fact, we propose an attention-based latent code alignment
module that explicitly aligns the appearance of images to selected reference
views during editing. Therefore, images are no longer edited independently;
instead, their appearances are unified to a common standard. This ensures
greater consistency across edited images and mitigates issues related to
appearance discrepancies.

Specifically, we first define the attention between two latent codes $z_{i}$
and $z_{j}$ as:

$\displaystyle\vspace{-0.5cm}\text{Attn}_{i,j}$
$\displaystyle=\text{Softmax}(\frac{W_{q}(z_{i})W_{k}(z_{j})^{\top}}{\sqrt{c}})W_{v}(z_{j}),$
(7)

where $W_{q}(\cdot)$, $W_{k}(\cdot)$, and $W_{v}(\cdot)$ are linear
projections to obtain query, key, and value for the attention operation, and
$c$ is a scaling factor. Given latent codes of $N_{r}$ reference images
$z^{t}_{r,i}$, where $i=1,2,...,N_{r}$ and the latent code of the to-be-edited
image $z^{t}_{e}$ at the time step $t$, our alignment module is defined as:

$\displaystyle\vspace{-0.5cm}\text{AttnAlign}_{e}$
$\displaystyle=\lambda\cdot\text{Attn}_{e,e}+{(1-\lambda)\cdot\frac{1}{N_{r}}\sum_{i=1}^{N_{r}}\text{Attn}_{e,i}}$
(8)

where $\lambda\in[0,1]$. This module blends the self-attention of $z^{t}_{e}$
with the cross-view attention between $z^{t}_{e}$ and each reference view
$z^{t}_{r,i}$. The cross-view attention aligns the appearance of all edited
images to the reference views while the self-attention helps each edited image
retain its distinctiveness. We find that this design significantly improves
the appearance consistency and minimises anomalies at challenging viewpoints.
We replace all image self-attention modules with the proposed alignment module
in the U-Net block $\mathcal{F}_{U}$ and the ControlNet block
$\mathcal{F}_{C}$. Therefore, Eq. 3 and Eq. 4 becomes:

$\displaystyle\epsilon_{p}^{t}$
$\displaystyle=\mathcal{F}_{U}(z_{e}^{t},z_{r}^{t},t,\hat{p}_{e},\mathcal{F}_{C}(z_{e}^{t},z_{r}^{t},t,\hat{p}_{e},\mathcal{D}))$
(9) $\displaystyle\epsilon_{\emptyset}^{t}$
$\displaystyle=\mathcal{F}_{U}(z_{e}^{t},z_{r}^{t},t,\emptyset,\mathcal{F}_{C}(z_{e}^{t},z_{r}^{t},t,\emptyset,\mathcal{D}))$
(10)

##  4 Experiments

In this section, we first introduce the setup of experiments, including the
testing data, method baselines, evaluation metrics, and implementation
details. We then provide both qualitative and quantitative results of our
method, followed by ablation studies on each proposed component and a
limitation discussion.

###  4.1 Experiment Setup

Dataset: To validate the effectiveness of GaussCtrl, we collect a variety of
scenes from multiple existing datasets for evaluation. Specifically, we
collected four 360-degree scenes from IN2N [9], Mip-NeRF360 [1], and
BlendedMVS [45] datasets, and two forward-facing scenes from IN2N [9] and
NeRF-Art [44] dataset. For each scene, we evaluate our method on multiple
text-based instructions.

Baselines: We mainly compare GaussCtrl with two state-of-the-art methods:
Instruct-GS2GS [43], a very recent update of Instruct NeRF2NeRF (IN2N) [9]
that replaces the NeRF in IN2N with a 3DGS model and ViCA-NeRF [5] for the
similarities shared between them. We denote Instruct-GS2GS as IN2N(GS) in the
following paragraphs for it being a variant of IN2N. Both IN2N(GS) and VICA-
NeRF employ Instruct-Pix2Pix, which takes an instruction-like prompt to edit
images. Our method is based on Stable Diffusion, using a description-like
prompt. Therefore, to edit a scene, we use editing instructions in IN2N(GS)
and VICA-NeRF while modifying the original scene description in our method. To
ensure a fair comparison, we preprocess all dataset images to $512\times 512$
resolution and evaluate all methods on the same data. We provide more visual
comparison results, including a comparison with IN2N in our supplementary.

Figure 3: Qualitative results. We show diverse results of text-guided editing
in various scenes, ranging from editing objects to adjusting environments,
_e.g._ , changing the appearance and age of the target human, and modifying
the environment.

Evaluation Criteria:  Following previous methods [9, 7, 2], we use CLIP Text-
Image Directional Similarity ($\text{CLIP}_{dir}$) to evaluate the alignment
of the 3D edit with text instructions. However, we notice that this metric may
not always reflect the true visual quality of editing, on which we will
elaborate in later paragraphs.

Figure 4: Qualitative comparison on 360-degree scenes. Our method generates
more consistent and higher-quality images than previous state-of-the-art
methods.

Implementation Details:  Our method is implemented by using the PyTorch
library. We use the "splatfacto" model, an improved implementation of 3D
Gaussian Splatting, in NeRFStudio [42] library for 3D reconstruction. We
employ Stable Diffusion v1.5 and its corresponding ControlNet for 2D image
editing using the Huggingface library [33]. For reference views, we sample
$N_{r}=4$ views from the dataset images randomly. We set $\lambda$ in Eqn. 8
as 0.6. We apply Language-based Segment Anything (Lang SAM) [17, 46, 24, 20,
23] as our mask segmentation backbone. Our method takes around 9 minutes to
edit one scene on an NVIDIA RTX A5000 with a GPU memory of 24GB.

Figure 5: Qualitative results on forward-facing scenes. Our method generates
more realistic results with better quality, consistency, and less artefact.

###  4.2 Qualitative Evaluation

Fig. 3 illustrates various editing results of our method, including in both
360-degree and forward-facing scenes. The text instructions range from editing
objects to adjusting environments. Fig. 4 and Fig. 5 show the qualitative
comparison of our method with previous SOTA alternatives in 360-degree and
forward-facing scenes, respectively. In Fig. 7, we sample $10$ different views
of a scene to visually compare the editing consistency of different methods.
We make more detailed analyses below.

Fig. 3 shows that our method can perform realistic edits globally ("a photo of
a dinosaur in the snow") and locally ("a photo of a polar bear in the forest")
with high quality and consistency. Also, it can change the colour of a
specific area ("a photo of the face of a man with red hair") and texture of
the object ("a photo of a bronze bust statue of a man"). Our method also shows
good consistency for complicated texture editing such as "a photo of a face of
a man with Chinese opera face paint".

Fig. 4 shows qualitative comparisons between our method and baselines in
360-degree scenes. Specifically, when editing objects, IN2N(GS) suffers from
incomplete editing shown by the face area indicated by view #2,4, the feet
area in view #1,2,4 of scene (a) and the neck area in view # 1, 3 of scene
(b). These show the inconsistent editing from using Instruct Pix2Pix when
editing local objects. When editing scene’s environments, IN2N(GS)
demonstrates relatively better quality than editing objects as shown by scenes
(c) and (d) VICA-NeRF suffers from blurry results in both local object and
scene environment editing. We attribute these artefacts to their blending of
inconsistently edited reference images. Thanks to the depth-conditioned
editing and proposed latent code alignment module, our method demonstrates
sharper, more consistent and realistic results in both object and environment
editing, indicating the superiority of our method over previous alternatives.

Fig. 5 shows qualitative comparisons in forward-facing scenes. Compared with
360-degree scenes, IN2N(GS) and ViCA-NeRF show better consistency in the
forward-facing setting. This is attributed to that the variation of image
viewpoints in the forward-facing setting is not as extreme as in the
360-degree setting, meaning that individual image editing may retain certain
consistency. However, compared with our method, IN2N(GS) and ViCA-NeRF still
suffer from artefacts such as blurry boundaries. Our method also generates
more realistic results.

To highlight the improvement in multi-view consistency, we render $10$ views
from different angles around the object for each method in Fig. 7. As shown in
view #1,2,4,8,10, both IN2N(GS) and ViCA-NeRF lose many details and are blurry
on side views of the polar bear, a direct result of inconsistent editing.
Second, ViCA-NeRF loses most of the details of the bear’s face as indicated in
view #1,2,3,8,9,10. What’s more, it can be observed in view #6 that IN2N(GS)
and ViCA both suffer from the face-on-the-back problem, which is caused by
Instruct Pix2Pix forcefully producing a polar bear and fitting it to the image
layout. Our approach greatly mitigates this problem through our latent code
alignment module by conditioning editing on reference views. More detailed
comparisons of editing consistency and quality are included in the
supplementary material.

Table 1: Quantitative Evaluation. $\text{CLIP}_{dir}$: CLIP Text-Image Direction Similarity  | Scene | IN2N | IN2N(GS) | ViCA-NeRF | Ours
---|---|---|---|---|---
| $\text{CLIP}_{dir}$ | Time | $\text{CLIP}_{dir}$ | Time | $\text{CLIP}_{dir}$ | Time | $\text{CLIP}_{dir}$ | Time
360 | Bear Statue | 0.1019 | $\sim$1.5h | 0.1165 | $\sim$13.5min | 0.1104 | $\sim$38.5min | 0.1388 | $\sim$9min
Dinosaur | 0.1466 | 0.1490 | 0.0723 | 0.1584
Garden | 0.3027 | 0.1663 | 0.2903 | 0.2891
Stone Horse | 0.1654 | 0.1947 | 0.1926 | 0.2268
Forward | Fangzhou | 0.1598 | 0.2032 | 0.1809 | 0.1887
Face | 0.1332 | 0.1357 | 0.1119 | 0.1503
Figure 6: Failure cases of $\text{CLIP}_{dir}$. $\text{CLIP}_{dir}$ reflects
the alignment between text instructions and editing results but ignores the
editing quality. In the top row, our red panda has a lighter colour, making it
score lower. Other methods have wrong face geometry but score higher than
ours. In the bottom row, previous methods are closer to newborn baby, making
them score higher. However, the first two methods have terrible results, and
ViCA’s result is unnatural in the baby’s eye areas.

###  4.3 Quantitative Evaluation

We calculate the average $\text{CLIP}_{dir}$ over text instructions for each
scene and summarize the results in Tab. 1. Our method outperforms other
approaches in four out of six scenes. However, we notice that
$\text{CLIP}_{dir}$ may not always reflect the editing quality, as it measures
more about the global similarity between the text prompt and the edited
images, ignoring the majority of local details. We illustrate two failure
cases of this metric in Fig. 6. Our method generates better visual results but
gets lower scores than previous methods. Therefore, we include as many
examples as we can in the paper and the supplementary material to reflect the
true visual quality of our method. We additionally provide the editing time
for each method, and our method is the fastest among them.

Figure 7: Editing consistency comparison on the bear scene (Polar bear).
IN2N(GS) and ViCA both suffer from editing inconsistency (View #1,2,4,8,10),
which results in artefacts and blurry of the bear’s face. Additionally, they
are affected by the face-on-the-back problem. Our method improves on both
problems.  Figure 8: Ablation studies on the consistent editing. (a): Sampled
images from the original dataset. (b): Editing results using Instruct Pix2Pix
[2]. (c): Our proposed depth-conditioned editing, which uses ControlNet with
the randomly initialised latent codes. (d): Consistent initial latent code is
applied by using DDIM inversion. (e): Attention-based latent code alignment is
added based on (d).

###  4.4 Ablation Study

We conduct ablation studies on our method to demonstrate the effectiveness of
each proposed component. We selected the scene “bear statue”, a 360-degree
scene, as our subject because the 360-degree scene can better illustrate the
effect of multi-view consistency. To ensure a fair comparison, the guidance
scale $\lambda$ is set to 7.5 in all cases. Lang-SAM [47, 19, 21] is not
applied either to highlight the effect of each component on the scene’s
environment. The result is provided in Fig. 8, where we show the original
images in (a).

One-time Instruct Pix2Pix Edit (b): The most significant limitation of
Instruct Pix2Pix (IPix2Pix) is that it fails to edit images in challenging
viewpoints. For example, IPix2Pix fails to produce noticeable effects in
challenging views #4, 6, 7, and 8, where the bear statue is viewed from
behind, and only partially alters views #2, 3, and 5. This limitation prompts
IN2N to choose iterative editing over one-time editing for the unstable
performance of IPix2Pix. Even in relatively simpler views #9,10, IPix2Pix
still exhibits artefacts around the face of the bear, limiting the performance
of IN2N and VICA-NeRF.

ControlNet with Random Noise (c): When ControlNet operates with random noise
instead of inverted latent codes, it performs a generation task instead of an
editing task. While the generated images maintain consistency in geometry,
owing to the incorporation of depth maps, their overall style diverges
significantly from that of the original images. Moreover, at challenging
viewpoints such as views #6,7,8, ControlNet forcefully generates front-facing
views of a bear with the geometry of the bottom of the bear, which damages the
eventual quality of 3D editing. Additionally, similar to IPix2Pix, it also
suffers from artefacts when editing local details, such as the face area in
view #10.

ControlNet with Inverted Latent Codes (w/o AttnAlign) (d): When employing
latent codes inverted from the original images, we observe a significant
improvement in the general style alignment compared to using random noise.
Additionally, texture and colour consistency are notably enhanced. We
attribute these enhancements to consistent initial latent codes brought by
latent inversion. However, artefacts experienced by random noise still exist,
such as the forceful front-facing views of the bear in view #6,7,8 and
artefacts around the facial area in view #10.

ControlNet with Inverted Latent Codes & AttnAlign (e): After applying our
proposed latent code alignment module, the artefacts presented in (c) and (d)
are notably mitigated. By conditioning the editing on reference views using
cross-view attention, the model searches for extra information about the to-
be-edited image from the reference views. Therefore, it has a better
understanding of the semantics and geometry of the image, avoiding producing
forceful results. Such conditioning also unifies the appearances of edited
images to a common ground, which further improves the quality of the final 3D
editing.

Figure 9: Failure cases. Left: Due to using depth guidance, our method cannot
work well when a significant geometry change is required. However, we find
that existing methods also cannot work well in this scenario even though they
do not use depths. Right: our method fails when the 2D pre-trained diffusion
model doesn’t work well. Nevertheless, it shows that our method can still
preserve the consistency.

###  4.5 Limitations

Some may be concerned about its ability to alter the scene’s original geometry
as we condition the editing on depth maps. However, We find that in the most
of 3D editing literature [9, 5, 50], significant changes to the original
geometry are not typically required. Instead, editing tasks often involve
adjusting object styles, modifying background environments, or adding
localized features, such as adding a moustache to a person. For the
completeness of the experiments, we include an example that requires geometric
changes. As illustrated on the left of Fig. 9, we fail to turn the bear statue
into a giraffe. However, the same failure is also observed in IPix2Pix and
baseline methods like IN2N and VICA-NeRF. Another limitation is that the final
result is not always faithful to the user’s intention. As demonstrated in the
right of Fig. 9, our method fails to transform the man into the comic
character Hulk. We suspect the root of this problem lies in the ControlNet,
which does not recognise the word “Hulk", and does not produce the correct
result. However, the consistency and sharp results demonstrate the
effectiveness of our method.

##  5 Conclusion

In this paper, we propose an efficient 3D-aware consistency control editing
method, GaussCtrl, which greatly mitigates the artefacts and blurry results
caused by the inconsistency in 2D editing, especially in 360-degree scenes.
Based on a pre-captured Gaussian model, our method controls multi-view
consistency by encouraging consistency in all the stages of editing, _i.e._ ,
Depth-conditioned Image Editing, and Attention-based Latent Code Alignment. We
evaluate the performance of GaussCtrl on diverse scenes, text prompts, and
objects. Our method outperforms other state-of-the-art methods through our
experiments.

####  Broader Impact:

Our method is one of the 3D editing methods that can be potentially misused
for creating deceptive or harmful content, which could erode trust in digital
media and exacerbate issues of misinformation and cyberbullying. By generating
hyper-realistic alterations to images, videos, or even deepfakes, 3D editing
technologies can be utilised to fabricate events, impersonate individuals, or
manipulate scenes in ways nearly indistinguishable from reality. This
capability not only leads to higher chances of confusion and misinformation
but also opens pathways for harassment and defamation. Hence, it is necessary
to enhance regulatory frameworks to mitigate these societal risks.

## References

  * [1] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR (2022)
  * [2] Brooks, T., Holynski, A., Efros, A.A.: Instructpix2pix: Learning to follow image editing instructions. In: CVPR (2023)
  * [3] Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., Zheng, Y.: Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. arXiv preprint arXiv:2304.08465 (2023)
  * [4] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. ArXiv abs/2105.05233 (2021), https://api.semanticscholar.org/CorpusID:234357997
  * [5] Dong, J., Wang, Y.X.: Vica-nerf: View-consistency-aware 3d editing of neural radiance fields. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)
  * [6] Epstein, D., Jabri, A., Poole, B., Efros, A.A., Holynski, A.: Diffusion self-guidance for controllable image generation (2023)
  * [7] Gal, R., Patashnik, O., Maron, H., Chechik, G., Cohen-Or, D.: Stylegan-nada: Clip-guided domain adaptation of image generators (2021)
  * [8] Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 2414–2423 (2016). https://doi.org/10.1109/CVPR.2016.265
  * [9] Haque, A., Tancik, M., Efros, A., Holynski, A., Kanazawa, A.: Instruct-nerf2nerf: Editing 3d scenes with instructions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)
  * [10] Hertz, A., Aberman, K., Cohen-Or, D.: Delta denoising score (2023)
  * [11] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control (2022)
  * [12] Ho, J.: Classifier-free diffusion guidance. ArXiv abs/2207.12598 (2022)
  * [13] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. arXiv preprint arxiv:2006.11239 (2020)
  * [14] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: LoRA: Low-rank adaptation of large language models. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=nZeVKeeFYf9
  * [15] Huang, Y.H., He, Y., Yuan, Y.J., Lai, Y.K., Gao, L.: Stylizednerf: Consistent 3d scene stylization as stylized nerf via 2d-3d mutual learning. In: Computer Vision and Pattern Recognition (CVPR) (2022)
  * [16] Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics 42(4) (July 2023), https://repo-sam.inria.fr/fungraph/3d-gaussian-splatting/
  * [17] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollár, P., Girshick, R.: Segment anything. arXiv:2304.02643 (2023)
  * [18] Li, B., Weinberger, K.Q., Belongie, S., Koltun, V., Ranftl, R.: Language-driven semantic segmentation. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=RriDjddCLN
  * [19] Li*, C., Liu*, H., Li, L.H., Zhang, P., Aneja, J., Yang, J., Jin, P., Lee, Y.J., Hu, H., Liu, Z., et al.: Elevater: A benchmark and toolkit for evaluating language-augmented visual models. arXiv preprint arXiv:2204.08790 (2022)
  * [20] Li, F., Zhang, H., Liu, S., Guo, J., Ni, L.M., Zhang, L.: Dn-detr: Accelerate detr training by introducing query denoising. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13619–13627 (2022)
  * [21] Li*, L.H., Zhang*, P., Zhang*, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J.N., Chang, K.W., Gao, J.: Grounded language-image pre-training. In: CVPR (2022)
  * [22] Liu, K., Zhan, F., Chen, Y., Zhang, J., Yu, Y., Saddik, A.E., Lu, S., Xing, E.: Stylerf: Zero-shot 3d style transfer of neural radiance fields (2023)
  * [23] Liu, S., Li, F., Zhang, H., Yang, X., Qi, X., Su, H., Zhu, J., Zhang, L.: DAB-DETR: Dynamic anchor boxes are better queries for DETR. In: International Conference on Learning Representations (2022), https://openreview.net/forum?id=oMI9PjOb9Jl
  * [24] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al.: Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 (2023)
  * [25] Luo, C.: Understanding diffusion models: A unified perspective. ArXiv abs/2208.11970 (2022), https://api.semanticscholar.org/CorpusID:251799923
  * [26] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis (2020)
  * [27] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv:2211.09794 (2022)
  * [28] Mou, C., Wang, X., Song, J., Shan, Y., Zhang, J.: Diffeditor: Boosting accuracy and flexibility on diffusion-based image editing. arXiv preprint arXiv:2402.02583 (2023)
  * [29] Mou, C., Wang, X., Song, J., Shan, Y., Zhang, J.: Dragondiffusion: Enabling drag-style manipulation on diffusion models. arXiv preprint arXiv:2307.02421 (2023)
  * [30] Nichol, A., Dhariwal, P.: Improved denoising diffusion probabilistic models. ArXiv abs/2102.09672 (2021), https://api.semanticscholar.org/CorpusID:231979499
  * [31] Nie, S., Guo, H.A., Lu, C., Zhou, Y., Zheng, C., Li, C.: The blessing of randomness: Sde beats ode in general diffusion-based image editing. arXiv preprint arXiv:2311.01410 (2023)
  * [32] Pan, X., Tewari, A., Leimkühler, T., Liu, L., Meka, A., Theobalt, C.: Drag your gan: Interactive point-based manipulation on the generative image manifold. In: ACM SIGGRAPH 2023 Conference Proceedings (2023)
  * [33] von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert, N., Rasul, K., Davaadorj, M., Wolf, T.: Diffusers: State-of-the-art diffusion models. https://github.com/huggingface/diffusers (2022)
  * [34] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 (2022)
  * [35] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. ArXiv abs/2204.06125 (2022), https://api.semanticscholar.org/CorpusID:248097655
  * [36] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10684–10695 (2022)
  * [37] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K.: Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2023)
  * [38] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E.L., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi, S.S., Lopes, R.G., Salimans, T., Ho, J., Fleet, D.J., Norouzi, M.: Photorealistic text-to-image diffusion models with deep language understanding. ArXiv abs/2205.11487 (2022), https://api.semanticscholar.org/CorpusID:248986576
  * [39] Shi, Y., Xue, C., Pan, J., Zhang, W., Tan, V.Y., Bai, S.: Dragdiffusion: Harnessing diffusion models for interactive point-based image editing. arXiv preprint arXiv:2306.14435 (2023)
  * [40] Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N., Ganguli, S.: Deep unsupervised learning using nonequilibrium thermodynamics. p. 2256–2265. ICML’15, JMLR.org (2015)
  * [41] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv:2010.02502 (October 2020), https://arxiv.org/abs/2010.02502
  * [42] Tancik, M., Weber, E., Ng, E., Li, R., Yi, B., Kerr, J., Wang, T., Kristoffersen, A., Austin, J., Salahi, K., Ahuja, A., McAllister, D., Kanazawa, A.: Nerfstudio: A modular framework for neural radiance field development. In: ACM SIGGRAPH 2023 Conference Proceedings. SIGGRAPH ’23 (2023)
  * [43] Vachha, C., Haque, A.: Instruct-gs2gs: Editing 3d gaussian splats with instructions (2024), https://instruct-gs2gs.github.io/
  * [44] Wang, C., Jiang, R., Chai, M., He, M., Chen, D., Liao, J.: Nerf-art: Text-driven neural radiance fields stylization. arXiv preprint arXiv:2212.08070 (2022)
  * [45] Yao, Y., Luo, Z., Li, S., Zhang, J., Ren, Y., Zhou, L., Fang, T., Quan, L.: Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. Computer Vision and Pattern Recognition (CVPR) (2020)
  * [46] Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M., Shum, H.Y.: Dino: Detr with improved denoising anchor boxes for end-to-end object detection (2022)
  * [47] Zhang, H., Zhang, P., Hu, X., Chen, Y.C., Li, L.H., Dai, X., Wang, L., Yuan, L., Hwang, J.N., Gao, J.: Glipv2: Unifying localization and vision-language understanding. arXiv preprint arXiv:2206.05836 (2022)
  * [48] Zhang, K., Kolkin, N., Bi, S., Luan, F., Xu, Z., Shechtman, E., Snavely, N.: Arf: Artistic radiance fields (2022)
  * [49] Zhang, L., Rao, A., Agrawala, M.: Adding conditional control to text-to-image diffusion models (2023)
  * [50] Zhuang, J., Wang, C., Liu, L., Lin, L., Li, G.: Dreameditor: Text-driven 3d scene editing with neural fields. arXiv preprint arXiv:2306.13455 (2023)
  * [51] Zwicker, M., Pfister, H., van Baar, J., Gross, M.: Ewa volume splatting. In: Proceedings Visualization, 2001. VIS ’01. pp. 29–538 (2001). https://doi.org/10.1109/VISUAL.2001.964490


# NNVISR: Bring Neural Network Video Interpolation and Super Resolution into
Video Processing Framework

Yuan Tong  ,  Mengshun Hu  and  Zheng Wang  National Engineering Research
Center for Multimedia Software, Hubei Key Laboratory of Multimedia and Network
Communication Engineering, School of Computer Science, Wuhan University, China

(2023)

###### Abstract.

We present NNVISR - an open-source filter plugin for the
VapourSynth111https://github.com/vapoursynth/vapoursynth video processing
framework, which facilitates the application of neural networks for various
kinds of video enhancing tasks, including denoising, super resolution,
interpolation, and spatio-temporal super-resolution. NNVISR fills the gap
between video enhancement neural networks and video processing pipelines, by
accepting any network that enhances a group of frames, and handling all other
network agnostic details during video processing. NNVISR is publicly released
at https://github.com/tongyuantongyu/vs-NNVISR.

open source, video super-resolution, video interpolation, spatio-temporal
super-resolution

††copyright: acmcopyright††journalyear: 2023††doi: XXXX/XXXX.XXXX††conference:
ACM MM ’23: ACM International Conference on Multimedia; October 29, 2023 –
November 3, 2023; Ottawa, Canada††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Human-centered computing Visualization††ccs:
Hardware Displays and imagers††ccs: Computing methodologies Computer vision

##  1\. Introduction

In recent years, deep learning has brought tremendous advancements in
fundamental computer vision tasks, such as image classification (Krizhevsky et
al., 2017; He et al., 2016; Tan and Le, 2019), object detection (Ren et al.,
2015; Redmon et al., 2016; Carion et al., 2020) and image super resolution
(Dong et al., 2016; Ledig et al., 2017; Wang et al., 2018a). These
advancements also shed light to video enhancing tasks, such as super
resolution (Caballero et al., 2017; Tao et al., 2017; Wang et al., 2019; Tian
et al., 2020; Chan et al., 2021), interpolation (Jiang et al., 2018; Bao et
al., 2019; Xu et al., 2019; Bao et al., 2021; Choi et al., 2020; Huang et al.,
2022), and spatio-temporal super resolution (Hu et al., 2022b, a; Xue et al.,
2019; Xiang et al., 2020; Xu et al., 2021; Haris et al., 2020; Dutta et al.,
2021).

However, different from the fundamental tasks which deal with single images,
video enhancing tasks deal with videos, which are indefinitely long sequences
of images (which is usually called frame), and can be extraordinarily large
when in uncompressed form. Such intrinsic characteristics of videos decided
that it is unfeasible to process videos like images which load the whole input
into computational memory. In fact, uncompressed videos are so large that they
can quickly fill up even storage devices, let along memory. To solve this
problem, many video enhancement methods (Caballero et al., 2017; Tao et al.,
2017; Xue et al., 2019; Wang et al., 2019; Hu et al., 2022b, a; Choi et al.,
2020; Haris et al., 2020; Bao et al., 2019; Xiang et al., 2020; Xu et al.,
2021) actually works on a definite and relatively small number of consecutive
frames, thus limiting the amount of data to be processed at a time, while
still being able to explore the extra temporal relation in video.

Such change to the task allowed applying deep learning method to video
enhancing tasks, but also left the ultimate task of enhancing videos in an
half-done state, compared to image tasks where task and network inputs and
outputs are aligned and can be directly integrated into current applications.
Most video enhancement network implementations expect input to be a tuple of
images, so to utilize them one have to first convert video to image file
sequence and separate them into tuples, then run the network inference, and
finally merge output tuples into sequence and convert to video, which is both
inefficient and resource-consuming. Some network implementations do try to
support process video by using Python libraries like OpenCV, but is barely
usable and far from the requirement of serious video processing, let along
performance. Moreover, by processing frame tuples instead of the whole video,
video enhancement networks avoided a non-trivial problem of detecting scene
changes, across which exploring temporal relation usually brings more noise
than information and should generally be avoided.

In developing a practical spatio-temporal video super resolution tool based on
our previous work CycMuNet (Hu et al., 2022a) and YOGO (Hu et al., 2022b), we
found that most of the work is not specific to our network, and even not
specific to the spatio-temporal video super resolution task. Instead, any
video enhancement network that produces output frames based on a consecutive
tuple of input frames can fit into our design. This motivated us to develop
NNVISR, an open-source tool, brings all kinds of video enhancement networks
into the video processing framework.

NNVISR is publicly released at https://github.com/tongyuantongyu/vs-NNVISR
under the 3-Clause BSD License. The repository contains all the source code
and detailed documentation including installation, compilation and network
intergration guide. NNVISR also provides binary release on Anaconda, which can
be easily installed along with all necessary dependencies using the conda
package manager. NNVISR is designed as a filter plugin for the open source
VapourSynth video processing framework. VapourSynth uses Python grammar to
declare processing pipeline, and has mature support for loading video,
traditional video processing and piping output to video encoders like FFmpeg
and x264. NNVISR uses NVIDIA’s
TensorRT222https://developer.nvidia.com/tensorrt framework to provide high
performance inference of deep learning network. Therefore, NNVISR is suitable
for both quick demo and large-scale deployment. To the best of our knowledge,
NNVISR is the first video processing tool to support inter-frame video
enhancement using arbitrary neural network.

##  2\. Related Work

###  2.1. Video Super Resolution

Video Super Resolution aims to convert video frames into higher resolution by
predicting high frequency detail. Compared to image super resolution, video
super resolution can utilize temporal information between frames to get better
result, but also faces an extra challenge to aggregating information among
misaligned frames. Early methods (Caballero et al., 2017; Tao et al., 2017)
use optical flow to guide spatial warping of frames. TDAN (Tian et al., 2020)
adopts deformable convolution (DCN) (Dai et al., 2017; Zhu et al., 2019) to
align features between frames, and EDVR (Wang et al., 2019) further designed a
DCN pyramid to perform multi-scale feature alignment. BasicVSR++ (Chan et al.,
2022) combined optical flow and DCN to generate high quality DCN offsets for
feature alignment.

###  2.2. Video Interpolation

Video Interpolation aims to generate non-existent pixels temporary between
known pixels, in contrast to Video Super Resolution which generates pixels
spatially between known pixels, so every output pixel need to be warped from
input frames. Early methods (Niklaus et al., 2017a, b) tried to use an
adaptive kernel to perform warping, but is very inefficient in term of
inference speed. Many flow-based methods (Jiang et al., 2018; Xue et al.,
2019) use optical flow to guide warping, but the inaccuracy of predicted
optical flow usually causes distortion to the result, so extra measures (Bao
et al., 2019, 2021) are usually taken to refine the warped frame. There are
also methods that generate intermediate frames directly, like CAIN (Choi et
al., 2020) using channel attention and RIFE (Huang et al., 2022) using
distillation to get good result one simple architecture.

###  2.3. Spatio-Temporal Video Super Resolution

Based on the similar nature of video super resolution and video interpolation,
Spatio-Temporal Video Super Resolution tries to perform spatial and temporal
(interpolation) super resolution in one network to reach both better result
and better efficiency. STARnet (Haris et al., 2020) and CycMuNet (Hu et al.,
2022a) jointly learns both spatial and temporal SR task using a mutual
learning strategy. Zooming Slow-Mo (Xiang et al., 2020) proposed deformable
ConvLSTM to explore global temporal contexts. TMNet (Xu et al., 2021) further
proposed a local temporal feature comparison module to extract short-term
motion cues. YOGO (Hu et al., 2022b) further compact ST-VSR by only perform
alignment once on all input frames to reach better efficiency.

###  2.4. Open Source Neural Network Video Enhancement Tool

Several open-source tools have been developed to perform video enhancement
tasks that are able to read and write video files.
MMagic333https://github.com/open-mmlab/mmagic implemented video enhancement
support using various video super-resolution and video interpolation models.
Practical-RIFE444https://github.com/hzwer/Practical-RIFE implemented video
interpolation using RIFE (Huang et al., 2022).
GMFSS_Fortuna555https://github.com/98mxr/GMFSS_Fortuna implemented video
interpolation based on GMFlow (Xu et al., 2022). But these tools simply use
OpenCV with its default settings, so they are only suitable for simple demo
purposes. Anime4K666https://github.com/bloc97/Anime4K implemented simple CNN-
based video super resolution and denoising networks using a shader to achieve
real-time enhancement during playback, but this approach is not suitable for
video processing, and doesn’t scale well for more complicated networks.

Under such conditions, VapourSynth gained much attention thanks to its ability
to write video process scripts using Python, and many VapourSynth scripts and
plugins are developed to perform video enhancement using neural networks.
VapourSynth-
Waifu2x-caffe777https://github.com/HomeOfVapourSynthEvolution/VapourSynth-
Waifu2x-caffe plugin integrated the Caffe
implementation888https://github.com/lltcggie/waifu2x-caffe of the famous
waifu2x999https://github.com/nagadomi/waifu2x image super resolution tool.
VSGAN101010https://github.com/rlaphoenix/VSGAN,
vsrife111111https://github.com/HolyWu/vs-rife and vs-
gmfss_fortuna121212https://github.com/HolyWu/vs-gmfss_fortuna each implemented
VapourSynth processing script using the PyTorch implementation of image super
resolution model ESRGAN (Wang et al., 2018b), video interpolation model RIFE
(Huang et al., 2022) and video interpolation tool GMFSS_Fortuna, and supports
TensorRT speed up using Torch-
TensorRT131313https://github.com/pytorch/TensorRT. vs-
mlrt141414https://github.com/AmusementClub/vs-mlrt plugin further integrates
multiple neural network inference engines into VapourSynth, with support for
arbitrary image enhancement models and RIFE interpolation model. VSGAN-
tensorrt-docker151515https://github.com/styler00dollar/VSGAN-tensorrt-docker
implemented VapourSynth processing script for various video enhancement
network, and packed into a docker image for easy use.

##  3\. Design

NNVISR provides simple interface for both end user to processing video, and
implementors to provide their network definition. On top of that, NNVISR also
integrates well into current VapourSynth ecosystem, and can cooperate with
other plugins to build an advanced processing pipeline.

###  3.1. Network Integration

NNVISR designed a simple yet expressive interface for neural network model to
implement, in order to be utilized during video processing.

NNVISR focus on supporting the kind of video enhancement neural networks that
explores temporal relationship between a definite number of consecutive frames
that we call as frame group. We separate the whole process into two stages:
the extract stage and the fusion stage. The extract stage operates on single
frame and produces a pyramid of features in different spatial dimensions
(Networks that do not use feature pyramid can be treated as a pyramid of only
one layer). The fusion stage mixes the features of all frames in a frame group
to produce output frames.

For convenience, we define $n$ to be the number of frames being “consumed” by
the network for each inference run. Note that for interpolation networks, the
last frame of a frame group is usually also the first frame of the next frame
group, so in this case $n$ is the number of input frames minus 1. Based on
this definition, we introduced several flags to precisely describe network
behavior:

  * •

“Interpolation” flag means the network is doing video interpolation, and
instruct NNVISR to produce output video in doubled framerate.

  * •

“Extra frame” flag means the network accepts $n+1$ frames rather than $n$.

  * •

“Double frame” flag means the network produces $2n$ frames for each inference
run, otherwise $n$. If “Double frame” flag is not set and “Interpolation” flag
is set, then input frames are also used as output frames as is, and network
outputs are all treated as intermediate frames.

To integrate neural network into NNVISR, researcher should provide their
network definition in ONNX161616https://github.com/onnx/onnx format, which is
widely supported by deep learning frameworks. Notably, PyTorch has built-in
support for automatically exporting model into ONNX format. Specially, we
recognized broad usage of Deformable Convolution (Dai et al., 2017),
especially the V2 (Zhu et al., 2019) variant, among video enhancement
networks, mainly due to it’s ability to efficiently adapt to arbitrary object
motion using its offset input, which is crucial in aligning object among
frames and warping frames based on optical flow information. However, while
Deformable Convolution plays an important role in video enhancement networks,
it does not have wide support, especially is not natively supported by
TensorRT. TensorRT has a plugin interface to support custom network operators,
so we implemented the V2 variant of Deformable Convolution for TensorRT. It is
also compatible with PyTorch’s Deformable Convolution implementation provided
in torchvision, so networks trained using PyTorch can be directly used.

NNVISR supports input and output in both RGB and YUV pixel format, and
implementors can choose to support either or both. RGB is the most common
pixel format used in computer vision, while the YUV420 variant of YUV pixel
format is the dominant format for video storage and processing. Output in YUV
pixel format has two advantages: one being more efficient since YUV420 halves
the spatial dimension of 2 in 3 of its channels, therefore has only half the
amount of data compared to RGB; another being higher quality because
convential conversion from RGB to YUV420 has been shown to be suboptimal,
especially for high dynamic range contents (Ström et al., 2016). By training
to output frame in YUV420 format neural networks can easily achieve better
visual quality compared to conventional conversion method.

NNVISR detects colorspace information in videos and loads the network trained
for the corresponding colorspace, since same pixel value can and usually mean
different colors under different colorspace. Therefore implementors should
train their networks on datasets in different colorspace separately, so that
the network can produce better result. Color preciseness is crucial to video
processing, especially on high dynamic range contents, which are gradually
gaining popularity recently.

###  3.2. End User Usage

The usage of NNVISR is simple. NNVISR follows the idiomatic design of
VapourSynth plugin, which provides a single function that accepts an input
video (conventionally called “clip” in VapourSynth) and parameters, and
returns the enhanced video.

By setting scale factor and flag parameters to different values, NNVISR can be
configured to perform various kinds of video enhancing tasks:

  * •

Setting scale factor to 1 to perform video denoising or .

  * •

Setting scale factor ¿ 1 to perform video super resolution.

  * •

Specially, setting horizontal scale factor to 1 and vertical scale factor to 2
to perform video deinterlacing.

  * •

Setting scale factor to 1 and “Interpolation” flag to perform video
interpolation.

  * •

Setting scale factor to 1, “Interpolation” flag and “Double frame” flag to
perform video interpolation along with denoising.

  * •

Setting scale factor ¿ 1, “Interpolation” flag and “Double frame” flag to
perform spatial-temporal video resolution.

Usually real-world videos contains more than one scene while video enhancement
models are only trained on a single scene. Therefore when crossing the scene
boundary, enhancement result is usually bad so this should be avoid.
VapourSynth framework has a standard way to signal the occurence of scene
change, therefore user can choose the most suitable plugin to detect scene
change based on the video content and user’s need. There are already multiple
filters that detects scene change using traditional method, and employing a
neural network to detect scene change can also be easily implemented.

In addition to source code, we also publish pre-built binaries for Windows and
Linux as conda packages, so that users can easily install NNVISR without going
through the complicated environment configuations.

###  3.3. Internal Design

NNVISR internally performs the grouping of frames, taking account of $n$ of
network and scene change. For interpolation networks, NNVISR uses a carefully
designed frame layout to reduce unnecessary memory usage as much as possible.

NNVISR always grouping frames within a scene, and has a complete set of rules
to always output required number of frames. For interpolation network, NNVISR
duplicates the last frame of the scene. If the last group doesn’t have enough
frames for network, last frames from the previous group is recycled to provide
enough input frames. If the whole scene does not contain enough input frames
then the last frame is further duplicated to satisfy network input count.

| 3 | 1 | 2 | 9 | 7 | 8 | …
---|---|---|---|---|---|---|---
0 | 6 | 4 | 5 | 12 | 10 | 11 | …
Figure 1. NNVISR frame store layout where $n=3$ and $b=2$. Numbers denote the
frame index. Actual store order is column first.

Interpolation networks usually require $n+1$ input frames, and for a batch of
$b$, the naive way need to provide $b*(n+1)$ frames to the network. However
for a batch of $b$, the total number of related frames is actually $b*n+1$,
and $b-1$ frames are duplicated and taking extra space. The nature of batching
requires the same input from different batch placed at contiguous location in
memory. Rather than placing the frames in the natural order, NNVISR use an
layout shown in Fig. 1. This layout eliminates the need to duplicate frames
and can be extended to any $b$. To process the next batch, only the last frame
need to be copied to the location of the second last frame (6 to 5 in the
figure).

##  4\. Conclusions

We have publicly released NNVISR, a versatile video processing tool that
supports arbitrary networks and various kinds of video enhancing tasks.
Moreover, it’s high performance, well documented, easy to use and integrates
well into current video processing toolchain. In this report, we introduced
the design of NNVISR, as well as the considerations behind. We hope NNVISR can
help the adoption of video enhancement research echievements into downstream
industrial applications.

## References

  * (1)
  * Bao et al. (2019) Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. 2019.  Depth-Aware Video Frame Interpolation. In _CVPR_.
  * Bao et al. (2021) Wenbo Bao, Wei-Sheng Lai, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. 2021.  MEMC-Net: Motion Estimation and Motion Compensation Driven Neural Network for Video Interpolation and Enhancement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 43, 3 (2021), 933–948.
  * Caballero et al. (2017) Jose Caballero, Christian Ledig, Andrew Aitken, Alejandro Acosta, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017.  Real-Time Video Super-Resolution With Spatio-Temporal Networks and Motion Compensation. In _CVPR_.
  * Carion et al. (2020) Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020.  End-to-End Object Detection with Transformers. In _ECCV_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 213–229.
  * Chan et al. (2021) Kelvin C.K. Chan, Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. 2021.  BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond. In _CVPR_. 4947–4956.
  * Chan et al. (2022) Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, and Chen Change Loy. 2022.  BasicVSR++: Improving Video Super-Resolution With Enhanced Propagation and Alignment. In _CVPR_. 5972–5981.
  * Choi et al. (2020) Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee. 2020.  Channel Attention Is All You Need for Video Frame Interpolation.  _AAAI_ 34, 07 (Apr. 2020), 10663–10671.
  * Dai et al. (2017) Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. 2017.  Deformable Convolutional Networks. In _ICCV_.
  * Dong et al. (2016) Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016.  Image Super-Resolution Using Deep Convolutional Networks.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 38, 2 (2016), 295–307.
  * Dutta et al. (2021) Saikat Dutta, Nisarg A. Shah, and Anurag Mittal. 2021.  Efficient Space-Time Video Super Resolution Using Low-Resolution Flow and Mask Upsampling. In _CVPR Workshop_. 314–323.
  * Haris et al. (2020) Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. 2020.  Space-Time-Aware Multi-Resolution Video Enhancement. In _CVPR_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition. In _CVPR_.
  * Hu et al. (2022a) Mengshun Hu, Kui Jiang, Liang Liao, Jing Xiao, Junjun Jiang, and Zheng Wang. 2022a.  Spatial-Temporal Space Hand-in-Hand: Spatial-Temporal Video Super-Resolution via Cycle-Projected Mutual Learning. In _CVPR_. 3574–3583.
  * Hu et al. (2022b) Mengshun Hu, Kui Jiang, Zhixiang Nie, and Zheng Wang. 2022b.  You Only Align Once: Bidirectional Interaction for Spatial-Temporal Video Super-Resolution. In _ACM MM_ (Lisboa, Portugal) _(MM ’22)_. Association for Computing Machinery, New York, NY, USA, 847–855.
  * Huang et al. (2022) Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. 2022.  Real-Time Intermediate Flow Estimation for Video Frame Interpolation. In _ECCV_.
  * Jiang et al. (2018) Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik Learned-Miller, and Jan Kautz. 2018.  Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolation. In _CVPR_.
  * Krizhevsky et al. (2017) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2017.  ImageNet Classification with Deep Convolutional Neural Networks.  _Commun. ACM_ 60, 6 (may 2017), 84–90.
  * Ledig et al. (2017) Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. 2017.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In _CVPR_.
  * Niklaus et al. (2017a) Simon Niklaus, Long Mai, and Feng Liu. 2017a.  Video Frame Interpolation via Adaptive Convolution. In _CVPR_.
  * Niklaus et al. (2017b) Simon Niklaus, Long Mai, and Feng Liu. 2017b.  Video Frame Interpolation via Adaptive Separable Convolution. In _ICCV_.
  * Redmon et al. (2016) Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016.  You Only Look Once: Unified, Real-Time Object Detection. In _CVPR_.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In _NeurIPS_ , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (Eds.), Vol. 28. Curran Associates, Inc.
  * Ström et al. (2016) Jacob Ström, Jonatan Samuelsson, and Kristofer Dovstam. 2016.  Luma adjustment for high dynamic range video. In _DCC_. IEEE, 319–328.
  * Tan and Le (2019) Mingxing Tan and Quoc Le. 2019.  EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In _ICML_ _(Proceedings of Machine Learning Research, Vol. 97)_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, 6105–6114.
  * Tao et al. (2017) Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, and Jiaya Jia. 2017.  Detail-Revealing Deep Video Super-Resolution. In _ICCV_.
  * Tian et al. (2020) Yapeng Tian, Yulun Zhang, Yun Fu, and Chenliang Xu. 2020.  TDAN: Temporally-Deformable Alignment Network for Video Super-Resolution. In _CVPR_.
  * Wang et al. (2019) Xintao Wang, Kelvin C.K. Chan, Ke Yu, Chao Dong, and Chen Change Loy. 2019.  EDVR: Video Restoration With Enhanced Deformable Convolutional Networks. In _CVPR Workshop_.
  * Wang et al. (2018a) Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018a.  ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. In _ECCV Workshop_.
  * Wang et al. (2018b) Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. 2018b.  ESRGAN: Enhanced super-resolution generative adversarial networks. In _ECCV Workshop_.
  * Xiang et al. (2020) Xiaoyu Xiang, Yapeng Tian, Yulun Zhang, Yun Fu, Jan P. Allebach, and Chenliang Xu. 2020.  Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution. In _CVPR_.
  * Xu et al. (2021) Gang Xu, Jun Xu, Zhen Li, Liang Wang, Xing Sun, and Ming-Ming Cheng. 2021.  Temporal Modulation Network for Controllable Space-Time Video Super-Resolution. In _CVPR_. 6388–6397.
  * Xu et al. (2022) Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. 2022.  GMFlow: Learning Optical Flow via Global Matching. In _CVPR_. 8121–8130.
  * Xu et al. (2019) Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. 2019.  Quadratic Video Interpolation. In _NeurIPS_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc.
  * Xue et al. (2019) Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T. Freeman. 2019.  Video Enhancement with Task-Oriented Flow.  _International Journal of Computer Vision_ 127, 8 (01 Aug 2019), 1106–1125.
  * Zhu et al. (2019) Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. 2019.  Deformable ConvNets V2: More Deformable, Better Results. In _CVPR_.