#  Videoshop: Localized Semantic Video Editing with Noise-Extrapolated
Diffusion Inversion

Xiang Fan1 Anand Bhattad2 Ranjay Krishna1000Equal advising.
1University of Washington 2Toyota Technological Institute at Chicago
https://videoshop-editing.github.io/

Figure 0: Videoshop is a training-free method for precise video editing. Given
an original video and user edits to the first frame, Videoshop automatically
propagates the changes to all the frames of the video while maintaining
semantic, geometric, and temporal consistency. To edit the first frame, users
can leverage image editing tools, including text-based inpainting and
professional editing software like Photoshop. As such, Videoshop supports
video edits congruent with possible image edits that Photoshop enables: users
can add new objects, remove objects or their parts, modify attributes, etc.
Equal advising.

###### Abstract

We introduce Videoshop, a training-free video editing algorithm for localized
semantic edits. Videoshop allows users to use any editing software, including
Photoshop and generative inpainting, to modify the first frame; it
automatically propagates those changes, with semantic, spatial, and temporally
consistent motion, to the remaining frames. Unlike existing methods that
enable edits only through imprecise textual instructions, Videoshop allows
users to add or remove objects, semantically change objects, insert stock
photos into videos, etc. with fine-grained control over locations and
appearance. We achieve this through image-based video editing by inverting
latents with noise extrapolation, from which we generate videos conditioned on
the edited image. Videoshop produces higher quality edits against 6 baselines
on 2 editing benchmarks using 10 evaluation metrics.

Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion
Inversion

Xiang Fan1 Anand Bhattad2††thanks: Equal advising. Ranjay Krishna1††Equal
advising. 1University of Washington 2Toyota Technological Institute at Chicago
https://videoshop-editing.github.io/ Figure 1: Videoshop is a training-free
method for precise video editing. Given an original video and user edits to
the first frame, Videoshop automatically propagates the changes to all the
frames of the video while maintaining semantic, geometric, and temporal
consistency. To edit the first frame, users can leverage image editing tools,
including text-based inpainting and professional editing software like
Photoshop. As such, Videoshop supports video edits congruent with possible
image edits that Photoshop enables: users can add new objects, remove objects
or their parts, modify attributes, etc.

Figure 1:

##  1 Introduction

Traditional video editing requires sophisticated direct manipulation and is a
manually exhaustive process Mackay and Pagani (1994); Santosa et al. (2013).
Software tools like Adobe Premiere and Apple Final Cut have a steep learning
curve and are still limited in their ease of use Adobe (2023); Apple (2023).
While they support operations like stitching together video clips, they
provide little to no support for propagating edit changes from one frame to
another. Consider wanting to change the wings of a swan to have white
features, add a crown on top of the swan’s head, or even add a rubber duck
wading along with the swan (Figure 1). Such edits are only possible with
painstaking per-frame manual curation.

Current video models fall short in facilitating precise, localized semantic
editing. Open-source video diffusion models Blattmann et al. (2023a) have
spurred new algorithmic developments Kara et al. (2023); Ceylan et al. (2023);
Qi et al. (2023); Yatim et al. (2023), yet they do not support the finesse
required for localized edits. Models typically require extensive fine-tuning
for individual videos or rely on coarse textual instructions that lack
specificity Zuo et al. (2023); Shin et al. (2023); Geyer et al. (2023).
Moreover, they struggle to introduce new objects with independent motions
Yatim et al. (2023); Kahatapitiya et al. (2024).

Image editing has improved control over still frames Zhang et al. (2023d), but
ensuring temporal consistency across video frames remains unresolved. Frame-
by-frame adjustments do not guarantee the continuity essential for coherent
video sequences, revealing a gap between static image manipulation and dynamic
video editing.

To address this limitation, we introduce Videoshop, a training-free video
editing algorithm that enables users to make localized semantic edits.
Videoshop allows users to make any modification they want to the first frame
of the video; it propagates those changes, with temporally consistent motion,
to all the remaining frames. The changes to the first frame can be made using
any image editing tool Michel et al. (2024); Bhattad and Forsyth (2022); Goel
et al. (2023); Chen et al. (2023); Zhang et al. (2023c); Yenphraphai et al.
(2024); Hertz et al. (2022). Users can use image editing models like
ControlNet Zhang et al. (2023d) or Text Inversion Gal et al. (2022); they can
even load the frame onto Adobe Photoshop to edit pixels manually, add clip
art, use content-aware fill, or any of Photoshop’s features. Videoshop does
not require finetuning and leverages Stable Video Diffusion Blattmann et al.
(2023a) to make the edits. In other words, Videoshop can edit 14-frame videos
within an average of 2 minutes, and supports edits within the large domain of
videos. As video diffusion models themselves improve to support longer videos,
Videoshop will be able to edit even longer videos.

Two technical insights enable Videoshop: (1) noticing that video latents
follow a near-linear trajectory during the denoising process, and that (2) the
VAE encoder is unnormalized, resulting in a high variance in the magnitude of
the latents. To contextualize these insights, let’s review how traditional
image inversion enables image editing. Input images are inverted using a
denoising diffusion implicit model (DDIM) to generate intermediate latents
across time steps deterministically. This process approximates the noise at
time step $t+1$ using the latent from time step $t$. We find that this
approximation reconstructs only the first frame accurately in image-to-video
diffusion, resulting in unstable latents due to cumulative approximation
errors. (1) Our investigations reveal that the latents are near-linear during
the denoising process. We capitalize on this observation and introduce
inversion with noise extrapolation, a mechanism for achieving faithful
reconstruction for any video. (2) Our investigations also reveal that the
video latents are unnormalized, leading to further instability. We introduce a
latent normalization technique to ensure consistency and quality.

Our extensive experiments show that Videoshop produces higher quality edits
against $6$ baselines on $2$ editing benchmarks using $10$ evaluation metrics.
This method empowers users to make direct pixel modifications, enabling a
spectrum of semantic edits (refer to Figure 1). Examples include “transforming
a specific cat into a dog”, “removing the ears of a cat”, “changing a cat’s
fur color to purple”, “adding dynamic objects such as a squirrel on the road”
or “police car lights” that respond naturally to the car’s movements.
Videoshop not only ensures geometric fidelity where the car lights
appropriately turn with the vehicle but also captures realistic motion, as
exemplified by a rubber duck floating in unison with the swan and a squirrel
that scurries away as a police car draws near.

On a broader scale, Videoshop equips users with video manipulation
capabilities akin to those provided by image editing software like Photoshop,
enabling new potential applications that previously would have been
prohibitively challenging.

##  2 Related Work

Image and video generation. Generative Adversarial Networks (GANs) Goodfellow
et al. (2014) initially advanced the quality of image generation, with notable
architectures such as StyleGAN Karras et al. (2019). Despite the community’s
efforts, GANs remain difficult to train, limiting their ability to generate
videos Kang et al. (2023); Sauer et al. (2023). Diffusion models, including
Denoising Score Matching Vincent (2011) and Noise-Contrastive Estimation
Gutmann and Hyvärinen (2010), are much easier to scale Karras et al. (2022);
Ho et al. (2020); Rombach et al. (2022). Today’s video generation models are
generally diffusion models Singer et al. (2022); Blattmann et al. (2023b, a);
Kondratyuk et al. (2023); Menapace et al. (2024); Mullan et al. (2023); Wang
et al. (2023a); Chen et al. (2024); Wang et al. (2023b); Zhang et al. (2023a).
We develop our technique using Stable Video Diffusion (SVD) Blattmann et al.
(2023a), which is based on the EDM Karras et al. (2022) framework and
generates high-quality videos conditioned on a first frame.

Text-based video editing. Current video editing methods are commonly text-
based, which modify video based on textual instructions. Such tasks include
motion transfer Yatim et al. (2023); Chan et al. (2019); Yin et al. (2023),
object editing Qi et al. (2023); Kahatapitiya et al. (2024); Kara et al.
(2023); Shin et al. (2023); Ren et al. (2024); Zuo et al. (2023), attribute
editing Kara et al. (2023); Jeong and Ye (2024), texture editing Couairon et
al. (2023); Kasten et al. (2021), and style editing Qi et al. (2023);
Kahatapitiya et al. (2024); Ceylan et al. (2023); Geyer et al. (2023); Yang et
al. (2023). A common approach is to use a text-to-video model Wang et al.
(2023a) and control the denoising process such that the generated result
satisfies the editing conditions. However, textual instructions alone provide
minimal specification, limiting most practical applications beyond modifying
object classes, attributes, and textures.

DDIM inversion. Denoising Diffusion Implicit Models (DDIMs) Song et al. (2022)
are a class of diffusion models that deterministically generate samples from a
random noise. Due to its determinism, DDIM models can be inverted to find the
corresponding noise given a sample. In the context of image editing, several
methods have been proposed to improve the quality the edited images, including
null-text inversion Mokady et al. (2022) and mathematically-exact inversion
methods Wallace et al. (2022); Meiri et al. (2023); Zhang et al. (2023b).
EDICT Wallace et al. (2022) and BDIM Zhang et al. (2023b) propose an
alternative denoising process that tracks more than one latent variables to
derive an exact solution to the inversion formula. Another proposes a fixed-
point iteration method to solve the inversion formula Meiri et al. (2023).
Unfortunately, applying diffusion inversion to text-to-image models can lead
to a loss of temporal consistency in the resulting video Qi et al. (2023).

Layered video editing. Another approach to video editing is through layered
atlases, which decompose video frames into several layers (often foregrounds
and backgrounds) and edit the layers corresponding to a target (e.g. NLA
Kasten et al. (2021) and DiffusionAtlas Chang et al. (2023)). However, atlases
do not generate new motions or support edits beyond changes to object class,
attribute, and texture Chang et al. (2023).

##  3 Method

Given an input video $\mathcal{V}$ and an edited first frame $\mathcal{I}$,
our goal is to generate a new video $\mathcal{J}$ that preserves the overall
motion and semantics of the original video $\mathcal{V}$, while propagating
the changes made to the first frame $\mathcal{I}$. Before we explain our
method, we revisit diffusion models, DDIM inversion, and the EDM framework.

###  3.1 Background on latent diffusion models

Denoising Diffusion Probabilistic Models (DDPM) Ho et al. (2020) are a class
of models that generate an image or video from a random Gaussian noise through
a sequence of $T$ denoising steps. The denoising process is defined as a
sequence of timesteps $T,T-1,...,0$. At timestep $T$, a $F\times H\times
W\times 3$-dimensional random noise is sampled from a multivariate normal
distribution, denoted as the initial noise $x_{T}$. DDPM applies a denoising
neural network to iteratively de-noise latents $x_{t+1}\rightarrow x_{t}$
until $x_{0}$. The final $x_{0}$ is a generated $F\times H\times W\times
3$-dimensional video with $F$ frames, and spatial size of $H\times W$.

Denoising in pixel space is computationally expensive as each denoising step
needs to produce a $F\times H\times W\times 3$-dimensional noise estimation.
Latent Diffusion Models Rombach et al. (2022) instead encode the sample image
into a much smaller latent space using a pretrained variational autoencoder
(VAE) with a $F\times h\times w\times c$-dimensional hidden representation,
such that $F\times h\times w\times c\ll F\times H\times W\times 3$. Latent
models sample $x_{T}$ in the latent space and iteratively denoise until
$x_{0}$. Finally, $x_{0}$ is decoded using the pretrained VAE decoder into the
pixel space.

Latent models are trained by first sampling a video encoded as $x_{0}$, a
timestep $t$, and a random noise $\epsilon$. From these, a noise
$\epsilon_{t}$ corresponding to the timestep $t$ is calculated and added to
$x_{0}$ to produce $x_{t}$. A denoising, timestep-conditioned U-Net
$\epsilon_{\theta}$, is trained to estimate $\epsilon_{t}$ given $x_{t}$:

$\displaystyle\mathbb{E}_{x,t,\epsilon\sim{\mathcal{N}}(0,1)}\lVert\epsilon_{t}-\epsilon_{\theta}(x_{t};t)\rVert^{2}$
(1)

###  3.2 Background on diffusion inversion

Denoising Diffusion Implicit Models (DDIMs) Song et al. (2022) are a class of
latent diffusion models with a deterministic denoising process. DDIMs
deterministically generate samples from a random noise when its denoising step
noise parameter is set to zero. This determinism allows for the diffusion
inversion process.

The inversion process is a technique used to generate the deterministic
latents $x_{T}$ given a video $\mathcal{V}$. From $\mathcal{V}$, we first
extract $x_{0}$ from the pretrained variational autoencoder’s encoder. This
process calculates the latents $\hat{x}_{t}$ for $t=1\ldots T$, given $x_{0}$.
If the inversion process is accurate, denoising $\epsilon_{\theta}(x_{t};t)$
iteratively starting from $\hat{x}_{T}$ should yield $\hat{x}_{0}$, such that
$\hat{x}_{0}\approx x_{0}$.

In diffusion-based image editing, it is common to first invert the image
latents $x_{0}$ into its corresponding noise $\hat{x}_{T}$, and apply the
denoising process with a modified conditioning text to obtain the edited image
Mokady et al. (2022). Common image diffusion models are conditioned on a text
prompt Rombach et al. (2022).

With deterministic sampling, the denoising step from $x_{t+1}\rightarrow
x_{t}$ is defined as:

$\displaystyle x_{t}$
$\displaystyle=\sqrt{\alpha_{t}}\underbrace{\left(\frac{x_{t+1}-\sqrt{1-\alpha_{t+1}}\epsilon_{\theta}(x_{t+1};t+1)}{\sqrt{\alpha_{t+1}}}\right)}_{\text{predicted
}x_{0}}$
$\displaystyle+\underbrace{\sqrt{1-\alpha_{t}}\cdot\epsilon_{\theta}(x_{t+1};t+1)}_{\text{direction
pointing to }x_{t+1}}\text{\vspace{-4em}}$ (2)

where $\epsilon_{\theta}$ is the noise-predicting U-Net and $\alpha_{t}$ is
the noise scheduling parameter.

Stable Video Diffusion Blattmann et al. (2023a) is a video diffusion model
that conditions on the first-frame image instead of text. Ideally, one would
invert the video latents similar to the inversion process mentioned above and
apply edits by conditioning on an edited first frame. However, we observe that
naively inverting the video latents often results in an incoherent video
(Figure 6), necessitating a new method.

###  3.3 Background on the EDM framework

Stable Video Diffusion Blattmann et al. (2023a) employs the EDM framework
Karras et al. (2022), which improves upon DDIM with a reparameterization to
the denoising process. A denoising step from $x_{t+1}\rightarrow x_{t}$ of
deterministic sampling in the EDM framework is:

$\displaystyle x_{t}$
$\displaystyle=x_{t+1}+\frac{\sigma_{t}-\sigma_{t+1}}{\sigma_{t+1}}\eta$ (3)
$\displaystyle\eta$
$\displaystyle=\underbrace{x_{t+1}-\overbrace{\left(c_{\text{skip}}^{t+1}x_{t+1}+c_{\text{out}}^{t+1}F_{\theta}\left(c_{\text{in}}^{t+1}x_{t+1};c_{\text{noise}}^{t+1}\right)\right)}^{\text{predicted
}x_{0}}}_{\text{noise removed at step }t\text{ before scaling}}$ (4)

where $\sigma_{t}$ is the scheduled noise level at step $t$.
$c_{\text{skip}}^{t}$, $c_{\text{in}}^{t}$, $c_{\text{out}}^{t}$, and
$c_{\text{noise}}^{t}$ are coefficients dependent on the noise schedule and
the current step $t$. $F_{\theta}$ is a neural network parametrized by
$\theta$.

To perform inversion in the EDM Framework, we can rewrite Equation 4 as an
inversion step $\hat{x}_{t}\rightarrow\hat{x}_{t+1}$, where $\hat{x}_{t+1}=$

$\displaystyle\frac{\sigma_{t+1}\hat{x}_{t}+\left(\sigma_{t}-\sigma_{t+1}\right)c_{\text{out}}^{t+1}F_{\theta}\left(c_{\text{in}}^{t+1}\hat{x}_{t+1};c_{\text{noise}}^{t+1}\right)}{\left(\sigma_{t}-\sigma_{t+1}\right)\left(1-c_{\text{skip}}^{t+1}\right)+\sigma_{t+1}}$
(5)

However, because the input to $F_{\theta}$ is dependent on the next inverted
latent $\hat{x}_{t+1}$, Equation 5 is not directly solvable. Naive inversion
methods approximate $\hat{x}_{t+1}$ with $\hat{x}_{t}$ such that:

$\displaystyle
F_{\theta}\left(c_{\text{in}}^{t+1}\hat{x}_{t+1};c_{\text{noise}}^{t+1}\right)\approx
F_{\theta}\left(c_{\text{in}}^{t}\hat{x}_{t};c_{\text{noise}}^{t+1}\right)$
(6)

###  3.4 Limitations with naive video inversion to EDM

While common in inverting image diffusion models Mokady et al. (2022); Ceylan
et al. (2023), naive inversion (Equation 6) leads to latents that only
correctly reconstruct the first video frame, when directly applied to Stable
Video Diffusion. We find that naive inversion (Equation 6) introduces a
compounding approximation error that we show quantitatively and qualitatively
in Section 4.1.

Several methods have been proposed to address this in image diffusion models
Wallace et al. (2022); Meiri et al. (2023); Zhang et al. (2023b). Amongst
them, EDICT Wallace et al. (2022) and FPI Meiri et al. (2023) require
additional passes through the model at each step, increasing computation cost.
While BDIA Zhang et al. (2023b) improves upon EDICT Wallace et al. (2022) by
eliminating additional passes, both of them modify the denoising process. We
find that BDIA destabilizes the latents in the inversion process and results
in undesirable artifacts in the resulting video (Figure 5).

Figure 2: Overview of Videoshop for localized semantic video editing. Our
contributions are highlighted with red boxes and arrows. Our method includes
four primary stages: (1) Encode & Norm, where the input video is encoded into
a latent space using a VAE encoder, followed by normalization to ensure
stability throughout inversion. (2) In the Inversion w/ Noise Extrapolation
phase, noise extrapolation is systematically applied at each step to provide a
corrective term that guides the inversion trajectory, ensuring the video is
mapped to correct latent noise. This step is key for aligning the latent space
trajectory at every timestep. (3) Diffusion then ensures user edits are
seamlessly integrated across the video sequence, enforcing consistency while
diffusing the initial modifications through time. (4) The last step is Rescale
& Decode, where the now-edited latent sequence is rescaled to align with the
original data’s statistical distribution and decoded back into the video,
resulting in an output video that reflects the desired semantic edits while
maintaining the natural flow of the original sequence.

###  3.5 Our contribution: Inverting with noise extrapolation

Our goal is to find a better approximation for
$F_{\theta}\left(c_{\text{in}}^{t+1}\hat{x}_{t+1};c_{\text{noise}}^{t+1}\right)$
in Equation 5. First, we observe that the latents in the denoising process
maintain a near-linear trajectory. We then propose noise extrapolation to
exploit this observation.

Figure 3: Cosine similarity matrix for pairs of latent vectors throughout the
denoising process. The latent vectors are approximately collinear, supporting
our linear noise extrapolation.

####  3.5.1 Near-linearity of $x_{t}$ trajectory.

It has been observed that the denoising trajectory of $x_{t}$ in image
diffusion models is approximately linear Karras et al. (2022) at low and high
noise levels. To investigate the latent trajectory of video diffusion models,
we measure the average cosine similarity between vectors $x_{t}\rightarrow
x_{0}$ and $x_{t^{\prime}}\rightarrow x_{0}$ for pairs of $t,t^{\prime}$ in
the denoising process of $100$ random videos and show the results in Figure 3.
We observe high cosine similarities throughout the denoising process after the
initial steps, suggesting that the trajectory of $x_{t}$ is approximately
linear after the low noise levels, which we exploit in our noise extrapolation
method to invert $\hat{x}_{t}$.

####  3.5.2 Noise extrapolation.

With this insight, we linearly extrapolate the noise at $\hat{x}_{t}$ to
obtain an approximation of $\bar{x}_{t+1}$ to provide to $F_{\theta}$, as
follows:

$\displaystyle\bar{x}_{t+1}$
$\displaystyle\approx\begin{cases}\underbrace{\frac{\sigma_{t+1}}{\sigma_{t}}\overbrace{(\hat{x}_{t}-x_{0})}^{\sim{\mathcal{N}}(0,\sigma_{t})}}_{\sim{\mathcal{N}}(0,\sigma_{t+1})}+x_{0}&\qquad(\sigma_{t}>\Sigma)\vspace{1em}\\\
{\mathcal{N}}(0,\sigma_{t+1})+x_{0}&\qquad(\sigma_{t}\leq\Sigma)\end{cases}$
(7)

where $\Sigma$ is a threshold noise level. The threshold $\Sigma$ is necessary
because at low $\sigma_{t}$, dividing by a small number results in a large
noise, which can destabilize the latents. We ablate noise threshold in Section
4.1.

We modify Equation 5 to estimate the inversion using $\bar{x}_{t+1}$. Putting
it together, we have $\hat{x}_{t+1}=$

$\displaystyle\frac{\sigma_{t+1}\hat{x}_{t}+\left(\sigma_{t}-\sigma_{t+1}\right)c_{\text{out}}^{t+1}F_{\theta}\left(c_{\text{in}}^{t+1}\bar{x}_{t+1};c_{\text{noise}}^{t+1}\right)}{\left(\sigma_{t}-\sigma_{t+1}\right)\left(1-c_{\text{skip}}^{t+1}\right)+\sigma_{t+1}}$
(8)

After obtaining the final $\hat{x}_{T}$ from the inversion process, we apply
the denoising process to $\hat{x}_{T}$ conditioned on the edited image
$\mathcal{I}$ to obtain the edited latents $x_{0}$.

###  3.6 Our contribution: Latent normalization and rescaling

Direct output from the VAE encoder is unnormalized, resulting in a large
variance in the magnitude of the final latent from the inversion process. We
observe that this leads to poor quality in generated videos. To address this,
we propose to normalize the latents before the start of the inversion process
to unit standard deviation to stabilize the latents. After denoising, we
rescale the latents with the mean and standard deviation of the latents of the
target image:

$\displaystyle\hat{x}_{0}$
$\displaystyle=\frac{x_{\text{in}}}{\sigma_{\text{in}}}$ (9) $\displaystyle
x_{\text{out}}$
$\displaystyle=\frac{\sigma_{\text{img}}}{\sigma_{0}}\left(x_{0}-\mu_{0}+\mu_{\text{img}}\right)$
(10)

where $x_{\text{in}}$ is the VAE-encoded latent sample, $\hat{x}_{0}$ is the
input to the inversion process, $x_{0}$ is the output of the denoising
process, and $x_{\text{out}}$ is the final output to be decoded by the VAE
into video $\mathcal{J}$. $\mu_{\text{img}}$ and $\sigma_{\text{img}}$ are the
mean and standard deviation of the VAE-encoded latents of the target image.
All normalization are done per-channel. $\sigma_{\text{in}}$ is calculated
across all frames; $\mu_{0}$, $\sigma_{0}$, $\mu_{\text{img}}$, and
$\sigma_{\text{img}}$ are calculated for the first frame (the only available
frame in the target image).

##  4 Experiments and Results

The key takeaways from our experiments are as follows: (1) Videoshop
successfully performs localized semantic video editing among a diverse set of
edit types. (2) Compared to Videoshop, existing methods demonstrate clear
limitations in maintaining visual fidelity to the source video and target
edit. (3) Videoshop achieves SOTA performance in localized editing, as
evaluated by edit fidelity and source faithfulness, while maintaining high
temporal consistency. (4) In our user study, Videoshop consistently
outperforms text-based video editing methods, while maintaining high video
generation quality. (5) Videoshop is efficient, with an average speedup of
2.23x compared to the baselines.

####  4.0.1 Datasets.

We utilize two datasets for our experiments: a large-scale generated video
editing dataset from the MagicBrush Zhang et al. (2023c) image-editing dataset
and an expert-curated video editing dataset with source videos from HD-
VILA-100M Xue et al. (2022).

The MagicBrush dataset is a manually annotated image editing dataset that
contains over 10,000 tuples of “(source image, instruction, edit mask, edited
image)”. These tuples cover a wide range of edit types, including object
addition, replacement, removal, and changes in action, color, texture, and
counting. To convert MagicBrush into a video dataset, we generate videos
conditioned on the source images using a video generation model Blattmann et
al. (2023a). The first frame of each generated video is conditioned to match
the corresponding source image. The resulting dataset consists of “(source
video, instruction, edit mask, edited image)” tuples.

Table 1: Types of edit in the expert dataset. Type of Edit |  Percentage %
---|---
Add object |  36%
Change appearance |  20%
Remove object |  18%
Replace object |  16%
Change action |  06%
Change color |  04%

The HD-VILA-100M dataset is a large-scale high-resolution video dataset
collected from YouTube, encompassing diverse open domains. From this dataset,
we sample 45 videos and ask editing experts to provide edits on the first
frame of each video, along with the corresponding edit instructions. The
resulting expert dataset consists of “(source video, instruction, edit mask,
edited image)” tuples. A summary of the edit types from the expert dataset is
shown in Table 1. All videos are resized to 14 frames with an aspect ratio of
16:9.

Figure 4: Examples of edited videos. Our method handles a diverse set of edit
types; examples shown include appearance editing, object removal, semantic
editing, and shape/texture editing. Videoshop successfully performs precise
local edits while maintaining high visual fidelity to the source video. Figure
5: Qualitative comparison against baselines. Videoshop successfully maintains
visual fidelity to the source video and target edit, while existing methods
fail to do so. Table 2: Quantitative results on MagicBrush. Videoshop
outperforms baselines on most metrics within edit fidelity and source
faithfulness, and is competitive on other metrics. (

T.C.

= Temporal Consistency.) Method |  Edit Fidelity |  Source Faithfulness |  T.C.
---|---|---|---
$\text{CLIP}_{\text{tgt}}$ $\uparrow$ |  $\text{CLIP}_{\text{tgt}}^{+}$ $\uparrow$ |  TIFA $\uparrow$ |  $\text{CLIP}_{\text{src}}$ $\uparrow$ |  $\text{CLIP}_{\text{src}}^{+}$ $\uparrow$ |  Flow $\downarrow$ |  $\text{Flow}^{+}$ $\downarrow$ |  FVD $\downarrow$ |  SSIM $\uparrow$ |  $\text{CLIP}_{\text{TC}}$ $\uparrow$
($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 1$) |  ($\times 1$) |  ($\times 1$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$)
BDIA Zhang et al. (2023b) |  82.12 |  82.19 |  57.67 |  82.48 |  87.10 |  2.83 |  1.43 |  3482.79 |  49.67 |  94.36
Pix2Video Ceylan et al. (2023) |  71.19 |  76.47 |  51.98 |  74.55 |  79.03 |  3.59 |  2.58 |  2993.95 |  59.08 |  94.48
Fate/Zero Qi et al. (2023) |  84.87 |  79.10 |  55.41 |  92.41 |  86.94 |  4.42 |  3.11 |  2205.03 |  48.59 |  95.71
Spacetime Yatim et al. (2023) |  63.85 |  75.20 |  46.33 |  65.74 |  71.91 |  8.24 |  5.62 |  4815.63 |  41.61 |  96.58
RAVE Kara et al. (2023) |  74.70 |  78.58 |  51.12 |  75.99 |  80.19 |  3.35 |  2.42 |  2354.09 |  62.21 |  96.59
SVD (no src. vid.) |  87.63 |  84.73 |  64.16 |  90.64 |  94.47 |  9.74 |  6.89 |  1894.16 |  47.50 |  95.07
Videoshop (Ours) |  88.80 |  85.58 |  64.40 |  90.95 |  94.77 |  1.90 |  0.78 |  1478.76 |  71.92 |  95.16
Table 3: Quantitative results on the expert dataset. Videoshop outperforms the
baselines on most metrics within edit fidelity and source faithfulness. There
is a noticeable increase in the overall optical flow errors in the expert
dataset, which can be attributed to the increased motion in the in-the-wild
videos. However, Videoshop still maintains competitive optical flow scores and
remains top-performing in preserving the source video flow in regions
unoccupied by edit masks. (

T.C.

= Temporal Consistency.) Method |  Edit Fidelity |  Source Faithfulness |  T.C.
---|---|---|---
$\text{CLIP}_{\text{tgt}}$ $\uparrow$ |  $\text{CLIP}_{\text{tgt}}^{+}$ $\uparrow$ |  TIFA $\uparrow$ |  $\text{CLIP}_{\text{src}}$ $\uparrow$ |  $\text{CLIP}_{\text{src}}^{+}$ $\uparrow$ |  Flow $\downarrow$ |  $\text{Flow}^{+}$ $\downarrow$ |  FVD $\downarrow$ |  SSIM $\uparrow$ |  $\text{CLIP}_{\text{TC}}$ $\uparrow$
($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 1$) |  ($\times 1$) |  ($\times 1$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$)
BDIA Zhang et al. (2023b) |  81.90 |  79.97 |  63.63 |  80.38 |  82.94 |  12.18 |  10.10 |  3048.53 |  36.42 |  92.86
Pix2Video Ceylan et al. (2023) |  68.28 |  71.18 |  49.66 |  72.35 |  76.81 |  7.92 |  6.46 |  1876.43 |  61.23 |  93.90
Fate/Zero Qi et al. (2023) |  71.51 |  72.53 |  48.65 |  74.59 |  82.68 |  9.70 |  8.30 |  2103.63 |  52.75 |  95.77
Spacetime Yatim et al. (2023) |  58.61 |  68.45 |  43.06 |  56.51 |  65.86 |  7.93 |  6.31 |  4497.18 |  40.12 |  97.50
RAVE Kara et al. (2023) |  71.59 |  71.39 |  57.78 |  74.37 |  76.35 |  5.59 |  4.54 |  1890.95 |  63.76 |  96.05
Videoshop (Ours) |  87.96 |  83.54 |  66.14 |  84.89 |  91.50 |  5.85 |  4.47 |  1718.31 |  65.63 |  94.71
Table 4: Human evaluation and execution time. For human evaluation, evaluators are asked to compare the editing quality and video generation quality of our method against each baseline. Videoshop outperforms all baselines in both editing quality and video generation quality and has a competitive runtime compared to the baseline methods. Videoshop (Ours) |  Editing Quality |  Video Generation Quality |  Execution Time
---|---|---|---
vs. … |  (preference in our favor %) |  (preference in our favor %) |  (as multiples of ours)
BDIA Zhang et al. (2023b) |  94.89% |  90.53% |  1.03x
Pix2Video Ceylan et al. (2023) |  98.30% |  96.97% |  4.70x
Fate/Zero Qi et al. (2023) |  98.67% |  92.23% |  0.71x
Spacetime Yatim et al. (2023) |  99.81% |  69.32% |  3.41x
RAVE Kara et al. (2023) |  92.99% |  74.05% |  1.31x
Average |  96.93% |  84.62% |  2.23x
Figure 6: Examples from ablations. The yellow vertical line tracks the
background movement and the yellow color picker shows the RGB value of a
background pixel at the same location in the first frame. As we can see from
the examples, the lack of noise extrapolation results in incoherent videos,
the lack of latent normalization results in a slow, incorrect background
movement that disregard the movement of the source video, and the lack of
latent rescaling results in shifted colors. Table 5: Ablations on a 10-video
subset. Our method outperforms all ablations except for a few scores when
removing latent norm (second row). However, it is important to note that the
lack of latent norm tends to result in static or slow movements that disregard
the movement in the source video, as evidenced by the low Flow scores.
Furthermore, removing the noise threshold (last row) leads to division by a
very small number, resulting in NaN values in the latents. (

T.C.

= Temporal Consistency.) Method |  Edit Fidelity |  Source Faithfulness |  T.C.
---|---|---|---
$\text{CLIP}_{\text{tgt}}$ $\uparrow$ |  $\text{CLIP}_{\text{tgt}}^{+}$ $\uparrow$ |  TIFA $\uparrow$ |  $\text{CLIP}_{\text{src}}$ $\uparrow$ |  $\text{CLIP}_{\text{src}}^{+}$ $\uparrow$ |  Flow $\downarrow$ |  $\text{Flow}^{+}$ $\downarrow$ |  FVD $\downarrow$ |  SSIM $\uparrow$ |  $\text{CLIP}_{\text{TC}}$ $\uparrow$
($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$) |  ($\times 1$) |  ($\times 1$) |  ($\times 1$) |  ($\times 10^{-2}$) |  ($\times 10^{-2}$)
Videoshop (Ours) |  92.12 |  88.89 |  100 |  88.82 |  97.92 |  4.57 |  2.42 |  1410.70 |  57.25 |  95.18
$-$ Norm. |  91.36 |  91.54 |  100 |  90.22 |  97.34 |  19.57 |  13.00 |  1324.45 |  41.74 |  96.58
$-$ Rescaling |  89.74 |  88.45 |  90 |  88.72 |  97.34 |  6.37 |  3.64 |  1784.10 |  52.52 |  94.60
$-$ Extrapolation |  73.36 |  75.42 |  70 |  87.19 |  95.12 |  17.92 |  10.47 |  5316.65 |  16.76 |  84.55
$-$ Threshold |  NaN |  NaN |  NaN |  NaN |  NaN |  NaN |  NaN |  NaN |  NaN |  NaN

####  4.0.2 Experimental setup.

We use the Stable Video Diffusion model Blattmann et al. (2023a) as the video
generation model ($F_{\theta}$ in Equation 8). We compare our method against
text-based video editing models including Pix2Video Ceylan et al. (2023),
Fate/Zero Qi et al. (2023), Spacetime Diffusion Yatim et al. (2023), and RAVE
Kara et al. (2023), as well as the exact inversion method BDIA Zhang et al.
(2023b). Pix2Video and RAVE are based on text-to-image models, while Fate/Zero
and Spacetime Diffusion are based on text-to-video models. All baselines are
implemented using their official codebases with videos resized to match the
maximum supported video size. The baseline text-based video editing models
either take a source and target prompt, or a single target prompt. To produce
such source-target prompt pairs, we caption the first frame of the source
video and update the modified concept to the target one (e.g., “a dog running”
$\rightarrow$ “a cat running”).

####  4.0.3 Video editing capabilities of Videoshop.

We demonstrate the video editing capabilities of Videoshop in Figure 1 and
Figure 4. We show a diverse example of video edits, including object addition
(Figure 1(a) [baseline=(char.base)] [shape=circle,draw,inner sep=0.5pt] (char)
3; [baseline=(char.base)] [shape=circle,draw,inner sep=0.5pt] (char) 4; ),
object removal (Figure 1(c) [baseline=(char.base)] [shape=circle,draw,inner
sep=0.5pt] (char) 2; , Figure 4(b)), color edits (Figure 1(c)
[baseline=(char.base)] [shape=circle,draw,inner sep=0.5pt] (char) 3; ),
semantic edits (Figure 1(c) [baseline=(char.base)] [shape=circle,draw,inner
sep=0.5pt] (char) 1; , Figure 4(c)(d)(e)), object attachment (Figure 1(b)
[baseline=(char.base)] [shape=circle,draw,inner sep=0.5pt] (char) 2; ) and
appearance edits (Figure 1(a) [baseline=(char.base)] [shape=circle,draw,inner
sep=0.5pt] (char) 1; , Figure 4(a)). We also show examples of multiple edits
at once (Figure 1). We observe that Videoshop successfully performs precise
local edits, appearance control, and independent object addition, while
maintaining high visual fidelity to the source video.

####  4.0.4 Limitations of existing methods.

We compare Videoshop with existing methods in Figure 5. We observe that
existing methods often fail to maintain visual fidelity to the source video
and target edit. For example, Fate/Zero does not correctly modify the source
video based on the edit, Spacetime Diffusion generates videos that are
structurally inconsistent with the source video, Pix2Video and RAVE can both
undesirably change the style of the source video (Figure 5(a)), and BDIA
demonstrates large visual inconsistencies from the source video.

####  4.0.5 Automated Evaluation.

To assess the quality of video generation, we adopt a comprehensive set of
metrics for evaluating the source video frames, the target image, and the
model-generated edited video frames. We use the terms “source video frames,”
“target image,” and “edited video frames” to refer to the frames from the
source video, the reference edited first frame, and the model-generated edited
video frames.

  1. 1.

Edit Fidelity: Building on prior text-based video editing work Kara et al.
(2023); Yatim et al. (2023); Ceylan et al. (2023), we use the CLIP similarity
Radford et al. (2021) metric. Our $\text{CLIP}_{\text{tgt}}$ metric measures
the similarity of CLIP embeddings between each edited frame and the target
image. The $\text{CLIP}_{\text{tgt}}^{+}$ score, utilizing CoTracker Karaev et
al. (2023), focuses only on the edited region. Furthermore, the TIFA score Hu
et al. (2023) evaluates semantic alignment between the target image and the
edited video frames.

  2. 2.

Source Faithfulness: We measure the $\text{CLIP}_{\text{src}}$ similarity
between the source and edited videos. To refine this,
$\text{CLIP}_{\text{src}}^{+}$ masks the edited region using CoTracker Karaev
et al. (2023) and measures only the unedited region. Motion faithfulness is
assessed via the end-point error (EPE) from optical flow comparisons using
RAFT Teed and Deng (2020), which we denote as Flow. We also report the EPE
within only unedited regions, denoted as $\text{Flow}^{+}$. FVD and SSIM
scores provide additional quality measures.

  3. 3.

Temporal Consistency: The average CLIP similarity between consecutive frames
($\text{CLIP}_{\text{TC}}$) is calculated following protocols from Kara et al.
(2023); Ceylan et al. (2023); Geyer et al. (2023); Yang et al. (2023).

We report these metrics for both Videoshop and baseline methods, detailed in
Table 2 for MagicBrush and Table 3 for the expert dataset. In Table 2,
Videoshop demonstrates superior performance over the baseline methods across
the majority of metrics related to edit fidelity and source faithfulness,
while showing competitive results in other evaluated areas, with marginal
differences from the leading method. The findings in Table 3 echo this
pattern, as Videoshop consistently ranks above the baselines in most metrics
and holds competitive in the rest. Notably, the expert dataset exhibits an
overall increase in optical flow errors, likely due to the more complex motion
dynamics in real-world videos. Despite this, Videoshop maintains competitive
scores on flow metrics and stands out particularly in preserving the source
video flow in regions unoccupied by the edit.

####  4.0.6 Human Evaluation.

We conduct human evaluation study on the dev set of the MagicBrush dataset.
For each baseline, we ask human evaluators to compare the editing quality and
video generation quality of our method against the baseline. The results are
presented in the first two columns of Table 4. From the results, we observe
that our method outperforms all baselines in both editing quality and video
generation quality.

####  4.0.7 Efficiency.

We assess Videoshop’s efficiency against baseline methods by measuring the
average execution time per video (last column in Table 4). Videoshop aligns
closely with the execution time of BDIA, which is known for its low overhead
due to not requiring extra U-Net steps. Additionally, Videoshop provides a
considerable speed advantage, operating at more than twice the speed (2.23x
faster) of the average baseline method.

###  4.1 Ablation Study

We ablate latent normalization (Section 3.6), latent rescaling (Section 3.6),
noise extrapolation (Section 3.5), and noise threshold (Section 3.5)
qualitatively in Figure 6 and quantitatively in Table 5. In Figure 6, the
yellow vertical line tracks the background movement and the yellow color
picker shows the RGB value of a background pixel at the same location in the
first frame. As we can see, the lack of noise extrapolation results in
incoherent videos, the lack of latent normalization results in a slow,
incorrect background movement that disregards motion in the source video, and
removing latent rescaling results in shifted colors. In Table 5, we observe
that our final method outperforms all ablations except for a few scores
without latent normalization. However, it is important to note that the lack
of latent normalization tends to result in static or slow movements that
disregard motion in the source video, and therefore is not suitable for video
editing. Furthermore, removing the threshold in noise extrapolation leads to
division by a very small number, resulting in numerical instability and NaN
values in the latents.

##  5 Discussion and Limitations

In this paper, we introduced Videoshop, a novel video editing approach that
lets users make localized semantic edits without any need for re-training.
Other video editing methods currently deal with this task by editing whole
videos with sparse textual instructions. Instead, we simplify the problem by
reducing it to image editing, a well-studied and widely applied task in the
image domain. We exploited the linear relationship in the latents during the
inversion process with our noise extrapolation approach and showed our latent
normalization and latent rescaling enables realistic localized semantic
editing.

Despite the strengths of our method, there are limitations: (a) The VAE used
in our method may lead to loss of information during video encoding, which
could obscure some fine details such as small text. (b) In cases where the
source video contains large movements or flickering, the temporal consistency
of our method may be compromised. (c) Our method is training-free, which means
that it only uses the knowledge contained in the base model. This limits the
introduction of new information (such as new motion) and also restricts the
length of video generation to the input length of the base model. However, we
expect that our method will also improve as image-to-video models continue to
improve. Finally, we can leverage video models to extend our method for
editing 3D meshes Decatur et al. (2023b, a).

In summary, our Videoshop reimagines video editing and could unlock
possibilities for new and creative applications. With our method, users can
edit videos with the same ease as they would edit images in Photoshop. Moving
forward, we plan to combine image editing with motion and trajectory controls
to ensure more seamless video results.

## Acknowledgments

This material was supported in part by the NSF through award IIS-2211133. Any
opinions, findings, conclusions, or recommendations expressed in this material
are those of the author(s) and do not necessarily reflect the views of the
NSF, and no official endorsement should be inferred.

## References

  * Adobe (2023) Adobe. 2023.  Adobe premiere pro.  https://www.adobe.com/products/premiere.html.  Accessed: 2024-03-03.
  * Apple (2023) Apple. 2023.  Final cut pro.  https://www.apple.com/final-cut-pro/.  Accessed: 2024-03-03.
  * Bhattad and Forsyth (2022) Anand Bhattad and David A Forsyth. 2022.  Cut-and-paste object insertion by enabling deep image prior for reshading.  In _2022 International Conference on 3D Vision (3DV)_ , pages 332–341. IEEE.
  * Blattmann et al. (2023a) Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. 2023a.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  _Preprint_ , arXiv:2311.15127.
  * Blattmann et al. (2023b) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023b.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22563–22575.
  * Ceylan et al. (2023) Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J. Mitra. 2023.  Pix2video: Video editing using image diffusion.  _Preprint_ , arXiv:2303.12688.
  * Chan et al. (2019) Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A. Efros. 2019.  Everybody dance now.  _Preprint_ , arXiv:1808.07371.
  * Chang et al. (2023) Shao-Yu Chang, Hwann-Tzong Chen, and Tyng-Luh Liu. 2023.  Diffusionatlas: High-fidelity consistent diffusion video editing.  _Preprint_ , arXiv:2312.03772.
  * Chen et al. (2024) Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. 2024.  Videocrafter2: Overcoming data limitations for high-quality video diffusion models.  _Preprint_ , arXiv:2401.09047.
  * Chen et al. (2023) Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. 2023.  Anydoor: Zero-shot object-level image customization.  _arXiv preprint arXiv:2307.09481_.
  * Couairon et al. (2023) Paul Couairon, Clément Rambour, Jean-Emmanuel Haugeard, and Nicolas Thome. 2023.  Videdit: Zero-shot and spatially aware text-driven video editing.  _Preprint_ , arXiv:2306.08707.
  * Decatur et al. (2023a) Dale Decatur, Itai Lang, Kfir Aberman, and Rana Hanocka. 2023a.  3d paintbrush: Local stylization of 3d shapes with cascaded score distillation.  _arXiv preprint arXiv:2311.09571_.
  * Decatur et al. (2023b) Dale Decatur, Itai Lang, and Rana Hanocka. 2023b.  3d highlighter: Localizing regions on 3d shapes via text descriptions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20930–20939.
  * Gal et al. (2022) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_.
  * Geyer et al. (2023) Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. 2023.  Tokenflow: Consistent diffusion features for consistent video editing.  _Preprint_ , arXiv:2307.10373.
  * Goel et al. (2023) Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi. 2023.  Pair-diffusion: Object-level image editing with structure-and-appearance paired diffusion models.  _arXiv preprint arXiv:2303.17546_.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27.
  * Gutmann and Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. 2010.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the thirteenth international conference on artificial intelligence and statistics_ , pages 297–304. JMLR Workshop and Conference Proceedings.
  * Hertz et al. (2022) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Preprint_ , arXiv:2006.11239.
  * Hu et al. (2023) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023.  Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.  _Preprint_ , arXiv:2303.11897.
  * Jeong and Ye (2024) Hyeonho Jeong and Jong Chul Ye. 2024.  Ground-a-video: Zero-shot grounded video editing using text-to-image diffusion models.  _Preprint_ , arXiv:2310.01107.
  * Kahatapitiya et al. (2024) Kumara Kahatapitiya, Adil Karjauv, Davide Abati, Fatih Porikli, Yuki M. Asano, and Amirhossein Habibian. 2024.  Object-centric diffusion for efficient video editing.  _Preprint_ , arXiv:2401.05735.
  * Kang et al. (2023) Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. 2023.  Scaling up gans for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10124–10134.
  * Kara et al. (2023) Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M. Rehg, and Pinar Yanardag. 2023.  Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models.  _Preprint_ , arXiv:2312.04524.
  * Karaev et al. (2023) Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. 2023.  Cotracker: It is better to track together.  _Preprint_ , arXiv:2307.07635.
  * Karras et al. (2022) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. 2022.  Elucidating the design space of diffusion-based generative models.  _Preprint_ , arXiv:2206.00364.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410.
  * Kasten et al. (2021) Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. 2021.  Layered neural atlases for consistent video editing.  _Preprint_ , arXiv:2109.11418.
  * Kondratyuk et al. (2023) Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. 2023.  Videopoet: A large language model for zero-shot video generation.  _arXiv preprint arXiv:2312.14125_.
  * Mackay and Pagani (1994) Wendy Mackay and Daniele Pagani. 1994.  Video mosaic: Laying out time in a physical space.  In _Proceedings of the second ACM international conference on Multimedia_ , pages 165–172.
  * Meiri et al. (2023) Barak Meiri, Dvir Samuel, Nir Darshan, Gal Chechik, Shai Avidan, and Rami Ben-Ari. 2023.  Fixed-point inversion for text-to-image diffusion models.  _Preprint_ , arXiv:2312.12540.
  * Menapace et al. (2024) Willi Menapace, Aliaksandr Siarohin, Ivan Skorokhodov, Ekaterina Deyneka, Tsai-Shien Chen, Anil Kag, Yuwei Fang, Aleksei Stoliar, Elisa Ricci, Jian Ren, et al. 2024.  Snap video: Scaled spatiotemporal transformers for text-to-video synthesis.  _arXiv preprint arXiv:2402.14797_.
  * Michel et al. (2024) Oscar Michel, Anand Bhattad, Eli VanderBilt, Ranjay Krishna, Aniruddha Kembhavi, and Tanmay Gupta. 2024.  Object 3dit: Language-guided 3d-aware image editing.  _Advances in Neural Information Processing Systems_ , 36.
  * Mokady et al. (2022) Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022.  Null-text inversion for editing real images using guided diffusion models.  _Preprint_ , arXiv:2211.09794.
  * Mullan et al. (2023) John Mullan, Duncan Crawbuck, and Aakash Sastry. 2023.  Hotshot-XL.
  * Qi et al. (2023) Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. 2023.  Fatezero: Fusing attentions for zero-shot text-based video editing.  _Preprint_ , arXiv:2303.09535.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  _Preprint_ , arXiv:2103.00020.
  * Ren et al. (2024) Yixuan Ren, Yang Zhou, Jimei Yang, Jing Shi, Difan Liu, Feng Liu, Mingi Kwon, and Abhinav Shrivastava. 2024.  Customize-a-video: One-shot motion customization of text-to-video diffusion models.  _Preprint_ , arXiv:2402.14780.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  _Preprint_ , arXiv:2112.10752.
  * Santosa et al. (2013) Stephanie Santosa, Fanny Chevalier, Ravin Balakrishnan, and Karan Singh. 2013.  Direct space-time trajectory control for visual media editing.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , pages 1149–1158.
  * Sauer et al. (2023) Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. 2023.  Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis.  _arXiv preprint arXiv:2301.09515_.
  * Shin et al. (2023) Chaehun Shin, Heeseung Kim, Che Hyun Lee, Sang gil Lee, and Sungroh Yoon. 2023.  Edit-a-video: Single video editing with object-aware consistency.  _Preprint_ , arXiv:2303.07945.
  * Singer et al. (2022) Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. 2022.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_.
  * Song et al. (2022) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2022.  Denoising diffusion implicit models.  _Preprint_ , arXiv:2010.02502.
  * Teed and Deng (2020) Zachary Teed and Jia Deng. 2020.  Raft: Recurrent all-pairs field transforms for optical flow.  _Preprint_ , arXiv:2003.12039.
  * Vincent (2011) Pascal Vincent. 2011.  A connection between score matching and denoising autoencoders.  _Neural computation_ , 23(7):1661–1674.
  * Wallace et al. (2022) Bram Wallace, Akash Gokul, and Nikhil Naik. 2022.  Edict: Exact diffusion inversion via coupled transformations.  _Preprint_ , arXiv:2211.12446.
  * Wang et al. (2023a) Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. 2023a.  Modelscope text-to-video technical report.  _Preprint_ , arXiv:2308.06571.
  * Wang et al. (2023b) Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. 2023b.  Videocomposer: Compositional video synthesis with motion controllability.  _Preprint_ , arXiv:2306.02018.
  * Xue et al. (2022) Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo. 2022.  Advancing high-resolution video-language representation with large-scale video transcriptions.  _Preprint_ , arXiv:2111.10337.
  * Yang et al. (2023) Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. 2023.  Rerender a video: Zero-shot text-guided video-to-video translation.  _Preprint_ , arXiv:2306.07954.
  * Yatim et al. (2023) Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. 2023.  Space-time diffusion features for zero-shot text-driven motion transfer.  _Preprint_ , arXiv:2311.17009.
  * Yenphraphai et al. (2024) Jiraphon Yenphraphai, Xichen Pan, Sainan Liu, Daniele Panozzo, and Saining Xie. 2024.  Image sculpting: Precise object editing with 3d geometry control.  _arXiv preprint arXiv:2401.01702_.
  * Yin et al. (2023) Wenjie Yin, Hang Yin, Kim Baraka, Danica Kragic, and Mårten Björkman. 2023.  Dance style transfer with cross-modal transformer.  _Preprint_ , arXiv:2208.09406.
  * Zhang et al. (2023a) David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. 2023a.  Show-1: Marrying pixel and latent diffusion models for text-to-video generation.  _Preprint_ , arXiv:2309.15818.
  * Zhang et al. (2023b) Guoqiang Zhang, J. P. Lewis, and W. Bastiaan Kleijn. 2023b.  Exact diffusion inversion via bi-directional integration approximation.  _Preprint_ , arXiv:2307.10829.
  * Zhang et al. (2023c) Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. 2023c.  Magicbrush: A manually annotated dataset for instruction-guided image editing.  _Preprint_ , arXiv:2306.10012.
  * Zhang et al. (2023d) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023d.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847.
  * Zuo et al. (2023) Zhichao Zuo, Zhao Zhang, Yan Luo, Yang Zhao, Haijun Zhang, Yi Yang, and Meng Wang. 2023.  Cut-and-paste: Subject-driven video editing with attention control.  _Preprint_ , arXiv:2311.11697.