# Edit Temporal-Consistent Videos with Image Diffusion Model

Yuanzhi Wang, Yong Li, Xiaoya Zhang, Xin Liu, Anbo Dai, Antoni B. Chan, Zhen
Cui  Yuanzhi Wang, Yong Li, Xiaoya Zhang, and Zhen Cui are with the PCA Lab,
Key Laboratory of Intelligent Perception and Systems for High-Dimensional
Information of Ministry of Education, School of Computer Science and
Engineering, Nanjing University of Science and Technology, Nanjing, 210094,
China. Xin Liu and Anbo Dai are with SeetaCloud, Nanjing, 210094, China.
Antoni B. Chan is with the Department of Computer Science, City University of
Hong Kong, Kowloon Tong, Hong Kong.

###### Abstract

Large-scale text-to-image (T2I) diffusion models have been extended for text-
guided video editing, yielding impressive zero-shot video editing performance.
Nonetheless, the generated videos usually show spatial irregularities and
temporal inconsistencies as the temporal characteristics of videos have not
been faithfully modeled. In this paper, we propose an elegant yet effective
Temporal-Consistent Video Editing (TCVE) method to mitigate the temporal
inconsistency challenge for robust text-guided video editing. In addition to
the utilization of a pretrained T2I 2D Unet for spatial content manipulation,
we establish a dedicated temporal Unet architecture to faithfully capture the
temporal coherence of the input video sequences. Furthermore, to establish
coherence and interrelation between the spatial-focused and temporal-focused
components, a cohesive spatial-temporal modeling unit is formulated. This unit
effectively interconnects the temporal Unet with the pretrained 2D Unet,
thereby enhancing the temporal consistency of the generated videos while
preserving the capacity for video content manipulation. Quantitative
experimental results and visualization results demonstrate that TCVE achieves
state-of-the-art performance in both video temporal consistency and video
editing capability, surpassing existing benchmarks in the field.

###### Index Terms:

Text-guided video editing, temporal Unet, spatial-temporal modeling, text-to-
image diffusion model.

##  I Introduction

Recently, diffusion-based generative models [1, 2, 3, 4] have shown remarkable
image [5, 6, 7, 8] and video [9, 10, 11, 12] generation capabilities via
diverse text prompts. It brings the large possibility to edit real-world
visual content by merely editing the text prompts.

Figure 1: Two frameworks for text-guided video editing. (a) shows the Tune-A-
Video [13] method. This pioneer work suffers from flickering artifacts and the
surfing stance of people is almost distorted. (b) illustrates our proposed
Temporal-Consistent Video Editing (TCVE) method. TCVE exploits a dedicated
temporal Unet to preserve the temporal consistency. As a comparison, TCVE
faithfully manipulates image content in accordance with the provided prompt
and shows encouraging temporal coherency.

Based on the publicly available large-scale pretrained text-to-image (T2I)
models, e.g., Stable Diffusion [5], researchers have developed various text-
guided diffusion-based image editing methods [14, 15]. To edit images, the
main idea is to leverage deterministic DDIM [16] for the image-to-noise
inversion, and then the inverted noise is gradually denoised to the edited
images under the condition of the edited prompt.

When it comes to text-guided video editing, a seemingly direct approach
involves an extension of the aforementioned paradigm to encompass video
content. Nevertheless, this paradigm is riddled with two formidable
challenges: firstly, the absence of readily accessible large-scale pretrained
text-to-video (T2V) diffusion models; and secondly, the typically resource-
intensive nature of training or refining T2V models for video editing
purposes. Consequently, an approach grounded in text-to-image (T2I) models
appears to hold greater potential value compared to one centered on video,
primarily owing to the plethora of open-source T2I models available within the
broader community.

Some researchers have exploited the pretrained T2I models for text-guided
video editing, e.g., Tune-A-Video [13] flattens the temporal dimensionality of
the source video and then manipulates the spatial content frame-by-frame using
the T2I model to generate the target video, as shown in Fig. 1 (a). In this
case, the additional temporal attention modules are incorporated into the T2I
model, while the source video and the corresponding prompt are used to train
these temporal attention modules to preserve the temporal consistency among
frames. Subsequently, Qi et al. [17] designed a fusing attention mechanism
based on Tune-A-Video to fuse the attention maps from the inversion and
generation process to preserve the motion and structure consistency. As
verified in Fig. 1 (a), these previous methods still suffer from flickering
artifacts and inconsistency among consecutive frames due to incomplete and
inconsistent temporal modeling. For the above video-editing paradigms, the
temporal attention modules are directly injected into each stage of the
spatial-only T2I Unet model for temporal modeling. This means that the input
of the temporal attention module is merely spatial-aware and temporal modeling
capability might not be reliable or faithfully.

In this paper, we aim to challenge the above limitations by proposing an
elegant yet effective Temporal-Consistent Video Editing (TCVE) method, as
shown in Fig. 1 (b). To model the temporal coherency, we construct a temporal
Unet model to facilitate temporal-focused modeling. In the temporal Unet
model, each residual block is built by stacked temporal convolutional layers.
The input video-based tensor is reshaped into a temporal-focused manner for
reliable temporal modeling. In particular, to bridge the temporal Unet and the
pretrained T2I 2D Unet, we establish a spatial-temporal modeling unit to
consolidate the temporal consistency while maintaining the video editing
capability. In contrast to prior work, TCVE can faithfully mitigate the
flickering artifacts between consecutive frames, as shown in the results of
Fig. 1 (b). In summary, the contributions of this work can be concluded as:

  * •

To mitigate the temporal inconsistency issue for reliable text-guided video
editing, we present a well-designed and efficient Temporal-Consistent Video
Editing (TCVE) method. TCVE strategically employs a dedicated temporal Unet
model to guarantee comprehensive and coherent temporal modeling.

  * •

To bridge the temporal Unet and pretrained T2I 2D Unet, we introduce a
cohesive spatial-temporal modeling unit. This unit is adept at capturing both
temporal and spatial information, thereby enhancing the temporal consistency
of the edited video while concurrently preserving the capacity for video
editing.

  * •

We perform extensive experiments on text-guided video editing datasets and
achieve superior or comparable results. Quantitative and visualization results
demonstrate that the flickering artifacts and temporal inconsistency are
effectively mitigated.

##  II Related Works

###  II-A Text-to-image Generation

Text-to-image (T2I) generation task aims to generate photorealistic images
that semantically match given text prompts [18, 19, 20, 21, 22, 23]. The main
idea of T2I generation is to utilize the current generative modeling paradigms
such as Generative Adversarial Networks (GANs) [24, 25], normalizing flows
[26, 27], diffusion models [1, 2], to construct a conditional generative model
conditioned on text embedding.

The pioneer work of T2I generation is AlignDRAW [18], which generated images
from natural language descriptions by applying sequential deep learning
techniques to conditional probabilistic models. Subsequently, Reed et al. [19]
proposed a Text-conditional GAN model that is the first end-to-end
differential architecture from the word-level to pixel-level. Further, some
researchers have developed several autoregressive methods to exploit large-
scale text-image data for T2I generation, such as DALL-E [21] and Parti [28].
Recently, due to the powerful capability of estimating data distribution and
the stable training process, diffusion-based generative models have achieved
unprecedented success in the T2I generation domain [29, 6, 5, 30, 7, 31]. For
example, Ramesh et al. [29] proposed the DALLE-2 that uses CLIP-based [32]
feature embedding to build a T2I diffusion model with improved text-image
alignments. Saharia et al. [6] designed robust cascaded diffusion models for
high-quality T2I generation. Rombach et al. [5] proposed a novel Latent
Diffusion Model (LDM) paradigm that projects the original image space into the
latent space of an autoencoder to improve training efficiency. Benefiting from
the excellent generation quality and efficiency of LDM, it has become the most
popular T2I generation model. Therefore, researchers have explored and
exploited LDM paradigm to develop various works [7, 31, 8, 33]. For instance,
Zhang et al. [7] proposed a ControlNet that appended additional conditions,
such as Canny edges, depth maps, human poses, to provide diverse generative
capabilities. Reco et al. [31] designed a region-controlled T2I diffusion
model based on pretrained Stable Diffusion to achieve controlled generation.
Considering the nice properties of LDM, we exploit it as a backbone model for
text-guided video editing.

###  II-B Text-to-video Generation

Despite major advances in T2I generation, text-to-video (T2V) generation is
still lagging behind due to the lack of large-scale text-video datasets and
the thousands of times harder to train compared to T2I diffusion models. To
achieve the T2V diffusion models, some researchers have attempted to propose
various methods [34, 35, 9, 11, 10, 36, 37, 38]. For instance, [34] proposed a
Video Diffusion Model (VDM) that is a naive extension of the standard image
diffusion models, and the original 2D Unet was replaced by space-only 3D Unet
to fit the video samples. Subsequently, Ho et al. [9] combined VDM with Imagen
[6] and designed an Imagen Video to generate high-definition videos. Blattmann
et al. [11] applied the LDM paradigm to high-resolution video generation,
called Video LDM. In addition to this, some works aim to utilize the
pretrained T2I diffusion models to conduct T2V generation, which can mitigate
the difficulty of training from scratch [12, 39, 40]. For example, Guo et al.
[12] proposed the Animatediff that inserts the motion (i.e., temporal) modules
into pretrained T2I 2D Unet to facilitate T2V generation. Although these T2V
methods are capable of generating high-quality videos, there are still many
issues that hinder the development of this field, such as large-scale
privatized training data, non-public well-trained models, and high training
costs, etc.

Figure 2: The framework of TCVE. Given a text-video pair as input, TCVE
leverages the pretrained 2D Unet from Stable Diffusion [5] and our proposed
temporal Unet for text-guided video editing. The input video is first diffused
into a noisy video $\mathbf{X}\in\mathbb{R}^{b\times c\times f\times h\times
w}$, where $b,c,f,h,w$ denote batch, channel, frame, height, and width
dimensionality, respectively. Then, $\mathbf{X}$ is reshaped into a spatial-
dominated tensor (i.e., $F_{\text{spa}}(\mathbf{X})\in\mathbb{R}^{(b\times
f)\times c\times h\times w}$) and a temporal-dominated tensor (i.e.,
$F_{\text{tem}}(\mathbf{X})\in\mathbb{R}^{(b\times h\times w)\times c\times
f}$) for subsequent spatial and temporal focused modeling. To bridge the
temporal Unet and pretrained 2D Unet, we establish a Spatial-Temporal modeling
Unit (STU) that adaptively fuses the spatial- and temporal-aware feature
($\mathbf{X}_{\text{spa}}$, $\mathbf{X}{{}_{\text{tem}}}$). During training,
we update the parameters of temporal Unet and STUs with the standard diffusion
training loss. For inference, we generate a new video from the source video
under the guidance of a modified prompt.

###  II-C Text-guided Video Editing

Recent text-guided diffusion-based image editing methods [41, 14, 15, 42, 43]
achieve promising image editing results. Despite the great success, text-
guided video editing is still lagging behind, as it faces the same
difficulties as the development of the T2V models. Some works attempt to
challenge this problem [44, 45, 46, 13, 17, 47]. For example, Text2Live [44]
and StableVideo [47] allowed some texture-based video editing with edited
prompts. However, Text2Live and StableVideo depend on the layered neural
atlases [48], thus the editing capabilities are often limited. Dreamix [49]
and Gen-1 [45] aimed to utilize VDM to conduct video editing, but training VDM
required large-scale datasets and tremendous computational resources.
Moreover, their training data and pretrained models are not publicly
available. Recently, some works have exploited the pretrained T2I diffusion
models to conduct efficient text-guided video editing with a single GPU device
[13, 17]. The first work is Tune-A-Video [13], which flattens the temporal
dimensionality of the source video and then edits it frame-by-frame using the
T2I diffusion model to generate the target video. Of these, the extra temporal
attention modules are incorporated into the T2I diffusion model to preserve
the temporal consistency among frames. FateZero [17] then improved the Tune-A-
Video by designing a fusing attention mechanism to preserve the motion and
structure. However, these previous methods still result in flickering
artifacts and inconsistency among frames due to incomplete and inconsistent
temporal modeling caused by alternating spatial and temporal modeling. Our
proposed TCVE has an essential difference from these prior works because we
design a temporal Unet as an independent temporal branch, which could
guarantee complete temporal awareness.

##  III Preliminaries

###  III-A Latent Diffusion Models

As one of the most popular diffusion-based generative paradigms, Latent
Diffusion Models (LDMs) [5] was proposed to diffuse and denoise the latent
space of an autoencoder to improve training efficiency. Specially, an encoder
$\mathcal{E}$ projects original image $x$ into a low-resolution latent state
$z=\mathcal{E}(x)$, and $z$ can be reconstructed back to the original image
$x\approx\mathcal{D}(z)$ by a decoder $\mathcal{D}$. Then, a denoising Unet
$\epsilon_{\theta}$ with cross-attention and self-attention [50] is trained to
denoise Gaussian noise into clean latent state $z$ using the following
objective:

$\mathcal{L}_{\text{LDM}}=\mathbb{E}_{z_{0},\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t\sim\mathcal{U}(1,T)}[\|\epsilon-\epsilon_{\theta}(z_{t},t,p)\|_{2}^{2}],$
(1)

where $p$ is the conditional text prompt embedding that is often extracted
from the CLIP text encoder [32]. $z_{t}$ is a diffused sample at timestep $t$,
$\mathcal{N}$ is a Gaussian distribution, and $\mathcal{U}$ is a Uniform
distribution.

###  III-B DDIM Sampler and Inversion

During inference, DDIM [16] sampler was employed to convert a Gaussian noise
$z_{T}$ to a clean latent state $z_{0}$ in a sequence of timestep
$t=T\rightarrow 1$ with the following iteration rule
$\text{DDIM}_{\text{smp}}:{z}_{t}\xrightarrow[]{\epsilon_{\theta}}{z}_{t-1}$,

$z_{t-1}=\sqrt{\alpha_{t-1}}\frac{z_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t-1}}\epsilon_{\theta},$
(2)

where $\alpha_{t}$ is a noise scheduling parameter defined by [1]. Next, the
DDIM inversion was proposed to project a clean latent state $z_{0}$ into a
noisy latent state $\hat{z}_{T}$ in a sequence of revered timestep
$t=1\rightarrow T$ with the following iteration rule
$\text{DDIM}_{\text{inv}}:\hat{z}_{t-1}\xrightarrow[]{\epsilon_{\theta}}\hat{z}_{t}$,

$\hat{z}_{t}=\sqrt{\alpha_{t}}\frac{\hat{z}_{t-1}-\sqrt{1-\alpha_{t-1}}\epsilon_{\theta}}{\sqrt{\alpha_{t-1}}}+\sqrt{1-\alpha_{t}}\epsilon_{\theta}.$
(3)

Intuitively, $\hat{z}_{T}$ can be denoised into a clean latent state
$\hat{z}_{0}=\text{DDIM}_{\text{smp}}(\hat{z}_{T},p)\approx z_{0}$ with a
classifier-free guidance whose scale factor set as 1. Current image editing
methods [51, 52] use a large classifier-free guidance scale factor ($\gg 1$)
to edit the latent with an edited prompt $p_{\text{edit}}$ as
$\hat{z}_{0}^{\text{edit}}=\text{DDIM}_{\text{smp}}(\hat{z}_{T},p_{\text{edit}})$,
and then use decoder $\mathcal{D}$ to map $\hat{z}_{0}^{\text{edit}}$ into an
edited image $x_{\text{edit}}$.

##  IV Method

###  IV-A Problem Formulation

Let $\mathcal{V}=(v_{1},v_{2},\cdots,v_{m})$ denotes a source video that
contains $m$ video frames. $p_{\text{sour}}$ and $p_{\text{edit}}$ denote the
the source prompt describing $\mathcal{V}$ and the edited target prompt,
respectively. The goal of text-guided video editing is to generate a new video
$\mathcal{V}_{\text{edit}}$ from source video $\mathcal{V}$ under the
condition of the edited prompt $p_{\text{edit}}$. For example, consider a
video and a source prompt “A man is surfing inside the barrel of a wave”, and
assume that the user wants to change the background of wave while preserving
the motion. The user can directly modify the source prompt such as “A man is
surfing on a wave made of aurora borealis”. Recent excellent works, e.g.,
Tune-A-Video [13], exploited the pretrained T2I diffusion models to conduct
video editing tasks. However, they mostly emphasize spatial content
generation, although the temporal attention modules are also used to
facilitate temporal awareness.

Our main idea is to build an independent temporal diffusion network through
using temporal convolutional layers to model the temporal information of
videos based on the T2I Unet model, as shown in the upper part in Fig. 2. In
addition to the utilization of a pretrained 2D Unet for spatial content
manipulation, we establish a dedicated temporal Unet to faithfully capture the
temporal coherence of the input video.

Concretely, the input video $\mathcal{V}$ is first encoded by $\mathcal{E}$
and inverted to the noise by DDIM inversion. Then, the inverted noise is
gradually denoised to the edited video frames through DDIM sampler under the
edited prompt $p_{\text{edit}}$ and decoded by $\mathcal{D}$. Among them, an
input video tensor is flattened into a spatial-dominated tensor via the
pretrained 2D Unet and a temporal-dominated tensor via the temporal UNet,
respectively. Then, the spatial-/temporal-dominated tensors are separately
injected into 2D and temporal Unet to enhance the spatial and temporal
awareness. Formally, the generation process of edited video frames is
abstractly defined as:

$\displaystyle\mathcal{V}_{\text{edit}}=\mathcal{D}\left[\\!\\!\\!\begin{array}[]{c}\text{DDIM}_{\text{smp}}(\text{DDIM}_{\text{inv}}(F_{\text{spa}}(\mathcal{E}(\mathcal{V})),\theta_{\text{spa}}),p_{\text{edit}},\theta_{\text{spa}})\\\
\text{DDIM}_{\text{smp}}(\text{DDIM}_{\text{inv}}(F_{\text{tem}}(\mathcal{E}(\mathcal{V})),\theta_{\text{tem}}),p_{\text{edit}},\theta_{\text{tem}})\end{array}\\!\\!\\!\right],$
(6)

where $F_{\text{spa}}$ and $F_{\text{tem}}$ denote the flattening operations
used to generate the spatial-/temporal-dominated tensors, as shown in Fig. 2.
$\theta_{\text{sps}}$ and $\theta_{\text{tem}}$ denote the parameters of T2I
2D Unet and temporal model, respectively.

###  IV-B Network Architecture

We now illustrate the proposed Temporal-Consistent Video Editing (TCVE)
network architecture, as shown in Fig. 2. The network architecture is mainly
composed of three parts: pretrained T2I 2D Unet, Temporal Unet, and Spatial-
Temporal modeling Unit. Below, we describe these modules in detail.

T2I 2D Unet. The common T2I diffusion model such as Stable Diffusion [5])
typically consists of a 2D spatial-only Unet model [53], which is a neural
network based on a spatial downsampling pass followed by an upsampling pass
with skip connections. In such 2D Unet architecture, several 2D convolutional
residual blocks and transformer blocks are stacked to encode the spatial
information. Each transformer block is mainly composed of a spatial self-
attention layer that leverages pixel locations to capture spatial dependency
and a cross-attention layer to capture correlations between embedded image
feature and embedded prompt feature. The latter cross-attention layer is the
core of condition generation, e.g., text prompt. Intuitively, the original 2D
Unet model cannot well encode continuous temporal variation information due to
the lack of dynamic sequence modeling. Hence, the generated videos from T2I
without expert sequence models would often result in flickering artifacts. To
suppress those artifacts effectively, we specifically design a temporal
diffusion model to compensate for the generated content information, which is
introduced in the next parts.

Figure 3: Visualization of style transfer. Tune-A-Video and FateZero may cause
some frame inconsistency, and StableVideo tends to disrupt the texture
information of foreground objects (i.e., horse and car). In contrast, our TCVE
generates temporally consistent videos while effectively transferring style.

Temporal Unet. As shown in Fig. 2, we design a temporal model to reliably
model the temporal consistency. To well-align the spatial T2I Unet, we also
choose the Unet architecture as the temporal model, whereas operating on
temporal axis with downsampling pass followed by an upsampling pass with skip
connections. Different from the 2D spatial Unet model, the temporal Unet is
composed of stacked temporal (i.e., 1D) convolutional residual blocks.
Considering an input video-based tensor $\mathbf{X}\in\mathbb{R}^{b\times
c\times f\times h\times w}$, where $b,c,f,h,w$ indicate batch size, channel
number, frame number, height, and width, respectively. The spatial dimensions
$h$ and $w$ will first be reshaped to the batch dimension, resulting in
$b\times h\times w$ sequences at the length of $f$, i.e.,
$F_{\text{tem}}(\mathbf{X})\in\mathbb{R}^{(b\times h\times w)\times c\times
f}$. Then, the reshaped tensor is injected into temporal convolutional
residual blocks for temporal axis downsampling and upsampling. Take a
downsampling stage as an example, the input tensor size is $(b\times h\times
w)\times c\times f$ and the output tensor size is $(b\times h\times w)\times
2c\times\frac{f}{2}$, and the upsampling is vice versa. Intuitively, the
temporal Unet can completely and consistently model temporal information due
to the temporal-aware input and output for each block.

Spatial-Temporal modeling Unit (STU). Another question is how to connect the
temporal Unet model and the 2D Unet model. To bridge the above two models, we
design a Spatial-Temporal modeling Unit (STU) to perceive both temporal and
spatial information. As shown in Fig. 2, the STU is mainly composed of a
temporal attention block and a 3D convolutional block. After performing the
spatial-/temporal-focused modeling, we are supposed to obtain the spatial-
aware feature $\mathbf{X}_{\text{spa}}$ and temporal-aware feature
$\mathbf{X}_{\text{tem}}$, respectively. The STU takes
$\mathbf{X}_{\text{spa}}$ and $\mathbf{X}_{\text{tem}}$ as input. In order to
facilitate subsequent feature fusion, $\mathbf{X}_{\text{tem}}$ is aligned to
the size of $\mathbf{X}_{\text{spa}}$ by resizing the same shape. Then, in
particular, a temporal attention block is used to enhance temporal awareness
of the resized $\mathbf{X}_{\text{tem}}$, formulated as:

$\mathbf{X}^{\text{att}}_{\text{tem}}=\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{softmax}(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}})\mathbf{V},$
(7)

where $\mathbf{Q}=\mathbf{W}_{q}\mathbf{X}_{\text{tem}}$,
$\mathbf{K}=\mathbf{W}_{k}\mathbf{X}_{\text{tem}}$, and
$\mathbf{V}=\mathbf{W}_{v}\mathbf{X}_{\text{tem}}$, and
$\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}$ are the learnable parameters.
This attention operation enables the module to capture the temporal
dependencies between features at the same spatial location across the temporal
axis. After that, $\mathbf{X}_{\text{spa}}$ and
$\mathbf{X}^{\text{att}}_{\text{tem}}$ are fused in a weighted manner:
$\mathbf{X}_{\text{fuse}}=\mathbf{X}_{\text{spa}}+\lambda\mathbf{X}^{\text{att}}_{\text{tem}}$,
where $\lambda=0.1$ is the balance factor. Finally, a 3D convolutional block
is utilized to conduct $\mathbf{X}_{\text{fuse}}$ for spatial-temporal
modeling due to its nice property for processing video-based context, thereby
improving the temporal consistency of the generated video while maintaining
the editing capability.

###  IV-C Training and Inference

The paradigms of training and inference are shown in Fig. 2. During the
training period, the source video and source prompt are used to train the
temporal Unet model and STUs with the original LDM objective in Eq. 1, and the
parameters of the pretrained T2I 2D Unet model (we use Stable Diffusion in
this work) are frozen and not trainable. Note that the training period does
not need to train for each edited prompt individually, thus our method is a
zero-shot video editing paradigm. During the inference period, we use the way
defined by Eq. 6 to edit the target video. Our experiments demonstrate such
training and inference strategy is effective in accurately delivering the
motion and structure from the source video to the edited videos.

##  V Experiments

###  V-A Implementation Details

Our TCVE is based on public pretrained Stable Diffusion
v1.4111https://huggingface.co/CompVis/stable-diffusion-v1-4. We conduct
experiments on several videos from the latest text-guided video editing
dataset LOVEU-TGVE-2023222https://github.com/showlab/loveu-tgve-2023 and the
video samples used in [47]. Each video has 4 different edited prompts for 4
applications: style transfer, object editing, background change, and multiple-
object editing. Style transfer aims to transfer videos into a variety of
styles. For example, we can transfer a real-world video into a vector art
style, as shown in Fig. 3. Object editing allows the users to edit the objects
of the video. As shown in Fig. 4, we can replace “Two gray sharks” with “Two
quadrotor drones”. Background change can enable users to change the video
background, i.e., the place where the object is, while preserving the
consistency in the movement of the object. For example, we can change the
background of the “shopping and entertainment center” to the “martian
landscape”, as shown in Fig. 5. Multiple-object editing aims to edit multiple
contents, e.g., perform object editing and background change, as shown in Fig.
6. We utilize Pytorch to implement all experiments with a RTX 3090 GPU. In the
training stage, the models are trained by Adam optimizer with learning rate
$3\times 10^{-5}$, and each video is fixed for 100 iterations. During
inference, we use DDIM inversion and DDIM sampler [16] with 50 steps and
classifier-free guidance with a guidance scale of 12.5 in our experiments.

Evaluation Metrics. We consider three evaluation metrics that are used by the
latest text-guided video editing dataset LOVEU-TGVE-2023 to measure the
quality of generated videos. Frame Consistency is to measure the temporal
consistency in frames by computing CLIP image embeddings on all frames of
output video and report the average cosine similarity between all pairs of
video frames. Textual Alignment is to measure textual faithfulness of edited
video by computing the average CLIP score between all frames of output video
and corresponding edited prompt. PickScore [54] is to measure human preference
for text-to-image generation models. We compute the average PickScore between
all frames of the output video and the corresponding edited prompt.

Figure 4: Qualitative comparison of object editing. Intuitively, our proposed
TCVE can faithfully alter the objects according to the edited prompts while
maintaining other video attributes.  Figure 5: Qualitative comparison of
background change. In contrast to these prior works, TCVE can faithfully
manipulate the background and preserve the smoothness between frames.

###  V-B Baseline Comparisons

We compare our method with the three latest baselines: 1) Tune-A-Video [13] is
a pioneer in efficient text-guided video editing using pretrained T2I
diffusion models. 2) FateZero [17] is an improved method with the fusing
attention mechanism based on Tune-A-Video. 3) StableVideo [47] is an atlas-
based method that exploits the pretrained T2I diffusion models to edit 2D
layered atlas images for text-guided video editing. Below, we analyze
quantitative and qualitative experiments.

TABLE I: Quantitative comparison with evaluated baselines.

Methods | Frame Consistency | Textual Alignment | PickScore
---|---|---|---
Tune-A-Video [13] | 91.99 | 26.52 | 20.29
FateZero [17] | 92.64 | 25.88 | 20.31
StableVideo [47] | 93.46 | 24.05 | 19.34
TCVE (Ours) | 94.44 | 27.58 | 20.50

Figure 6: Qualitative comparison of multiple-object editing. TCVE can generate
temporally-coherent videos, and both background and objects are well edited.

####  V-B1 Quantitative results

Tab. I lists the quantitative results of different methods. From these
results, we can observe that TCVE achieves the best video editing performance
under three evaluation metrics. In particular, TCVE gains considerable
performance improvements in the Frame Consistency and Textual Alignment. This
occurrence can be ascribed to the exhaustive and uniform modeling of spatial-
temporal information accomplished by TCVE. This accomplishment is facilitated
by the deployment of an dedicated temporal Unet as well as STU, thereby
significantly amplifying the temporal coherence of the generated video.
Further visualization analysis for the generated video will be provided in the
next part.

####  V-B2 Qualitative results

We provide some visual comparison of our TCVE against three baselines in four
editing tasks.

Style transfer. Fig. 3 shows the visual results of style transfer. All methods
seem to transfer the video style at first glance. However, the results of
three baselines suffer from frame inconsistency or texture distortions. For
Tune-A-Video w.r.t the left sample, the man disappears in the $4^{th}$ frame.
For the right sample, the car has been distorted. For FateZero w.r.t the left
sample, the object disappears in the $1^{th}$ frame and the horse color shows
inconsistency. For the right sample, two cars appear in the $4^{th}$ frame.
For StableVideo, the texture information of foreground objects (i.e., horse
and car) in two samples are severely distorted and disrupted, and the “flat
design, vector art” style is also unsatisfactorily transferred in the left
sample. In contrast, TCVE can produce temporally smooth videos while
successfully editing the video style.

Object editing. Fig. 4 exhibits the comparison of object editing task. Tune-A-
Video shows obvious limitations: the solitary drone is not consistent with the
goal of two drones (for the left sample) and the sheep orientation have not
been preserved (for the right sample). For the FateZero, evident flickering
and temporal inconsistency also exist, e.g., sudden distortion of the two
drones (for the left sample). StableVideo does not have the ability to edit
and deform foreground objects due to the intrinsic limitations of atlas-based
methods, i.e., restrictions on the foreground opacity values [48]. Compared
with them, TCVE faithfully alters the object according to the prompt while
maintaining other video attributes.

Background change. The visualization results of background change are shown in
Fig. 5, we can discover that Tune-A-Video, FateZero, and StableVideo fail to
change the “shopping and entertainment center” into “desolate martian
landspace” for the left sample. In contrast, TCVE consistently changes the
background according to the target prompt. For the right sample, TCVE adeptly
alters the background depicting the wave, whilst effectively preserving the
original surfing postures of the individuals.

Multiple-object editing. Besides the above single object editing, we also
explore the challenging multiple-object editing task, as shown in Fig. 6. From
these results, we observe that the all the methods except StableVideo can
successfully change the background for the two illustrated samples.
Nevertheless, Tune-A-Video and FateZero show evident shortcomings concerning
the coherence of foreground objects. For the left sample, the astronaut and
horse encounter a substantial reduction in visibility; For the right one, the
car shows conspicuous inconsistency across consecutive frames. In contrast,
our proposed TCVE demonstrates the ability to produce videos with enhanced
temporal coherence, showcasing proficient editing of both backgrounds and
objects.

###  V-C Ablation Studies

####  V-C1 Exploring effects of the key components in TCVE

We evaluate the effects of the key components in TCVE, including Temporal Unet
(TU) and the STU. The results are illustrated in Tab. II, please note that
TCVE w/o STU means removing the STU and directly fusing
$\mathbf{X}_{\text{spa}}$ and $\mathbf{X}_{\text{tem}}$ with a simple element-
wise summation operation. From these results, we can draw the following
conclusions: 1) TU is effective and brings considerable performance
improvements in frame consistency due to its promising property for temporal
modeling. 2) STU brings further benefits, which proves that bridging T2I 2D
Unet and TU using STU can further align the edited video with the target
prompt while enhancing temporal consistency.

TABLE II: Ablation studies of the key components in TCVE.

Methods | Frame Consistency | Textual Alignment | PickScore
---|---|---|---
TCVE w/o TU | 90.65 | 27.34 | 20.42
TCVE w/o STU | 92.74 | 26.98 | 20.38
TCVE | 94.44 | 27.58 | 20.50

Figure 7: Visualization results of ablation studies. TCVE w/o TU causes some
flickering artifacts and inconsistency due to the fact that the escalator is
deformed to varying degrees in different frames. TCVE w/o STU can generate
smooth video frames, but it fails to achieve the purpose of the targeted
editing. In contrast, TCVE could simultaneously maintain inter-frame
consistency and editing capabilities.

We provide the visualization results in Fig. 7 to further analyze the effects
of each component. For TCVE w/o TU, there exists some flickering artifacts and
inconsistency due to the fact that the escalator is deformed to varying
degrees in different frames. In contrast, TCVE preserves consistent structure
for the escalator among frames, which proves the effectiveness of TU in
modeling temporal information. For TCVE w/o STU, the generated video frames
are smooth. However, it fails to achieve the purpose of the targeted editing.
This phenomenon suggests that directly fusing temporal-aware and spatial-aware
features may lead to a compromise in the editing capabilities. In contrast,
TCVE demonstrates the ability to produce a seamlessly flowing video while
effectively conveying the intended editing objectives. This substantiates the
efficacy of STU in preserving both temporal coherence and the capacity for
video editing.

####  V-C2 Exploring effects of the key components in STU

Although the effectiveness of STU is verified in the previous part, the
effectiveness of key components in STU remains to be explored. Therefore, we
now evaluate the effects of the key components in STU, including temporal
attention (TA) and 3D convolution (3DConv). The results are reported in Tab.
III, we observe the following conclusions: 1) STU w/o TA degrades more frame
consistency due to the lack of temporal awareness. 2) STU w/o 3DConv reduces
more textual alignment due to the lack of spatio-temporal modeling. The above
results demonstrate the effectiveness of TA and 3DConv in maintaining frame
consistency and textual alignment, respectively.

TABLE III: Ablation studies of the key components in STU.

Methods | Frame Consistency | Textual Alignment | PickScore
---|---|---|---
STU w/o TA | 93.51 | 27.41 | 20.48
STU w/o 3DConv | 93.92 | 27.04 | 20.41
TCVE | 94.44 | 27.58 | 20.50

##  VI Conclusion

In this paper, we challenge the temporal inconsistency issue in text-guided
video editing by proposing a straightforward and effective Temporal-Consistent
Video Editing method. To model the temporal information, we construct a
temporal Unet model inspired by the pretrained T2I 2D Unet to facilitate
temporal-focused modeling. To bridge the temporal Unet and pretrained T2I 2D
Unet, we design a spatial-temporal modeling unit to perceive both temporal and
spatial information, thereby maintaining both the temporal consistency of
video and the desired editing capability. Quantitative and qualitative
experiments prove the validity of TCVE. A limitation is that our TCVE may
cause some unsatisfactory results when simultaneously manipulating style,
objects, and backgrounds. This could be attributed to the fact that the text
conditioning embedding stems from the CLIP text encoder, which aligns
predominantly with image-based embeddings and may not seamlessly correspond
with video samples. A potential solution is to use an additional video-based
CLIP model as text embedding. This avenue of research is left as future work.

## References

  * [1] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” _Advances in neural information processing systems_ , vol. 33, pp. 6840–6851, 2020.
  * [2] Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-based generative modeling through stochastic differential equations,” in _International Conference on Learning Representations_ , 2021\.
  * [3] Y. Wang, Y. Li, and Z. Cui, “Incomplete multimodality-diffused emotion recognition,” in _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * [4] S. Cao, W. Chai, S. Hao, Y. Zhang, H. Chen, and G. Wang, “Difffashion: Reference-based fashion design with structure-aware transfer by diffusion models,” _IEEE Transactions on Multimedia_ , pp. 1–13, 2023.
  * [5] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [6] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * [7] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 3836–3847.
  * [8] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 500–22 510.
  * [9] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet _et al._ , “Imagen video: High definition video generation with diffusion models,” _arXiv preprint arXiv:2210.02303_ , 2022.
  * [10] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni _et al._ , “Make-a-video: Text-to-video generation without text-video data,” in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [11] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis, “Align your latents: High-resolution video synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 563–22 575.
  * [12] Y. Guo, C. Yang, A. Rao, Y. Wang, Y. Qiao, D. Lin, and B. Dai, “Animatediff: Animate your personalized text-to-image diffusion models without specific tuning,” _arXiv preprint arXiv:2307.04725_ , 2023.
  * [13] J. Z. Wu, Y. Ge, X. Wang, W. Lei, Y. Gu, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * [14] O. Avrahami, D. Lischinski, and O. Fried, “Blended diffusion for text-driven editing of natural images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 208–18 218.
  * [15] N. Tumanyan, M. Geyer, S. Bagon, and T. Dekel, “Plug-and-play diffusion features for text-driven image-to-image translation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 1921–1930.
  * [16] J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in _International Conference on Learning Representations_ , 2021.
  * [17] C. Qi, X. Cun, Y. Zhang, C. Lei, X. Wang, Y. Shan, and Q. Chen, “Fatezero: Fusing attentions for zero-shot text-based video editing,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * [18] E. Mansimov, E. Parisotto, J. L. Ba, and R. Salakhutdinov, “Generating images from captions with attention,” in _International Conference on Learning Representations_ , 2016.
  * [19] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee, “Generative adversarial text to image synthesis,” in _International conference on machine learning_. PMLR, 2016, pp. 1060–1069.
  * [20] B. Li, X. Qi, T. Lukasiewicz, and P. Torr, “Controllable text-to-image generation,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [21] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 8821–8831.
  * [22] R. Li, N. Wang, F. Feng, G. Zhang, and X. Wang, “Exploring global and local linguistic representations for text-to-image synthesis,” _IEEE Transactions on Multimedia_ , vol. 22, no. 12, pp. 3075–3087, 2020.
  * [23] Q. Cheng, K. Wen, and X. Gu, “Vision-language matching for text-to-image synthesis via generative adversarial networks,” _IEEE Transactions on Multimedia_ , 2022.
  * [24] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [25] A. Creswell, T. White, V. Dumoulin, K. Arulkumaran, B. Sengupta, and A. A. Bharath, “Generative adversarial networks: An overview,” _IEEE signal processing magazine_ , vol. 35, no. 1, pp. 53–65, 2018.
  * [26] D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [27] Y. Wang, Z. Cui, and Y. Li, “Distribution-consistent modal recovering for incomplete multimodal learning,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 22 025–22 034.
  * [28] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan _et al._ , “Scaling autoregressive models for content-rich text-to-image generation,” _arXiv preprint arXiv:2206.10789_ , vol. 2, no. 3, p. 5, 2022.
  * [29] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, “Hierarchical text-conditional image generation with clip latents,” _arXiv preprint arXiv:2204.06125_ , 2022.
  * [30] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen, “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 16 784–16 804.
  * [31] Z. Yang, J. Wang, Z. Gan, L. Li, K. Lin, C. Wu, N. Duan, Z. Liu, C. Liu, M. Zeng _et al._ , “Reco: Region-controlled text-to-image generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 14 246–14 255.
  * [32] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [33] O. Avrahami, T. Hayes, O. Gafni, S. Gupta, Y. Taigman, D. Parikh, D. Lischinski, O. Fried, and X. Yin, “Spatext: Spatio-textual representation for controllable image generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 370–18 380.
  * [34] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” _Advances in Neural Information Processing Systems_ , 2022\.
  * [35] D. Zhou, W. Wang, H. Yan, W. Lv, Y. Zhu, and J. Feng, “Magicvideo: Efficient video generation with latent diffusion models,” _arXiv preprint arXiv:2211.11018_ , 2022.
  * [36] Z. Qing, S. Zhang, J. Wang, X. Wang, Y. Wei, Y. Zhang, C. Gao, and N. Sang, “Hierarchical spatio-temporal decoupling for text-to-video generation,” _arXiv preprint arXiv:2312.04483_ , 2023.
  * [37] J. Wang, H. Yuan, D. Chen, Y. Zhang, X. Wang, and S. Zhang, “Modelscope text-to-video technical report,” _arXiv preprint arXiv:2308.06571_ , 2023\.
  * [38] S. Ge, S. Nah, G. Liu, T. Poon, A. Tao, B. Catanzaro, D. Jacobs, J.-B. Huang, M.-Y. Liu, and Y. Balaji, “Preserve your own correlation: A noise prior for video diffusion models,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 22 930–22 941.
  * [39] L. Khachatryan, A. Movsisyan, V. Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi, “Text2video-zero: Text-to-image diffusion models are zero-shot video generators,” _arXiv preprint arXiv:2303.13439_ , 2023\.
  * [40] B. Liu, X. Liu, A. Dai, Z. Zeng, Z. Cui, and J. Yang, “Dual-stream diffusion net for text-to-video generation,” _arXiv preprint arXiv:2308.08316_ , 2023\.
  * [41] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon, “Sdedit: Guided image synthesis and editing with stochastic differential equations,” in _International Conference on Learning Representations_ , 2021.
  * [42] G. Couairon, J. Verbeek, H. Schwenk, and M. Cord, “Diffedit: Diffusion-based semantic image editing with mask guidance,” in _The Eleventh International Conference on Learning Representations_ , 2023.
  * [43] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang, T. Dekel, I. Mosseri, and M. Irani, “Imagic: Text-based real image editing with diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 6007–6017.
  * [44] O. Bar-Tal, D. Ofri-Amar, R. Fridman, Y. Kasten, and T. Dekel, “Text2live: Text-driven layered image and video editing,” in _European Conference on Computer Vision_. Springer, 2022, pp. 707–723.
  * [45] P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis, “Structure and content-guided video synthesis with diffusion models,” _arXiv preprint arXiv:2302.03011_ , 2023.
  * [46] S. Loeschcke, S. Belongie, and S. Benaim, “Text-driven stylization of video objects,” in _European Conference on Computer Vision_. Springer, 2022, pp. 594–609.
  * [47] W. Chai, X. Guo, G. Wang, and Y. Lu, “Stablevideo: Text-driven consistency-aware diffusion video editing,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 23 040–23 050.
  * [48] Y. Kasten, D. Ofri, O. Wang, and T. Dekel, “Layered neural atlases for consistent video editing,” _ACM Transactions on Graphics (TOG)_ , vol. 40, no. 6, pp. 1–12, 2021.
  * [49] E. Molad, E. Horwitz, D. Valevski, A. R. Acha, Y. Matias, Y. Pritch, Y. Leviathan, and Y. Hoshen, “Dreamix: Video diffusion models are general video editors,” _arXiv preprint arXiv:2302.01329_ , 2023.
  * [50] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [51] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” _arXiv preprint arXiv:2208.01626_ , 2022.
  * [52] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or, “Null-text inversion for editing real images using guided diffusion models,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 6038–6047.
  * [53] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer, 2015, pp. 234–241.
  * [54] Y. Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy, “Pick-a-pic: An open dataset of user preferences for text-to-image generation,” _arXiv preprint arXiv:2305.01569_ , 2023.