# ZeroSmooth: Training-free Diffuser Adaptation for High Frame Rate Video
Generation

Shaoshu Yang  School of Artificial Intelligence, University of Chinese Academy
of Sciences  New Laboratory of Pattern Recognition (NLPR), Institute of
Automation, Chinese Academy of Sciences  Yong Zhang  Tencent AI Lab  Xiaodong
Cun  Tencent AI Lab  Ying Shan  Tencent AI Lab  Ran He  School of Artificial
Intelligence, University of Chinese Academy of Sciences  New Laboratory of
Pattern Recognition (NLPR), Institute of Automation, Chinese Academy of
Sciences

###### Abstract

Video generation has made remarkable progress in recent years, especially
since the advent of the video diffusion models. Many video generation models
can produce plausible synthetic videos, e.g., Stable Video Diffusion (SVD).
However, most video models can only generate low frame rate videos due to the
limited GPU memory as well as the difficulty of modeling a large set of
frames. The training videos are always uniformly sampled at a specified
interval for temporal compression. Previous methods promote the frame rate by
either training a video interpolation model in pixel space as a postprocessing
stage or training an interpolation model in latent space for a specific base
video model. In this paper, we propose a training-free video interpolation
method for generative video diffusion models, which is generalizable to
different models in a plug-and-play manner. We investigate the non-linearity
in the feature space of video diffusion models and transform a video model
into a self-cascaded video diffusion model with incorporating the designed
hidden state correction modules. The self-cascaded architecture and the
correction module are proposed to retain the temporal consistency between key
frames and the interpolated frames. Extensive evaluations are preformed on
multiple popular video models to demonstrate the effectiveness of the propose
method, especially that our training-free method is even comparable to trained
interpolation models supported by huge compute resources and large-scale
datasets.

††footnotetext:  ${{}^{~{}\textrm{{\char 0\relax}}}}$Corresponding author

figure Our method enables pretrained video diffusion models for high frame
rate (4$\times$ more than during training) generation without extra training
data and parameter updates.

##  1 Introduction

Video generation has undergone rapid evolution recently, benefiting from the
development of diffusion models [15, 29]. There are numerous popular video
diffusion models [17] that can generate plausible synthetic videos. They are
trained on large-scale datasets, including videos and images, using extensive
compute resources. Several models are trained in the pixel space, e.g., Imagen
Video [14] and Make-A-Video [27]. The majority are trained in the latent
space, e.g., Stable Video Diffusion (SVD) [2], Align-your-latent [3],
VideoCrafter [5], LaVie [34], ModelScope [33], etc.

Although those video models can produce realistic videos, most of them cannot
generate high frame rate videos with smooth transitions. One reason is that
most video generation models use uniformly sampled videos as training data to
avoid exceeding the GPU memory limit; i.e., models are trained to generate key
frames. The other reason is that capturing the distribution of a long video is
challenging. In this work, we focus on improving the frame rate of those
generative video models.

Conventional video frame interpolation methods can be grouped into two
categories: i.e., flow-based [8, 20, 26, 37] and kernel-based methods [21,
41]. The former follows a two-stage paradigm that first estimates flows
between neighboring frames and then applies forward or backward warping to the
pixels or latent features for intermediate frame generation. The performance
relies on the accuracy of the estimated flow, which is still a quite
challenging task. The latter considers pixel synthesis for the interpolated
frame as local convolution over input frames and uses a network to predict a
convolution kernel for each pixel. However, it has difficulty handling large
spatial displacements.

Recently, several methods have applied diffusion models for video
interpolation, e.g., LDMVFI [9], MCVD [31], VIDIM [18], and CBBD [23]. These
methods treat video interpolation as a conditional generation or temporal
inpainting task. Given key frames, the diffusion models are trained to
generate intermediate frames through the denoising process. In some video
generation methods, video interpolation is integrated into the video model as
a module, e.g., ImagenVideo, Make-A-Video, LaVie, and Show-1. The
interpolation module is deeply integrated with other modules, which makes it
difficult to incorporate the interpolation module into other video models.

Although conventional methods, diffusion-based models, and bounded
interpolation modules can generate intermediate frames for constructing a
smooth video, all these methods require a training process involving millions
or even billions of parameters. Furthermore, since conventional and diffusion-
based models are always trained independently on real data, a domain gap
exists when applying them to synthetic videos produced by generative video
models. As for bounded interpolation modules, they need to be retrained if the
base video model is updated, and they are not generalizable across different
video models.

In this work, we propose a training-free method to improve the frame rate of
various existing generative video models, e.g., SVD [2] and VideoCrafter [4].
The proposed method can be applied in a plug-and-play manner across various
video models. We investigate the temporal correlation of features learned by
the video UNet and observe that latent space back-projection fails to inject
content and appearance to adjacent frames. Fortunately, we find the temporal
correlation learned in transformer hidden states is strong. Back-projection in
transformer hidden states achieves strong visual content control while
preserving great inter-frame consistency. Based on this observation, we
transform the target video model into a self-cascaded architecture containing
two branches (see Fig. 1). One branch retains the architecture of the target
model for short video inference, while the other is adapted for long video
inference by placing the proposed hidden state correction modules into the
transformer blocks. The correction modules use both hidden states from the two
branches as input to calibrate the hidden states of the long branch for inter-
frame consistency. Additionally, we design a strategy for the controllability
of the correction strength.

Our contributions are summarized as follows:

  * •

We propose a training-free method to enhance generative video models to
produce videos with a higher frame rate, resulting in visually smooth videos.
The method can be applied to various video models in a plug-and-play manner.

  * •

We propose the self-cascaded architecture and the hidden state correction
modules for inter-frame consistency.

  * •

We conduct extensive experiments with various popular generative video models
to demonstrate the effectiveness of the proposed method, inclusing SVD,
VideoCrafter, and LaVie.

##  2 Related Works

#### Video Diffusion Models

Diffusion models now prevail in the video generation community. They model the
video data distribution with a sequence of iterative denoising steps [15, 28,
29]. Diffusion models are used for a variety of video tasks, including action
category conditioned video generation [31, 38, 22, 11], text conditioned video
generation [14, 3, 13], image conditioned video generation [2, 40], and video
translation [36, 39]. Among them, VideoCrafter[4, 5], LaVie[34], and
StableVideoDiffusion[2] approximate the data in a compact video latent space
computed by a VAE[19]. Meanwhile, Imagen-Video [14], Make-A-Video [27], and
PYoCo [11] learn the pixel space video distribution. Some efforts have been
made in long video generation [32, 25]. Gen-L-Video [32] bootstraps video
diffusers to generate beyond the video length limit. FreeNoise [25] proposes a
noise schedule to alleviate content shifting.

#### Zero-shot Visual Restoration and Video Interpolation

With the power of modern pre-trained diffusion models, zero-shot visual
restoration has made considerable progress [10, 35, 7, 42]. ILVR [6] adopts a
trained diffuser for zero-shot image super-resolution. DDNM [35] proposes
theoretical insights into null-space and range-space decomposition of visual
restoration. DDPG [10] combines the back-projection method and least square
method using an optimization preconditioner. However, limited efforts have
been made to deliver zero-shot visual restoration to the video domain.
ScaleCrafter [12] proposes a tuning-free method for inferring at a higher
resolution. In this paper, we investigate the potential of zero-shot methods
for higher frame rate generation.

Video interpolation is a long-standing problem in computer vision. Traditional
video interpolation models adopt a two-stage paradigm to estimate the optical
flow and use it to aid frame interpolation [8, 20, 37, 26]. Recently, progress
has been made in deploying new architectures [21, 41] and training paradigms
[9, 18, 23]. VFIformer [21] adopts a transformer for frame interpolation.
LDMVFI[9] and VIVDM[18] propose training diffusion models for the task.

##  3 Preliminary

###  3.1 Video Diffusion Models

Given a video $\mathbf{x}_{0}\sim p(\mathbf{x}_{0})$ with $S$ frames, let
$\mathbf{x}_{0}=\\{\boldsymbol{x}_{0}^{0},\boldsymbol{x}_{0}^{1},\cdots,\boldsymbol{x}_{0}^{S}\\}$.
Generative models approximate a sequence of diffusion denoising transitions
[17, 15, 24]

$\displaystyle q(\mathbf{z}_{t}|\mathbf{x}_{0})$
$\displaystyle=\mathcal{N}(\mathbf{z}_{t}|\alpha_{t}\mathbf{x}_{0},\sigma_{t}^{2}\mathbf{I})$
(1) $\displaystyle q(\mathbf{z}_{s}|\mathbf{z}_{t},\mathbf{x}_{0})$
$\displaystyle=\mathcal{N}(\mathbf{z}_{s}|\bm{\mu}_{s|t}(\mathbf{z}_{t},\mathbf{x}_{0}),\sigma_{s|t}^{2}\mathbf{I}),$
(2)

where $\mathbf{z}_{t}$ is the noisy data at a noise scale $t=1,2,\cdots,T$ and
$0\leq s<t\leq T$. Let $\lambda_{t}=\log(\alpha_{t}^{2}/\sigma_{t}^{2})$ be
the monotonically decreasing signal-to-noise ratio along noise scale $t$. The
mean and variance of backward diffusion process
$q(\mathbf{z}_{s}|\mathbf{z}_{t},\mathbf{x}_{0})$ is

$\displaystyle\bm{\mu}_{s|t}(\mathbf{z}_{t},\mathbf{x}_{0})$
$\displaystyle=e^{\lambda_{t}-\lambda_{s}}(\frac{\alpha_{s}}{\alpha_{t}})\mathbf{z}_{t}+(1-e^{\lambda_{t}-\lambda_{s}})\alpha_{s}\mathbf{x}_{0}$
(3) $\displaystyle\sigma_{s|t}^{2}$
$\displaystyle=(1-e^{\lambda_{t}-\lambda_{s}})\sigma_{t}^{2}.$ (4)

Starting from Gaussian noise
$\mathbf{z}_{T}\sim\mathcal{N}(\mathbf{z}_{T}|\bm{0},\mathbf{I})$, we train a
video diffusion UNet to denoise $\mathbf{z}_{T}$ iteratively. Specifically, it
estimates
$p_{\theta}(\mathbf{z}_{s}|\mathbf{z}_{t},y)=\mathcal{N}(\mathbf{z}_{s}|\bm{\mu}_{\theta}^{s|t}(\mathbf{z}_{t},y),\bm{\Sigma}_{\theta}^{s|t}(\mathbf{z}_{t}),y)$,
which approximates the reverse diffusion process with condition $y$, i.e.
textual prompts. The diffusion UNet $\boldsymbol{\epsilon}_{\theta}(\cdot)$ is
parameterized to estimate the added noise in $\mathbf{x}_{t}$. One can
rearrange the $\boldsymbol{\epsilon}_{\theta}(\cdot)$ to compute the noiseless
prediction $\mathbf{x}_{0|t}=\mathbf{f}_{\theta}(\mathbf{z}_{t},y)$ that
approximates $\mathbf{x}_{0}$ [28].

###  3.2 Visual Restoration with Diffusion Models

The back-projection method [35, 10] is prevalent for zero-shot image
restoration. It considers a linear measurement operator $\mathbf{A}$, such
that the degenerated sample is $\mathbf{y}=\mathbf{A}\mathbf{x}_{0}$.
Meanwhile, the high-resolution image can be decomposed into two parts

$\mathbf{x}_{0}=\underbrace{\mathbf{A}^{\dagger}\mathbf{A}\mathbf{x}_{0}}_{\mathrm{rangespace}}+\underbrace{(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{x}_{0}}_{\mathrm{nullspace}}$
(5)

where $\mathbf{A}^{\dagger}\mathbf{A}\mathbf{x}_{0}$ is in the range space of
$\mathbf{A}$, $(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{x}_{0}$ is
in the null space of $\mathbf{A}$. $\mathbf{A}^{\dagger}$ is the pseudo
inverse of $\mathbf{A}$ such that
$\mathbf{A}\mathbf{A}^{\dagger}\mathbf{A}=\mathbf{A}$. DDNM [35] proposes to
back-project the observed part of a sample $\mathbf{y}$ to replace the range
space part in denoising process

$\hat{\mathbf{x}}_{0|t}=(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{x}_{0|t}+\mathbf{A}^{\dagger}\mathbf{y}$
(6)

where $\hat{\mathbf{x}}_{0|t}$ is the corrected estimate. Therefore, the
diffusion model only predicts in the null space of $\mathbf{A}$ and ensures
the consistency to condition $\mathbf{y}$ in the output.

##  4 Method

Figure 1:  An overview of our method. (a) We build cascaded video diffusion
model by adapting the base generator to generate at a higher frame rate. (b) A
sketch for hidden states correction in transformers in ZeroSmooth.

We aim to adapt a video diffusion model to generate at a higher frame rate
while preserving its original generation capability. Given a pretrained video
diffusion model $\epsilon_{\theta}(\cdot):\mathbb{R}^{c\times t_{0}\times
h\times w}\rightarrow\mathbb{R}^{c\times t_{0}\times h\times w}$, which is
trained on $t_{0}$ frame videos. Our method creates
$\hat{\epsilon}_{\theta}(\cdot):\mathbb{R}^{c\times t\times h\times
w}\rightarrow\mathbb{R}^{c\times t\times h\times w}$ from
$\epsilon_{\theta}(\cdot)$ in a training-free manner, where $t>t_{0}$. Once
the key frames $\mathbf{y}$ are observed, generating the corresponding high
frame rate video can be considered video frame interpolation (VFI). Meanwhile,
the measurement operator $\mathbf{A}^{t_{0}\times t}$, which acquires key
frames from a full video $\mathbf{x}_{0}$ is linear and follows the
formulation of $\mathbf{y}=\mathbf{A}\mathbf{x}_{0}$. An intuitive idea to
achieve this is to apply DDNM [35] to the $t_{0}$-frame base video generated
by $\epsilon_{\theta}(\cdot)$, interpolating between the key frames and get
the $t$-frame high frame rate video. During this process, one can build a
$t$-frame generator $\hat{\epsilon}_{\theta}(\cdot)$ using the method proposed
by Qiu et al. [25] After each denoising step, the key frames are back-
projected to correct the estimate.

However, as seen in Fig. 3, we find it challenging for this intuitive approach
to maintain temporal consistency between observed and interpolated frames. We
attribute this to the weak temporal correlation of the video latent space
learned by diffusers. Correcting the estimate in the latent space can hardly
inject content and appearance into the interpolated frames. Alternatively, we
propose to correct the hidden states of transformer modules where the temporal
correlation is strong. Our method is summarized in three parts. In the
following subsections, we introduce the overall architecture of our method,
the hidden state correction method in transformers, and our remedy to
alleviate joint distribution mismatch after correction.

###  4.1 Self-cascaded Video Diffusion Model

As shown in Fig. 1(a), we construct a cascaded video diffusion model
consisting of $\epsilon_{\theta}(\cdot)$ and $\hat{\epsilon}_{\theta}(\cdot)$.
In our method, $\epsilon_{\theta}(\cdot)$ and $\hat{\epsilon}_{\theta}(\cdot)$
share the same neural network modules and parameters. Therefore, we call this
paradigm a _self-cascaded video diffusion model_. In order to correct the high
frame rate estimate, we first generate key frame transformer hidden states.
The hidden state distribution in a diffusion model varies along denoising
timesteps. Instead of using only a final key frame video for back-projection
like DDNM, our method uses the key frame hidden states at the corresponding
step to match the data distribution at different timesteps.

Mathematically, we denote the input to spatial or temporal transformer as
$\mathbf{h}\in\mathbb{R}^{t\times l\times d}$ in
$\hat{\boldsymbol{\epsilon}}_{\theta}(\cdot)$. $l$ is the stacked spatial
dimension (i.e., feature width times feature height), and $d$ is the hidden
state channel. The hidden state input for the corresponding transformer in
$\boldsymbol{\epsilon}_{\theta}(\cdot)$ is
$\mathbf{h}^{k}\in\mathbb{R}^{t_{0}\times l\times d}$. As shown in Fig. 1(b),
we apply back-projection to correct the hidden state estimation with key frame
hidden states $\mathbf{h}^{k}$ before computing attention. We denote the
query, key and value with $\mathbf{Q}\in\mathbb{R}^{t\times l\times
d},\mathbf{K}\in\mathbb{R}^{t\times l\times
d},\mathbf{V}\in\mathbb{R}^{t\times l\times d}$. After hidden state
correction, we use $\hat{\mathbf{Q}},\hat{\mathbf{K}}$ and $\hat{\mathbf{V}}$
to represent them. Note that the self-cascaded video diffusion model can
include multiple stages to achieve higher frame rate results. For instance,
consider $t=nt_{0}$, where $n$ is the video interpolation scale. One can
construct another model $\tilde{\epsilon}_{\theta}(\cdot):\mathbb{R}^{c\times
n^{2}t_{0}\times h\times w}\rightarrow\mathbb{R}^{c\times n^{2}t_{0}\times
h\times w}$ from $\epsilon_{\theta}(\cdot)$. The self-cascaded model
$\\{\epsilon_{\theta},\hat{\epsilon}_{\theta},\tilde{\epsilon}_{\theta}\\}$
achieves $n^{2}$ times more frames than the key frame generator.

#### Temporal Attention for High Frame Rate Generation

A pretrained video diffusion model is trained on videos of length $t_{0}$.
Therefore, $t$-frame video is unseen for the temporal transformer modules.
Direct inference without adaptation will result in severe sample quality
degradation. We propose _ZeroSmooth temporal attention_ to adapt the modules
for a longer sequence. Video diffusers use different categories of temporal
attention. Text-to-video diffusion models, like LaVie[34] and VideoCrafter1
[4], use relative positional embedding (RPE). The sequence length should be
kept unchanged when processing high frame rate input. We improve upon the
attention fusion technique in FreeNoise [25] by adding RPE to every attention
window, ensuring correct relative time perception. Image-to-video diffusion
models, like StableVideoDiffusion [2], use absolute positional embedding (APE)
to provide temporal distance between the current frame and the reference
image. Therefore, we interpolate the position index to maintain the temporal
distance in the high frame rate video.

###  4.2 Hidden State Correction for Transformers

We adopt back-projection method in transformer modules of video diffusion to
correct hidden states estimate. Without loss of generality, we consider a
$2\times$ frame interpolation in this subsection. The measurement for video
frame interpolation can be defined in two ways. First, we can use sampling
operator $\mathbf{A}\in\mathbb{R}^{t_{0}\times 2t_{0}}$ to draw key frames
from a full video. Second, one can use the interpolation operator
$\\{\mathbf{A}_{1},\mathbf{A}_{2}\\}$ to perform temporal downsampling to get
key frames, in which the align corners are different in $\mathbf{A}_{1}$ and
$\mathbf{A}_{2}$. The measurement matrices are shown below for the $2\times$
interpolation case:

$\displaystyle\mathbf{A}=\left[\begin{matrix}1&0&0&0&\cdots&0&0\\\
0&0&1&0&\cdots&0&0\\\ \vdots&\vdots&\vdots&\vdots&&\vdots&\vdots\\\
0&0&0&0&\cdots&1&0\end{matrix}\right]\qquad\qquad\qquad\qquad\qquad\quad$ (7)
$\displaystyle\mathbf{A}_{1}=\left[\begin{matrix}0.5&0.5&0&0&\cdots&0&0\\\
0&0&0.5&0.5&\cdots&0&0\\\ \vdots&\vdots&\vdots&\vdots&&\vdots&\vdots\\\
0&0&0&0&\cdots&0.5&0.5\end{matrix}\right],\mathbf{A}_{2}=\left[\begin{matrix}1&0&0&0&\cdots&0&0\\\
0&0.5&0.5&0&\cdots&0&0\\\ \vdots&\vdots&\vdots&\vdots&&\vdots&\vdots\\\
0&0&0&0&\cdots&0.5&0\end{matrix}\right].$ (8)

Intuitively, sampling operator $\mathbf{A}$ is less likely to cause blur in
key frames since it directly copies corresponding frames. However, we find
$\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ especially useful for the key and value
in spatial transformer modules. Within these modules, the query determines the
scene structure of a frame while the key and value provide textures and
appearances. As a result, using interpolation operator for back-projection in
the spatial transformer key and value offers additional temporal consistency
in visual details without causing blurry scene structures. Examples of hidden
state correction is shown in Fig. 2(a). Mathmatically, the correction for
temporal transformer hidden states is

$\hat{\mathbf{h}}^{\mathrm{temp}}=(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{h}^{\mathrm{temp}}+\mathbf{A}^{\dagger}\mathbf{h}^{k},$
(9)

where $\mathbf{h}^{\mathrm{temp}}$ and $\hat{\mathbf{h}}^{\mathrm{temp}}$ are
the temporal hidden states and the corrected ones. In spatial transformers,
the correction is

$\displaystyle\qquad\qquad\qquad\qquad\qquad\hat{\mathbf{Q}}^{\mathrm{spatial}}=(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{Q}^{\mathrm{spatial}}+\mathbf{A}^{\dagger}\mathbf{W}_{q}\mathbf{h}^{k}$
(10)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\hat{\mathbf{K}}^{\mathrm{spatial}}=\mathbbm{1}_{p>0.5}\hat{\mathbf{K}}^{\mathrm{spatial}}_{1}+(1-\mathbbm{1}_{p>0.5})\hat{\mathbf{K}}^{\mathrm{spatial}}_{2}$
(11)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\hat{\mathbf{V}}^{\mathrm{spatial}}=\mathbbm{1}_{p>0.5}\hat{\mathbf{V}}^{\mathrm{spatial}}_{1}+(1-\mathbbm{1}_{p>0.5})\hat{\mathbf{V}}^{\mathrm{spatial}}_{2}$
(12)
$\displaystyle\qquad\hat{\mathbf{K}}^{\mathrm{spatial}}_{1}=(\mathbf{I}-\mathbf{A}_{1}^{\dagger}\mathbf{A}_{1})\mathbf{K}^{\mathrm{spatial}}+\mathbf{A}_{1}^{\dagger}\mathbf{W}_{k}\mathbf{h}^{k},\hat{\mathbf{K}}^{\mathrm{spatial}}_{2}=(\mathbf{I}-\mathbf{A}_{2}^{\dagger}\mathbf{A}_{2})\mathbf{K}^{\mathrm{spatial}}+\mathbf{A}_{2}^{\dagger}\mathbf{W}_{k}\mathbf{h}^{k}$
(13)
$\displaystyle\qquad\hat{\mathbf{V}}^{\mathrm{spatial}}_{1}=(\mathbf{I}-\mathbf{A}_{1}^{\dagger}\mathbf{A}_{1})\mathbf{V}^{\mathrm{spatial}}+\mathbf{A}_{1}^{\dagger}\mathbf{W}_{v}\mathbf{h}^{k},\hat{\mathbf{V}}^{\mathrm{spatial}}_{2}=(\mathbf{I}-\mathbf{A}_{2}^{\dagger}\mathbf{A}_{2})\mathbf{V}^{\mathrm{spatial}}+\mathbf{A}_{2}^{\dagger}\mathbf{W}_{v}\mathbf{h}^{k}.$
(14)

where $p$ is a random value drawn from standard Gaussian.
$\mathbbm{1}_{p>0.5}$ equals 1 if $p>0.5$ and equals 0 otherwise. We randomly
use $\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ to correct $\mathbf{K}$ and
$\mathbf{V}$, since using only a single interpolation function will cause
biased time stamp in interpolated frames (i.e. the 2-nd frame will look like
1.5-th frame if use $\mathbf{A}_{1}$ only) due to the align corner.

Figure 2:  (a) Examples for hidden states correction in 2$\times$ higher frame
rate generation case, showcasing the queries, keys and values are calibration
in temporal transformer (Temporal), and in spatial transformers using
different interpolation operators (Spatial A1, Spatial A2). (b) We adapt
temporal transformers to generate longer sequences in different ways. For the
temporal module with relative positional embedding (RPE), we use windowed
attention and apply RPE within each window. For absolute positional embedding
(APE) modules, we interpolate the position index to get APE before applying
attention operation.

###  4.3 Controlling Correction Strength

In vanilla DDNM, if $\hat{\epsilon}_{\theta}(\cdot)$ approximates the data
distribution well, then $\mathbf{A}^{\dagger}\mathbf{y}$ and
$(\mathbf{I}-\mathbf{A}^{\dagger}\mathbf{A})\mathbf{x}_{0|t}$ satisfy the
marginal data distribution in the range-space and null-space respectively.
Unfortunately, when combining them together. $\hat{\mathbf{x}}_{0|t}$ may fail
to match the joint distribution. This problem also presents in our transformer
hidden state back-projection. We propose a simple yet effective remedy for
this shown in Fig. 1(b). Instead of using the output $\hat{\mathbf{o}}$ from
corrected hidden states alone. Our method utilize a linear interpolation of
the original output $\mathbf{o}$ and the corrected output $\hat{\mathbf{o}}$
as the final result. Mathmatically, the controlled corrected output is

$\tilde{\mathbf{o}}=\mathbf{o}+w_{t}(\hat{\mathbf{o}}-\mathbf{o}).$ (15)

$w_{t}$ is a predefined control scale that lies between 0 and 1 for denoising
timestep $t$. With Eqn.15, we are able to control how much a denoising step
relies on the corrected output. By introducing $\mathbf{O}$ to the final
result, we alleviate the artifact caused by the joint distribution mismatch of
$\hat{\mathbf{h}}$.

##  5 Experiments

#### Experiment Settings

We deploy our method on three prevalent video diffusers including
VideoCrafter2[5], LaVie[34] and StableVideoDiffusion[2]. The base model of
VideoCrafter2 and LaVie generates 16 frames 320$\times$512 videos using
textual prompt. StableVideoDiffusion generates 14 frames 576$\times$1024
videos given a reference start frame. The testing tasks includes text-to-video
generation and image-to-video generation. Specifically, we use self-cascaded
VideoCrafter2 and LaVie for text-to-video generation, and use self-cascaded
StableVideoDiffusion for image-to-video. For all the experiments, we use our
method to inference in two untrained frame rate settings including 2$\times$
and $4\times$ higher frame rate without any further model tuning. The noise in
key frame is set to be identical to the base generation stage in every
timesteps to alleviate the stochasticity of denoising. Please see the detailed
inference hyperparameters in our appendix.

#### Testing Datasets and Metrics

For text-to-video generation, we use UCF-101[30] as the testing dataset.
UCF-101 is a video dataset that includes 101 action categories. To use it as a
text-to-video benchmark, we follow the text prompt proposed by Gu et al.[11]
For each category of action, we generate 20 videos and getting 2,020 videos in
total for every method. For both VideoCrafter2 and LaVie, we generate
320$\times$ 512 videos with 16, 32 and 64 frames. VideoCrafter2 uses a frame-
per-second condition and we set it to 8, 16, and 32 respectively. The sample
quality is evaluated with Inception Score (IS) and Frechet Video Distance
(FVD). The consistency to key frame condition is measured by SSIM and PSNR. To
get these metrics, we compute the image similarity between key frames in the
high frame rate video and the base video. For image-to-video generation, we
use WebVid-10M[1] as the testing dataset. It is a high definition video
dataset within the domain of StableVideoDiffusion. WebVid-10M contains over 10
million videos in various scenarios. We randomly sample 2,048 video clips from
the dataset and use the first frame of each video to serve as reference image.
We use StableVideoDiffusion to generate $576\times 1024$ videos with 14, 28
and 56 frames. The motion strength is 180, and the frame-per-second is set to
7 in the model. For every method in image-to-video, we generate 2,048 videos.
FVD, SSIM and PSNR are used to evaluate the video quality and the similarity
of key frames compared to the base video condition.

###  5.1 Comparison to Tuning-free Methods

Model | Method | 2$\times$ | 4$\times$
---|---|---|---
FVD$\downarrow$ | IS$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FVD$\downarrow$ | IS$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
VC2 | Dir. Inf. | 1202.3 | 30.31 | 10.90 | 0.426 | 1183.9 | 21.63 | 10.81 | 0.409
DDNM | 1139.6 | 40.99 | 23.50 | 0.841 | 1100.4 | 35.90 | 22.21 | 0.813
Ours | 1085.8 | 43.92 | 24.12 | 0.852 | 1020.6 | 43.77 | 22.68 | 0.837
LaVie | Dir. Inf. | 682.0 | 28.11 | 10.21 | 0.438 | 834.7 | 24.23 | 10.19 | 0.447
DDNM | 729.5 | 35.59 | 24.39 | 0.824 | 721.0 | 29.42 | 26.47 | 0.817
Ours | 703.2 | 39.23 | 26.90 | 0.876 | 697.6 | 38.67 | 24.17 | 0.831
SVD | Dir. Inf. | 1350.7 | - | 17.33 | 0.651 | 1090.2 | - | 15.75 | 0.604
DDNM | 846.6 | - | 29.12 | 0.875 | 895.2 | - | 28.12 | 0.864
Ours | 779.6 | - | 30.92 | 0.927 | 752.1 | - | 30.74 | 0.921
Table 1:  Quantitative comparisons between tuning-free methods to generate
2$\times$ and 4$\times$ higher frame rate.

Figure 3: Visual comparison between our method and other tuning-free
baselines. The green rectangles capture the abrupt changes between adjacent
frames.

We compare our method to the vanilla model (Direct inference), a tuning-free
method for visual restoration via back-projection in latent space named DDNM
[35]. Our results are shown in Tab.1. A qualitative comparison is shown in
Fig.3. For the best visual experience, please see the videos in our
supplementary material. We surpass the baselines in most of the metrics. The
similarity metrics PSNR and SSIM show our method is able to preserve key frame
when generating in high frame rate. Meanwhile, our method achieves the best
FVD and IS. ZeroSmooth generates visual plausible and temporal consistency
high frame rate videos. We show the method is effective in VideoCrafter2,
LaVie and StableVideoDiffusion. It illustrates our method’s potential to be an
model agnostic method and can be generalized to other video generative
diffusers.

The result of DDNM shows it is hard to apply back-projection in video latent
space for temporal consistent video interpolation. Generating higher frame
rate video directly is out of the domain of a pretrained video diffuser.
Therefore inference directly deteriorates in visual quality and text-video
alignment. Due to the stochasticity introduced by interpolated frames. Direct
inference fails to maintain key frame contents even when the key frame noise
is identical to that during the base video generation.

###  5.2 Comparison to Training-based Methods

Method | 2$\times$ | 4$\times$
---|---|---
FVD$\downarrow$ | IS$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FVD$\downarrow$ | IS$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
LDMVFI | 1304.6 | 41.40 | 34.60 | 0.948 | - | - | - | -
LaVie Intp. | - | - | - | - | 1172.1 | 35.78 | 28.44 | 0.874
Ours | 1085.8 | 43.92 | 24.12 | 0.852 | 1020.6 | 43.77 | 22.68 | 0.837
Table 2:  Quantitative comparisons between our method and training-based video
interpolators.

Despite ZeroSmooth is zero-shot method and no additional tuning or training
data is required. We find our method competitive with those training-based
video interpolation models. We compare our method with video interpolation
models including LDMVFI[9] and LaVie interpolation model [34]. Since our
method is aimed for text-to-video and image-to-video generation, but not for
real world video interpolation. We use the methods to interpolate the
generated videos of VideoCrafter2 base generator for a fair comparison. The
results are shown in Tab.2. Our method surpasses training-based method in
visual quality and is competitive in content preserving. Please see more
visual comparisons in our supplementary material.

###  5.3 Ablation Study

Setting | 2$\times$ | 4$\times$
---|---|---
FVD$\downarrow$ | IS$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | FVD$\downarrow$ | IS$\uparrow$ | PSNR$\uparrow$ | SSIM$\uparrow$
Vanilla | 1202.3 | 30.31 | 10.90 | 0.426 | 1183.9 | 21.63 | 10.81 | 0.409
\+ Spatial | 1251.2 | 42.40 | 22.96 | 0.739 | 1116.8 | 41.19 | 22.31 | 0.721
\+ Temopral | 1221.0 | 43.25 | 24.03 | 0.826 | 1156.1 | 42.54 | 22.63 | 0.817
\+ Spatial A1/A2 | 1199.8 | 43.09 | 23.22 | 0.768 | 1108.4 | 42.43 | 22.14 | 0.755
\+ CCS | 1085.8 | 43.95 | 24.12 | 0.852 | 1020.6 | 43.77 | 22.68 | 0.837
Table 3:  Quantitative results of ablation experiments.

We conduct an ablation experiment of our method on VideoCrafter2 to generate
$320\times 512$ videos with 32 and 64 frames. It includes three crucial
technical components of our method, including spatial transformers hidden
state correction (+ Spatial), spatial hidden state correction with
interpolation operator (+ Spatial A1/A2), temporal hidden state correction (+
Temporal), and controlling correction strength (+ CCS). The result is shown in
Tab.3. It shows the technical components of ZeroSmooth ensures high quality
and content preserving high frame rate video generation. Without the key
designs of our method, the visual quality and consistency to key frame content
degenerate dramatically.

##  6 Conclusion

We propose a training-free method for promoting generative video diffusion
models to generate videos with a high frame rate, i.e., producing more smooth
videos. It can be applied to different video models in a plug-and-play manner.
Since most video models are trained on videos consisting of sampled key frames
due to limited GPU memory, their performance will degenerate sharply if
generating a video with more frames than those used in training. We design the
self-cascaded framework and the hidden state correction module to avoid the
degeneration and achieve temporal consistency between frames. Extensive
experiments are conducted with various video models to demonstrate the
effectiveness of the proposed method.

#### Limitations

The proposed method is designed for generative video diffusion models. The
interpolation performance heavily depends on the capability of the target
video model in frame consistency and visual quality. If the generated key
frames are inconsistent or blurry, our method will fail to improve the video
smoothness.

#### Boarder Impact

The proposed interpolation method have a broad impact, enhancing video quality
and enabling applications like slow-motion effects, and video compression.
They’re beneficial in industries like virtual reality, gaming, and film
production. However, they also raise ethical concerns, such as the potential
for creating misleading deepfake videos, necessitating their responsible use.

## References

  * [1] Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1728–1738 (2021)
  * [2] Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., Jampani, V., Rombach, R.: Stable video diffusion: Scaling latent video diffusion models to large datasets (2023)
  * [3] Blattmann, A., Rombach, R., Ling, H., Dockhorn, T., Kim, S.W., Fidler, S., Kreis, K.: Align your latents: High-resolution video synthesis with latent diffusion models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2023)
  * [4] Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q., Wang, X., Weng, C., Shan, Y.: Videocrafter1: Open diffusion models for high-quality video generation (2023)
  * [5] Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., Shan, Y.: Videocrafter2: Overcoming data limitations for high-quality video diffusion models (2024)
  * [6] Choi, J., Kim, S., Jeong, Y., Gwon, Y., Yoon, S.: Ilvr: Conditioning method for denoising diffusion probabilistic models. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)
  * [7] Chung, H., Kim, J., Mccann, M.T., Klasky, M.L., Ye, J.C.: Diffusion posterior sampling for general noisy inverse problems. In: The Eleventh International Conference on Learning Representations (2023), https://openreview.net/forum?id=OnD9zGAGT0k
  * [8] Danier, D., Zhang, F., Bull, D.: St-mfnet: A spatio-temporal multi-flow network for frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3521–3531 (June 2022)
  * [9] Danier, D., Zhang, F., Bull, D.: Ldmvfi: Video frame interpolation with latent diffusion models. arXiv preprint arXiv:2303.09508 (2023)
  * [10] Garber, T., Tirer, T.: Image restoration by denoising diffusion models with iteratively preconditioned guidance. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2024)
  * [11] Ge, S., Nah, S., Liu, G., Poon, T., Tao, A., Catanzaro, B., Jacobs, D., Huang, J.B., Liu, M.Y., Balaji, Y.: Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305.10474 (2023)
  * [12] He, Y., Yang, S., Chen, H., Cun, X., Xia, M., Zhang, Y., Wang, X., He, R., Chen, Q., Shan, Y.: Scalecrafter: Tuning-free higher-resolution visual generation with diffusion models. In: The Twelfth International Conference on Learning Representations (2024)
  * [13] He, Y., Yang, T., Zhang, Y., Shan, Y., Chen, Q.: Latent video diffusion models for high-fidelity video generation with arbitrary lengths. arXiv preprint arXiv:2211.13221 (2022)
  * [14] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D.P., Poole, B., Norouzi, M., Fleet, D.J., et al.: Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 (2022)
  * [15] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, 6840–6851 (2020)
  * [16] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 (2022)
  * [17] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. arXiv preprint arXiv:2204.03458 (2022)
  * [18] Jain, S., Watson, D., Tabellion, E., Hołyński, A., Poole, B., Kontkanen, J.: Video interpolation with diffusion models (2024)
  * [19] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013)
  * [20] Kong, L., Jiang, B., Luo, D., Chu, W., Huang, X., Tai, Y., Wang, C., Yang, J.: Ifrnet: Intermediate feature refine network for efficient frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [21] Lu, L., Wu, R., Lin, H., Lu, J., Jia, J.: Video frame interpolation with transformer. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2022)
  * [22] Luo, Z., Chen, D., Zhang, Y., Huang, Y., Wang, L., Shen, Y., Zhao, D., Zhou, J., Tan, T.: Videofusion: Decomposed diffusion models for high-quality video generation (2023)
  * [23] Lyu, Z., Li, M., Jiao, J., Chen, C.: Frame interpolation with consecutive brownian bridge diffusion (2024)
  * [24] Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: International Conference on Machine Learning. pp. 8162–8171. PMLR (2021)
  * [25] Qiu, H., Xia, M., Zhang, Y., He, Y., Wang, X., Shan, Y., Liu, Z.: Freenoise: Tuning-free longer video diffusion via noise rescheduling (2023)
  * [26] Reda, F., Kontkanen, J., Tabellion, E., Sun, D., Pantofaru, C., Curless, B.: Film: Frame interpolation for large motion. In: European Conference on Computer Vision (ECCV) (2022)
  * [27] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al.: Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 (2022)
  * [28] Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 (2020)
  * [29] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)
  * [30] Soomro, K., Zamir, A.R., Shah, M.: Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
  * [31] Voleti, V., Jolicoeur-Martineau, A., Pal, C.: Masked conditional video diffusion for prediction, generation, and interpolation. arXiv preprint arXiv:2205.09853 (2022)
  * [32] Wang, F.Y., Chen, W., Song, G., Ye, H.J., Liu, Y., Li, H.: Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305.18264 (2023)
  * [33] Wang, J., Yuan, H., Chen, D., Zhang, Y., Wang, X., Zhang, S.: Modelscope text-to-video technical report (2023)
  * [34] Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., Guo, Y., Wu, T., Si, C., Jiang, Y., Chen, C., Loy, C.C., Dai, B., Lin, D., Qiao, Y., Liu, Z.: Lavie: High-quality video generation with cascaded latent diffusion models (2023)
  * [35] Wang, Y., Yu, J., Zhang, J.: Zero-shot image restoration using denoising diffusion null-space model. The Eleventh International Conference on Learning Representations (2023)
  * [36] Xing, J., Xia, M., Liu, Y., Zhang, Y., Zhang, Y., He, Y., Liu, H., Chen, H., Cun, X., Wang, X., et al.: Make-your-video: Customized video generation using textual and structural guidance. arXiv preprint arXiv:2306.00943 (2023)
  * [37] Xue, T., Chen, B., Wu, J., Wei, D., Freeman, W.T.: Video enhancement with task-oriented flow. International Journal of Computer Vision 127(8), 1106–1125 (Feb 2019). https://doi.org/10.1007/s11263-018-01144-2, http://dx.doi.org/10.1007/s11263-018-01144-2
  * [38] Yan, W., Zhang, Y., Abbeel, P., Srinivas, A.: Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104.10157 (2021)
  * [39] Yang, S., Zhou, Y., Liu, Z., , Loy, C.C.: Rerender a video: Zero-shot text-guided video-to-video translation. In: ACM SIGGRAPH Asia Conference Proceedings (2023)
  * [40] Yu, J., Cun, X., Qi, C., Zhang, Y., Wang, X., Shan, Y., Zhang, J.: Animatezero: Video diffusion models are zero-shot image animators. arXiv preprint arXiv:2312.03793 (2023)
  * [41] Zhang, G., Zhu, Y., Wang, H., Chen, Y., Wu, G., Wang, L.: Extracting motion and appearance via inter-frame attention for efficient video frame interpolation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5682–5692 (2023)
  * [42] Zhu, Y., Zhang, K., Liang, J., Cao, J., Wen, B., Timofte, R., Gool, L.V.: Denoising diffusion models for plug-and-play image restoration. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (NTIRE) (2023)

##  Appendix A Appendix

###  A.1 ZeroSmooth Temporal Attention

Detailed computation of ZeroSmooth temporal attention is shown in this
subsection. As shown in the main text, we have different strategies for
temporal attention modules with relative positional embedding (RPE) and
absolute positional embedding (APE). Specifically, in our experiment, LaVie
uses RPE in temporal transformers. StableVideoDiffusion uses APE in temporal
transformers. VideoCrafter2 learns relative position information within its
temporal convolution blocks and no positional embedding is used in temporal
transformers. We treat VideoCrafter2 as it has empty RPE.

#### Absolute positional embedding

We interpolate the position index before computing absolute positional
embedding. In the base video model, the video length is $\mathbf{t_{0}}$.
Consider a adapted model that generates $t$ frames. StableVideoDiffusion uses
sinusoidal absolute postition embedding. Mathmatically, the APE is then
interpolated

$\displaystyle\mathrm{APE}_{(\mathrm{pos},2i)}$
$\displaystyle=\mathrm{sin}\left(\frac{\mathrm{pos}\cdot
t_{0}}{10000^{2i/d}\cdot t}\right),$ (16)
$\displaystyle\mathrm{APE}_{(\mathrm{pos},2i+1)}$
$\displaystyle=\mathrm{cos}\left(\frac{\mathrm{pos}\cdot
t_{0}}{10000^{2i/d}\cdot t}\right).$ (17)

$\mathrm{pos}\in\\{1,2,\cdots,t\\}$ is the position of the current token. Then
a linear mapping layer is applied to get the final absolute positional
embedding $\mathbf{U}\in\mathbb{R}^{t\times d}$. The APE is added to hidden
state $\mathbf{h}$ before attention operation. We then use windowed attention
where the window size equals to sequence length during model training to
maintain the perception field of attention operation. Our method ensures there
are overlaps between attention windows, such that we can apply the attention
fusion technique proposed by Qiu et al[25]. to achieve smooth transition
between adjacent attention windows. We use an overlap of $\lfloor
t_{0}/2\rfloor$ in our experiment. The the $i$-th attention window starts at
position $i_{\mathrm{start}}=\mathrm{min}(it_{0},t-t_{0})$ and ends at
$i_{\mathrm{end}}=\mathrm{min}((i+1)t_{0},t)$. We use the subscript $i$ to
denote the tokens within position $i_{\mathrm{start}}$ and $i_{\mathrm{end}}$.
The output of the $i$-th attention window is:

$\displaystyle\mathbf{o}_{i}$
$\displaystyle=\mathrm{Attention}(\mathbf{h}_{i}+\mathbf{U}_{i})$ (18)
$\displaystyle=\mathrm{Softmax}\left(\frac{\mathbf{W}_{q}(\mathbf{h}_{i}+\mathbf{U}_{i})(\mathbf{h}_{i}+\mathbf{U}_{i})^{T}\mathbf{W}_{k}^{T}}{\sqrt{d}}\right)\mathbf{W}_{v}(\mathbf{h}_{i}+\mathbf{U}_{i}),$
(19)

where $\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}$ are the projection weight
for query, key and value. The final output is

$\mathbf{o}=\mathrm{AttentionFusion}(\mathbf{o}_{1},\mathbf{o}_{2},\cdots,\mathbf{o}_{n}).$
(20)

$\mathrm{AttentionFusion}(\cdot)$ is the attention fusion operator in [25].
$n$ is the number of windows, one can compute it through
$n=\lceil(t-t_{0})/(t_{0}-\lfloor t_{0}/2\rfloor)\rceil$, where
$\lceil\cdot\rceil$ is the ceil operator.

#### Relative positional embedding

There are many forms of RPEs. Without loss of generality, we consider
trainable relative positional embedding in the followings. Let
$\mathbf{p}_{k}\in\mathbb{R}^{t_{0}\times d}$ and
$\mathbf{p}_{v}\in\mathbb{R}^{t_{0}\times d}$ denote the learned relative
position. Then we also apply the windowed attention and attention fusion
strategy. Mathmatically, the attention output is

$\displaystyle\mathbf{o}_{i}$
$\displaystyle=\mathrm{Softmax}\left(\frac{\mathbf{W}_{q}\mathbf{h}_{i}(\mathbf{h}_{i}+\mathbf{p}_{k})^{T}\mathbf{W}_{k}}{\sqrt{d}}\right)\mathbf{W}_{v}(\mathbf{h}_{i}+\mathbf{p}_{v})$
(21) $\displaystyle\mathbf{o}$
$\displaystyle=\mathrm{AttentionFusion}(\mathbf{o}_{1},\mathbf{o}_{2},\cdots,\mathbf{o}_{n})$
(22)

###  A.2 Implementation Details

Figure 4:  An illustration of three stages self-cascaded model.

We describe the implementation details that is not included in the main text
in this subsection, showcasing the adaptation methods for all the used models
including VideoCrafter2, LaVie and StableVideoDiffusion.

#### Modules to apply ZeroSmooth

Modern video diffusion UNet includes 6 categories of neural network modules.
ZeroSmooth is applied to spatial self-attention, spatial cross-attention,
temporal self-attention and temporal cross-attention. The remaining modules
including spatial ResNet block and temporal ResNet block are kept the same to
the base generator. We use our method to all the modules in the selected
categories for all denoising timesteps. When computing the key and the value,
cross-attention modules uses text prompts or image embedding in text-to-video
and image-to-video respectively. Since the condition is repeated for all
frames, our hidden state correction will not change the value of these
conditions.

#### Stochasticity control

We use a noise scheduling strategy to reduce the stochasticity during
denoising sampling. Consider a $N$-stage self-cascaded ZeroSmooth model that
includes $\\{\boldsymbol{\epsilon}^{1}_{\theta}(\cdot):\mathbb{R}^{t_{0}\times
hwc}\rightarrow\mathbb{R}^{t_{0}\times
hwc},\boldsymbol{\epsilon}^{2}_{\theta}(\cdot):\mathbb{R}^{nt_{0}\times
hwc}\rightarrow\mathbb{R}^{nt_{0}\times
hwc},\cdots,\boldsymbol{\epsilon}^{N}_{\theta}(\cdot):\mathbb{R}^{n^{N-1}t_{0}\times
hwc}\rightarrow\mathbb{R}^{n^{N-1}t_{0}\times hwc}\\}$. $n$ is the frame
interpolation scale. Let $\epsilon^{N}\in\mathbb{R}^{n^{N-1}t_{0}\times hwc}$
denote the additional standard Gaussian noise introduced to the denoising
iteration of $\boldsymbol{\epsilon}_{\theta}^{N}(\cdot)$ for a specific
denoising timestep. Let the lower script $i$ denote the noise in $i$-th frame.
Then, the additional noised used for
$\boldsymbol{\epsilon}_{\theta}^{s}(\cdot)$ is sampled from $\epsilon^{N}$ via

$\epsilon_{i}^{s}=\epsilon_{in^{N-s}}^{N}.$ (23)

This strategy ensures the noise at the corresponding frame to be the same in
all self-cascaded stages.

#### VideoCrafter2

VideoCrafter2 uses an additional frame-per-second condition to control the
motion speed presents in the video. Given a fps condition for the base
generator. We multiply it with the interpolation scale to get the fps
condition of a spefic self-cascaded stage.

#### StableVideoDiffusion

StableVideoDiffusion uses two additional conditions including fps and motion
bucket id. Motion bucket id controls the the motion amplitude in the video.
Since the overall motion is kept unchanged in high frame video, the motion
bucket id is changed in all stages. We apply a similar strategy to set fps
like in VideoCrafter2.

Unlike text-to-image video diffusers, StableVideoDiffusion concatenate a noisy
reference image to every frame to provide a guidance for visual appearance.
The noise strength added to the reference image is callded augmentation
strength. In our method, we use a uniform augmentation strength for all
stages. StableVideoDiffusion uses classifier-free guidance [16] during
sampling. The guidance scale is linear monotonically decreasing along the
frame number. To adapt this feature in ZeroSmooth, we fix the highest and the
lowest guidance scale and use linear interpolation to get the guidance scale
for the high frame rate video.

#### Self-cascaded model

Our experiment includes 2$\times$ and 4$\times$ higher frame rate generation.
We use two stages for the 2$\times$ experiment and use three stages for the
$4\times$ experiment. Every stage will generate $2\times$ higher frame rate
videos. A illustration of the three stages model is shown in Fig.4. We pass
the corrected states of the second stage
$\hat{\mathbf{h}},\hat{\mathbf{Q}},\hat{\mathbf{K}},\hat{\mathbf{V}}$ to
calibrate the hidden states in the third stage.

#### Controlling correction strength

The weight controlling correction strength is defined as

$w_{t}=0.8\sqrt{t/T},$ (24)

where $T$ is the maximum denoising timestep. We find in the early stage of
denoising, the model is more tolerant to hidden states correction. Therefore
we build a monotonically decreasing $w_{t}$. In the late stage of denoising,
the strength of hidden states correction becomes small.

#### Correcting color tone

An additional normalization is applied to avoid abrupt color tone change in
the interpolated frames. We find this helps in some special cases.
Specifically, after each denoising iteration. We use adaptive instance
normalization to calibrate the statistics of frames in the high frame rate
video to match the corresponding key frames.

###  A.3 Experiment Settings

Hyperparameter | VideoCrafter2 | LaVie | StableVideoDiffusion
---|---|---|---
Frame resolution | $320\times 512$ | $320\times 512$ | $576\times 1024$
Frame number | (16, 32, 64) | (16, 32, 64) | (14, 28, 56)
FPS condition | 8 | - | 7
Motion bucket id | - | - | 180
Aug. strength | - | - | 0.02
Inference steps | 50 | 50 | 25
Eta | 1.0 | 1.0 | -
Temp. attn. overlap | 8 | 8 | 7
Sampler | DDIM samplre | DDIM sampler | Euler discrete sampler
Guidance scale | 12.0 | 7.5 | 7.5
Table 4: Hyperparameters of ZeroSmooth.

The hyperparameter used for the experiment is shown in Tab.4. Aug. strength
denotes noise augmentation strength used for StableVideoDiffusion in image-to-
video. FPS condition is an input for VideoCrafter2 and StableVideoDiffusion.
Temporal attention overlap determines how many frames are shared between
attention windows in temporal transformers. Eta controls the noise added for
every denoising step in DDIM sampler. We use a stochastic samping trajectory
for StableVideoDiffusion, no additional randomness is added after the
initialization during sampling.

When using LDMVFI, we apply DDIM sampler. The inference timestep to 50 and eta
is set to 0. Classifier-free guidance scale is 0 for LDMVFI. The model
achieves $2\times$ frame interpolation. When using LaVie interpolation model,
we also use DDIM sampler. The inference timestep is 25 and eta is 0.
Classifier-free guidance scale for LaVie interpolator is 4.

We deploy all the models in fp16 precision during inference. 32 NVIDIA V100
32GB GPUs are used to sample the results. Due to the memory limit, we use
flash attention to reduce the VRAM requirements. The transformer hidden states
in the previous stage are offloaded to CPU memory and is thrown to VRAM when
it is need.

###  A.4 Additional Visual Comparisons

More visual comparisons between tuning-free methods and comparions between
ZeroSmooth and training-based methods are shown below.

Figure 5: Visual comparison between our method and other tuning-free baselines
on VideoCrafter2.

Figure 6: Visual comparison between our method and other tuning-free baselines
on LaVie[34].

Figure 7: Visual comparison between our method and other tuning-free baselines
on StableVideoDiffusion.

Figure 8: Visual comparison between our method and LDMVFI [9].

Figure 9: Visual comparison between our method and LaVie interpolation model
[34].