# AnimateAnything: Fine-Grained Open Domain Image Animation with Motion
Guidance

Zuozhuo Dai , Zhenghao Zhang , Yao Yao , Bingxue Qiu , Siyu Zhu , Long Qin ,
Weizhi Wang
Alibaba Group

###### Abstract

Image animation is a key task in computer vision which aims to generate
dynamic visual content from a static image. Recent image animation methods
employ neural based rendering technique to generate realistic animations.
Despite these advancements, achieving fine-grained and controllable image
animation guided by text remains challenging, particularly for open-domain
images captured in diverse real environments. In this paper, we introduce an
open domain image animation method that leverages the motion prior of video
diffusion model. Our approach introduces targeted motion area guidance and
motion strength guidance, enabling precise control the movable area and its
motion speed. This results in enhanced alignment between the animated visual
elements and the prompting text, thereby facilitating a fine-grained and
interactive animation generation process for intricate motion sequences. We
validate the effectiveness of our method through rigorous experiments on an
open-domain dataset, with the results showcasing its superior performance.
Project page can be found at https://animationai.github.io/AnimateAnything.

##  1 Introduction

Figure 1:  Examples of our method for image animation. The first three rows
illustrate the use of prompt text to animate the reference image, with the
6th, 11th, and 16th frames of the generated animation visualized. The last
three rows demonstrate the precise control of movable objects using motion
mask guidance in images containing multiple objects. The last two rows show
the iterative generation of animation using red and green prompts to animate
objects in the corresponding masks. Additional examples are provided in the
supplemental material.

The field of image animation has recently gained attention, especially for
creating short videos and animated GIFs for social media and photo-sharing
platforms. However, current image animation methods [9, 18, 33, 28, 3, 34] are
limited in animating specific object types, such as fluid [18, 33, 38, 28,
34], natural scenes [59, 29, 7, 23, 46], human hair [58], portraits [54, 53,
12], and bodies [54, 55, 25, 47, 4, 3], limiting their practical application
in open domain scenarios. Recent advancements in video diffusion models [17,
11, 52, 48, 26, 14, 65, 32] have enabled the generation of diverse and
realistic videos based on reference texts and images. In this paper, we aim to
address the open domain image animation problem by leveraging the motion
priors of video diffusion models. We propose a controllable diffusion-based
image animation method capable of animating arbitrary objects within an image
while preserving their details. Our practical experience has shown that
creating animations solely through prompt text is laborious and challenging,
and results in limited control over finer details. To enhance user control
over the animation process, we introduce the motion area guidance and motion
strength guidance, allowing for precise and interactive control of the motion
speed of multiple objects, significantly improving controllable and fine-
grained image animation.

To accurately identify movable objects and their corresponding movable regions
within an image, we introduce motion area masks. Inspired by ControlNet [63],
we append the mask along the channel dimension of the video latent
representation and initialize the convolutional weights to zero, allowing them
to adjust incrementally during the training process. This approach enables
fine-grained and precise control over multiple movable areas in the input
image, even when using multiple prompting texts. Training the model to follow
the guidance of the motion area mask presents a significant challenge, as it
is difficult to amass and annotate a substantial corpus of real videos in
which only specific regions are in motion. To address this issue, we propose
an unsupervised technique to generate synthetic videos with motion area masks
derived from actual videos. The model is trained on both synthetic and real
videos to ensure the generation of realistic videos directed by the motion
area mask.

To effectively control the speed of moving objects in image animation, we
introduce the metric of motion strength. Frame rate per second (FPS)
represents the number of frames displayed in one second and previous methods
[21, 52, 5] relied on FPS to control motion speed. However, it is important to
note that different object types may exhibit varying motion speeds, and FPS
primarily serves as a global scaling factor to indirectly adjust the motion
speed of multiple objects. For instance, a video featuring a sculpture may
have a high FPS but zero motion speed. To enable direct learning of motion
speed by the video diffusion model, we propose a novel motion strength loss to
supervise the model in learning inter-frame variances in the latent space.

In recent developments, image-to-video diffusion models [17, 52, 5] have
emerged for generating realistic videos from input images. These models employ
the CLIP vision encoder [41] to encode the reference image, capturing its
semantic structure and style. However, they often struggle to preserve fine-
grained visual details of the reference image. To address this limitation and
apply video diffusion models to image animation tasks, we propose encoding the
reference image into a latent space using a VAE [27]. This latent
representation can then serve as the first frame of the generated video,
effectively preserving image details without introducing additional parameters
to existing video diffusion models, thereby enabling efficient training and
inference for image animation tasks.

The combination of prompt text, motion area mask, and motion strength guidance
enables the generation of complex image animations in diverse real-world
scenarios. A sequential approach can be adopted, where a specific object is
initially animated with a prompt text, followed by the animation of another
object with a different prompt text. This iterative process allows users to
progressively and interactively modify the image animation until a
satisfactory outcome is achieved. An example of interactive image animation is
illustrated in the last row of Figure 1. Initially, the actions of the boy are
manipulated using a motion area mask marked in red together with the prompting
text: “the boy is running on the beach”. Subsequently, one of the two girls is
animated through another motion area mask marked in green, accompanied by the
prompting text: “the girl runs to the sea”. In summary, our contributions in
this work are threefold:

  * •

We introduce a highly flexible motion guidance mechanism that allows for fine-
grained open domain image animation guided by motion area and motion strength.
We propose a synthetic motion area mask generation method and a novel motion
strength loss for effective training.

  * •

Our image animation framework supports combined guidance from text, motion
area, and motion strength, enabling an interactive animation generation
process for creating complex animations.

  * •

By enhancing the image guidance mechanism of the current video diffusion
model, we achieve a state-of-the-art FVD score of 443 on the zero-shot MSRVTT
dataset.

##  2 Related Work

Figure 2:  Overview of our pipeline. We adopt the widely used 3D U-Net based
video diffusion model [11, 52] for image animation. Given a noisy video latent
with shape (frames, height, width, channel), we concatenate the clean latent
of the reference image and the noisy frames in the temporal dimension.
Additionally, we concatenate the motion area mask with the video latent in the
channel dimension. This results in the input latent with shape (frames+1,
height, width, channel+1) for the 3D U-Net. To control the motion strength of
the generated video, we project the motion strength as positional embedding
and concatenate it with the time step embedding.

###  2.1 Image Animation

Image animation involves creating a video sequence from a static image.
Initial approaches manually simulated motion for specific object types, such
as mass simulation for vegetation and 2D rigid transformations for inanimate
objects [8, 23]. Later research predicted inter-frame flows and utilized
neural rendering to produce cinemagraphs, primarily focusing on fluid dynamics
[18, 33, 38, 28, 34], natural environments [59, 29, 7, 23, 46], and human
features such as hair [58], faces [54, 53, 12], and body movements [54, 55,
25, 47, 4, 3]. Recent advancements have seen diffusion-based models that
generate diverse video content guided by a still image. The Make-it-move
framework [20] captures motion patterns through motion anchors, which are then
used to condition a VQ-VAE model. Despite its innovation, the model is limited
by its dependency on the specific motion-object pairs from the training data,
restricting its efficacy on open-domain images. The LFDM model [36] adopts a
two-stage process for human-centric video generation, while Generative
Dynamics [29] focuses on modeling oscillatory motion in natural scenes.
However, these methods are domain-specific. In contrast, our approach
addresses the challenge of open-domain image animation.

###  2.2 Image generation with diffusion models

The evolution of image generation research has transitioned from traditional
frameworks such as Generative Adversarial Networks (GANs) [13], Variational
Autoencoders (VAEs) [27], and autoregressive transformer models (ARMs) [6], to
the more recent diffusion models (DMs) [15]. This shift is attributed to the
stability, superior sample quality, and conditional generation capabilities of
DMs. DALLE-2 [42] represents a significant advancement by integrating the CLIP
model [41] for text-image feature alignment, enabling text-prompted image
synthesis. GLIDE [37] introduces classifier-free guidance to refine image
quality, while Imagen [45] utilizes a sequence of diffusion models for high-
resolution image creation. Latent Diffusion Models (LDMs) [42] employ an
autoencoder [10] to manage the diffusion process in latent space, enhancing
efficiency. Subsequent models like DiTs [39] and SDXL [40] further concentrate
on latent space manipulation. For conditional generation, T2I-Adapter [35] and
ControlNet [63] have been developed to integrate spatial conditions such as
depth maps and sketches into LDMs. Composer [22] extends this by training LDMs
with multiple conditions for more precise control. Building on these
developments, our work introduces motion area and motion strength guidance to
provide fine-grained control of image animation.

###  2.3 Video generation with diffusion models

Recent advancements in diffusion models (DMs) have shown great promise in
video generation [37, 45, 43, 39, 40]. The Video Diffusion Model (VDM) [17]
pioneers this domain by adapting the image diffusion U-Net architecture [44]
into a 3D U-Net structure for joint image and video training. Imagen Video
[16] employs a series of video diffusion models for high-resolution,
temporally coherent video synthesis. Make-A-Video [48] innovatively learns
motion patterns from unlabeled video data, while Tune-A-Video [57] explores
one-shot video generation by fine-tuning LDMs with a single text-video pair.
Text2Video-Zero [26] tackles zero-shot video generation using pretrained LDMs
without further training. ControlVideo [64] introduces a hierarchical sampler
and memory-efficient framework to craft extended videos swiftly. Concurrently,
VideoCrafter1 [5] integrates image guidance from CLIP into diffusion models
via cross-attention. Despite these innovations, capturing complex motion and
camera dynamics remains challenging. VideoComposer [52] and DragNUWA [62]
propose motion trajectory-based control for video generation, yet they fall
short in interactive animation with multiple objects. Our approach addresses
this by incorporating motion masks and motion strength parameters to precisely
manipulate individual objects within an image, thus facilitating more user-
centric interactive animation generation.

##  3 Method

We provide an overview of the video diffusion model in Section 3.1, followed
by its adaptation for the image animation task in Section 3.2. Sections 3.3
and 3.4 offer detailed information on the integration of motion guidance into
the generation process. Lastly, Sections 3.5 and 3.6 present the inference
process under different guidance.

###  3.1 Background

In this section, we introduce the preliminary knowledge of the latent
diffusion-based model (LDM) [43]. Given an image sample
$x_{0}\in\mathbb{R}^{3\times H\times W}$, the LDM initially utilizes a pre-
trained VAE to encode $x_{0}$ into a down-scaled latent representation
$z_{0}\in\mathbb{R}^{c\times h\times w}$. The forward process of the LDM can
be described as a Markov chain that incrementally introduces Gaussian noise to
the latent representation:

$q(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I),$
(1)

where $t=1,...,T$ and $T$ denotes the total number of timesteps. $\beta_{t}$
is a coefficient that controls the noise strength in step $t$. The iterative
noise adding can be simplified as:

$z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I),$
(2)

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{t})$. During training, the
LDM learns the latent space distribution of the real data by predicting the
noise $\epsilon$ added on $z_{t}$, resulting in a reduction of computational
complexity for diffusion models. The objective function can be written as:

$l_{\epsilon}=||\epsilon-\epsilon_{\theta}(z_{t},t,c)||^{2}_{2},$ (3)

where $\epsilon_{\theta}(\cdot)$ denotes the noise prediction function of
diffusion models, which is implemented by an U-Net [44] architecture. To
control the generation process flexibly, LDM employs a domain-specific encoder
to map the user-input condition $c$ into an intermediate representation, which
is then injected into the UNet via a cross-attention layer.

Video diffusion models [17, 48, 11, 52] expand upon the image LDM by
incorporating a 3D U-Net, enabling them to effectively handle video data. The
3D U-Net incorporates an extra temporal convolution following each spatial
convolution and a temporal attention block following each spatial attention
block. To inherit the generation capacity from image data, the 3D U-Net is
trained concurrently with both image and video data.

Figure 3: Motion mask guidance examples. The first column and second column
are the input mask and motion mask respectively. The user can specify one or
multiple movable areas in the motion mask to fine grained control the video
generation.

###  3.2 Image Animation with Video Diffusion Model

We employ the LDM VAE [43] to encode the reference image into a latent
representation, denoted as $z_{ref}$, in order to retain more appearance
details. The VAE is trained for image reconstruction, thus $z_{ref}$ contains
rich low-level image features. Although it may contain less semantic
information compared to CLIP [41] vision tokens, the diffusion model itself
has demonstrated powerful semantic understanding capabilities in tasks such as
semantic segmentation [24, 30]. As illustrated in Figure 2, our training
pipeline uses the reference image as the initial frame and adopts an auto-
regressive strategy to forecast subsequent frames, facilitating image
animation without extra model parameters. The first frame’s content is
propagated to later frames via temporal convolution and attention mechanisms.
Consequently, only the temporal layers are fine-tuned, while the spatial
layers remain frozen. At each time step $t$, we concatenate the clean
$z_{ref}$ with the noisy latent $z_{t}$, which contains $N$ frames, resulting
in a latent representation with $(N+1)$ frames as input. We then select only
the last $N$ frames from the denoised $z_{t}$.

###  3.3 Motion Area Guidance

we introduce motion area guidance to provide users with precise control over
the movable area of the input image. As shown in Figure 2, we concatenate the
motion area mask with the video latent in the channel dimension. Drawing
inspiration from ControlNet [63], we initialize the convolution kernel of the
mask channel with zeros to maintain the original video generation capability.

We construct the training pairs of video and corresponding motion area masks
from real videos using the following approach. Firstly, we convert the given
video sample with $N$ frames to gray-scale. Then, we calculate frame
differences that exceed a threshold value $T_{m}$. These differences are then
combined to create the difference binary mask $d$:

$d=\bigcup\limits_{i=1}^{N-1}(|x_{gray}^{i}-x_{gray}^{i-1}|>T_{m}),$ (4)

where $x_{gray}^{i}$ is the gray-scale of the $i$-th frame, the threshold
$T_{m}$ determines the intensity of motion in both movable and non-movable
areas. If $T_{m}$ is set too high, objects in non-movable areas may still
appear to move. Conversely, if $T_{m}$ is set too low, objects in non-movable
areas might be completely frozen, potentially causing image artifacts near the
boundary of the motion area mask. Subsequently, we identify the contours of
these difference areas in $d$ and construct the motion area mask $m$ by
assigning label 1 to the pixels contained within these contours, indicating
the movable area. Finally, considering the motion area mask $m$, we post-
process the video latent $z_{0}$ such that pixels in the non-movable area are
reset to the values of the first frame $z_{0}^{0}$:

$z_{0}^{\prime}=(1-m)\cdot z_{0}^{0}+m\cdot z_{0}.$ (5)

We use $z_{t}^{i}$ to denote the $i$-th frame of the video latent at time step
$t$. As illustrated in Section 4.4, the post-processing step significantly
enhances the effectiveness of the motion area guidance. To address subtle
movements imperceptible to the human eye, which should not be marked as the
movable area, we explicitly instruct the model to keep these pixels unchanged.
The motion threshold $T_{m}$ is adjusted to ensure that the visual differences
between the reconstructed video $z_{0}^{\prime}$ and $z_{0}$ remain reasonably
small. The impact of motion area guidance on is demonstrated in Figure 3.

###  3.4 Motion Strength Guidance

During our training process, we observed that the sampling FPS affects the
motion speed of the movable objects in the generated videos. However, using
FPS alone as the motion speed guidance for video generation is inadequate, as
videos with the same FPS may exhibit varying motion speeds based on their
content. Therefore, FPS alone cannot effectively regulate the speed of object
motion. Consequently, we propose a metric called motion strength $s$ to
quantitatively measure the motion speed of the target motion area:

$s(z)=\frac{1}{N-1}\sum_{i=1}^{N}|z^{i}-z^{i-1}|.$ (6)

Here, the motion strength quantifies the differences between frames in the
latent space. Similar to the time step, we project the motion strength into a
positional embedding and add it to each frame in the residual block to ensure
uniform application of the motion strength to every frame. The impact of
motion strength guidance on the animation results is illustrated in Figure 4.

Training our motion strength guided pipeline directly with the noise
prediction loss defined in Equation 3 is hard to converge. This is likely due
to the fact that the noise prediction loss is primarily influenced by the
frame-level image difference and does not directly supervise the inter-frame
difference. Therefore, we introduce a motion strength loss to directly
supervise the inter-frame difference:

$l_{s}=||s(z_{0})-s(\hat{z}_{0})||^{2}_{2},$ (7)

where $\hat{z}_{0}$ represents the model’s estimated clean video latent
$z_{0}$, which can be obtained by transforming Equation 2 as:

$\hat{z}_{0}=\frac{z_{0}-\sqrt{1-\bar{\alpha}_{t}}~{}\epsilon_{\theta}(z_{t},t,c)}{\sqrt{\bar{\alpha}_{t}}}.$
(8)

Finally, we combine the noise prediction loss and the motion strength loss
with the scaling factor $\lambda$.

$l=l_{\epsilon}+\lambda\cdot l_{s}.$ (9)

Figure 4: Motion strength guidance examples. Augmenting the motion strength
accelerates the alteration of Mona Lisa’s expression, but excessive motion
strength may lead to the loss of fine-grained facial details.

###  3.5 Guidance Composition

Our image animation model integrates guidance from reference images, text,
motion areas, and motion strength. During training, we vary the textual prompt
and motion area to allow the model to accept different combinations of
guidance during inference. However, conflicting guidance inputs can diminish
their individual effects. For instance, if the text prompt does not align with
the content of the reference image, the model prioritizes fidelity to the
image. Fortunately, with motion area guidance, objects outside the motion area
mask are completely frozen, allowing for interactive editing of the generated
animation. As demonstrated in the last row of Figure 1, different objects can
be animated with different texts.

###  3.6 Shared Noise Inference

During training, we construct the input latent by adding noise on the clean
video latent. This noise schedule leaves some residual signal even at the
terminal diffusion timestep T. As a consequence, the diffusion model fails to
generalize faithful image animation during test time when we sample from
random Gaussian noise without any real data signal. To solve this train-test
discrepancy problem, during testing, we obtain the base noise by adding noise
on $z_{ref}$ using the forward process of DDPM [15]. As the DDPM independently
samples random noise $\epsilon^{i}$ for each frame, it allows for frame
diversity. The DDPM independently samples random noise $\epsilon^{i}$ for each
frame, allowing for frame diversity. The noise latent for frame $i$ can be
expressed as:

$z_{T}^{i}=\sqrt{\bar{\alpha_{T}}}z_{ref}+\sqrt{1-\bar{\alpha_{T}}}\epsilon^{i},$
(10)

where $\bar{\alpha_{T}}$ denotes the diffusion factor. This approach combines
the base noise with $z_{ref}$ to achieve a balance between preserving the
reference image information and introducing frame-specific diversity through
the random noise $\epsilon^{i}$. This design decision is critical for high
image fidelity animation.

##  4 Experiment

Figure 5: Qualitative Results. Comparing to open sourced methods Video
Composer and VideoCrafter1, our method achieves higher image fidelity.
Comparing to the commercial product Gen-2, our method achieves higher frame
consistency.

###  4.1 Experimental Setup

Datasets. Our model is initialized from VideoComposer [22] which pretrains on
WebVid10M [2]. Then we finetune it on 20K videos randomly sampled from HD-
VILA-100M [61] to remove watermark. Following [56, 48, 51], we conduct
evaluation on MSR-VTT dataset [60] in a zero-shot setting. MSR-VTT is an open
domain video retrieval dataset, where each video clip is accompanied by 20
natural sentences for description. Typically, the textual descriptions
corresponding to the 2,990 video clips in its test set are utilized as prompts
to generate videos.

Evaluation Metrics. Following previous methods [48, 11], we use Frechet Video
Distance (FVD) [50] to measure the video generation quality. We also use the
Frame Consistency measured via CLIP cosine similarity of consecutive frames to
show the temporal consistency of video. We propose the metric Motion Mask
Precision to measure the effect of motion mask guidance, which calculates the
percentage of the moving area of generated video within the given motion area
mask. To measure the effect of motion strength guidance, we propose Motion
Strength Error, which is the mean squared error of the generated video motion
strength and the given video motion strength. In this section, all statistics
are collected by generating 16-frame videos in 256 $\times$ 256 resolution
with DDIM [49] algorithm in 50 steps.

Implements Details. We employ the AdamW [31] optimizer with a learning rate of
$5\times 10^{-5}$ for training our model. All experiments are conducted on a
single NVIDIA A10 GPU, requiring approximately 20 GB of vRAM for training and
6 GB of vRAM for inference. To enhance the model performance, we conduct
multiple frame rate sampling during training, utilizing various frame rates
(e.g., 4, 8, 12) to obtain an 8-frame training clip with a resolution of 384
$\times$ 384\. We train the model for 10,000 iterations with a batch size of
2, which costs about 1 day. The motion area mask threshold $T_{m}$ is set to 5
and the motion strength loss scaling factor $\lambda$ is set to 0.001.

###  4.2 Quantitative Results

Table 1 presents the quantitative results of the zero-shot video generation
ability on MSR-VTT compared to existing methods. To ensure a fair comparison,
all models are evaluated on a resolution of 256 $\times$ 256\. With the same
text and first frame condition, our method achieves lower FVD comparing to
VideoComposer and concurrent work VideoCrafter1, which is evident that
integrating fine-grained spatial information into the conditions leads to a
significant improvement. This improvement suggests that our method is capable
of generating more coherent videos than previous approaches.

Method | Conditions | Params(B) |  FVD($\downarrow$)
---|---|---|---
CogVideo [19] | text | 15.5 | 1294
LVDM [14] | text | 1.16 | 742
MagicVideo [65] | text | - | 998
VideoComposer [52] | text | 1.85 | 580
VideoFusion [32] | text | 1.83 | 581
ModelScope [51] | text | 1.70 | 550
VideoComposer [52] | text&image | 2.49 | 551
VideoCrafter1 [5] | text&image | 3.24 | 465
Ours | text&image | 1.81 | 443
Table 1: Video generation performance on the test set of MSR-VTT. “Conditions”
denotes the type of condition for generation.

###  4.3 Qualitative Results

We present several visual examples of our method with three baselines:
VideoComposer [52], Gen-2 [1] and VideoCrafter1 [5] in Figure 5. All the
methods have the ability to generate videos based on reference image, thus
showcasing the significant progress in image-conditioned video generation.
VideoComposer and VideoCrafter1 achieves satisfactory levels of fluency but
lost details of the reference image. In contrast, our proposed method
preserves the latent representation of the first frame and denoises the
subsequent frames, which plays a fundamental role in ensuring faithful video
generation. Despite Gen-2’s relatively successful preservation of temporal
consistency, it falls short in effectively controlling the motion amplitudes
of the generated videos. In contrast, our method offers the flexibility of
adjusting motion strength, where larger strengths correspond to greater motion
amplitudes.

###  4.4 Ablation Study

#### Image condition

We conduct an analysis of various design strategies for injecting reference
image information in Table 2. The “CLIP Vision Global Token” method used by
Video Composer [52] encodes the input image to vision tokens using CLIP image
encoder and only injects the global [CLS] token into the U-Net through cross
attention. The “CLIP Vision Full Tokens” method, used by VideoCrafter1 [5],
projects and concatenates all vision tokens with the text embedding. As shown
in Figure 5, CLIP vision tokens contain rich semantic information by
pretraining on text and image pairs, but image details such as background
texture and face details may be lost. Using all vision tokens also requires
more computation cost to compute the cross attention between the latent and
additional vision tokens. To preserve more image details, we first try the
“Concat Latent Spatial” method, which concatenates the reference image latent
from VAE with every frame on the channel dimension. However, we notice that
this method limits the diversity of video motion when generating long videos.
Finally, we concatenate the input image latent and the noise latent on the
temporal dimension. As shown in “Concat Latent Temporal”, this method produces
videos with better image fidelity and achieves better inference efficiency
without using the additional ViT vision encoder.

Method | FVD | Consistency | Time(s) | Mem(G)
---|---|---|---|---
CLIP Vision Global Token | 551 | 0.879 | 22.13 | 11.1
CLIP Vision Full Tokens | 457 | 0.911 | 24.12 | 11.7
Concat Latent Spatial | 450 | 0.917 | 20.38 | 5.7
Concat Latent Temporal | 443 | 0.916 | 21.54 | 5.9
Table 2: The performance comparison of image condition designs.

#### Motion area guidance

Table 3 illustrates different design choices for motion area guidance. A
simple and training-free approach to freeze the area outside the motion area
is to apply the motion mask on the noise video latent, therefore the latent
values outside motion area are to the same. However, this method produces
videos nearly not moves, since the freeze noise latent is not Gaussian noise,
which is inconsistent with the training setting. In Table 3, the ”Mask
Guidance” method concatenates the motion area mask and the noise latent as the
input to the U-Net. ”Mask Guidance + Freeze” is our proposed method that the
latent values in non-movable areas are frozen and set to be the same as the
first frame. We can observe the strategy of freezing non-movable areas helps
the model to easily learn the guidance between the motion area mask and the
target video.

Method | Motion Mask Precision
---|---
No Control | 0.21
Mask Guidance | 0.52
Mask Guidance + Freeze | 0.82
Table 3: Ablation study of the motion mask guidance. Our proposed strategy demonstrates effective capability in following designated motion areas to generate corresponding animation videos. Method | Motion Strength Error
---|---
No Control | 14.19
Adding Noise | 12.74
FPS Guidance | 8.37
Motion Strength Guidance | 4.82
Motion Strength Guidance + Loss | 2.36
Table 4: Ablate the design of the motion strength guidance. Comparing to FPS
guidance, our method offers greater flexibility in incorporating motion into
animation videos.

#### Motion strength guidance

We compare different design choices of motion strength guidance in Table 4.
Since the initial noise video latent is generated by adding noise on
$z_{ref}$, we can perform limited control over motion strength by the amount
of added noise, as indicated by “Adding Noise”. The performance of FPS
guidance is largely depend on the video content, so its variance is large.
Comparing to FPS, our motion strength guidance performs more stable and
achiever lower motion strength error.

##  5 Conclusion and Future Work

In this paper, we have presented an open domain image animation pipeline with
a video diffusion model, incorporating motion area and motion strength
guidance. While our method has shown promising results in achieving fine-
grained and interactive image animation, it is important to acknowledge that
our model has not been trained on high-resolution videos due to limited
training resources. This limitation constrains the applicability and
performance of our method in generating high-resolution animations.

## References

  * [1] Gen-2.  Accessed October 22, 2023 [Online] https://research.runwayml.com/gen2.
  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gul Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1728–1738, 2021.
  * Bertiche et al. [2023] Hugo Bertiche, Niloy J Mitra, Kuldeep Kulkarni, Chun-Hao P Huang, Tuanfeng Y Wang, Meysam Madadi, Sergio Escalera, and Duygu Ceylan.  Blowing in the wind: Cyclenet for human cinemagraphs from still images.  In _CVPR_ , 2023.
  * Blattmann et al. [2021] Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Bjorn Ommer.  Understanding object dynamics for interactive image-to-video synthesis.  In _CVPR_ , 2021.
  * Chen et al. [2023] Haoxin Chen, Menghan Xia, Yin-Yin He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, Chao-Liang Weng, and Ying Shan.  Videocrafter1: Open diffusion models for high-quality video generation.  _ArXiv_ , abs/2310.19512, 2023.
  * Chen et al. [2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _International conference on machine learning_ , pages 1691–1703. PMLR, 2020.
  * Cheng et al. [2020] Chia-Chi Cheng, Hung-Yu Chen, and Wei-Chen Chiu.  Time flies: Animating a still image with time-lapse video as reference.  In _CVPR_ , 2020.
  * Chuang et al. [2005] Yung-Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David H Salesin, and Richard Szeliski.  Animating pictures with stochastic motion textures.  In _ACM SIGGRAPH_ , 2005.
  * Endo et al. [2019] Yuki Endo, Yoshihiro Kanamori, and Shigeru Kuriyama.  Animating landscape: self-supervised learning of decoupled motion and appearance for single-image video synthesis.  _ACM TOG_ , 38(6):1–19, 2019.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7346–7356, 2023.
  * Geng et al. [2018] Jiahao Geng, Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou.  Warp-guided gans for single-photo facial animation.  _ACM TOG_ , 37(6):1–12, 2018.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * He et al. [2022] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen.  Latent video diffusion models for high-fidelity video generation with arbitrary lengths.  _arXiv preprint arXiv:2211.13221_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. [2022a] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022a.
  * Ho et al. [2022b] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  In _NeurIPS_ , 2022b.
  * Holynski et al. [2021] Aleksander Holynski, Brian L Curless, Steven M Seitz, and Richard Szeliski.  Animating pictures with eulerian motion fields.  In _CVPR_ , 2021.
  * Hong et al. [2022] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  _arXiv preprint arXiv:2205.15868_ , 2022.
  * Hu et al. [2021] Yaosi Hu, Chong Luo, and Zhenzhong Chen.  Make it move: Controllable image-to-video generation with text descriptions.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18198–18207, 2021.
  * Hu et al. [2022] Yaosi Hu, Chong Luo, and Zhenzhong Chen.  Make it move: controllable image-to-video generation with text descriptions.  In _CVPR_ , 2022.
  * Huang et al. [2023] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou.  Composer: Creative and controllable image synthesis with composable conditions.  _arXiv preprint arXiv:2302.09778_ , 2023.
  * Jhou and Cheng [2015] Wei-Cih Jhou and Wen-Huang Cheng.  Animating still landscape photographs through cloud motion creation.  _IEEE TMM_ , 18(1):4–13, 2015.
  * Karazija et al. [2023] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and C. Rupprecht.  Diffusion models for zero-shot open-vocabulary segmentation.  _ArXiv_ , abs/2306.09316, 2023.
  * Karras et al. [2023] Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman.  Dreampose: Fashion image-to-video synthesis via stable diffusion.  _arXiv preprint arXiv:2304.06025_ , 2023.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint arXiv:2303.13439_ , 2023.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Li et al. [2023a] Xingyi Li, Zhiguo Cao, Huiqiang Sun, Jianming Zhang, Ke Xian, and Guo-Shing Lin.  3d cinemagraphy from a single image.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4595–4605, 2023a.
  * Li et al. [2023b] Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski.  Generative image dynamics.  _arXiv preprint arXiv:2309.07906_ , 2023b.
  * Li et al. [2023c] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie.  Open-vocabulary object segmentation with diffusion models.  2023c.
  * Loshchilov and Hutter [2019] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019_ , 2019.
  * Luo et al. [2023] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan.  Videofusion: Decomposed diffusion models for high-quality video generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10209–10218, 2023.
  * Mahapatra and Kulkarni [2022] Aniruddha Mahapatra and Kuldeep Kulkarni.  Controllable animation of fluid elements in still images.  In _CVPR_ , 2022.
  * Mahapatra et al. [2023] Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin-Ying Lee, S. Tulyakov, and Jun-Yan Zhu.  Synthesizing artistic cinemagraphs from text.  _ArXiv_ , abs/2307.03190, 2023.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Ni et al. [2023] Haomiao Ni, Changhao Shi, Kai Li, Sharon X Huang, and Martin Renqiang Min.  Conditional image-to-video generation with latent flow diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18444–18455, 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Okabe et al. [2009] Makoto Okabe, Ken Anjyo, Takeo Igarashi, and Hans-Peter Seidel.  Animating pictures of fluid using video examples.  In _Comput. Graph. Forum_ , pages 677–686, 2009.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4195–4205, 2023.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022.  _URL https://arxiv. org/abs/2204.06125_ , 7, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Shaham et al. [2019] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli.  Singan: Learning a generative model from a single natural image.  In _ICCV_ , 2019.
  * Siarohin et al. [2021] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov.  Motion representations for articulated animation.  In _CVPR_ , 2021.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint arXiv:2209.14792_ , 2022.
  * Song et al. [2021] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _ICLR_ , 2021.
  * Unterthiner et al. [2019] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly.  Fvd: A new metric for video generation.  2019\.
  * Wang et al. [2023a] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.  Modelscope text-to-video technical report.  _arXiv preprint arXiv:2308.06571_ , 2023a.
  * Wang et al. [2023b] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  _arXiv preprint arXiv:2306.02018_ , 2023b.
  * Wang et al. [2020] Yaohui Wang, Piotr Bilinski, Francois Bremond, and Antitza Dantcheva.  Imaginator: Conditional spatio-temporal gan for video generation.  2020\.
  * Wang et al. [2021] Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva.  Latent image animator: Learning to animate images via latent space navigation.  In _ICLR_ , 2021.
  * Weng et al. [2019] Chung-Yi Weng, Brian Curless, and Ira Kemelmacher-Shlizerman.  Photo wake-up: 3d character animation from a single photo.  In _CVPR_ , 2019.
  * Wu et al. [2022] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan.  Nüwa: Visual synthesis pre-training for neural visual world creation.  In _European conference on computer vision_ , pages 720–736. Springer, 2022.
  * Wu et al. [2023] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7623–7633, 2023.
  * Xiao et al. [2023] Wenpeng Xiao, Wentao Liu, Yitong Wang, Bernard Ghanem, and Bing Li.  Automatic animation of hair blowing in still portrait photos.  In _ICCV_ , 2023.
  * Xiong et al. [2018] Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo.  Learning to generate time-lapse videos using multi-stage dynamic generative adversarial networks.  In _CVPR_ , 2018.
  * Xu et al. [2016] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5288–5296, 2016.
  * Xue et al. [2022] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo.  Advancing high-resolution video-language representation with large-scale video transcriptions.  In _International Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * Yin et al. [2023] Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan.  Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.  _arXiv preprint arXiv:2308.08089_ , 2023.
  * Zhang et al. [2023a] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023a.
  * Zhang et al. [2023b] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian.  Controlvideo: Training-free controllable text-to-video generation.  _arXiv preprint arXiv:2305.13077_ , 2023b.
  * Zhou et al. [2022] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  _arXiv preprint arXiv:2211.11018_ , 2022.