# Animate3D: Animating Any 3D Model with
Multi-view Video Diffusion

Yanqin Jiang1, Chaohui Yu211footnotemark: 1, Chenjie Cao2, Fan Wang2, Weiming
Hu1 Jin Gao1
1CASIA 2DAMO Academy, Alibaba Group
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
https://animate3d.github.io/ Equal contribution. Work done during Yanqin’s
internship at DAMO Academy, Alibaba Group.Corresponding author.

###### Abstract

Recent advances in 4D generation mainly focus on generating 4D content by
distilling pre-trained text or single-view image-conditioned models. It is
inconvenient for them to take advantage of various off-the-shelf 3D assets
with multi-view attributes, and their results suffer from spatiotemporal
inconsistency owing to the inherent ambiguity in the supervision signals. In
this work, we present Animate3D, a novel framework for animating any static 3D
model. The core idea is two-fold: 1) We propose a novel multi-view video
diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D
object, which is trained on our presented large-scale multi-view video dataset
(MV-Video). 2) Based on MV-VDM, we introduce a framework combining
reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the
multi-view video diffusion priors for animating 3D objects. Specifically, for
MV-VDM, we design a new spatiotemporal attention module to enhance spatial and
temporal consistency by integrating 3D and video diffusion models.
Additionally, we leverage the static 3D model’s multi-view renderings as
conditions to preserve its identity. For animating 3D models, an effective
two-stage pipeline is proposed: we first reconstruct motions directly from
generated multi-view videos, followed by the introduced 4D-SDS to refine both
appearance and motion. Qualitative and quantitative experiments demonstrate
that Animate3D significantly outperforms previous approaches. Data, code, and
models will be open-released.

##  1 Introduction

3D content creation has garnered significant attention due to its wide
applicability in AR/VR, gaming, and movie industry. With the development of
diffusion models [48, 47, 42, 20, 54, 6, 19] and large-scale 3D object
datasets [13, 12, 39, 69, 74, 14], recent 3D foundational generations have
seen extensive exploration through fine-tuned text-to-image (T2I) diffusion
models [33, 43, 44, 55, 34, 52], as well as training large reconstruction
models from scratch [22, 64, 50, 72, 57], leading the 3D assets creation to a
new era. Despite the significant progress in static 3D representation, this
momentum has not been paralleled in the realm of dynamic 3D content
generation, also known as 4D generation.

4D generation is more challenging due to the difficulty of simultaneously
maintaining spatiotemporal consistency in visual appearance and dynamic
motion. In this paper, we mainly focus on two challenges: 1) _No foundational
4D generation models to unify both spatial and temporal consistency._ Though
recent 4D generation works [46, 24, 5, 73, 31, 40, 67, 62, 4, 60, 70, 16]
separately distill knowledge from pre-trained T2I and 3D diffusion models [42,
44, 33] and video diffusion models [54, 2, 6] to model multi-view spatial
appearance and temporal motions respectively, we clarify that such a detached
learning way suffers from inevitable accumulation of appearance degradation as
the motion changed, as shown in Fig. LABEL:fig:teaser (SVD+Zero123). 2)
_Failing to animate existing 3D assets through multi-view conditions._ With
the development of 3D generations, animating existing high-quality 3D content
becomes a common demand. However, previous works about 4D modeling from video
[24, 60] or based on generated 3D assets [5, 31, 40, 70] are all based on text
[42, 44] or single-view [33, 43] conditioned models, struggling to faithfully
preserve multi-view attributes during the 4D generation, such as the back of
butterfly in Fig. LABEL:fig:teaser is ignored by Zero123 [33].

To address these issues, we advocate for an approach better suited for 4D
generation, that is, animating any off-the-shelf 3D models with unified
spatiotemporal consistent supervision. In this way, it would be convenient to
directly take advantage of various fast-developing 3D generation and
reconstruction approaches based on a single foundational model, eliminating
the accumulation of errors in modeling appearance and motion.

To this end, we propose a novel 4D generation framework called Animate3D in
this paper, which can be divided into a foundational 4D generation model and a
joint 4D Gaussian Splatting (4DGS) optimization. Formally, the foundational 4D
model is a Multi-View Video Diffusion Model (MV-VDM) built upon the 3D
generation model, MVDream [44], which can synchronously synthesize multi-view
images with various temporal motions. Specifically, to better inherit the
prior in previous 3D and video diffusion models trained on large-scale data,
we propose a learnable plug-and-play spatiotemporal attention module, building
upon the motion module in video diffusion [18, 6, 20] to expand the attention
learning from the temporal domain to the spatial and temporal domain.
Moreover, MV-VDM also includes the ability to refer from multi-view images,
sufficiently preserving the identity and details of off-the-shelf 3D assets.
Inspired by adaptive image-to-video works [17, 68], given multi-view images
rendered from existing 3D assets or collected from real-world objects, we
propose the multi-view version of I2V-Adapter, called MV2V-Adapter,
incorporating multi-view conditions to 4D learning through additionally
spatial features and text embeddings. Enhanced by the multi-view appearance
injection, we disentangle the appearance learning from the motion learning,
ensuring MV-VDM focuses on learning natural and coherent dynamic motions.

To further enable impressive animations from 3D objects that can be observed
at any viewpoint and time, we jointly optimize the 4DGS [58] through both
reconstruction and 4D Score Distillation Sampling (4D-SDS) losses based on our
unified MV-VDM. Benefiting from the spatiotemporal consistent multi-view video
generations, 4DGS can be roughly converged to proper results with only
reconstruction loss, while 4D-SDS further improves the details and fine-
grained motions.

The main dilemma in building a foundational 4D generation model lies in the
rarity of large-scale 4D datasets, which is non-trivial to collect but the key
factor to drive our MV-VDM. In this work, we make the first attempt to build a
large-scale multi-view video (4D) dataset, dubbed MV-Video. Specifically, MV-
Video comprises about 115K animations that are available under a public
license, consisting of about 53K animated 3D objects at all, which are
rendered into over 1.8M multi-view videos with minigpt4-video [3] generated
prompts, to serve as the training dataset for our 4D foundation model.

We highlight the contribution of this paper as follows: 1) Animate3D is the
first 4D generation framework to animate any 3D objects with detailed multi-
view conditions. 2) We propose the foundational 4D generation model, MV-VDM,
to jointly model spatiotemporal consistency. 3) We present the largest high-
quality 4D datasets MV-Video collected with about 115K animations with over
1.8M multi-view videos. Extensive experiments demonstrate that our data-driven
approach can generate spatiotemporal consistent 4D objects, significantly
outperforming previous counterparts.

##  2 Related Work

3D Generation. Early 3D generation works optimized single 3D object with CLIP
loss [38] or Score Distillation Sampling (SDS) [35] from 2D text-to-image
(T2I) diffusion models. Since the models providing supervision lacked 3D
prior, those works usually suffered from spatial inconsistency, i.e., multi-
face Janus problem [53, 30, 56]. To tackle this problem, on the one hand, some
works [33, 43, 44, 34, 37] lifted the T2I diffusion to multi-view image
diffusion by injecting new spatial attention layers and fine-tuning on large-
scale 3D synthetic datasets [13, 12]. Although 3D consistency was improved,
these optimization-based methods still required a relatively long time to
optimize a 3D object. On the other hand, some feed-forward 3D generation
foundation models [22, 64, 50, 72, 57, 63, 32], also trained on large-scale 3D
datasets, were able to produce a good-quality 3D object in several seconds in
an inference-only way. Inspired by the success of the data-driven approaches
in 3D generation, we aim to construct a large-scale 4D generation dataset and
take the pioneering step towards developing foundation models for 4D
generation.

Video Generation. Video generation works started with text-to-video (T2V)
generation [20, 45, 2, 18, 6, 19, 7, 10], subsequently followed by image-to-
video (I2V) approaches [68, 17, 61, 6]. Previous T2V works usually built upon
T2I diffusion models [20, 45, 19, 18, 21], leveraging their pre-trained
weights by leaving the spatial blocks unchanged and inserting new temporal
blocks to model the temporal camera or object motions. The I2V works [61, 17,
68], building upon the aforementioned T2V methods, typically incorporate image
semantics into video models. This is achieved through cross-attention
mechanisms between noisy frames and the conditional image, while retaining the
motion module design from the T2V models unaltered. We draw inspiration from
the development paradigm of video generation to design our 4D generation
foundation model, which is a multi-view image conditioned multi-view video
diffusion model building upon pre-trained multi-view 3D and video diffusion
models.

4D Generation. The pioneer work of 4D generation is MAV3D [46], which is a
text and image-conditioned 4D generation framework. MAV3D first proposed a
multi-stage pipeline to optimize the static 3D object generation through the
T2I model and subsequently learn motions from the T2V model [45]. Following
works [5, 31, 73, 4, 62] adopted a similar pipeline, and they further found
that employing T2I [42] and 3D-SDS [44] are crucial for both object generation
and motion learning stages. Without them, the quality of the generated
object’s appearance suffered a remarkable decline, and the motion-learning
process was prone to failure. Very recently, Consistent4D [24] proposed a
video-to-4D generation task, which used single-view video reconstruction and
SDS from Zero123 [33] for motion and appearance learning. This paradigm was
adopted by following works [40, 67, 70, 60, 16, 11] and extended to
text/image-to-video then video-to-4D generation. All these manners
aforementioned heavily depend on the foundational model for SDS to preserve
objects’ appearance and attributes. However, existing 3D diffusion models
struggle to refer to multi-view conditions, restricting their broader
applications to animate various off-the-shelf 3D assets without losing their
multi-view attributes.

Furthermore, it is worth noting that existing 4D generation methods suffer
from another issue, i.e., spatial and temporal inconsistency [46, 5, 31, 24,
40]. Because the diffusion models used for SDS were never trained with multi-
view video (4D) datasets, missing the critical capacity to formulate spatial
and temporal consistency simultaneously. Thus previous methods failed to
properly trade off a good balance between appearance and motion learning.

In this work, we resort to disentangle 3D object generation/reconstruction and
motion learning through a foundational 4D generation model, and propose a
novel framework to animate any static 3D object with consistent multi-view
attributes.

Concurrent with our work, there are some attempts towards building 4D
foundational models [41, 71, 29, 65, 26]. Diffusion2 [65] and Diffusion4D [29]
propose to generate multi-view/spaito-temporal videos for reconstruction
purpose. 4Diffusion [71] further introduces SDS to the generation pipeline.
Different from above optimization-based methods, L4GM [41] proposes to
generate per-frame 3D Gaussian in feed-forward way, based on large 3D
reconstruction model [50]. Our method has three advantages over them: 1) We
advocate the animation of any 3D model, thus directly benefiting from high-
quality 3D data and advanced 3D generation works. 2) We meticulously collect
and render large-scale high-quallity 4D data. Our training data comprises the
highest quality part of Objaverse [29] (6949 animated models), which is used
in above works, while the rest are collected by ourselves (46391 animated
models). 3) We show impressing results and we provide high-resolution videos
(1024 $\times$ 1024) on our project page.

##  3 Method

Figure 1: Illustration of our proposed multi-view video diffusion model—MV-VDM
(upper part) and our Animate3D framework (lower part). MV-VDM, trained on our
presented large-scale 4D dataset MV-Video, can generate spatiotemporal
consistent multi-view videos. Animate3D, based on MV-VDM, combines
reconstruction and 4D-SDS optimization to animate any static 3D models.

Given a static 3D model, our goal is to animate it with a text prompt and use
its multi-view renderings as image condition. This 4D generation task is
particularly challenging as it requires ensuring spatial and temporal
consistency of the appearance and motion, compatibility with the prompt, and
preserving the identity of the static object. To address these challenges more
fundamentally, we propose a novel framework, Animate3D, to animate any static
3D object. As depicted in Fig. 1, we divide the task into two parts: learning
a multi-view video diffusion model (MV-VDM), and learning 3D object animation
with MV-VDM.

###  3.1 Multi-view Video Diffusion Model (MV-VDM)

We propose a novel multi-view image conditioned multi-view video diffusion
model, named MV-VDM. To inherit prior knowledge acquired by spatial consistent
3D models and temporal consistent video models trained on large-scale
datasets, we advocate a baseline architecture by integrating them to utilize
their pre-trained weights. In this work, we take MVDream [44] and AnimateDiff
[18] for the 3D and the video diffusion model, respectively. To enhance the
spatiotemporal consistency and ensure compatibility with the prompts and the
object’s multi-view images, we propose an efficient plug-and-play
spatiotemporal attention module combined with an image-conditioning approach.
Our MV-VDM is trained on our presented large-scale multi-view video dataset,
MV-Video, which is introduced in Sec. 4.

Spatiotemporal Attention Module. As illustrated in Fig. 1, the proposed
spatiotemporal attention module comprises two parallel branches: the left
branch is for spatial attention, and the right branch is for temporal
attention. For spatial attention, we adopt the same architecture as the multi-
view 3D attention in MVDream [44]. Specifically, the original 2D self-
attention layer is converted into 3D by connecting $n$ different views. In
addition, we incorporate 2D spatial encoding, specifically sinusoidal
encoding, into the latent features to enhance spatial consistency. As for
temporal attention, we keep all designs of the temporal motion module from the
video diffusion model [18] unchanged in order to reuse their pre-trained
weights. Based on the features of these two branches, we employ an alpha
blender layer with a learnable weight to achieve features with enhanced
spatiotemporal consistency. It is worth noting that we do not apply
spatiotemporal attention across all views and frames due to the prohibitive
GPU memory requirements that render training infeasible. Instead, our
parallel-branch design offers an efficient and practical alternative. The
spatiotemporal attention can be formulated as:

$\displaystyle X_{out}={}$
$\displaystyle\mu\cdot\mathrm{Attention}_{\mathrm{spatial}}(X_{l}W_{Q}^{s},X_{l}W_{K}^{s},X_{l}W_{V}^{s})W_{O}^{s}~{}+$
(1)
$\displaystyle(1-\mu)\cdot\mathrm{Attention}_{\mathrm{temporal}}(X_{r}W_{Q}^{t},X_{r}W_{K}^{t},X_{r}W_{V}^{t})W_{O}^{t},$

where $\mu$ denotes the learnable weight,
$W^{s/t}_{Q},W^{s/t}_{K},W^{s/t}_{V},W^{s/t}_{O}$ represent the corresponding
projection matrices, $X_{l}\in\mathbb{R}^{(b\times f)\times(n\times h\times
w)\times c}$ and $X_{r}\in\mathbb{R}^{(b\times n\times h\times w)\times
f\times c}$ are the reshaped inputs of the spatial and temporal branches,
respectively. $b,n,f,h,w,c$ are the batch size, views, frames, height, width,
and channels of the image features, respectively.

Multi-view Images Conditioning. Inspired by I2V-Adapter [17], we add a new
attention layer, termed MV2V-Adapter, parallel to the existing frozen multi-
view 3D self-attention layer within the proposed spatiotemporal block, as
shown in Fig. 1. Concretely, noisy frames are first concatenated along the
spatial dimension. These are then used to query the rich contextual
information from the multi-view conditional frames, which are extracted using
the frozen 3D diffusion model. Next, we add the output of the MV2V-Adapter
layer to that of the original multi-view 3D attention layer of MVDream. Thus,
for each frame $i\in\\{1,...,f\\}$, denoting the multi-view input, output, and
conditional frames’ features as $X^{1:n,i}$, ${X^{1:n,i}}_{out}$, and
$X^{1:n,1}$, we have:

$\displaystyle{X^{1:n,i}_{out}}={}$
$\displaystyle\mathrm{Attention}(X^{1:n,i}W_{Q},X^{1:n,i}W_{K},X^{1:n,i}W_{V})W_{O}~{}+$
(2)
$\displaystyle\mathrm{Attention}(X^{1:n,i}{W_{Q}}^{\prime},X^{1:n,1}W_{K},X^{1:n,1}W_{V}){W_{O}}^{\prime},$

where $W_{Q}$, $W_{K}$, $W_{V}$ and ${W_{O}}$ are projection matrices in
original self-attention layer, while ${W_{Q}}^{\prime}$ and ${W_{O}}^{\prime}$
are those in the newly added layer. We find this simple cross-attention
operator can effectively improve the object’s appearance consistency in the
generated video. After that, as shown in the spatiotemporal block in Fig. 1,
we employ two cross-attention layers to align the text prompt and preserve the
object’s identity, respectively. The left one is inherited from MVDream, while
the right one is pre-trained in IP-Adapter [66].

Training Objectives. The training process of our multi-view video diffusion
model is similar to Latent Diffusion Model [42]. Specifically, the sampled
multi-view video data $q_{0}^{1:n,1:f}$ are first encoded into latent feature
$z_{0}^{1:n,1:f}$ via encoder $\mathcal{E}$ frame by frame and view by view.
Then we add noise using the forward diffusion scheduler:
$z_{t}^{1:n,2:f}=\sqrt{\bar{\alpha}_{t}}z_{0}^{1:n,2:f}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$,
where $\alpha_{t}$ is a weighted parameter and $\epsilon$ is Gaussian noise.
Note that, following I2V-Adapter, we keep the first frame, i.e., the condition
multi-view frames clean, and only add noise to the rest of the frames. During
training, the proposed MV-VDM takes as input the clean latent code
$z_{0}^{1:n,1}$, noisy latent code $z_{0}^{1:n,2:f}$, text prompt embedding
$y$, and the camera parameters $\Sigma^{1:n}$, and outputs the noise strength,
supervised by $\mathcal{L}_{2}$ loss. The training objective of our MV-VDM is
calculated as:

$\mathcal{L}_{\mathrm{MV-
VDM}}=\mathbb{E}_{\mathcal{E}(q_{0}),y,\epsilon\in\mathcal{N}(0,I),t,}[||\epsilon-\epsilon_{\theta}(z_{0}^{1:n,1},z_{t}^{1:n,2:f},t,y,\Sigma^{1:n})||_{2}^{2}],$
(3)

where $\theta$ denotes the diffusion model. It is important that we keep the
entire multi-view 3D attention module frozen and only train the MV2V-Adapter
layer and the spatiotemporal attention module to conserve GPU memory and
accelerate training. Moreover, as the multi-view images of the first frame,
$z_{0}^{1:n,1}$, serves as the condition images, we calculate the loss only
for the latter $f-1$ frames, i.e., $z_{0}^{1:n,2:f}$.

###  3.2 Reconstruction and Distillation of 4DGS

Based on our 4D generation foundation model MV-VDM, we propose to animate any
off-the-shelf 3D object. For efficiency, we take 3D Gaussian Splatting (3DGS)
[25] as the static 3D object representation, and animate it by learning motion
fields represented by Hex-planes, as in [58].

4D Motion Fields. As in 4D Gaussian Splatting (4DGS) [58], we represent the
motion fields by Hex-planes [15, 8]. Denoting the static 3DGS as
$\mathcal{G}=\\{\mathcal{X},\mathcal{C},\alpha,r,s\\}$, where $\mathcal{X}$,
$\mathcal{C}$, $\alpha$, $r$, and $s$ represent the position, color, opacity,
rotation, and scale, respectively. The motion module $\mathcal{D}$ predicts
changes in position, rotation, and scale for each Gaussian point in frame $i$
by interpolating the Hex-planes $R$. The motion fields computation can be
formulated as:

$\mathcal{F}=\bigcup_{l}\prod_{\zeta}\texttt{interp}(R^{\zeta},(\mathcal{X},i)),$
(4) $\Delta\mathcal{X}=\phi_{\mathcal{X}}(\mathcal{F}),\Delta
r=\phi_{r}(\mathcal{F}),\Delta s=\phi_{s}(\mathcal{F}),$ (5)

where $l$ equals to the scales in Hex-plane, and interp() denotes
interpolating the Gaussian points on the specific plane $\zeta$ to obtain
corresponding motion features. We have
$\zeta\in\\{(x,y),(x,z),(y,z),(x,t),(y,t),(z,t)\\}$. $\phi$ denotes offset
prediction layers. Therefore, Gaussian $\mathcal{G}^{\prime}$ at time $t$ is
updated as follows:

$\mathcal{G}^{\prime}=\\{\mathcal{X}+\Delta\mathcal{X},\mathcal{C},\alpha,r+\Delta
r,s+\Delta s\\}.$ (6)

To better preserve the appearance of static 3D objects, we keep certain
attributes, specifically opacity $\alpha$ and color $\mathcal{C}$, unchanged.

Motion Reconstruction. Based on the spatiotemporal consistent multi-view
videos generated by MV-VDM, we first leverage a 4DGS reconstruction stage to
directly reconstruct the coarse motions. Specifically, we use a simple but
effective $\mathcal{L}_{2}$ loss of image and mask as our
$\mathcal{L}_{\mathrm{rec}}$, which is calculated as:

$\mathcal{L}_{\mathrm{rec}}=\sum^{n}_{i=1}\sum^{f}_{j=1}(||~{}\mathcal{C}-\widehat{\mathcal{C}}~{}||^{2}+||~{}\mathcal{M}-\widehat{\mathcal{M}}~{}||^{2}),$
(7)

where $\mathcal{C}$ and $\widehat{\mathcal{C}}$ denote the multi-view and
multi-frame renderings and the corresponding ground truth, and $\mathcal{M}$
and $\widehat{\mathcal{M}}$ denotes masks. As verified in Fig. 2, this
reconstruction stage can already learn high-quality coarse motions by
leveraging the generated multi-view videos of MV-VDM.

4D-SDS Optimization. To better model the fine-level motions, we introduce a
4D-SDS optimization stage to distill the knowledge of our multi-view video
diffusion model. The 4D-SDS loss $\mathcal{L}_{\mathrm{4D-SDS}}$ is a variant
of $\mathbf{z}_{0}$-reconstruction SDS loss and can be formulated as:

$\mathcal{L}_{\mathrm{4D-SDS}}(\mathcal{G},\mathcal{D},z=\mathcal{E}(g(\mathcal{D}(\mathcal{G}))))=\mathbb{E}_{t,\Sigma,\epsilon}[||z-\hat{z_{0}}||_{2}^{2}],\quad\hat{z_{0}}=\frac{z_{t}-\sigma_{t}\epsilon_{\theta}}{\alpha_{t}},$
(8)

where $z$ and $z_{0}$ are latent feature of the rendered image and the
estimation of clean latent feature from current noise prediction
$\epsilon_{\theta}$, respectively, $g$ represents the rendering function.
$\alpha_{t}$ and $\sigma_{t}$ are the signal and noise scale controlled by the
noise scheduler, respectively.

Training Objectives. In addition to $\mathcal{L}_{\mathrm{rec}}$ and
$\mathcal{L}_{\mathrm{4D-SDS}}$, we introduce a variant of As-Rigid-As-
Possible (ARAP) loss [49] to facilitate the rigid movement learning as well as
the maintenance of the high-quality appearance of the static object. The ARAP
loss $\mathcal{L}_{\mathrm{arap}}$ in our work is defined as:

$\mathcal{L}_{\mathrm{arap}}(p_{j})=\sum_{i=2}^{f}\sum_{k\in\mathcal{N}_{c_{i}}}w_{j,k}||(p_{j}^{i}-p_{k}^{i})-R_{j}((p_{j}^{1}-p_{k}^{1})||^{2},$
(9)

where $\hat{R}_{j}$ is estimated from a rigid transformation using Singular
Value Decomposition (SVD) according to [49]:

$\hat{R}_{j}=\mathrm{argmin}_{R\in\mathbf{SO}(3)}\sum_{k\in\mathcal{N}_{c_{i}}}w_{j,k}||(p_{j}^{i}-p_{k}^{i})-\hat{R}_{j}((p_{j}^{1}-p_{k}^{1})||^{2}.$
(10)

$\mathcal{N}_{c_{j}}$ denotes the set of points within a fixed radius of
$p_{j}$, and $w_{j,k}=\mathrm{exp}(-\frac{d_{jk}}{\bar{d}})$ where $d_{jk}$ is
the distance between center of $p_{j}$ and $p_{k}$, measuring the impact of
$p_{k}$ on $p_{j}$. This loss encourages the generated dynamic object to be
locally rigid, and it enhances the learning with rigid movement.

In summary, the training objectives for animating off-the-shelf 3DGS object
is:

$\mathcal{L}=\lambda_{1}\mathcal{L}_{\mathrm{rec}}+\lambda_{2}\mathcal{L}_{\mathrm{4D-SDS}}+\lambda_{3}\mathcal{L}_{\mathrm{arap}},$
(11)

where $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are weighted parameters.

##  4 Experiment

###  4.1 Setup

Training Dataset.

To train our MV-VDM, we build a large-scale multi-view video dataset, MV-
Video. Concretely, we render multi-view videos of 53,340 animated 3D models
collected from Sketchfab [1].111Our training dataset overlaps with Objaverse
[13], comprising the highest quality 6,949 objects from that dataset. Each
model has 2.2 animations on average, resulting in 115,566 animations in total.
Each animation is 2 seconds long at 24 fps. Note that animated models that are
not allowed to be used to generate AI programs are filtered. The statistical
information of our MV-Video dataset is reported in Table 1. For more details
about the rendering settings and data examples, please refer to our Appendix
(D). We will release this dataset to further advance the field of 4D
generative research.

Table 1: Statistical information for our multi-view video (MV-Video) dataset.

Model ID | Animations | Avg. Animations per ID | Max Animations per ID | Multi-view Videos
---|---|---|---|---
53,340 | 115,566 | 2.2 | 6.0 | 1,849,056

Implementation Details. We sample 16 frames evenly for each animation to train
our MV-VDM. We use the Adamw optimizer with a learning rate of $2e-4$ and a
weight decay $0.01$, and train the model for 20 epochs with a batch size of
1024. When inference, we set the sampling step to 25 and adopt freeinit [59]
to get stable results when animating 3D objects. As for 4D generation, the
resolution and feature dimension of the Hex-planes are set to $[100,100,8]$
and 16, respectively. We perform motion reconstruction in progressive way for
the first 750 iterations with a batch size of 64 (4 views, 16 frames), and
then add 4D-SDS optimization for another 250 iterations. Learning rate is 0.01
and 0.0001 for Hex-plane and offset prediction layers. $\lambda_{1}$,
$\lambda_{2}$ and $\lambda_{3}$ in Eq. 11 are set to $100.0$, $0.01$ and
$10.0$ , respectively. It costs 10 days to train our MV-VDM on 32 80G A800
GPUs, and the optimization for 4D generation takes around 40 minutes (20
minutes for motion reconstruction, and 20 minutes for 4D-SDS) on a single A800
GPU per object.

Evaluation Dataset. For the evaluation of MV-VDM, we render multi-view images
from the first frame of 128 held-out animated 3D objects and then generate
multi-view videos conditioned on them. Following the evaluation setting of
VBench [23], we use four different random seeds for each object and report the
average results. For the evaluation of 4D generation, we generate 10 objects
across various categories using the large 3DGS reconstruction model GRM [64].
Besides, we reconstruct 10 high-quality objects from Sketchfab and take them
as evaluation data. Input images for GRM, visualization of reconstructed
objects, and animation prompts used in this work are provided in our Appendix
(E.1).

Evlaution Metrics. We adopt the evaluation protocol proposed in VBench [23],
which is a popular video generation benchmark consisting of both T2V and I2V
evaluation tools. The I2V evaluation protocol contains 9 evaluation metrics,
and we choose 4 for our evaluation, i.e., I2V Subject, Motion Smoothness,
Dynamic Degree, and Aesthetic Quality, measuring the consistency with the
given image, the motion smoothness, the motion degree, and the appearance
quality, respectively. We abbreviate them as I2V, M. Sm., Dy. Deg. and Aest.
Q. in the experiment section. Values of all metrics are the higher, the
better, except for Dynamic Degree, since we observe that sometimes completely
failed results present extremely high dynamic degree. For more details about
the introduction and calculation of the evaluation metrics, please refer to
our Appendix (E.2).

Comparison Methods. We compare our work with 4Dfy [5] and DreamGaussian4D
(DG4D) [40] on the task of animating any given 3D object using their official
implementations. They represent the state-of-the-art in 4D generation methods,
starting by generating a static 3D object using 3D-SDS in the initial stage,
and subsequently animating it via video SDS and single-view video
reconstruction in later stages. For 4D representation, 4Dfy and DG4D adopt
dynamic NeRF [36, 27] and 4DGS [58], respectively. For a fair comparison, we
replace the dynamic NeRF in 4Dfy with 4DGS used in both our work and DG4D, and
also apply ARAP loss for motion regularization. For DG4D, we keep the 4DGS
representation and motion regularizations in their work unchanged.

###  4.2 Comparison with State-of-the-Art

In this section, we perform comprehensive comparisons with previous works,
including quantitative and qualitative comparisons and user studies.

Table 2: Quantitative comparisons with state-of-the-art methods.

(a) Comparison on video generation metrics. | I2V $\uparrow$ | M. Sm. $\uparrow$ | Dy. Deg. | Aest. Q. $\uparrow$
---|---|---|---|---
4Dfy (Gau.) [5] | 0.741 | 0.995 | 0.0 | 0.478
DG4D [40] | 0.910 | 0.990 | 0.363 | 0.491
Ours | 0.983 | 0.993 | 0.388 | 0.552

(b) Comparison via user study. | Align. Text | Align. 3D. | Mot. | Appr.
---|---|---|---|---
4Dfy(Gau.) [5] | 2.062 | 1.553 | 1.535 | 1.818
DG4D [40] | 2.697 | 3.316 | 2.124 | 2.997
Ours | 4.388 | 4.735 | 4.312 | 4.503

Figure 2: Qualitative comparison with state-of-the-art methods. Best viewed by
zooming in.

Quantitative Comparison. As shown in Tab. 2(a), our method significantly
outperforms 4Dfy and DG4D in terms of I2V, Dy. Deg., and Aest. Q.. This
indicates our generation results have good alignment with the given static 3D
object (I2V Subject), dynamic motion (Dynamic Degree), and superior appearance
(Aesthetic Quality). For Motion Smoothness, we slightly lag behind 4Dfy, since
4Dfy always generates nearly static results, as illustrated by the 0.0 Dynamic
Degree in the first row of Tab 2(a). Generally, our method is able to animate
3D object with smooth and dynamic motion, at almost no cost of sacrificing
their high-quality appearance, facilitating customized and high-quality
dynamic 3D object creation.

Qualitative Comparison. As shown in Fig. 2, it is obvious that 4Dfy’s results
are blurred and deviate much from the given 3D object, owing to the use of
text-conditioned diffusion models to optimize the motion and appearance.
Additionally, its generated objects are almost static. This is because at the
beginning of the training process, the noisy rendered image sequence, i.e.,
the input to the T2V model, has no temporal changes, which misleads the video
diffusion model to generate almost static supervisions, as illustrated in Fig
LABEL:fig:teaser. For DG4D, its results align relatively well with the given
3D object in front view, i.e., the view used for generating guided video.
However, it doesn’t align with the object in the novel view, as indicated by
the the missing patterns on Spider-man and Superman’s suits, and blurred back
and side views in Fig 2. This is because it adopts Zero123 to optimize the
novel views. Zero123 only conditions on the front view, leading to NVS
optimization favoring pre-trained data distributions, which could lead to
potential appearance degradation. More importantly, DG4D fails when the object
in the guided video is assigned with movements toward the camera. For example,
the dog is moving towards in the guided video, however, DG4D interprets it as
enlargement of the object. This misinterpretation usually results in blurry
effect and strange appearance.

In contrast, our method, levering the spatiotemporal consistent muti-view
prior, manage to deal with motion towards the camera, as demonstrated by
Spiderman’s walking legs(our model takes the front view and its orthogonal
views as the condition view). Besides, we successfully maintain the high
quality appearance of the given 3D object when generating natural motion.
Please refer to the videos in our supplementary material for a more intuitive
comparison.

User Study. We conduct a user study among 20 people on the 20 dynamic objects
and report the averaged results in Tab. 2(b). The participants are asked to
score the generated dynamic objects from 1 to 5, according to the alignment
with the given text (Align. Text) and static object (Align. 3D), motion
quality (Mot.), and appearance quality (Appr.). The user study proves the
superiority of our method. Please refer to the Appendix (E.3) for more
details.

###  4.3 Ablation

Table 3: Ablation Studies

(a) Ablation of Multi-View Diffusion | I2V $\uparrow$ | M. Sm. $\uparrow$ | Dy. Deg. | Aest. Q. $\uparrow$
---|---|---|---|---
w/o S.T. Att. | 0.915 | 0.980 | 0.958 | 0.531
w/o Pre-train | 0.910 | 0.981 | 0.944 | 0.531
Ours | 0.935 | 0.988 | 0.710 | 0.532

(b) Ablation of 4D Generation | I2V $\uparrow$ | M. Sm. $\uparrow$ | Dy. Deg. | Aest. Q. $\uparrow$
---|---|---|---|---
w/o SDS loss | 0.982 | 0.991 | 0.404 | 0.551
w/o ARAP loss | 0.968 | 0.992 | 0.360 | 0.509
Ours | 0.983 | 0.993 | 0.388 | 0.552

Figure 3: Ablation for multi-view video diffusion. Figure 4: Ablation for 3D
object animation. Best viewed by zooming in.

Multi-view Video Diffusion. In Tab 3(a), we validate the effectiveness of the
proposed SpatioTemporal Attention (short as S.T. Att.) and the pre-trained
weight from video diffusion model (short as Pre-train). This albation is
conducted using 8-frame version of our model trained on a subset of the
training dataset. We replace the proposed spatiotemporal block with temporal
block from AnimateDiff, and this leads to performance drop in I2V Subject,
Motion Smoothness and Aesthetic Quality. Dynamic Degree seems to be enhanced,
but that is caused by the increase of unstable failure cases. The same
tendency could be observed in experiments w/o pre-trained video diffusion
weight. Therefore, we think both designs are necessary for generating multi-
view videos consistent with the given multi-view images and with high-quality
appearance and motion. Qualitative ablations in Fig. 3 further demonstrate
this point.

4D Object Optimization. The ablations for 4D object optimization are shown in
Tab. 3(b) and Fig. 4. The quantitative results in Tab. 3(b) indicate SDS
slightly decreases Dynamic Degree while has almost no impact on other metrics.
That is because our motion reconstruction results are relatively good, with
only small floaters and slightly blurry effect around the edge of the object,
which are removed by 4D-SDS later, as shown in Fig. 4. (Since the image
resolution here is limited, please visit our project page to see high-
resolution videos of ablation for 4D SDS). ARAP loss improves all four
metrics, and it is important for maintaining high-quality appearance.

##  5 Conclusion

In this work, we present Animate3D, a novel framework for animating any off-
the-self 3D object. Animate3D disentangles the 4D object generation into a
foundational 4D generation model, MV-VDM, and a joint 4DGS optimization
pipeline based on MV-VDM. MV-VDM is the first 4D foundation model, which can
generate spatiotemporal consistent multi-view videos conditioned on multi-view
renderings of a static 3D object. To train MV-VDM, we present the largest
multi-view video (4D) dataset, MV-Video, containing about 115K animations with
over 1.8M multi-view videos. Based on MV-VDM, we propose an effective pipeline
to animate any static 3D object by jointly optimizing the 4DGS via both
reconstruction and 4D-SDS. Animate3D is a highly practical solution for
downstream 4D applications since it can animate any generated or reconstructed
3D objects. Data, codes, and pre-trained weights will be released to
facilitate the research in 4D generation.

## References

  * [1] Sketchfab.  https://sketchfab.com/.  Accessed: 2024-5-21.
  * [2] Zeroscope text-to-video model.  https://huggingface.co/cerspense/zeroscope_v2_576w.  Accessed: 2023-10-31.
  * [3] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny.  Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens.  arXiv preprint arXiv:2404.03413, 2024.
  * [4] Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al.  Tc4d: Trajectory-conditioned text-to-4d generation.  arXiv preprint arXiv:2403.17920, 2024.
  * [5] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B. Lindell.  4d-fy: Text-to-4d generation using hybrid score distillation sampling.  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
  * [6] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  arXiv preprint arXiv:2311.15127, 2023.
  * [7] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023.
  * [8] Ang Cao and Justin Johnson.  Hexplane: A fast representation for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130–141, 2023.
  * [9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [10] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al.  Videocrafter1: Open diffusion models for high-quality video generation.  arXiv preprint arXiv:2310.19512, 2023.
  * [11] Wen-Hsuan Chu, Lei Ke, and Katerina Fragkiadaki.  Dreamscene4d: Dynamic multi-object scene generation from monocular videos.  arXiv preprint arXiv:2405.02280, 2024.
  * [12] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al.  Objaverse-xl: A universe of 10m+ 3d objects.  Advances in Neural Information Processing Systems, 36, 2024.
  * [13] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi.  Objaverse: A universe of annotated 3d objects.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142–13153, 2023.
  * [14] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke.  Google scanned objects: A high-quality dataset of 3d scanned household items.  In 2022 International Conference on Robotics and Automation (ICRA), pages 2553–2560. IEEE, 2022.
  * [15] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa.  K-planes: Explicit radiance fields in space, time, and appearance.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479–12488, 2023.
  * [16] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and Ulrich Neumann.  Gaussianflow: Splatting gaussian dynamics for 4d content creation.  arXiv preprint arXiv:2403.12365, 2024.
  * [17] Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Chongyang Ma, Weiming Hu, Zhengjun Zha, Haibin Huang, Pengfei Wan, et al.  I2v-adapter: A general image-to-video adapter for video diffusion models.  arXiv preprint arXiv:2312.16693, 2023.
  * [18] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [19] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  Advances in Neural Information Processing Systems, 35:8633–8646, 2022.
  * [21] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  arXiv preprint arXiv:2205.15868, 2022.
  * [22] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan.  Lrm: Large reconstruction model for single image to 3d.  arXiv preprint arXiv:2311.04400, 2023.
  * [23] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu.  VBench: Comprehensive benchmark suite for video generative models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
  * [24] Yanqin Jiang, Li Zhang, Jin Gao, Weiming Hu, and Yao Yao.  Consistent4d: Consistent 360° dynamic object generation from monocular video.  In The Twelfth International Conference on Learning Representations, 2024.
  * [25] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ACM Transactions on Graphics, 42(4):1–14, 2023.
  * [26] Bing Li, Cheng Zheng, Wenxuan Zhu, Jinjie Mai, Biao Zhang, Peter Wonka, and Bernard Ghanem.  Vivid-zoo: Multi-view video generation with diffusion model.  arXiv preprint arXiv:2406.08659, 2024.
  * [27] Sixu Li, Chaojian Li, Wenbo Zhu, Boyang Yu, Yang Zhao, Cheng Wan, Haoran You, Huihong Shi, and Yingyan Lin.  Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction.  In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1–13, 2023.
  * [28] Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng.  Amt: All-pairs multi-field transforms for efficient frame interpolation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
  * [29] Hanwen Liang, Yuyang Yin, Dejia Xu, Hanxue Liang, Zhangyang Wang, Konstantinos N Plataniotis, Yao Zhao, and Yunchao Wei.  Diffusion4d: Fast spatial-temporal consistent 4d generation via video diffusion models.  arXiv preprint arXiv:2405.16645, 2024.
  * [30] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
  * [31] Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis.  Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models.  arXiv preprint arXiv:2312.13763, 2023.
  * [32] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su.  One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization.  Advances in Neural Information Processing Systems, 36, 2024.
  * [33] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023.
  * [34] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang.  Syncdreamer: Generating multiview-consistent images from a single-view image.  arXiv preprint arXiv:2309.03453, 2023.
  * [35] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  arXiv preprint arXiv:2209.14988, 2022.
  * [36] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [37] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han.  Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d.  arXiv preprint arXiv:2311.16918, 2023.
  * [38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [39] Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny.  Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction.  In International Conference on Computer Vision, 2021.
  * [40] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu.  Dreamgaussian4d: Generative 4d gaussian splatting.  arXiv preprint arXiv:2312.17142, 2023.
  * [41] Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, et al.  L4gm: Large 4d gaussian reconstruction model.  arXiv preprint arXiv:2406.10324, 2024.
  * [42] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [43] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su.  Zero123++: a single image to consistent multi-view diffusion base model, 2023.
  * [44] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  arXiv preprint arXiv:2308.16512, 2023.
  * [45] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792, 2022.
  * [46] Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al.  Text-to-4d dynamic scene generation.  arXiv preprint arXiv:2301.11280, 2023.
  * [47] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  * [48] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [49] Olga Sorkine and Marc Alexa.  As-rigid-as-possible surface modeling.  In Symposium on Geometry processing, volume 4, pages 109–116. Citeseer, 2007.
  * [50] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu.  Lgm: Large multi-view gaussian model for high-resolution 3d content creation.  arXiv preprint arXiv:2402.05054, 2024.
  * [51] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.
  * [52] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani.  Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion.  arXiv preprint arXiv:2403.12008, 2024.
  * [53] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12619–12629, 2023.
  * [54] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.  Modelscope text-to-video technical report.  arXiv preprint arXiv:2308.06571, 2023.
  * [55] Peng Wang and Yichun Shi.  Imagedream: Image-prompt multi-view diffusion for 3d generation.  arXiv preprint arXiv:2312.02201, 2023.
  * [56] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation.  Advances in Neural Information Processing Systems, 36, 2024.
  * [57] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu.  Meshlrm: Large reconstruction model for high-quality mesh.  arXiv preprint arXiv:2404.12385, 2024.
  * [58] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang.  4d gaussian splatting for real-time dynamic scene rendering.  arXiv preprint arXiv:2310.08528, 2023.
  * [59] Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu.  Freeinit: Bridging initialization gap in video diffusion models.  arXiv preprint arXiv:2312.07537, 2023.
  * [60] Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, and Xiang Bai.  Sc4d: Sparse-controlled video-to-4d generation and motion transfer.  arXiv preprint arXiv:2404.03736, 2024.
  * [61] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan.  Dynamicrafter: Animating open-domain images with video diffusion priors.  arXiv preprint arXiv:2310.12190, 2023.
  * [62] Dejia Xu, Hanwen Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang.  Comp4d: Llm-guided compositional 4d scene generation.  arXiv preprint arXiv:2403.16993, 2024.
  * [63] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan.  Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models.  arXiv preprint arXiv:2404.07191, 2024.
  * [64] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein.  Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation.  arXiv preprint arXiv:2403.14621, 2024.
  * [65] Zeyu Yang, Zijie Pan, Chun Gu, and Li Zhang.  Diffusion 2: Dynamic 3d content generation via score composition of orthogonal diffusion models.  arXiv preprint arXiv:2404.02148, 2024.
  * [66] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  arXiv preprint arXiv:2308.06721, 2023.
  * [67] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei.  4dgen: Grounded 4d content generation with spatial-temporal consistency.  arXiv preprint arXiv:2312.17225, 2023.
  * [68] Jiwen Yu, Xiaodong Cun, Chenyang Qi, Yong Zhang, Xintao Wang, Ying Shan, and Jian Zhang.  Animatezero: Video diffusion models are zero-shot image animators.  arXiv preprint arXiv:2312.03793, 2023.
  * [69] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, and Xiaoguang Han.  Mvimgnet: A large-scale dataset of multi-view images.  In CVPR, 2023.
  * [70] Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao.  Stag4d: Spatial-temporal anchored generative 4d gaussians.  arXiv preprint arXiv:2403.14939, 2024.
  * [71] Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao.  4diffusion: Multi-view video diffusion model for 4d generation.  arXiv preprint arXiv:2405.20674, 2024.
  * [72] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu.  Gs-lrm: Large reconstruction model for 3d gaussian splatting.  arXiv preprint arXiv:2404.19702, 2024.
  * [73] Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello.  A unified approach for text-and image-guided 4d scene generation.  arXiv preprint arXiv:2311.16854, 2023.
  * [74] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  arXiv preprint arXiv:1805.09817, 2018.

##  Appendix A More Visualizations

Please visit our project page to see high-resolution videos of 100+ animation
results.

##  Appendix B More Ablations

Figure 5: More ablation for multi-view video diffusion.

We provide more ablations for MV-VDM in Fig. 5.

##  Appendix C Limitations

Despite the promising performance in generating spatiotemporal consistent 4D
objects, our method still has a few limitations. First, it takes a relatively
long time (about 40 minutes) to animate an existing 3D object. Second, there
is a trade-off between temporal coherence and motion amplitude in the multi-
view videos generated from the proposed MV-VDM. Specifically, the larger the
motion amplitude, the higher the risk of temporal incoherence. Third, our
model sometimes fails to animate realistic scenes due to the domain gap
between synthetic training data and real-world test data. At last, current
evaluation metrics in 4D generation are not sufficient, as they mainly rely on
video generation metrics and user studies. Designing more suitable metrics for
4D generation will be an important future work.

##  Appendix D More Details of MV-Video Dataset

###  D.1 Rendering Details.

For the rendering settings, we first centralize the model according to the
bounding box of the object in the first frame. Then, we adjust the camera
distance to make sure the object stays in the scope of view during the
animation process. Sixteen views are evenly sampled in terms of azimuth,
starting from values randomly selected between $-11.25^{\circ}$ and
$11.25^{\circ}$. The elevation angle is randomly sampled within the range of
$0^{\circ}$ to $30^{\circ}$. To stabilize training, we manually filter out
objects that we identify as challenging to learn due to factors such as large
movements, complex environmental interactions, high speed, or sudden changes
in appearance.

###  D.2 Data Examples.

As shown in Fig. 9 and Fig. 10, we showcase more examples of our multi-view
video dataset (MV-Video).

Furthermore, as shown in Fig. 6, we extracted all nouns from the text captions
of our MV-Video dataset, which are generated by minigpt4-video [3], and
plotted a word cloud of the Top-1000 nouns. We can see that our MV-Video
dataset contains diverse categories of animated 3D objects, including humans,
characters, animals, plants, mechanical structures, and ect..

Figure 6: Illustration of the word cloud of the top 1000 nouns extracted from
the text captions of our MV-Video dataset.

##  Appendix E 4D Generation Evaluation

###  E.1 Evaluation Dataset

Figure 7: Illustration of the evaluation dataset. The first two rows are input
images for image-to-3D generation and the last two rows are renderings of
reconstructed objects. Corresponding prompts for 4D animation are provided
below the images.

In Fig. 7, we provide the input images for image-to-3D generation, renderings
of reconstructed objects, and corresponding text prompts for 4D animation used
in Sec. 4.2.

###  E.2 Evaluation Metrics

VBench [23] provides six evaluation metrics for I2V
evaluation222https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v,
i.e., I2V Subject, I2V Background, Camera Motion, Subject Consistency,
Background Consistency, Motion Smoothness, Dynamic Degree, Aesthetic Quality
and Imaging Quality. Since our generated results have no background, and the
evaluation cameras are fixed, metrics related to background and camera motion
are not used. The metric Imaging Quality, which is affected by ambient light,
is also not used here. Subject Consistency is also omitted since its
calculation process is similar to I2V Subject, except for the choice of
reference frame. It takes the first generated frame, instead of the input
image used in I2V Subject, as the reference frame for similarity score
calculation, which is not suitable for our task of animating 3D objects. We
briefly introduce the evaluation metrics used in our work as below:

I2V Subject assess whether the appearance of the object in the generated video
remains consistent with that in the input image. To this end, DINO [9] feature
similarity across frames is calculated.

Motion Smoothness evaluates whether the motion in the generated video is
smooth, and follows the physical law of the real world. The motion prior in
the video frame interpolation model [28] is utilized for evaluation.

Dynamic Degree employs RAFT [51] to estimate the degree of dynamics in
synthesized videos.

Aesthetic Quality is calculated by the LAION aesthetic predictor, which
reflects the artistic and beauty value perceived by humans towards each frame.

###  E.3 User Study Template

As illustrated in Fig. 8, a picture of the user study page is depicted. The
survey contains 20 dynamic objects, which are shown in Fig. 7. The
participants are asked to score the generated 4D objects from 1 to 5,
according to the alignment with the given static object and text prompt,
appearance quality, and motion quality.

Figure 8: The layout of our user study.

##  Appendix F Border Impacts

This paper exploited 4D generation based on our proposed multi-view video
diffusion model, which can generate spatiotemporal consistent multi-view
videos. Because of the advanced generative capacity, our models may output
misinformation or fake videos. Thus, we sincerely remind the users to pay
attention to it. Note that our method only focuses on the technical aspect.
All the code, dataset, and trained models will be released.

Figure 9: More examples of our MV-Video dataset. Figure 10: More examples of
our MV-Video dataset.