# Training-free Camera Control for Video Generation

Chen Hou1 , Guoqiang Wei2, Yan Zeng2, Zhibo Chen1
1University of Science and Technology of China, 2ByteDance
<EMAIL_ADDRESS>
{weiguoqiang.9<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Work done during internship at ByteDance.

###### Abstract

We propose a training-free and robust solution to offer camera movement
control for off-the-shelf video diffusion models. Unlike previous work, our
method does not require any supervised finetuning on camera-annotated datasets
or self-supervised training via data augmentation. Instead, it can be plugged
and played with most pretrained video diffusion models and generate camera
controllable videos with a single image or text prompt as input. The
inspiration of our work comes from the layout prior that intermediate latents
hold towards generated results, thus rearranging noisy pixels in them will
make output content reallocated as well. As camera move could also be seen as
a kind of pixel rearrangement caused by perspective change, videos could be
reorganized following specific camera motion if their noisy latents change
accordingly. Established on this, we propose our method CamTrol, which enables
robust camera control for video diffusion models. It is achieved by a two-
stage process. First, we model image layout rearrangement through explicit
camera movement in 3D point cloud space. Second, we generate videos with
camera motion using layout prior of noisy latents formed by a series of
rearranged images. Extensive experiments have demonstrated the robustness our
method holds in controlling camera motion of generated videos. Furthermore, we
show that our method can produce impressive results in generating 3D rotation
videos with dynamic content. Project page at
https://lifedecoder.github.io/CamTrol/.

##  1 Introduction

Figure 1: Training-free video camera control by CamTrol. CamTrol could handle
basic camera motions, hybrid motions and complicated trajectories with precise
camera coordinates. Besides, CamTrol produces impressive results on generating
3D rotation videos in various styles. Note that all human faces presented in
this paper are synthesized using text-to-image models.

As a more appealing and content-richer modality, videos differ from images by
including an extra temporal dimension. This temporal aspect provides increased
versatility for depicting diverse and dynamic movements, which can be
decomposed into object motion, background transitions and perspective changes.
Recent years have witnessed the rapid development and splendid breakthrough of
video generation with text prompt or images as input instructions
li2023videogen ; hong2022cogvideo ; ho2022imagenvideo ; luo2023videofusion ;
zeng2023PixelDance ; SVD ; sora ; ge2023preserve ; fei2023DysenVDM , and
demonstrated the inestimable potential of diffusion models to synthesis
realistic videos. While these video generation models have made progress in
generating videos with highly dynamic objects and backgrounds
zeng2023PixelDance ; SVD ; li2023videogen , most of them fail to provide
camera control for the generated videos.

The difficulty of controlling camera trajectory in videos primarily arises
from two aspects. The initial challenge lies in the inadequacy of annotated
data. Most video annotations lack of descriptions, especially precise
descriptions of video’s camera movements. As a result, video generation models
trained on these data often fail to interpret text prompts related to camera
motions and generate correct outputs. One solution to mitigate the data
insufficiency problem is to mimic videos with camera movements through simple
data augmentation yang2024direct . However, these methods could only handle
simple camera motions like zoom or truck, and have trouble in dealing with
more complicated ones. The second challenge lies in the effort of additional
finetuning required for controlling camera movements. As camera trajectories
could be sophisticated, they sometimes cannot be accurately elaborated using
naïve text prompts alone. Common solutions wang2023motionctrl ;
he2024cameractrl proposed to embed camera parameters into diffusion models
through learnable encoders and perform extensive finetuning on large-scale
datasets with detailed camera trajectories. However, such datasets like
RealEstate10k realestate10k and MVImageNet yu2023mvimgnet are intensively
limited in scale and diversity due to the difficulty associated with data
collection, in this way, these finetuning methods demand substantial resources
but exhibit limited generalizability to other types of data. Lack of
annotations and heavy finetuning effort make camera control a challenging task
in video generations.

In this work, we attempt to address these issues through a training-free
solution to offer camera control for off-the-shelf video diffusion models. We
begin by introducing two core observations underpin that video diffusion
models can achieve camera movement control in a training-free manner. First,
we find that base video models could produce results with rough camera moves
by integrating specific camera-related text into input prompts, such as camera
zooms in or camera pans right. This simple implementation, though not very
accurate and always leads to static or wrong motions, shows the natural prior
knowledge learnt by pretrained models about following different camera
trajectories. The other observation is the effectiveness video models have
exhibited in adapting to 3D generation tasks. Recent works sv3d ; im3d ;
shi2023mvdream find that leveraging pretrained video models as initialization
helps drastically improve the performance of multi-view generations,
demonstrating their strong ability of handling perspective change. The two
crucial observations reveals the hidden power of video models for camera
motion control, thus, we seek to find a way to evoke this innate ability, as
it already exists in the model itself.

We propose CamTrol, which offers camera control for off-the-shelf video
diffusion models in a training-free but robust manner. CamTrol is inspired by
the layout prior that noisy latents hold towards generation results: As pixels
in noisy latents change their positions, corresponding rearrangement will also
occur to the output and leads to layout modification. Considering camera moves
could also be seen as a kind of layout rearrangement, this prior can serve as
an efficacious hint providing video model with information of specific camera
motions. Specifically, CamTrol consists of a two-stage procedure. In stage I,
explicit camera movements are modeled in 3D point cloud representations and
produce a series of rendered images indicating specific camera movements. In
stage II, layout prior of noisy latents are utilized to guide video
generations with camera movements. We conduct extensive experiments to
validate the effectiveness of our proposed CamTrol. Both quantitative and
qualitative results demonstrate the robustness of CamTrol as a useful tool of
controlling camera motion for video diffusion models. Furthermore, we show
that CamTrol produces impressive results of dynamic 3D rotation videos in
various styles.

##  2 Related Work

#### Camera Control for Video Generation

While methods aim for controlling video foundation models constantly emerge
ma2024magic ; liu2023video ; feng2023ccedit , there are few works explore how
to manipulate camera motion of generated videos. Earlier work
hao2018controllable controls motion trajectory via warping image through
densified sparse flow and pixel fusion, similar ideas also appear later in
chen2023motion ; yin2023dragnuwa . Besides utilizing optical flow, two main
techniques for implementing video camera control are via self-supervised
augmentation or additional finetuning. yang2024direct disentangles object
motion with camera movement and incorporates extra layers to embed camera
motions, where model is trained in a self-supervised manner by augmenting
input videos to stimulate simple camera movements. he2024cameractrl and
wang2023motionctrl train an additional camera encoder and integrate the output
into temporal attention layers of U-Net. guo2023animatediff learns new motion
pattern via LoRA hu2021lora and finetuning with multiple reference videos.

#### Noise Prior of Latents in Diffusion Model

One of the most natural advantages of diffusion model comes from its pixel-
wise noisy latents formed during denoising process. These latents hold strong
causuality towards output and directly determine what the result looks like,
meanwhile have robust error-resilience as they are perturbed by Gaussian
noises across different scales. Numerous work have exploited the convenience
of this noise prior to attain controllable generation, such as image-to-image
translation meng2021sdedit , pixel-level manipulation nichol2022glide ;
avrahami2022blended , image inpainting lugmayr2022repaint and semantic editing
choi2021ilvr ; kim2022diffusionclip ; hou2024high . Recent research has shown
that even sampled from Gaussian distribution, the initial noise of diffusion
process still have significantly influence to the layout of generated contents
mao2023guided . In other work, noise prior is used to guarantee temporal
consistency among video frames luo2023videofusion , or to trade-off between
fidelity and diversity of image editing kim2022diffusionclip .

#### Video Model for 3D Generation

Similar to how most video generation models using the groundwork laid by image
foundation models blattmann2023alignVideo-LDM ; esser2023GEN1 ; singer2022make
; tuneavideo , training of 3D generation model also relies heavily on
pretrained 2D video models sv3d ; im3d ; shi2023mvdream ; chen2024v3d ;
han2024vfusion3d . These methods either finetunes with rendered videos
directly SVD ; chen2024v3d ; im3d ; han2024vfusion3d , or adds camera
embedding for each view as extra condition sv3d ; shi2023mvdream . Video
foundation models have shown to be particularly beneficial in generating
consistent multi-view rendering of 3D objects, demonstrates their inherent
abundant prior knowledge for handling camera pose change.

##  3 Training-free Camera Control for Video Generation

CamTrol takes two stages to evoke the innate camera control ability hidden in
base video models. In Sec. 3.1, we will describe how to model explicit camera
motion in point cloud space. In Sec. 3.2, we will elaborate on the camera-
controllable video generation with the guidance of noise layout prior.

Figure 2: Pipeline of CamTrol. In stage I, camera movements are modeled
through explicit 3D point cloud. In stage II, layout prior of noisy latents
are utilized to guide video generation.

###  3.1 Camera Motion Modeling

To evoke pretrained video diffusion model’s ability of dealing with camera
perspective changes, hints of camera motion should be injected to diffusion
model in a proper way. While simply concatenating camera trajectories with
text prompt is incomprehensible for original model, previous works
wang2023motionctrl ; he2024cameractrl introduce additional embedder to encode
camera parameters and finetune with limited annotated data realestate10k ;
yu2023mvimgnet , which are data-hungry yet lack of generalization ability.
Other methods yang2024direct construct camera motions by self-supervised
augmentations, but could only handle a few easy camera controls. Thus, we seek
a more efficient and robust way to guide the model towards camera
controllable.

Considering perspective change of video is originally caused by camera
movements in 3D space, we resort to 3D representation for providing explicit
motion hints to video diffusion models. Specifically, we choose point cloud as
the intermediate representation, in which space we can expediently manipulate
camera poses and positions for simulating diverse camera movements. Besides
explicit camera modeling, introducing point cloud brings extra benefits: The
first is its data-efficiency. By utilizing inpainting techniques, only one
single input image is required for the whole point cloud reconstruction, this
sidesteps the effort of large-scale finetuning. Second, consistency between
multi-view renderings can be easily ensured, as the known points will remain
unchanged once the reconstruction finished.

#### Point Cloud Initialization

We start by lifting the pixels in input image plane to 3D point cloud
representations. In practice, the input image can be either user-defined or
created by image generators like Stable Diffusion rombach2022high . Given an
input image $\mathbf{I}_{0}\in\mathbb{R}^{3\times H\times W}$, we first
estimate its depth map $\mathbf{D}_{0}$ using off-the-shelf monocular depth
estimator ZeoDepth bhat2023zoedepth . By combining image and its depth map,
point cloud $\mathcal{P}_{0}$ can be initialized as:

$\mathcal{P}_{0}=\phi([\mathbf{I}_{0},\mathbf{D}_{0}],\mathbf{K},\mathbf{P}_{0}),$
(1)

where $\phi$ denotes the mapping function from RGBD to 3D point cloud,
$\mathbf{K}$ and $\mathbf{P}_{0}$ represent camera’s intrinsic and extrinsic
matrices set by convention chung2023luciddreamer as they’re usually
intractable.

#### Camera Trajectories

To get consistent images from multiple viewpoints, we set camera motion as a
pre-defined trajectory of extrinsic matrix
$\left\\{\mathbf{P}_{1},...,\mathbf{P}_{N-1}\right\\}$, each of which includes
a rotation matrix and translation matrix representing camera’s pose and
position. At each step $i$, we project the point cloud back to camera plane
using function $\psi$ and get a rendered image with perspective change:
$\mathbf{I}_{i}=\psi(\mathcal{P}_{i},\mathbf{K},\mathbf{P}_{i})$. By
calculating extrinsic matrices of corresponding movement, we obtain a series
of camera motions including zoom, tilt, pan, pedestal, truck, roll and rotate,
enabling flexible camera movements. Detailed definitions of these movements
are elaborated in Appendix A.1. By combining basic trajectories, hybrid camera
movements can be attained and produce videos with cinematic charm. What’s
more, benefit from explicit camera motion modeling, our method could support
trajectories with precise coordinates, which means it can generate videos with
any complicated camera motion (Fig. 1).

#### Multi-view Rendering

When perspective changes, there can be vacancies appear as some areas are
unoccupied within the point cloud. To get more reasonable results, we employ
image inpainting model rombach2022high to fill up the holes for new
renderings, with a mask distinguishing the known points from nonexistent ones.
After inpainted in 2D space, image is lifted again onto 3D space and gradually
complete the whole point cloud representation. During this process,
misalignment between adjacent views may occur since depth estimator only
estimates relative depth, further leads to inconsistency of rendered images.
We adopt depth coefficient optimization chung2023luciddreamer to avoid this
situation, which can be formed as:

$d_{i}=\operatorname*{argmin}_{d}\left(\sum_{M}\left\|\phi([\tilde{\mathbf{I}}_{i},d\tilde{\mathbf{D}}_{i}],\mathbf{K},\mathbf{P}_{i})-\mathcal{P}_{i-1}\right\|\right),$
(2)

where $\tilde{\mathbf{I}}_{i}$ and $\tilde{\mathbf{D}}_{i}$ refer to the
inpainted image and its depth map respectively, $d_{i}$ denotes depth
coefficient to be optimized, and $M$ refers to the overlapping region between
$\mathcal{P}_{i}$ and $\mathcal{P}_{i-1}$, as other areas are not shared for
calculating $\ell_{1}$ loss.

Thus, we get a set of images refer to the input and indicate specific camera
movement:

$\left\\{\mathbf{I}_{0},...,\mathbf{I}_{N-1}\right\\}=\left\\{\psi(\mathcal{P}_{i},\mathbf{K},\mathbf{P}_{i})|i\in\left[0,N-1\right]\right\\}.$
(3)

###  3.2 Layout Prior of Noise

With camera motion modeling, we obtain a sequence
$\mathbf{V}_{0}=\left[\mathbf{I}_{0},...,\mathbf{I}_{N-1}\right]\in\mathbb{R}^{N\times
3\times H\times W}$ of rendered images adhering to a specific camera
trajectory. Note that quality of rendered images are not perfect as single
input image only leads to sparse point cloud reconstruction, besides, these
renderings are static, thus they could not use directly as video frames. To
form an ideal video, we need to find a way that satisfies the following three
requirements: 1) camera motions should be maintained; 2) video should be
encouraged with more dynamics; and 3) quality imperfection should be
compensated.

#### Camera Motion Inversion

Recent work on diffusion models have demonstrated the strong controllability
of its noisy latents meng2021sdedit ; mao2023guided , the causality and error-
resilience they hold towards final output make them a convenient yet powerful
tool for controllable generation of diffusion models. Particularly for initial
noise, even sampled from Gaussian distribution, it still have significant
influence on the layout of generated image mao2023guided , so that rearranging
the noise pixels will make content in output relocate as well. For instance,
if all pixels in initial noise shift to right by a certain distance, it is
likely that generated output reflect a similar shift. This reminds us that the
impact of camera movement on images could also be regarded as a kind of layout
rearrangement, where pixels change their positions caused by viewpoint change.
In a similar way, videos can be reorganized following camera motion if their
noisy latents change accordingly.

Inspired by this, we first construct a series of noisy latents indicating
corresponding camera movements. It can be intuitively done by employing
diffusion model’s inversion process on rendered image sequence
$\mathbf{V}_{0}$. Latent at timestep $t_{0}$ can be calculated as:

$\mathbf{V}_{t_{0}}=\sqrt{\bar{\alpha}_{t_{0}}}\mathbf{V}_{0}+\sqrt{1-\bar{\alpha}_{t_{0}}}\epsilon,\quad\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$
(4)

where $\bar{\alpha}_{t}$ are variances used in DDIM song2020DDIM scheduler.
Because the rendered images $\mathbf{V}_{0}$ share common pixels in certain
regions, their latents also have relevance to each other in a way indicating
pixels’ move. Moreover, while being perturbed with random noise, blank spaces
and flawed regions in $\mathbf{V}_{0}$ can be further filled with randomness,
providing video model with more possibilities to generate and correct them.

#### Video Generation

After camera motion inversion, noisy latents presenting camera movements are
then passed through the backward process of video diffusion model, utilizing
their layout controllability to guide video generation. Leveraging prior
knowledge of base video model, the generation process also bestows video with
rational dynamic information. In this way, explicit camera movements are
injected into video diffusion model in an appropriate and training-free
fashion. Starting from noisy motion latents at timestep $t_{0}$, the
generation step can be represented as:

$\hat{\mathbf{V}}_{t-1}=\sqrt{\alpha_{t-1}}\left(\frac{\mathbf{V}_{t}-\sqrt{1-\alpha_{t}}\epsilon^{(t)}_{\theta}(\mathbf{V}_{t})}{\sqrt{\alpha_{t}}}\right)+\sqrt{1-\alpha_{t-1}-\sigma^{2}_{t}}\epsilon^{(t)}_{\theta}(\mathbf{V}_{t})+\sigma_{t}\epsilon,\quad
t\in\left[1,t_{0}\right].$ (5)

Here we use DDIM as sampling scheduler song2020DDIM and $\epsilon_{\theta}$
denotes the video model for noise prediction. $\sigma_{t}$ determines whether
the denoising process is deterministic or probabilistic, we set $\sigma=1$ to
encourage diversity of generation results.

#### Trade-off Between Fidelity and Diversity

Leveraging noise prior guidance in diffusion model could lead to trade-off
problem between generation’s fidelity and diversity meng2021sdedit ;
kim2022diffusionclip ; hou2024high , where results that hold more faithfulness
towards guidance tend to decline in generation quality or diversity. In this
task, similar circumstance also occurs as model is required to receive
guidance from a series of imperfect renderings while generate a reasonable and
dynamic video. The key factor to balance the trade-off problem lies in the
choice of $t_{0}$. When larger $t_{0}$ applied, generation bears more
resemblance to original guidance $\mathbf{V}_{0}$, yet lacks of rationality
and dynamics to be an appealing video. Instead, smaller $t_{0}$ leads to well-
generated video, but is less aligned with desired camera motion. In our
experiments, we find larger $t_{0}$ works better for motions with moderate
intensity, and for those with relatively drastic move, smaller $t_{0}$ shows
preferable performance.

##  4 Experiments

###  4.1 Experimental Settings

#### Implementation Details

Our work is based on zeng2023PixelDance with 16 frames at resolution
256$\times$256, which is trained on public data. Note that our method could be
incorporated with most video diffusion models to achieve camera control, thus
we also include results performed on other backbones SVD ; luo2023videofusion
. For fair comparison with other methods, we set all model’s DDIM steps as 25.
Our method doesn’t require any additional training utilizing camera
trajectories.

#### Evaluation Details

We compare CamTrol with state-of-the-art works: AnimateDiff guo2023animatediff
, MotionCtrl wang2023motionctrl and CameraCtrl he2024cameractrl , all three
methods finetune on specific data. For quantitative evaluation, FVD
unterthiner2018FVD , FID heusel2017FID and IS saito2020IS are used to assess
video generation quality, while CLIPSIM wu2021CLIPSIM quantifies the
similarity between generated video and input prompt. The evaluations are
performed on 8 basic camera trajectories, details could be found in Appendix
A.1. For evaluation dataset, we follow similar setting in MotionCtrl
wang2023motionctrl , but with more complex text prompt. Concretely, we apply
UCF-101 soomro2012UCF as the reference of calculating FID and FVD, where we
randomly sample 125 prompts per camera motion type and form 1000 samples in
total for assessment. We also conduct a user study to compare different
methods with human evaluation. Details can be found in Appendix A.5.

###  4.2 Comparisons with State-of-the-art Methods

#### Quantitative Evaluation

Quantitative evaluations are shown in Table 1. Compared to previous state-of-
the-art work, CamTrol demonstrates superior performance across most
quantitative metrics. As large-stride camera move sometimes causes deformation
on objects, there appears little decrease in single frame’s quality, i.e. FID.
While method like AnimateDiff guo2023animatediff exhibits better performance
on this term, it takes as sacrifice the dynamics of generated video, which we
will see in qualitative evaluations. For camera motion control, human
evaluations from user study indicate that training-free CamTrol attains
comparable performance even compared to other training-based methods, where
the video diffusion models are heavily finetuned on specific camera motion
data.

Table 1: Quantitative comparison. Experiments are conducted on 8 basic camera trajectories. Method | Metric | User Study
---|---|---
FVD $\left(\downarrow\right)$ | FID $\left(\downarrow\right)$ | IS $\left(\uparrow\right)$ | CLIP-SIM $\left(\uparrow\right)$ |  Camera $\left(\uparrow\right)$ | Quality $\left(\uparrow\right)$
AnimateDiff guo2023animatediff | 3699.34 | 65.81 | 11.61 | 0.3073 | 1.12 | 0.82
MotionCtrl wang2023motionctrl | 1768.25 | 78.26 | 14.60 | 0.3143 | 2.00 | 2.03
CameraCtrl he2024cameractrl | 1383.68 | 76.18 | 14.04 | 0.3123 | 1.01 | 1.08
CamTrol | 1134.71 | 69.70 | 15.87 | 0.3253 | 1.86 | 2.06
Figure 3: Qualitative comparison with finetune methods. CamTrol generates
videos that adhere to desired camera movements, without compromising their
dynamics and rationality.

#### Qualitative Analysis

Qualitative comparison of different methods is illustrated in Fig. 3. All
methods generate videos align well with desired camera motion, however, the
constraints imposed by camera trajectories are sometimes too strong, that
output videos have to compromise their quality and rationality to satisfy
these movements. Some examples that demonstrate this are results of Zoom Out
and Roll Anticlockwise, which looks more like flat images being zoomed or
rolled on 2D plane instead of videos recorded in 3D spaces. The reason might
lies on the limitations of proprietary dataset used for finetuning
realestate10k ; yu2023mvimgnet , where most scenes are static, making
pretrained model forget about knowledge of dynamic contents. In contrast,
CamTrol makes full use of the powerful dynamics in base model and generate
plausible videos with camera perspective change utilizing layout prior
guidance, handles these circumstances gracefully. More comparisons at Appendix
A.3.

###  4.3 Ablation Study

Figure 4: Comparisons with base model. Controlling camera motion via prompt
engineering doesn’t work at most times. Instead, CamTrol offers robust control
towards video’s camera movement in a training-free manner. Figure 5:
Effectiveness of layout prior.  Layout prior guidance compensates for the
vacancies (Left) and static contents (Right) caused by point cloud rendering.

#### Comparison to Base Model

To demonstrate that changes of camera motion are attributed to our method
rather than the innate capability of video base model, we conduct ablation
study to assess its effectiveness. We add prompts describing relevant camera
moves such as camera rotates around it or camera zooms out and let video model
understand by itself. The results are shown in Fig. 4. We can observe that
even provided with prompts indicating how camera should move, base model fail
to produce correct results. Instead, CamTrol is able to achieve corredponding
motion control without any instructions from input prompts.

#### Effectiveness of Layout Prior

We employ ablation study to validate the effectiveness of layout prior
guidance, and illustrate its necessity from two aspects: the completeness and
dynamic of generated video. In Fig. 5, we showcase frames before and after
noise prior guidance. With the position of camera changes, there appears
regions unfilled in point cloud and causes blank spaces in rendered images
(left part); Besides, due to the static nature of point cloud, rendered images
remain stationary too (right part). Noise layout prior could compensate for
these flaws, finally produce videos with inpainted vacancies and rationalized
dynamics.

#### Effect of Timestep $t_{0}$

$t_{0}$ is a crucial factor that influences the trade-off between generated
video’s diversity and its faithfulness to camera motions requirements. To
investigate its effect on the output, we conduct experiments with various
$t_{0}$ values, relevant results are shown in Fig. 7. As illustrated, videos
generated with larger $t_{0}$ tend to conform better to camera motion
requirements, but suffer from decrease in dynamics; On the contrary, smaller
$t_{0}$ leads to more plausible generations but fails to meet camera’s
requirements, as latents at these timesteps carry more randomness.

#### Generalization to Diverse Base Models

Our proposed CamTrol can be seamlessly plugged and played with most pretrained
video diffusion models to achieve training-free camera control. We test its
performance on different base video models, including SVD SVD and VideoFusion
luo2023videofusion , the results are illustrated in Fig. 8. Our approach
remains effective applied to other video base models, demonstrating its strong
robustness and generalization ability. More results with other base models can
be found in Appendix A.4.

###  4.4 Further Applications

Figure 6: Effect of $t_{0}$. Smaller $t_{0}$ encourages
dynamics while larger $t_{0}$ preserves camera
movements (Pedestal Down).

Figure 7: Camera control at different scales. CamTrol supports camera
movements over various scales.

Figure 8: Applied onto SVD SVD and VideoFusion luo2023videofusion . CamTrol
could be plugged and played with most video diffusion models to offer camera
movement control.

#### 3D Rotation Videos

One of the most advantages of our method is it can generate videos rotating
around some objects and produce outputs similar to 3D generation models sv3d ;
im3d . While these 3D models need large-scale training on 3D dataset and could
only handle inputs in specific styles, our approach is able to deal with any
type of images and achieve this in a completely zero-shot manner. Some results
are shown in 1. We offer more results on multi-view synthesis in Appendix A.2.

#### Hybrid and Complex Camera Movements

By combining different basic camera trajectories, CamTrol can support hybrid
camera movements and endow generated video with cinematic charm. Besides
hybrid control, benefit from explicit camera motion modeling in 3D space,
CamTrol could support trajectories with precise coordinates, which means it
can generate videos with any complicated motion. Results about hybrid and
complex camera motions are shown in Fig. 1 and Appendix A.2.

#### Camera Motion at Different Scales

CamTrol supports camera movements at controllable scales. We provide some
results in Fig. 7. By specifying different magnitudes of camera’s extrinsic
matrix within point cloud spaces, rendered images will exhibit varying degrees
of motion, leading to videos with distinct scales of camera movements. This
further demonstrate the powerful controllability of CamTrol, and provides new
pathway for video’s customized camera control.

##  5 Conclusion

In this paper, we propose a training-free and robust method CamTrol to offer
camera control for off-the-shelf video diffusion models. It consists of two-
stage procedure including explicit camera motion modeling in 3D point cloud
space and video generation utilizing layout prior of noisy latents. Compared
to previours work, CamTrol does not require any additional finetuning on
camera-annotated datasets, or self-supervised training via data augmentation,
instead, it could be plugged and played with most video diffusion models to
generate camera controllable videos with single image or text prompt as input.
Comprehensive experimental results demonstrate the effectiveness and
robustness of CamTrol for controlling camera motion for videos. Besides, we
show that CamTrol produces impressive results in generating videos at
different scales, with 3D rotations and more complicated trajectories.

## References

  * [1] O. Avrahami, D. Lischinski, and O. Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [2] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  arXiv preprint arXiv:2302.12288, 2023.
  * [3] A. Blattmann, T. Dockhorn, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti, A. Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  arXiv preprint arXiv:2311.15127, 2023.
  * [4] A. Blattmann, R. Rombach, H. Ling, T. Dockhorn, S. W. Kim, S. Fidler, and K. Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023.
  * [5] T. Brooks, B. Peebles, C. Homes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, et al.  Video generation models as world simulators, 2024.
  * [6] T.-S. Chen, C. H. Lin, H.-Y. Tseng, T.-Y. Lin, and M.-H. Yang.  Motion-conditioned diffusion model for controllable video synthesis.  arXiv preprint arXiv:2304.14404, 2023.
  * [7] Z. Chen, Y. Wang, F. Wang, Z. Wang, and H. Liu.  V3d: Video diffusion models are effective 3d generators.  arXiv preprint arXiv:2403.06738, 2024.
  * [8] J. Choi, S. Kim, Y. Jeong, Y. Gwon, and S. Yoon.  Ilvr: Conditioning method for denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14367–14376, 2021.
  * [9] J. Chung, S. Lee, H. Nam, J. Lee, and K. M. Lee.  Luciddreamer: Domain-free generation of 3d gaussian splatting scenes.  arXiv preprint arXiv:2311.13384, 2023.
  * [10] P. Esser, J. Chiu, P. Atighehchian, J. Granskog, and A. Germanidis.  Structure and content-guided video synthesis with diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7346–7356, 2023.
  * [11] H. Fei, S. Wu, W. Ji, H. Zhang, and T.-S. Chua.  Empowering dynamics-aware text-to-video diffusion with large language models.  arXiv preprint arXiv:2308.13812, 2023.
  * [12] R. Feng, W. Weng, Y. Wang, Y. Yuan, J. Bao, C. Luo, Z. Chen, and B. Guo.  Ccedit: Creative and controllable video editing via diffusion models.  arXiv preprint arXiv:2309.16496, 2023.
  * [13] S. Ge, S. Nah, G. Liu, T. Poon, A. Tao, B. Catanzaro, D. Jacobs, J.-B. Huang, M.-Y. Liu, and Y. Balaji.  Preserve your own correlation: A noise prior for video diffusion models.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22930–22941, 2023.
  * [14] Y. Guo, C. Yang, A. Rao, Z. Liang, Y. Wang, Y. Qiao, M. Agrawala, D. Lin, and B. Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  In The Twelfth International Conference on Learning Representations, 2023.
  * [15] J. Han, F. Kokkinos, and P. Torr.  Vfusion3d: Learning scalable 3d generative models from video diffusion models.  arXiv preprint arXiv:2403.12034, 2024.
  * [16] Z. Hao, X. Huang, and S. Belongie.  Controllable video generation with sparse trajectories.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7854–7863, 2018.
  * [17] H. He, Y. Xu, Y. Guo, G. Wetzstein, B. Dai, H. Li, and C. Yang.  Cameractrl: Enabling camera control for text-to-video generation.  arXiv preprint arXiv:2404.02101, 2024.
  * [18] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  Advances in neural information processing systems, 30, 2017.
  * [19] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [20] W. Hong, M. Ding, W. Zheng, X. Liu, and J. Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  arXiv preprint arXiv:2205.15868, 2022.
  * [21] C. Hou, G. Wei, and Z. Chen.  High-fidelity diffusion-based image editing.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2184–2192, 2024.
  * [22] E. J. Hu, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al.  Lora: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2021.
  * [23] G. Kim, T. Kwon, and J. C. Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [24] X. Li, W. Chu, Y. Wu, W. Yuan, F. Liu, Q. Zhang, F. Li, H. Feng, E. Ding, and J. Wang.  Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation.  arXiv preprint arXiv:2309.00398, 2023.
  * [25] S. Liu, Y. Zhang, W. Li, Z. Lin, and J. Jia.  Video-p2p: Video editing with cross-attention control.  arXiv preprint arXiv:2303.04761, 2023.
  * [26] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022.
  * [27] Z. Luo, D. Chen, Y. Zhang, Y. Huang, L. Wang, Y. Shen, D. Zhao, J. Zhou, and T. Tan.  Videofusion: Decomposed diffusion models for high-quality video generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10209–10218, 2023.
  * [28] Z. Ma, D. Zhou, C.-H. Yeh, X.-S. Wang, X. Li, H. Yang, Z. Dong, K. Keutzer, and J. Feng.  Magic-me: Identity-specific video customized diffusion.  arXiv preprint arXiv:2402.09368, 2024.
  * [29] J. Mao, X. Wang, and K. Aizawa.  Guided image synthesis via initial image editing in diffusion model.  In Proceedings of the 31st ACM International Conference on Multimedia, pages 5321–5329, 2023.
  * [30] L. Melas-Kyriazi, I. Laina, C. Rupprecht, N. Neverova, A. Vedaldi, O. Gafni, and F. Kokkinos.  Im-3d: Iterative multiview diffusion and reconstruction for high-quality 3d generation.  arXiv preprint arXiv:2402.08682, 2024.
  * [31] C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [32] A. Q. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. Mcgrew, I. Sutskever, and M. Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In International Conference on Machine Learning, pages 16784–16804. PMLR, 2022.
  * [33] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [34] M. Saito, S. Saito, M. Koyama, and S. Kobayashi.  Train sparsely, generate densely: Memory-efficient unsupervised training of high-resolution temporal gan.  International Journal of Computer Vision, 128(10-11):2586–2606, 2020.
  * [35] Y. Shi, P. Wang, J. Ye, M. Long, K. Li, and X. Yang.  Mvdream: Multi-view diffusion for 3d generation.  arXiv preprint arXiv:2308.16512, 2023.
  * [36] U. Singer, A. Polyak, T. Hayes, X. Yin, J. An, S. Zhang, Q. Hu, H. Yang, O. Ashual, O. Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792, 2022.
  * [37] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [38] K. Soomro, A. R. Zamir, and M. Shah.  A dataset of 101 human action classes from videos in the wild.  Center for Research in Computer Vision, 2(11), 2012.
  * [39] T. Unterthiner, S. Van Steenkiste, K. Kurach, R. Marinier, M. Michalski, and S. Gelly.  Towards accurate generative models of video: A new metric & challenges.  arXiv preprint arXiv:1812.01717, 2018.
  * [40] V. Voleti, C.-H. Yao, M. Boss, A. Letts, D. Pankratz, D. Tochilkin, C. Laforte, R. Rombach, and V. Jampani.  Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion.  arXiv preprint arXiv:2403.12008, 2024.
  * [41] Z. Wang, Z. Yuan, X. Wang, T. Chen, M. Xia, P. Luo, and Y. Shan.  Motionctrl: A unified and flexible motion controller for video generation.  arXiv preprint arXiv:2312.03641, 2023.
  * [42] C. Wu, L. Huang, Q. Zhang, B. Li, L. Ji, F. Yang, G. Sapiro, and N. Duan.  Godiva: Generating open-domain videos from natural descriptions.  arXiv preprint arXiv:2104.14806, 2021.
  * [43] J. Z. Wu, Y. Ge, X. Wang, S. W. Lei, Y. Gu, Y. Shi, W. Hsu, Y. Shan, X. Qie, and M. Z. Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7623–7633, October 2023.
  * [44] S. Yang, L. Hou, H. Huang, C. Ma, P. Wan, D. Zhang, X. Chen, and J. Liao.  Direct-a-video: Customized video generation with user-directed camera movement and object motion.  arXiv preprint arXiv:2402.03162, 2024.
  * [45] S. Yin, C. Wu, J. Liang, J. Shi, H. Li, G. Ming, and N. Duan.  Dragnuwa: Fine-grained control in video generation by integrating text, image, and trajectory.  arXiv preprint arXiv:2308.08089, 2023.
  * [46] X. Yu, M. Xu, Y. Zhang, H. Liu, C. Ye, Y. Wu, Z. Yan, C. Zhu, Z. Xiong, T. Liang, et al.  Mvimgnet: A large-scale dataset of multi-view images.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9150–9161, 2023.
  * [47] Y. Zeng, G. Wei, J. Zheng, J. Zou, Y. Wei, Y. Zhang, and H. Li.  Make pixels dance: High-dynamic video generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
  * [48] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely.  Stereo magnification: Learning view synthesis using multiplane images.  arXiv preprint arXiv:1805.09817, 2018.

##  Appendix A Appendix

###  A.1 Definitions of Basic Camera Motions

We refer to the terminology in cinematography to describe different camera
motions, definitions of each type are detailed in Table 2.

Table 2: Definitions of basic camera motions. Camera Motion | Directions | Definition
---|---|---
Zoom | In | Camera moves towards or away from a subject.
Out
Tilt | Up | Rotating the camera vertically from a fixed position.
Down
Pan | Left | Rotating the camera horizontally from a fixed position.
Right
Pedestal | Up | Moving a camera vertically in its entirety.
Down
Truck | Left | Moving a camera horizontally in its entirety.
Right
Roll | Clockwise | Rotating a camera in its entirety in a horizontal manner.
Anticlockwise
Rotate | Clockwise | Moving a camera around a subject.
Anticlockwise
Hybrid | Arbitrary | Combination of other motions.

###  A.2 More Results on Camera Control

In this section, we showcase additional qualitative results of CamTrol on
various camera movements. Fig. 9 shows video frames with camera move Rotate
Anticlockwise and Rotate Clockwise. These outputs share sort of similarity
with outputs of 3D generation models, as they all exhibit in a turning-table
like way, which camera rotates around some objects. The difference here is
that 3D model, as only trained on specific datasets, could only generate
outputs in certain styles, e.g., single static object with no background.
Instead, our model could handle arbitrary image as input, and generate a
rotating-around video with proper dynamics. From this aspect, our method could
be seen as a infinite source of attaining 3D data. And by utilizing our method
with stronger backbones, video foundation models could truly become the
largest source of 3D data as it should be.

Besides, we provide additional results on hybrid motions and basic motions,
which is illustrated in Fig. 10, Fig. 11 and Fig. 13, respectively. We also
present pre-loaded complex camera motions in Fig. 12.

###  A.3 More Comparisons to State-of-the-Art Methods

More comparison results with state-of-the-art methods are shown in Fig. 14 and
Fig. 15. Compared to other works, our method matches well with input texts and
camera requirements, meanwhile has wide-ranging diversities on generated
contents.

###  A.4 More Results on other Base Video Models

Our methods could be plugged and played in arbitrary video diffusion models to
achieve camera control. We test its performance on two other video generation
backbones including SVD SVD and VideoFusion luo2023videofusion , relevant
results are shown in Fig. 16.

###  A.5 Details of User Study

In user study, we collect 7 text prompts for each of the 8 basic camera
trajectories, yielding 56 samples in total. Users are asked to sort randomly
displayed four videos generated by four methods separately, and rate them from
the highest score to the lowest score (belongs to {3,2,1}, where higher score
indicates better performance). The assess metrics including two aspects:
alignment of video move with input camera motion, and video quality (such as
the deformation, rationality and diversity) of generated contents during
camera moves. After rating, we calculate the average score attained by each
methods.

The prompts applied in user study are ensured with balance between different
categories. We involve prompts from various contents and styles including
natural scenes, human, animations, objects, animals and etc.

Prompts applied in user study are listed as below:

  1. 1.

Blueberries were squeezed by fingers.

  2. 2.

The bear dolls were dancing in a circle, with lights flashing behind them,
Christmas atmosphere.

  3. 3.

The wind is blowing, and the flames are gradually extinguishing.

  4. 4.

A young hero sets out on a journey to find their missing parent, encountering
friends and foes along the way.

  5. 5.

A toy robot discovers it can move on its own and sets off on an adventure
around the house.

  6. 6.

A determined ant leads its colony on a journey to find a new home.

  7. 7.

The Great Wall of China winds its way across rugged mountains and through lush
valleys, a testament to ancient engineering.

  8. 8.

The bear doll in the basket is jumping out from the basket.

  9. 9.

My armor pieces kept fluctuating and turned into scale-like armor.

  10. 10.

A little boy turned a page of the book.

  11. 11.

Jellyfish swim in the deep sea.

  12. 12.

The big tree growing by the sea, its leaves slowly turn yellow and begin to
wither and fall.

  13. 13.

At the top of a mountain, a hiker is greeted by a breathtaking view of snow-
capped peaks and rolling valleys below.

  14. 14.

A group of squirrels gather around a picnic blanket, nibbling on sandwiches
and stealing sips of lemonade.

  15. 15.

The road is changing strangely and disappearing and twisting.

  16. 16.

Clouds and fog linger on the tops of mountains.

  17. 17.

Easter eggs showed different expressions, some happy, some confused, some sad,
and some angry.

  18. 18.

A skilled dancer twirls gracefully across the stage, captivating the audience
with every movement.

  19. 19.

A curious cat discovers a hidden treasure in the backyard.

  20. 20.

At the beach, children build sandcastles while seagulls swoop overhead, and
families relax under colorful umbrellas.

  21. 21.

In a bustling marketplace, vendors call out their wares as shoppers weave
through the crowds, browsing colorful stalls.

  22. 22.

On the busy streets, people come and go.

  23. 23.

Sunlight filters through dense foliage in a tranquil grove.

  24. 24.

A toy monkey sits at a pretend computer, typing away furiously as it pretends
to work on important documents.

  25. 25.

The Eiffel Tower stands tall against a backdrop of Parisian rooftops, its iron
lattice illuminated by the setting sun.

  26. 26.

The Colosseum in Rome stands as a majestic reminder of ancient gladiator
battles and Roman history.

  27. 27.

A cozy reading nook by a window.

  28. 28.

A regal statue of a historical figure stands tall in a city square, gazing out
over the bustling streets.

  29. 29.

Let them perform a dance.

  30. 30.

The goose is playing trumpet. slow motion. 2D animation.

  31. 31.

The machine is making Spider-Man’s hood.

  32. 32.

The girl danced in the water, splashing water and her hair fluttering the
wind.

  33. 33.

The man looked up at the peak.

  34. 34.

A woman reading and taking notes.

  35. 35.

A group of penguins are walking towards you.

  36. 36.

A shimmering portal appears in the midst of a moonlit forest, leading to
realms unknown.

  37. 37.

In a bustling port, cranes swing cargo containers onto waiting ships,
orchestrating a dance of logistics and trade.

  38. 38.

At dawn, the sky ignites in a riot of colors over the tranquil expanse of a
mist-covered lake.

  39. 39.

A touch of enchanted ink brings the pages of an ancient spellbook to life,
casting spells with each turn.

  40. 40.

A travel blogger documents their journey around the world, sharing tales of
exotic destinations and unforgettable experiences.

  41. 41.

A playful puppy learns to fetch a stick for the first time.

  42. 42.

In a cozy living room, friends gather around a crackling fireplace, sipping
hot cocoa and sharing stories on a chilly evening.

  43. 43.

A man fell asleep amid the flowers, roses swaying in the wind.

  44. 44.

A tutorial video guides viewers through the step-by-step process of creating
intricate origami designs.

  45. 45.

Underneath a starry sky, the Northern Lights dance in shimmering curtains of
green and purple, painting the night with magic.

  46. 46.

A cluttered desk in a home office, with papers strewn about and a computer
humming softly.

  47. 47.

A towering skyscraper gleams in the sunlight, its sleek glass facade
reflecting the surrounding cityscape.

  48. 48.

A modernist home with clean lines and geometric shapes, surrounded by
minimalist landscaping.

  49. 49.

A children’s playground is a kaleidoscope of colors, with slides, swings, and
climbing structures painted in playful shades.

  50. 50.

Thunder and rain, ancient oriental dragons hover in the air.

  51. 51.

A dog barking at you.

  52. 52.

Multiple people raised their arms and struggled upward in the lava of hell.

  53. 53.

An astronaut aboard a space station encounters a mysterious anomaly while
conducting experiments.

  54. 54.

A guitarist strums their instrument with gentle fingers, the melody flowing
naturally from their movements.

  55. 55.

A snow-capped mountain peak towers above a tranquil alpine lake, mirrored
perfectly in its glassy surface.

  56. 56.

A rainbow arcs across the sky after a refreshing rain shower, its vibrant
colors painting a stunning backdrop against the clouds.

Figure 9: Results with camera move Rotate Anticlockwise and Rotate Clockwise.
Our method could generated 3D rotation-like videos with dynamics. Figure 10:
Results with camera move Hybrid I. Here we combine Zoom Out, Truck Left,
Pedestal Up, Pan Right and Tilt Down together, form a cinematic like effect on
generated videos. Figure 11: Results with camera move Hybrid II. Here the
hybrid motion is defined as Zoom In first, then Pedestal Up. Figure 12:
Results with complex camera motions. Figure 13: Results with basic camera
motions. Figure 14: Comparison to state-of-the-art methods on Truck Left,
Truck Right, Pedestal Up, Pedestal Down. Figure 15: Comparison to state-of-
the-art methods on Zoom In, Zoom Out, Roll Anticlockwise, Roll Clockwise.
Figure 16: Results with other video generation base models including SVD SVD
and VideoFusion luo2023videofusion . Our method works well with these
backbones.