# Snowmass 2021 White Paper
Computational needs of quantum mechanical calculations of materials for high-
energy physics

Sinéad M. Griffin  Materials Sciences Division, Lawrence Berkeley National
Laboratory, Berkeley, California 94720, USA  Molecular Foundry Division,
Lawrence Berkeley National Laboratory, Berkeley, California 94720, USA

###### Abstract

Searches for new physics in high-energy physics experiments commonly rely on
interactions with materials. A burgeoning direction is the accurate
calculation and design of materials for high-energy physics (HEP)
applications. In this Snowmass contribution, I briefly motivate the science
need for quantum mechanical calculations of materials for HEP and outline the
range of questions that such calculations can address. With this information,
I assess the computational needs for ab initio calculations in HEP, the
specific computational resources and workflows used by state-of-the-art
methods, and finally identify promising future directions such as the use of
machine learning and strongly-correlated quantum mechanical calculations
moving towards materials calculations on quantum computers.

Submitted to the Proceedings of the US Community Study

on the Future of Particle Physics (Snowmass 2021)

###### Contents

  1. I Science Need for Advanced Calculations of Quantum Materials in HEP
  2. II Computational Efforts in Quantum Materials for HEP
    1. II.1 Accurate description of quantum materials and phenomena
    2. II.2 Materials informatics for accelerated discovery and inverse design
    3. II.3 Machine learning (ML) for quantum mechanical calculations
  3. III General Patterns of HPC Usage
    1. III.1 Resource requirements
    2. III.2 Data generation, transfer, curation and dissemination
  4. IV Outlook

Snowmas 2021 White Paper
Computational needs of quantum mechanical calculations of materials for HEP
applications

##  I Science Need for Advanced Calculations of Quantum Materials in HEP

Next-generation HEP experiments are firmly entering the ‘quantum’ realm where
the energy scales and interactions of interest are approaching quantum
limitsFarhi  _et al._ (2015); Fradkin  _et al._ (2015); Chattopadhyay  _et
al._ (2016). This push towards quantum is aptly exemplified in searches for
new physics (such as dark matter (DM)) that lie below the weak mass scale both
in proposing new modes of coupling to the standard model, and fresh strategies
for sensors and readout that surpass current sensing limits. Because of this,
an in-depth understanding of the quantum behavior of the materials that make
up target/sensing systems is now needed for:

  * •

Accurate calculations of the material’s form factor for specific couplings to
new physics

  * •

Target/sensing efficiency calculations that incorporate decay channels,
transduction, and carrier transport

  * •

The role of materials inhomogeneities, defects and other decoherence channels
for interpretation of near-threshold measurements

  * •

Materials design for the systematic improvement of detector/sensor regimes

  * •

Novel detection/sensing platforms based on (exotic) materials properties

As a brief motivational case study, we consider the need for materials theory
and design through state-of-the-art quantum mechanical calculations in sub-GeV
DM detection based on quasiparticle scattering (e.g. phonons)Essig  _et al._
(2016); Griffin  _et al._ (2020). In such proposals, DM interacts with the
standard model through quasiparticle scattering, with the type and profile of
quasiparticle generated depending on the DM-matter couplingTrickle  _et al._
(2020). With the known DM constraints on masses and interactions, the
challenge lies in the accurate calculation of the material’s response, the
evolution of these quasiparticles before and during readout, and their
transduction efficiency for readout. Therefore, accurate quantum mechanical
descriptions and design of materials’ properties and function are required for
any next-generation HEP detector or sensor relying on low-threshold
phenomenaKahn and Lin (2022).

First-principles descriptions of such materials’ properties, responses and
energy transfer processes is possible using state-of-the-art Density
Functional Theory (DFT) calculationsHafner  _et al._ (2006). DFT is the work-
horse for the ab initio description of materials at a quantum mechanical level
and has recently been applied to accurately describe a range of HEP science
problems such as DM-electron and DM-phonon interactions in a variety of
materialsBlanco  _et al._ (2021); Catena  _et al._ (2021); Griffin  _et al._
(2020); Marsh  _et al._ (2019); Røising  _et al._ (2021). Crucially, being
entirely first-principles these robust methods eliminate the need for
empirical parameters, and when carefully applied, can describe and predict the
behavior of materials within a few percent of accuracy. The power and accuracy
of these approaches can also enable the inverse design of target functional
properties, such as that recently carried out for the design of spin-orbit
semiconductors for electron-based DM detectionInzani  _et al._ (2021).

Applications of ab initio approaches in HEP include the prediction of novel
materials systems/phenomena for targets and sensors including new
chemistries/materials, combinations of materials, and material architectures
that can both maximize current detector/sensing efficiency and propose
entirely new paradigms and systems with improved functionality. They can also
be employed to accurately quantify material-based origins of excess signals,
diagnose decoherence in quantum systems, and suggest mitigation strategies –
for example these methods have recently identified the origin of spin-based
decoherence channels in superconducting qubits – the resulting materials-based
improvements led to a factor of 6 increase in resonator quality factorAltoé
_et al._ (2022). Current applications of quantum materials in HEP focus on
single-particle phenomena that are typically described in a non-interacting
framework. However, these proposals do not take advantage of the thriving
field of correlated quantum phenomena where small perturbations can result in
threshold or cascade events, apt for small mass/energy detectionKeimer and
Moore (2017). Looking to the future, contemporary discoveries in quantum
materials of novel correlated phases and phenomena including spin, topology
and orbital degrees of freedom should be explored for HEP applications, with
the corresponding quantum mechanical calculations going beyond DFT to
accurately include strongly-correlated physics (e.g. DFT+DMFT).

##  II Computational Efforts in Quantum Materials for HEP

###  II.1 Accurate description of quantum materials and phenomena

Calculations of quantum materials including their single-particle spectra are
currently used to estimate electronic and phononic properties of materials for
HEP applicationsGriffin  _et al._ (2020). Standard first-principles methods
for this offer a balance of accuracy with computational efficiency, and can be
extended to include higher-order diagrammatic expansions of quasiparticle
interactions (e.g. multiphonon responses). This can also include the
prediction of quasiparticle transduction and coherence between disparate
degrees of freedom (e.g. spin to charge) to propose novel detection and
sensing schemesRøising  _et al._ (2021). However, beyond-DFT approaches are
required when phenomena beyond their mean-field, ground state properties are
relevant, sometimes at a much higher computational cost. For example,
underlocalization in commonly-employed DFT functionals results in bandgaps and
defect properties that can deviate significantly from experiment – advanced
methods such as the use of hybrid functionals, that include the calculation of
exact exchange, can improve this with a higher compute cost. The accurate
description of quasiparticle excitations requires calculation of the self-
energy of the many-body electron problem through the GW approximationHybertsen
and Louie (1986), for example, resulting in a significant increase in
computational cost. Strong correlations beyond those captured with simple
extensions to DFT can be incorporated through the construction of embedding
theories such as Dynamical Mean-Field Theory (DMFT)Paul and Birol (2019),
which again come at a high computational cost. Looking to the future, quantum
computers offer great potential for simulations of such quantum embedding
models (e.g. the Hubbard model), with nearer-term applications using hybrid
quantum-classical approaches to simulate materials’ propertiesMa  _et al._
(2020).

###  II.2 Materials informatics for accelerated discovery and inverse design

Materials theory and design is enhanced by the construction of materials
databases (the Materials Genome InitiativeSubcommittee on the Materials Genome
Initiative Committee on Technology of the National Science and Technology
Council (2018)) facilitated by both improved quantum mechanical methodology
and high-performance computing (HPC) capabilities. Using materials informatics
approaches, novel chemistries and materials can be identified from these large
databases (e.g. the Materials ProjectJain  _et al._ (2013)) by combining
targeted high-throughput calculations, materials theory, data science, and
machine learning, and have made key discoveries in fields as disparate as
batteries, superconductivity and photovoltaics. Nascent efforts to use such
computational materials informatics approaches are being explored for novel
HEP detector materials Geilhufe  _et al._ (2018); Inzani  _et al._ (2021), and
have a huge potential for materials and systems optimization in HEP. Such
identification and prediction of materials can then be used as a guide for the
synthesis of optimal sensor and detector materials, with a theory/experiment
feedback loop for improved predictions and synthesis. For example, following
the proposal of Marsh et al. to detect axion DM in antiferromagnetic
topological insulators (AFTIs)Marsh  _et al._ (2019), several new AFTIs were
identified by combining a high-throughput search of the Materials Project with
DFT calculations and symmetry analysisFrey  _et al._ (2020). Automation of
high-throughput calculations and searches can accelerate discovery through
workflows developed for materials theory applications, towards optimization of
target and response properties. These results can be made available to the
community through the publication of online databases for general reference
datasets.

###  II.3 Machine learning (ML) for quantum mechanical calculations

Full ab initio treatment of materials properties encompasses a range of
established methods as described above, with the computational cost being
proportional to the level of complexity and accuracy needed (see Section
III.1). A rapidly emerging approach is the development of machine learning
models to reproduce the essential features of these complex calculations but
with significantly reduced computational overheadvon Lilienfeld and Burke
(2020). Training data for ML models will require the production and curation
of high-quality calculations (10s-100s of GBs) which can then be centralized
and disseminated for a variety of applications in quantum materials in
addition to HEP.

##  III General Patterns of HPC Usage

###  III.1 Resource requirements

The most used quantum mechanical first principles method is DFT which, for
example, accounts for over 70% of NERSC allocation time in materials
science/solid state physics, and is the most used algorithm at NERSC (for
comparison see DFT and Lattice QCD statistics for NERSC AY 2018 in Fig.1). A
standard DFT calculation solves the many-body Schrödinger equation using
careful approximations – most common DFT algorithms have O(N3) scaling where N
is the number of electrons in the system. Therefore, computational
requirements can reach very large scales with increasing system sizes and/or
complexity (e.g. accurate simulations of interfaces of multi-component
systems). While a typical calculation solves the many-body Hamiltonian for
total energy in a single self-consistent calculation, more complex
calculations needing several self-consistent calculations and/or derivatives
of energy to obtain forces (e.g. to find optimal atomic geometries) can
rapidly increase the computational resources needed. Commonly, DFT
practitioners are limited to a few hundred atoms for standard simulations
because of the O(N3) scaling – many strategies for dealing with more complex
larger systems are used including splitting up large models into smaller
subsystems, or developing effective models derived from ab initio parameters.
However, for some cases, petascale (and even exascale) resources may be
required, for example when studying complex heterostructures that cannot be
simplified further, or for materials and phenomena that require higher-level
theory and thus higher compute resources (e.g. the GW approximation has
applications with O(N4) scalingdel Ben (2019)). More recently, O(N) scaling
algorithms – linear scaling DFT – have been developed and implemented in code
packagesGoedecker (1999); Prentice  _et al._ (2020), but at the expense of
some of the range of applicability and accuracy for the systems under study,
and are generally not currently suitable for correlated quantum materials that
are the main focus of HEP applications.

Figure 1: Top algorithms among NERSC codes for allocation year 2018.
Reproduced from Ref. Austin and al. (2018)

Workflows for calculations of materials make use of a wide variety of HPC
resources, ranging from few-node local clusters and mid-sized institution
compute clusters to larger scale national facilities such as NERSC and XSEDE.
In addition to conventional CPU architectures, several standard code packages
are now optimized and scale well with GPUs. Most practitioners do not use any
of these resources exclusively with typical workflows requiring a combination
for file preparation and testing (local), and larger production runs (national
HPC facilities). While some analysis can be performed on-the-fly, most
workflows will make extensive use of a combination of HPC resources for
preparation, analysis and curation of datasets. Because of this, efficient
transfer of large data sets between these different compute resources is
needed both during production runs and following the completion of the project
for storage and archiving (often datasets may not reside on the HPC facilities
where they have been created for future access, curation and re-use). Globus
file transfer, for example, is often used for transfer between national
facilities and local university clusters.

###  III.2 Data generation, transfer, curation and dissemination

Workflows using DFT and beyond-DFT methods to calculate materials properties
typically generated 10s-100s TB of data per user, depending on the complexity
of the system and the targeted properties. Much of this will be high-quality
materials data that can be used for a wide range of applications both within
and outside of HEP. To avoid duplication of efforts, and to maximize the
utility of these computationally demanding calculations, they should be
disseminated and maintained in databases that can be easily accessed and
interpreted across disciplines. For example, the storage and dissemination of
raw data can be done with the DOE’s Materials Data Facility (MDF)Blaiszik  _et
al._ (2016), which will also attach a unique identifier to each dataset for
property attribution and provenance. Higher-level raw data which often
comprises of extracted physically-relevant information can be curated and
preserved through MPContribsHuck  _et al._ (2016) as part of the Materials
Project. Codes, models, model parameters and algorithms used in the
simulations, theoretical models and ML approaches should be made open source
through common repositories such as github, with source datasets linked at
their location in the MDF. ML products (code, models and networks) can be
disseminated through the MLExchangeHexemer  _et al._ (2021), a DOE platform
for storing, maintaining and tracking ML-related data.

##  IV Outlook

Given the vast potential for the calculation, prediction and design of
materials for HEP applications, this whitepaper outlines the computational
needs for such efforts, and considerations for planning future HPC resources
that take these into account. This effort also highlights the need to
multidisciplinary collaboration across HEP, condensed matter physics and
computational materials science, and how common datasets, codes and materials
properties can be an effective means of accelerated progress in the design and
interpretation of HEP experiments.

## References

  * Farhi  _et al._ (2015) E. Farhi, S. Jordan, P. Hayden, M. Lukin, J. Maldacena, J. Preskill, P. Shor, J. Taylor, and C. Williams, “Grand challenges at the interface of quantum information science, particle physics, and computing,” DoE Community Report (2015).
  * Fradkin  _et al._ (2015) E. Fradkin, J. Maldacena, L. Chatterjee, and J. W. Davenport, “BES-HEP connections: Common problems in condensed matter and high energy physics, Round Table Discussion,” https://www.osti.gov/biblio/1275474 (2015).
  * Chattopadhyay  _et al._ (2016) S. Chattopadhyay, R. Falcone, and R. Walsworth, “Quantum Sensors at the Intersections of Fundamental Science, Quantum Information Science & Computing,” https://www.osti.gov/biblio/1358078 (2016).
  * Essig  _et al._ (2016) R. Essig, M. Fernandez-Serra, J. Mardon, A. Soto, T. Volansky, and T.-T. Yu, “Direct detection of sub-GeV dark matter with semiconductor targets,” Journal of High Energy Physics 46 (2016).
  * Griffin  _et al._ (2020) S. M. Griffin, K. Inzani, T. Trickle, Z. Zhang, and K. M. Zurek, “Multichannel direct detection of light dark matter: Target comparison,” Physical Review D 101, 055004 (2020).
  * Trickle  _et al._ (2020) T. Trickle, Z. Zhang, K. M. Zurek, K. Inzani, and S. M. Griffin, “Multi-channel direct detection of light dark matter: theoretical framework,” Journal of High Energy Physics 2020, 1–36 (2020).
  * Kahn and Lin (2022) Y. Kahn and T. Lin, “Searches for light dark matter using condensed matter systems,” Reports on Progress in Physics (2022).
  * Hafner  _et al._ (2006) J. Hafner, C. Wolverton, and G. Ceder, “Toward computational materials design: the impact of density functional theory on materials research,” MRS bulletin 31, 659–668 (2006).
  * Blanco  _et al._ (2021) C. Blanco, Y. Kahn, B. Lillard, and S. D. McDermott, “Dark matter daily modulation with anisotropic organic crystals,” Physical Review D 104, 036011 (2021).
  * Catena  _et al._ (2021) R. Catena, T. Emken, M. Matas, N. A. Spaldin, and E. Urdshals, “Crystal responses to general dark matter-electron interactions,” Physical Review Research 3, 033149 (2021).
  * Marsh  _et al._ (2019) D. J. Marsh, K. C. Fong, E. W. Lentz, L. Šmejkal, and M. N. Ali, “Proposal to detect dark matter using axionic topological antiferromagnets,” Physical Review Letters 123, 121601 (2019).
  * Røising  _et al._ (2021) H. S. Røising, B. Fraser, S. M. Griffin, S. Bandyopadhyay, A. Mahabir, S.-W. Cheong, and A. V. Balatsky, “Axion-matter coupling in multiferroics,” Physical Review Research 3, 033236 (2021).
  * Inzani  _et al._ (2021) K. Inzani, A. Faghaninia, and S. M. Griffin, “Prediction of tunable spin-orbit gapped materials for dark matter detection,” Physical Review Research 3, 013069 (2021).
  * Altoé  _et al._ (2022) M. V. P. Altoé, A. Banerjee, C. Berk, A. Hajr, A. Schwartzberg, C. Song, M. Alghadeer, S. Aloni, M. J. Elowson, J. M. Kreikebaum, _et al._ , “Localization and mitigation of loss in niobium superconducting circuits,” PRX Quantum 3, 020312 (2022).
  * Keimer and Moore (2017) B. Keimer and J. Moore, “The physics of quantum materials,” Nature Physics 13, 1045–1055 (2017).
  * Hybertsen and Louie (1986) M. S. Hybertsen and S. G. Louie, “Electron correlation in semiconductors and insulators: Band gaps and quasiparticle energies,” Phys. Rev. B 34, 5390–5413 (1986).
  * Paul and Birol (2019) A. Paul and T. Birol, “Applications of DFT+DMFT in materials science,” Annual Review of Materials Research 49, 31–52 (2019).
  * Ma  _et al._ (2020) H. Ma, M. Govoni, and G. Galli, “Quantum simulations of materials on near-term quantum computers,” npj Computational Materials 6, 1–8 (2020).
  * Subcommittee on the Materials Genome Initiative Committee on Technology of the National Science and Technology Council (2018) Subcommittee on the Materials Genome Initiative Committee on Technology of the National Science and Technology Council, “Materials genome initiative strategic plan,” https://www.mgi.gov/sites/default/files/documents/MGI-2021-Strategic-Plan.pdf (2018).
  * Jain  _et al._ (2013) A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter, D. Skinner, G. Ceder, _et al._ , “Commentary: The Materials Project: A materials genome approach to accelerating materials innovation,” APL Materials 1, 011002 (2013).
  * Geilhufe  _et al._ (2018) R. M. Geilhufe, B. Olsthoorn, A. D. Ferella, T. Koski, F. Kahlhoefer, J. Conrad, and A. V. Balatsky, “Materials informatics for dark matter detection,” Physica Status Solidi (RRL)–Rapid Research Letters 12, 1800293 (2018).
  * Frey  _et al._ (2020) N. C. Frey, M. K. Horton, J. M. Munro, S. M. Griffin, K. A. Persson, and V. B. Shenoy, “High-throughput search for magnetic and topological order in transition metal oxides,” Science Advances 6, eabd1076 (2020).
  * von Lilienfeld and Burke (2020) O. A. von Lilienfeld and K. Burke, “Retrospective on a decade of machine learning for chemical discovery,” Nature Communications 11, 1–4 (2020).
  * del Ben (2019) M. del Ben, “BerkeleyGW scaling at GPUs for science day at NERSC,” https://www.nersc.gov/assets/GPUs-for-Science-Day/05-mauro-del-ben.pdf (2019), accessed: 2022-05-17.
  * Goedecker (1999) S. Goedecker, “Linear scaling electronic structure methods,” Reviews of Modern Physics 71, 1085–1123 (1999).
  * Prentice  _et al._ (2020) J. C. Prentice, J. Aarons, J. C. Womack, A. E. Allen, L. Andrinopoulos, L. Anton, R. A. Bell, A. Bhandari, G. A. Bramley, R. J. Charlton, _et al._ , “The ONETEP linear-scaling density functional theory program,” The Journal of Chemical Physics 152, 174111 (2020).
  * Austin and al. (2018) B. Austin and al., “NERSC 2018 workload analytis,” https://portal.nersc.gov/project/m888/nersc10/workload/N10_Workload_Analysis.latest.pdf (2018), accessed: 2022-05-15.
  * Blaiszik  _et al._ (2016) B. Blaiszik, K. Chard, J. Pruyne, R. Ananthakrishnan, S. Tuecke, and I. Foster, “The Materials Data Facility: Data services to advance materials science research,” JOM 68, 2045–2052 (2016).
  * Huck  _et al._ (2016) P. Huck, D. Gunter, S. Cholia, D. Winston, A. N’Diaye, and K. Persson, “User applications driven by the community contribution framework MPContribs in the Materials Project,” Concurrency and Computation: Practice and Experience 28, 1982–1993 (2016).
  * Hexemer  _et al._ (2021) A. Hexemer, P. H. Zwart, D. McReynolds, A. Green, T. A. Chavez Esparza, and USDOE, “MLExchange v1,” https://www.osti.gov/biblio/1812187 (2021).


# UniVG: Towards UNIfied-modal Video Generation

Ludan Ruan, Lei Tian, Chuanwei Huang, Xu Zhang, Xinyan Xiao
Baidu Inc.
Beijing, China
{ruanludan<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
{zhangxu44<EMAIL_ADDRESS>

###### Abstract

Diffusion based video generation has received extensive attention and achieved
considerable success within both the academic and industrial communities.
However, current efforts are mainly concentrated on single-objective or
single-task video generation, such as generation driven by text, by image, or
by a combination of text and image. This cannot fully meet the needs of real-
world application scenarios, as users are likely to input images and text
conditions in a flexible manner, either individually or in combination. To
address this, we propose a Unified-modal Video Genearation system that is
capable of handling multiple video generation tasks across text and image
modalities. To this end, we revisit the various video generation tasks within
our system from the perspective of generative freedom, and classify them into
high-freedom and low-freedom video generation categories. For high-freedom
video generation, we employ Multi-condition Cross Attention to generate videos
that align with the semantics of the input images or text. For low-freedom
video generation, we introduce Biased Gaussian Noise to replace the pure
random Gaussian Noise, which helps to better preserve the content of the input
conditions. Our method achieves the lowest Fréchet Video Distance (FVD) on the
public academic benchmark MSR-VTT, surpasses the current open-source methods
in human evaluations, and is on par with the current close-source method Gen2.
For more samples, visit https://univg-baidu.github.io.

Figure 1: UniVG is a unified video generation framework that supports various
video generation tasks, such as Text-to-Video, Image-to-Video, and Text&Image-
to-Video. Here displays two sets of examples. Row 1: Input text to generate
semantically consistent videos; Row 2: Input image to produce pixel-aligned
videos; Row 3: Combine the semantic of input text and image to create
semantically aligned videos. All videos are shown on https://univg-
baidu.github.io.

##  1 Introduction

In recent years, diffusion-based generative models [1, 2, 3] have significant
progress in image generation [4, 5, 6, 7, 8, 9] with applications rapidly
expanding to video generation [10, 11, 12, 13, 14]. The majority of video
generation models employ textual descriptions as conditional inputs [15, 16,
17, 18, 10, 11, 19]. However, recent studies have begun to explore the use of
image conditions to improve the detail of generated videos [20] or for pixel-
level controlling [21, 22, 13, 12, 23, 24]. Additionally, to enhance the
temporal smoothness and spatial resolution of the generated videos, current
approaches often incorporate modules for frame interpolation and super-
resolution [10, 11]. However, existing works focus exclusively on single-
objective or single-task video generation, where the input is limited to text
[10, 11, 9, 18], an image [13], or a combination of text and image [12, 24].
This single-objective or single-task pipeline lacks the necessary flexibility
to satisfy all user needs. In practice, users may not have the requisite text
or image conditions for input, rendering the model unusable. Alternatively,
the introduction of conflicting text-image pairs may lead to the generation of
static videos or videos with abrupt transitions (similar conclusion is
proposed in [24]).

In essence, all models used in video generation are conditional generative
models that accept one or more conditions to produce a corresponding video.
These conditions can be text, images, low-resolution videos, even control
signals. In order to construct a versatile video generation system capable of
handling multiple video generation tasks, we revisit existing methods and
categorize the relevant methods based on generative freedom rather than the
task itself. The concept of generative freedom that we propose corresponds to
the range of solution space for video generation models given certain
conditions. In this paper, we categorize various video generation tasks as
either high-freedom or low-freedom video generation.  Specifically, high-
freedom video generation is characterized by input conditions, i.e., text and
image, that are weakly constrained at the semantic level, so that the
generative model in this scenario has a larger solution space, providing a
higher degree of freedom.  Conversely, low-freedom video generation typically
involves strongly constrained conditions at the low-level information (i.e.,
pixel), such as in image animation and video super-resolution. These
constraints limit the solution space available to the generative model,
resulting in a lower degree of freedom.

In order to better match the characteristics of various video generation
tasks, different strategies with varying degrees of generative freedom should
be taken for video generation. For high-freedom video generation, the standard
diffusion Generation Paradigm is appropriate and has been extensively utilized
in existing research some refs should be provided @ludan. Specifically, during
training stage, the diffusion model learns the added noise in the forward
processing, and predicts the target distribution by reversing from a purely
random Gaussian distribution during inference stage. Classifier guidance [4]
and classifier free guidance [25] are employed to align the predicted
distribution with the one specified by the input conditions. For low-freedom
video generation, the Editing Paradigm is more suitable. Taking image editing
[26] as a case in point, a prevalent practice involves adding noise to the
original image up to a certain level and then using text as the editing signal
to steer the distribution toward the intended outcome. This approach, compared
to generation from scratch, offers better retention of the original input’s
content. Video super-resolution has utilized a similar technique to that of
image editing [23]. However, the Editing Paradigm has a limitation in the form
of a discrepancy between training stage and inference one. Specifically, the
model is trained solely to approximate the target distribution without
learning the transition from the conditional distribution to the target
distribution. This discrepancy results in a trade-off-related issue, i.e., the
less noise that is introduced, the weaker the model’s ability to edit, whereas
the more noise that is added, the less capable the model is of preserving the
input.  In extreme cases, when the noise level approaches that of a completely
random Gaussian distribution, editing paradigm becomes analogous to generation
one, significantly diminishing the model’s capability to preserve the content
of the original input. How to reconcile the training and inference stages of
editing models to balance their editing capabilities while preserving the
input is also a problem that needs to be addressed but has been overlooked in
previous work.

In this paper, we propose a unified system Unified-modal Video Generation
(i.e.UniVG), designed to support flexible video generation conditioned on the
arbitrary combination of image and text. To achieve this, we categorize all
models within the system into two groups: high-freedom video generation and
low-freedom video generation. For high-freedom video generation, we present a
base model that is capable of the requirements of handling arbitrary
combinations of text and image conditions. We accomplish this by enhancing the
original cross-attention module of the UNet architecture with a multi-
condition cross-attention module. With regard to low-freedom video generation,
we propose two corresponding models that are individually tailored for image
animation and video super-resolution task. These models utilize the editing
paradigm, as opposed to the generation paradigm. To reconcile the differences
between the training process based on generation paradigm and the inference
process based on editing one,  in this paper, we predict Biased Gaussian Noise
(shorted as BGN) that is directed towards the target distribution, instead of
standard Gaussian noise, by refining the objective function during training
stage.

The proposed UniVG system comprises a Base model, an Image Animation model and
a Super Resolution model. The Base model is capable of handling arbitrary
combinations of text and image conditions and outputs a video sequences of
$24\times 320\times 576$ that are semantically aligned with the input
conditions at 8 frames per second (fps). The Image Animation model that fine-
tuned from the Base model with the additional condition of image
concatenation, generates video frames of $24\times 320\times 576$ that are
pixel-aligned with the input image. The Super Resolution model enhances the
resolution of each frame to $720\times 1280$ pixels. Compared to previous
works, Our UniVG demonstrates better tasks adaptability for video generation,
i.e., handling various video generation tasks within an unified system, but
also significantly improvements on the generation details and frame
consistency. Experiments have proven the effectiveness of our method. On
objective metrics, our method significantly surpasses other existing methods,
and in manual evaluations, our approach is on par with Gen2 and exceeds the
other methods.

Our contributions can be summarized as follows:

  1. 1.

We propose UniVG, the first video generation system that is capable of
handling multiple video generation tasks, such as semantically aligned
text/image-to-video generation, image animation.

  2. 2.

We introduce Biased Gaussian Noise and confirm its effectiveness for low-
freedom video generation tasks, such as image animation and super-resolution.

  3. 3.

Experiments demonstrate that our method surpasses existing text/image-to-video
generation methods in terms of objective metrics and is on par with Gen2 in
subjective evaluations.

##  2 Related Work

###  2.1 Text-to-Video Generation

Early works on Text-to-Video generation utilized GANs [27, 28, 29], VQ-VAEs
[30, 31], auto-regressive models [30, 18], or transformer structure [32], but
were limited by low resolution and suboptimal visual quality. Following the
success of diffusion models in image generation [4, 5, 6, 7], audio generation
[33, 34, 35], and other domains [36, 37, 38], VideoDiffusion [39] marked the
first application of diffusion models in video generation. Subsequently, Make-
A-Video [10] and ImagenVideo [11] expanded video generation into the open
domain by extending the 2D U-Net from text-to-image generation to 3D U-Nets.
Until then, researchers had been studying video modeling in the pixel space,
which requires massive GPU memory consumption and high training costs. To
address this issue, many researchers shifted their focus to conducting the
diffusion process in the latent space instead of pixel space [8, 15, 40, 16],
and to improving the sampling efficiency by learning-free sampling [2, 41, 42,
43] or learning-based sampling [44, 45]. Additionally, some work has
concentrated on reducing the training cost to that of a single video [46] or
to no training cost at all [47].

###  2.2 Image-to-Video Generation

Generating video directly from text is a challenging task with high
complexity. A natural thought is to use images as an intermediate bridge.
Similar to Text-to-Video generation, early works on video prediction used non-
diffusion methods [48, 49, 50], which were often limited in low resolutions or
specific domains. With the significant advancements in diffusion-based methods
in Text-to-Video tasks, I2VGen-XL [23] is, to our knowledge, the first to
utilize diffusion for open-domain Image-to-Video generation. It replaces the
textual CLIP features with image CLIP features within the text-to-video
framework, achieving video generation semantically aligned with the input
image. Similarly, SVD [13] also fine-tunes from a text-to-video model to an
image-to-video model but further concatenates the image’s VAE features as a
stronger controlling signal. Concurrently, videogen [21], VideoCrafter1 [20],
EMU Video [12] and Make Pixels Dance [24] remain their objective of text-to-
video generation, but they introduce Text-to-Image synthesis as an
intermediate step. The generated images are incorporated into the video
generation framework either through concatenation or by CLIP features.

As can be inferred from the above, although text-to-video generation and
image-to-video generation serve different applications, they share many
similarities in their technical approaches. Therefore, this paper explores
whether a single framework can unify these two objectives. The primary
distinction of our UniVG from earlier works is that we differentiate various
models included in video generation from the perspective of generative freedom
rather than task.

##  3 Method

This section presents our proposed Unified-modal Video Generation (i.e. UniVG)
for flexibly conditional video generation. Before diving into specific
designs, we first briefly recap the preliminary knowledge of diffusion models
in Sec 3.1. We then illustrate the overview of the whole system UniVG in Sec
3.2, the Multi-condition Cross Attention (i.e. MCA) used for high-freedom
generation in Sec 3.3, and the Biased Guassian Noise (i.e. BGN) used for low-
free generation in Sec 3.4.

Figure 2: Overview of the proposed UniVG system. (a) displays the whole
pipeline of UniVG, which includes the Base Model $\mathcal{F}_{B}$, the
Animation model $\mathcal{F}_{A}$, and the Super Resolution model
$\mathcal{F}_{SR}$. (b) illustrates the Multi-condition Cross Attention
involved in $\mathcal{F}_{B}$ and $\mathcal{F}_{A}$.

###  3.1 Preliminaries

Diffusion Models [1] are a class of generative models that are trained to
generate the samples by iteratively denoising from Gaussian noise. During
training, timestep $t(0<t\leq N)$ determined noise is added at the original
input $x$ to get noisy input
$x_{t}=\sqrt{\overline{\alpha}_{t}}x_{0}+\sqrt{1-\overline{\alpha}_{t}}\epsilon$
($\overline{\alpha}$ refers to noise schedule and $\epsilon$ refers to the
noise that sampled from standard Gaussian distribution
$\mathcal{N}(\mathbf{0},\mathbf{I})$), the model is trained to predict the
added noise by either $\epsilon$-prediction [1] or v-prediction [45]. During
inference, samples are generated from pure noise
$x_{N}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ by iteratively denoising.
Furthermore, Conditional Diffusion Models [4, 25] introduce extra conditional
signals to bias the predicted distribution by
$x_{t}=p_{\theta}(x_{t+1})+w_{c}(p_{\theta}(x_{t+1},c)-p_{\theta}(x_{t+1}))$,
where $\theta$ defines the diffusion model, $c$ defines input condition, and
$w_{c}$ defines guidance scale of control intensity. Another mainstream
adopted diffusion models are Latent Diffusion Models (LDM) [8], which consists
of a Variational AutoEncoder (VAE) [51] and a latent diffusion model that
denoising in latent hidden space. This approach reduces the complexity of
fitting distributions at high resolution. In this paper, each single model of
UniVG is a Conditional Latent Diffusion Model. That is, the video $V$ consists
of $F$ RGB frames is first compressed into latent space
$X\in\mathbb{R}^{F\times C\times H\times W}$ with an image auto encoder, then
input into UNet with one or multiple conditions (text condition $T$, image
condition $I$, and low resolution video $V^{lr}$).

###  3.2 UniVG

As illustrated in Figure 2-(a), our entire UniVG consists of three models: (1)
A Base model $\mathcal{F}_{B}$ accepts any combination of text and image
conditions for high-freedom video generation. (2) An Image Animation
$\mathcal{F}_{A}$ model accepts text-image pairs to generated video aligned
with input image in pixel level, and (3) a Super-resolution model
$\mathcal{F}_{SR}$ for improving spatial resolution. Each model is a latent
diffusion model with 3D UNet architecture composed of Spatial Layers, Temporal
Layers, and Cross Attention Layers. Following previous works [10, 13], the
Spatial Layer consists of 2D Convolution layer and spatial transformers, while
the Temporal Layer consists of 1D temporal Convolution layer and temporal
transformers. The cross attention module is used to process semantic control
signals, such as text and image feature.

(1) For the Base Model $\mathcal{F}_{B}$, we employ an image encoder that
matches the text encoder of CLIP [52] inspired by VideoCrafter1 [20]. To fully
utilize the global semantics and local details of input image and text, we
utilize all $K_{I}$ visual tokens $F_{I}=\\{f^{I}_{i}\\}_{i=0}^{K_{I}}$ and
all $K_{T}$ text tokens $F_{T}=\\{f^{T}_{i}\\}_{i=0}^{K_{T}}$ from the last
layer of CLIP ViT. To enable the model with the ability of processing more
than one semantic features, we extend the original Cross Attention to Multi-
condition Cross Attention and introduce its mechanism in Sec 3.3. (2) In order
to further generate videos that aligned with the input image at the pixel
level, we train the Image Animation model $\mathcal{F}_{A}$ by finetuning
$\mathcal{F}_{B}$ and concatenating the hidden space feature of the first
frame as an additional condition. Because of the additional condition, the
corresponding channel dimension of the initial convolution layer’s kernel
changes from $C$ to $2C$. We initialize the extra parameters to zero to
preserve the performance of the original model. Using either $\mathcal{F}_{B}$
or $\mathcal{F}_{A}$, we can obtain video frames of $24\times 320\times 576$.
(3) To upscale the clarity of the generated videos, we further finetune a
Super-Resolution model $\mathcal{F}_{SR}$ from $\mathcal{F}_{B}$. Since super-
resolution tasks have no image condition, the multi-condition cross attention
module reverts to a regular cross-attention module that only accepts the text
condition. During training, $\mathcal{F}_{SR}$ accepts videos of low
resolution $V^{lr}$, which are obtained by destroying high-definition videos
through RamdomBlur, RandomResize, JPEG Compression and so on. As we classify
the tasks corresponding to $\mathcal{F}_{A}$, and $\mathcal{F}_{SR}$ as low-
freedom generation, we present the Biased forward and backward processes from
conditional distribution to target distribution by adjusting the standard
Gaussian Noise to Biased Gaussian Noise (BGN that is introduced in Sec 3.4).

###  3.3 Multi-condition Cross Attention

Since our base model $\mathcal{F}_{B}$ and Image Animation model
$\mathcal{F}_{A}$ accept text and image CLIP features, we use Multi-condition
Cross Attention instead of the standard Cross Attention. This module’s
architecture mainly follows VideoCrafter [20], which computes $F_{\text{out}}$
by:

$\displaystyle
F_{out}=\text{Softmax}\left(\frac{Q_{in}K_{T}^{\intercal}}{\sqrt{d}}\right)$
$\displaystyle\cdot
V_{T}+\text{Softmax}\left(\frac{Q_{in}K_{I}^{\intercal}}{\sqrt{d}}\right)\cdot
V_{I}$ $\displaystyle Q_{in}=W_{Q}\cdot F_{in},\ K_{T}=W_{K_{T}}\cdot F_{T},\
V_{T}$ $\displaystyle=W_{V_{T}}\cdot F_{T},\ K_{I}=W_{K_{I}}\cdot F_{I},\
V_{I}=W_{V_{I}}\cdot F_{I}$

where $d_{k}$ is the dimensionality of the key/query vectors and
$Q_{\text{in}}$ is shared between $F_{I}$ and $F_{T}$. The weights $W_{K_{I}}$
and $W_{V_{I}}$ are initialized from $W_{K_{T}}$ and $W_{V_{T}}$,
respectively. Unlike VideoCrafter1 that treats image as an additional input
enhancement, we regard the image as an equally significant control signal
along with the text. This is achieved by applying a certain proportion of
image dropout throughout the training process. By extension, MCA can
accommodate more than two conditions by increasing the number of cross-
attention units, without the necessity for retraining (e.g. stronger text
features). This flexibility greatly reduces the cost of extending the model’s
training to handle new conditions.

Figure 3:  The forward & backward diffusion process with Random Gaussian Noise
and Biased Gaussian Noise.

###  3.4 Biased Gaussian Noise

Our proposed Biased Gaussian Noise is used to transfer condition distribution
to target distribution for low-freedom video generation. As illustrated in
Figure 3-(a), the standard forward diffusion process transitions from the
target distribution $v^{T}$ to the standard Gaussian distribution $\epsilon$
via
$v^{T}_{t}=\sqrt{\overline{\alpha}_{t}}v^{T}+\sqrt{1-\overline{\alpha}_{t}}\epsilon$.
However, typically in the backward process, these are the only two
distributions involved. This can result in suboptimal editing outcomes when
the samples are introduced from a condition distribution $v^{C}$ during
inference. To account for the condition distribution in both forward and
backward processes, we segment the original diffusion into three parts, as
illustrated in Figure 3-(b). For timestep between $0$ to $t_{m}$, $v_{t}$ is
calculated by the target sample with
$q(v_{t}|v^{T},t)=\sqrt{\overline{\alpha}_{t}}v^{T}_{0}+\sqrt{1-\overline{\alpha}_{t}}\epsilon(0\leq
t<t_{m})$ that followed the original forward process. For timestep between
$t_{n}$ to $N$, $v_{t}$ is calculated by the condition sample with
$q(v_{t}|v^{C},t)=\sqrt{\overline{\alpha}_{t}}v^{C}+\sqrt{1-\overline{\alpha}_{t}}\epsilon(t_{n}\leq
t<N)$. The core problem is how to design $q(v_{t}|v^{C},v^{T},t)$ that can
smoothly transition from $v_{t_{m}}$ to $v_{t_{n}}$. To preserve the original
diffusion schedule, we introduce a variable for the noise $\epsilon$, denoted
as $\epsilon^{\prime}$. Assume that for timesteps between $t_{m}$ and $t_{n}$,
we have
$q(v_{t}|v^{C},v^{T},t)=\sqrt{\overline{\alpha}_{t}}v^{T}+\sqrt{1-\overline{\alpha}_{t}}\epsilon^{\prime}$,
which meets the conditions
$v_{t_{m}}=\sqrt{\overline{\alpha}_{t_{m}}}v^{T}_{0}+\sqrt{1-\overline{\alpha}_{t_{m}}}\epsilon$
and
$v_{t_{n}}=\sqrt{\overline{\alpha}_{t_{n}}}v^{C}_{0}+\sqrt{1-\overline{\alpha}_{t_{n}}}\epsilon$.
Thus, the corresponding $\epsilon^{\prime}$ should satisfy the following
formulas at timestep $t_{m}$ and $t_{n}$.

$\displaystyle\epsilon_{t_{m}}^{\prime}=\epsilon,\ \
\epsilon_{t_{n}}^{\prime}=\epsilon+\frac{\sqrt{\overline{\alpha}_{t_{n}}}}{\sqrt{1-\overline{\alpha}_{t_{n}}}}\times\left(v^{C}-v^{T}\right)$

In theory, there are an infinite number of solutions to $\epsilon^{\prime}$.
In this paper, we simply define $\epsilon^{\prime}$ as a linear transformation
following

$\displaystyle\epsilon_{t}^{\prime}=\epsilon+\frac{\sqrt{\overline{\alpha}_{t}}}{\sqrt{1-\overline{\alpha}_{t}}}\times\frac{t-t_{m}}{t_{n}-t_{m}}\times\left(v^{C}-v^{T}\right),\
(t_{m}\leq t<t_{n})$

The $\epsilon^{\prime}$ is sampled from a Biased Gaussian distribution, with
its mean value shifted by a weighted combination of $v^{C}$ and $v^{T}$. This
bias is crucial to bridging the diffusion process from the condition
distribution to the target distribution. Alternative solutions for
$\epsilon^{\prime}$ will be explored in our future work.

##  4 Experiments

###  4.1 Implementation Details

#### Dataset

Our training datasets include publicly available academic datasets such as
WebVid-10M [53] and LAION-COCO [54], along with self-collected data.
WebVid-10M is a large and diverse text-video dataset containing approximately
10 million open-domain videos with a resolution of $336\times 596$ pixels.
LAION-COCO is a substantial text-image dataset comprising 600 million high-
quality images, filtered from LAION-2B and scored using the Aesthetic and
Semantic Estimate (ASE). To further enhance the quality of the generated
videos and to address the issue of watermarks present in WebVid-10M, we
continue training on our own curated datasets of videos and images, which
contain high-quality visual content. We prepare the self-collected videos by
first proportionally compressing them to 720p resolution along their shorter
edge and then segmenting them into 10-second clips. This process yielded 5
million high-quality text-video pairs. Additionally, our self-curated image
dataset includes 1.3 million high-quality text-image pairs, with a focus on
artistic styles.

#### Training

Our $\mathcal{F}_{B}$ is trained with an image:video:video frame ratio of
1:1:1, where the training video frames were sampled with equal probability
from 8 to 24 frames. We set the text dropout to 0.5 and the image dropout to
0.1. In addition, we utilize offset noise [55] with a strength of 0.1 and zero
terminal Signal-to-Noise Ratio (SNR) [12]. Offset noise has been proven
helpful to be helpful in generating extremely dark or bright images. Zero
terminal-SNR has been shown to be beneficial for generating high-quality and
high-resolution visual content by adding noise to pure Gaussian noise
following a rescaled schedule. Both techniques have proven useful in our
experiments. Subsequently, we continue finetuning $\mathcal{F}_{B}$ to obtain
$\mathcal{F}_{A}$ and $\mathcal{F}_{SR}$, using Biased Gaussian Noise (BGN) on
our self-curated video dataset only. For $\mathcal{F}_{A}$, we set the text
dropout to 0.1 and the image dropout to 0.1, the BGN is experimentally set
during timesteps $t_{m}=600$ to $t_{n}=990$ since the earlier steps determine
the content [26]. For $\mathcal{F}_{SR}$, the text dropout is set to 0.1, and
the BGN is applied during timesteps $t_{m}=0$ to $t_{n}=700$ since the later
steps deciding the details [26]. We incorporate $\epsilon-$prediction [1] for
$\mathcal{F}_{B}$ and $\mathcal{F}_{A}$, $v-$prediction for
$\mathcal{F}_{SR}$. The learning rate of all models is fixed at $1\times
10^{-5}$. We use DPM Solver [56] for accelerating sampling: 50 steps for
$\mathcal{F}_{B}$ and $\mathcal{F}_{A}$, and 7 steps for $\mathcal{F}_{SR}$
since we set initial weight to 0.7.

#### Evaluation

We use both objective metrics and human evaluation as the assessment criteria
for UniVG. In terms of objective metrics, we follow the previous work [21, 24]
to use the test set of MSR-VTT [57] as the standard benchmark. This test set
comprises $2,990$ test videos, each corresponding to $20$ prompts, totaling
$59,800$ prompts. For efficiency in our ablation study, we randomly selected
one prompt for each test video, ultimately obtaining $2,990$ prompts as the
evaluation set. We calculate the CLIPSIM [30] between the generated videos and
the corresponding text, and FVD [58] between the generated videos and the
original videos as comparative metrics. Since some studies [12] have pointed
out that objective metrics may not always align with human perception, we
primarily employ human evaluation. Specifically, we adopt the categorization
of video generation metrics from EMU video [12], which includes Visual Quality
(including Visual Quality consists of pixel sharpness and recognizable
objects/scenes), Motion Quality (including frame consistency, motion
smoothness and amount of motion), Text Faithfulness (Includes text-spatial
alignment and text-temporal alignment). Since UniVG supports conditional
generation for any combination of text and image, we further introduce Image
Faithfulness (Includes text-spatial alignment and text-temporal alignment) to
measure the alignment performance of generated videos with given images.
Evaluators also provide their Overall Likeness of the two videos being
compared, serving as a complement to the aforementioned sub-indicators. The
prompts used for human evaluation were collected from the webpages of previous
work [10, 11, 21, 24, 16], totaling $131$ in number. To simplify the
annotation process, annotators need only select G (our method is better), S
(equally good), or B (other methods are better) for each indicator. To ensure
fairness, the videos being compared are randomized during the annotation
process. Our six annotators provide a total of $6\times 131$ (prompts) $\times
10$ (sub-metrics) = $7,860$ evaluation results.

###  4.2 Comparison with SOTA

Table 1: Zero-shot performance comparison on MSR-VTT. T refers that the input
condition contains text and I refers to image. UniVG-HG refers to high-freedom
generation within our UniVG, UniVG-LG refers to low-freedom generation within
our UniVG. Best in bold

Method | Input | CLIPSIM$\uparrow$ | FVD$\downarrow$
---|---|---|---
CogVideo(En) [18] | T | 0.2631 | 1294
MagicVideo [59] | T | - | 1290
LVDM [19] | T | 0.2381 | 742
Video-LDM [15] | T | 0.2929 | -
InternVid [60] | T | 0.2951 | -
Modelscope [17] | T | 0.2939 | 550
Make-a-Video [10] | T | 0.3049 | -
Latent-Shift [61] | T | 0.2773 | -
VideoFactory [9] | T | 0.3005 | -
PixelDance [24] | T+I | 0.3125 | 381
Videogen [21] | T+I | 0.3127 | -
UniVG-HG | T | 0.3014 | 336
UniVG-HG | T+I | 0.3136 | 331
UniVG-LG | T+I | 0.3140 | 291

Figure 4: Percentage(%) of Overall Preference of UniVG-LG generated videos
compared with other SOTA methods.

#### Automatic Metrics

Due to the previous work involving both plain text-to-video and image-to-video
generations, we adopt aligned settings to conduct a fair comparison with them.
For text-to-video generation, we use only text as the input condition to
generate videos($\mathcal{F}_{B}$\+ $\mathcal{F}_{SR}$). For image-to-video
generation, we start by creating images from prompts using SDXL1.0+refiner
[62] and then proceed with both high-free generation ($\mathcal{F}_{B}$\+
$\mathcal{F}_{SR}$) and low-free generation ($\mathcal{F}_{A}$\+
$\mathcal{F}_{SR}$) using UniVG for the combination of text and images. Since
the i3d model [63] used for testing FVD can only accept 16 frames, we random
sample 16 frames from our generated 24 frame and the test videos in MSR-VTT.
The results are shown in Table 4.2. Whether utilizing only text as the input
condition or using both text and image together, our method generates videos
that outperform those created by other methods under the same settings. Even
when using only text as the condition, the videos generated by our method
surpass in the FVD metric those generated by others that use both text and
image. This demonstrates the capability of UniVG to concurrently process text
and image conditions and generate high-quality videos with both good visual
content and text alignment.

Table 2: The winning rate (%) of UniVG-LG compared to other methods in human
evaluations across 10 sub-dimensions(The abbreviations include VQ: Visual
Quality, MQ: Motion Quality, TF: Text Faithfulness, IF:Image Faithfulness, OL:
Overall Likeness, PS: Pixel Sharpness, RO/S: Recognizable Objects/Scenes, FC:
Frame Consistency, Motion Smoothness, AM: Amount of Motion, TSA: Text-Spatial
Alignment, TTA: Text-Temporal Alignment, ISA: Image-Spatial Alignment,
ITA:Image-Temporal Alignment)

Method | resolution | VQ | MQ | TF | IF | OL
---|---|---|---|---|---|---
PS | RO/S | MS | FC | AM | TSA | TTA | ISA | ITA
I2VGen-XL [23] | $32\times 720\times 1280$ | 98.79 | 72.85 | 87.63 | 63.20 | -11.61 | 26.24 | 27.30 | 97.13 | 73.76 | 85.22
VideoCrafter1 [20] | $16\times 576\times 1024$ | 73.74 | 11.45 | 80.61 | 20.92 | -12.52 | -3.66 | -3.05 | 92.82 | 54.35 | 53.59
SVD [13] | $25\times 576\times 1024$ | 28.11 | 4.41 | 79.06 | 12.59 | -41.43 | -0.44 | -4.39 | -14.79 | -1.76 | 11.86
Pika beta [64] | $72\times 576\times 1024$ | 55.11 | 2.44 | 16.34 | 9.62 | 8.09 | 3.76 | 6.26 | 0.92 | 2.14 | 27.33
Gen2 [65] | $96\times 1536\times 2816$ | -34.86 | -2.19 | -3.72 | 1.75 | -14.64 | -1.09 | 4.04 | -14.54 | 3.17 | -2.73

#### Human Evaluation

111Done in December 18th. The compared I2VGen is the version released in
September.

Due to the fact that automatic metrics are not able to fully reflect an
individual’s subjective perception of video quality, we further conduct human
evaluations. Since many projects are close sourced, in this paper, we chose to
compare with accessible works, including open-source works I2VGen-XL [23],
VideoCrafter1 [20], SVD [13], and closed-source works Pika beta [64] and Gen2
[65] that we can obtain the results from website or discord. All of these are
recent works and represent the current best level in text/image-to-video
generation. For a fair comparison, except for SVD and Pika beta which only
support image input, all other works were kept consistent in terms of text and
image inputs (The images are generated from text prompt by SDXL1.0 and
refiner). The comparison results are shown in Figure 4 and Table 2. Figure 4
shows a comparison of Overall Likeness between videos generated by our model
($\mathcal{F}_{A}+\mathcal{F}_{SR}$) and those produced by other methods. We
find that the videos generated by our method outperform open-source
Text/Image-to-video models and the closed-source method Pika beta, and are on
par with the closed-source method Gen2. Table 2 records the winning rates for
other sub-metrics. The formula for calculating the winning rate from GSB is
$(G-B)/(G+S+B)$. The number>0 indicates our method is better, and the number<0
indicates the other method is better. We found that the prominent advantage of
our method lies in its FC, which is due to our adoption of an editing paradigm
for low-freedom video generation, benefiting $\mathcal{F}_{A}$ in producing
more stable videos. Additionally, our generated videos exhibit superior PS
compared to videos of similar resolution (except for gen2 that generates
videos of much larger resolution). This is because we employ BGN, ensuring
consistency between training and inference by directly predicting high-
resolution videos from low-resolution ones. One significant drawback of our
generated videos is the AM, due to our current lack of filtering for static
videos in the training data. Addressing this will be part of our future work.

###  4.3 Ablation Studies

Figure 5: FVD Scores on MSR-VTT during the Training Process of
$\mathcal{F}_{B}$.

Table 3: FVD scores on MSR-VTT of $\mathcal{F}_{A}$ and $\mathcal{F}_{SR}$
that w/ or w/o BGN

model | BGN | FVD$\downarrow$
---|---|---
$\mathcal{F}_{A}$ | w/o BGN | 393.53
$\mathcal{F}_{A}$ | w/ BGN | 369.27
$\mathcal{F}_{SR}$ | w/o BGN | 648.68
$\mathcal{F}_{SR}$ | w/ BGN | 491.32

Figure 6: The generation cases of $\mathcal{F}_{B}$ with different classifier
free guidance scale of text $w_{T}$ and $w_{I}$ and different text prompts.

#### Training Process of Base Model

As our base model $\mathcal{F}_{B}$ emphasizes the conditional video
generation with arbitrary combinations of text and images, a core question is
whether the base model can maintain capabilities in text-to-video, image-to-
video, and text/image-to-video generation simultaneously. Therefore, we take
the checkpoints from the training process of $\mathcal{F}_{B}$ and test their
performance in text-to-video, image-to-video, and text&image-to-video
generation with FVD. The results are shown in Figure 5, where the overall
trends of three curves are downward, indicating that the training process
enhances the base model’s ability to generate videos from text or images. This
proves that for high-freedom video generation, multi-condition video
generation can be integrated into one single model.

Figure 7: The generation cases of $\mathcal{F}_{SR}$ w/o or w/ BGN.

#### Biased Gaussian Noise

To demonstrate that Biased Gaussian Noise (BGN) better suits low-freedom video
generation tasks, we conducted ablation studies on the Animation Model
$\mathcal{F}_{A}$ and the Video Super Resolution model $\mathcal{F}_{SR}$. The
results, shown in Table 4.3, indicate that BGN enhances video quality in both
Image Animation and Super Resolution, as evidenced by lower FVDs. It proves
more beneficial for Super Resolution, a task with less freedom than Image
Animation. Figure 7 visualizes $\mathcal{F}_{SR}$’s performance with and
without BGN. The first row shows the original, low-resolution input video.
Rows 2 and 3 depict the outputs from $\mathcal{F}_{SR}$ without BGN, processed
from the upscaled low-resolution input and subjected to $700$ and $900$
denoising steps, respectively. The fourth row presents the output from
$\mathcal{F}_{SR}$ using BGN at timestep $t_{m}=700$ to $t_{n}=0$,
illustrating how a low-resolution video upscaled to high-resolution can be
denoised effectively after $700$ steps. Each row’s far right offers a
magnified view to better showcase the detail in the model-generated content.
Our observations indicate that absent BGN, a smaller initial noise step count
results in less clarity (second row), while a larger count produces a clear
yet inconsistent output due to noise overpowering the original content (third
row). With BGN, the model directly predicts high-resolution videos from low-
resolution inputs, achieving clarity and preserving original features (fourth
row). We also acknowledge that BGN’s application can extend to other low-
freedom video generation tasks, such as frame interpolation and video editing,
which we aim to explore in future work.

#### Text&Image Conditions

Since our system is capable of generating videos that align both image and
text flexibly, we explore the videos generated under different inference
weights for these two conditions. Given text prompt $T$ and image condition
$I$, the inference formula we use is
$V_{out}=\mathcal{F}_{B}(\varnothing)+w_{T}(\mathcal{F}_{B}(T)-\mathcal{F}_{B}(\varnothing))+w_{I}(\mathcal{F}_{B}(I)-\mathcal{F}_{B}(\varnothing))$.
We adjust the classifier free guidance scale of text $w_{T}$ and image
$w_{I}$, the generating videos are shown in Figure 6-(a), we find that
adjusting the $w_{T}$ and $w_{I}$ can bias the generated video towards the
text or image conditions. Figure 6-a shows that in row 1, $w_{I}=0$,
$\mathcal{F}_{B}$ generates a video that is almost unrelated to the input
image, while in row 3, $w_{T}=0$, $\mathcal{F}_{B}$ produces a video that is
almost unrelated to the text. By adjusting both $w_{T}$ and $w_{I}$ to
appropriate values, the second row’s generated video retains the
characteristics of the input image and is also aligned with the textual
semantics. Based on this feature, our $\mathcal{F}_{B}$ can achieve different
video generation with the same input image combined with different text
prompts, as shown in Figure 6-(b). We have also explored whether
$\mathcal{F}_{A}$ possesses similar properties. However, due to the
concatenated image features having much more stronger constraints than text,
the generated videos mainly rely on image semantics. Nevertheless, inputting
consistent text helps to enhance the dynamic effects of the generated videos.

##  5 Conclusion

In this paper, we propose the UniVG system, designed for multi-task
conditional video generation that leverages both text and images. We propose a
novel categorization of models within our system based on generative freedom,
distinguishing between high-freedom and low-freedom video generation. The
high-freedom component of UniVG features a base model capable of modulating
the influence of text and images to produce videos under varying semantic
conditions. For low-freedom video generation, UniVG includes an Image
Animation model and a Super Resolution model, which generate videos closely
pixel-aligned with the input. In low-freedom generation, we propose Biased
Gaussian Noise to replace the standard random Gaussian noise, facilitating a
more direct connection between the conditional and the target distributions.
Our experiments show that our system outperforms existing methods in objective
assessments and matches Gen2 in subjective evaluations.

## References

  * [1] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [2] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In ICLR, 2021.
  * [3] Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  In NeurIPS, 2020.
  * [4] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In NeurIPS, 2021.
  * [5] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In ICML, 2022.
  * [6] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [7] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In NeurIPS, 2022.
  * [8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [9] Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu.  Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation.  arXiv preprint arXiv:2305.10874, 2023.
  * [10] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  In ICLR, 2023.
  * [11] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [12] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra.  Emu video: Factorizing text-to-video generation by explicit image conditioning.  arXiv preprint arXiv:2311.10709, 2023.
  * [13] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  arXiv preprint arXiv:2311.15127, 2023.
  * [14] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  arXiv preprint arXiv:2307.04725, 2023.
  * [15] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In CVPR, 2023.
  * [16] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji.  Preserve your own correlation: A noise prior for video diffusion models.  In CVPR, 2023.
  * [17] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.  Modelscope text-to-video technical report.  arXiv preprint arXiv:2308.06571, 2023.
  * [18] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  In ICLR, 2022.
  * [19] Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen.  Latent video diffusion models for high-fidelity video generation with arbitrary lengths.  arXiv preprint arXiv:2211.13221, 2022.
  * [20] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al.  Videocrafter1: Open diffusion models for high-quality video generation.  arXiv preprint arXiv:2310.19512, 2023.
  * [21] Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding, and Jingdong Wang.  Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation.  arXiv preprint arXiv:2309.00398, 2023.
  * [22] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou.  Show-1: Marrying pixel and latent diffusion models for text-to-video generation.  arXiv preprint arXiv:2309.15818, 2023.
  * [23] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou.  I2vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models.  arXiv preprint arXiv:2311.04145, 2023.
  * [24] Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li.  Make pixels dance: High-dynamic video generation.  arXiv preprint arXiv:2311.10982, 2023.
  * [25] Xihui Liu, Dong Huk Park, Samaneh Azadi, Gong Zhang, Arman Chopikyan, Yuxiao Hu, Humphrey Shi, Anna Rohrbach, and Trevor Darrell.  More control for free! image synthesis with semantic diffusion guidance.  In WACV, 2023.
  * [26] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In ICLR, 2022.
  * [27] Yitong Li, Martin Renqiang Min, Dinghan Shen, David E. Carlson, and Lawrence Carin.  Video generation from text.  In AAAI, 2017.
  * [28] Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei.  To create what you tell: Generating videos from captions.  In ACM MM, 2017.
  * [29] Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin.  Generating videos with dynamics-aware implicit generative adversarial networks.  In ICLR, 2022.
  * [30] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan.  Godiva: Generating open-domain videos from natural descriptions.  arXiv preprint arXiv:2104.14806, 2021.
  * [31] Gaurav Mittal, Tanya Marwah, and Vineeth N. Balasubramanian.  Sync-draw: Automatic video generation using deep recurrent attentive architectures.  In ACM MM, 2017.
  * [32] Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh.  Long video generation with time-agnostic VQGAN and time-sensitive transformer.  In ECCV, 2022.
  * [33] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  In ICLR, 2021.
  * [34] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J. Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  In ICLR, 2021.
  * [35] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov.  Grad-tts: A diffusion probabilistic model for text-to-speech.  In ICML.
  * [36] Jiatao Gu, Alex Trevithick, Kai-En Lin, Joshua M. Susskind, Christian Theobalt, Lingjie Liu, and Ravi Ramamoorthi.  Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion.  In ICML, 2023.
  * [37] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In CVPR, 2021.
  * [38] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo.  Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation.  In CVPR, 2023.
  * [39] Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.  Video diffusion models.  In NeurIPS, 2022.
  * [40] Chin-Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron C Courville.  Riemannian diffusion models.  In NeurIPS, 2022.
  * [41] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In ICLR, 2021.
  * [42] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo numerical methods for diffusion models on manifolds.  In ICLR, 2022.
  * [43] Qinsheng Zhang and Yongxin Chen.  Fast sampling of diffusion models with exponential integrator.  In ICLR, 2023.
  * [44] Daniel Watson, William Chan, Jonathan Ho, and Mohammad Norouzi.  Learning fast samplers for diffusion models by differentiating through sample quality.  In ICLR, 2023.
  * [45] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In ICLR, 2022.
  * [46] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [47] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  arXiv preprint arXiv:2303.13439, 2023.
  * [48] Ruslan Rakhimov, Denis Volkhonskiy, Alexey Artemov, Denis Zorin, and Evgeny Burnaev.  Latent video transformer.  In Giovanni Maria Farinella, Petia Radeva, José Braz, and Kadi Bouatouch, editors, VISIGRAPP, 2021.
  * [49] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.  Videogpt: Video generation using VQ-VAE and transformers.  arXiv preprint arXiv:2104.10157, 2021.
  * [50] Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Metaxas, and Sergey Tulyakov.  A good image generator is what you need for high-resolution video synthesis.  In ICLR, 2021.
  * [51] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  In ICLR, 2014.
  * [52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning. PMLR, 2021.
  * [53] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In ICCV, 2021.
  * [54] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [55] Nicholas Guttenberg.  Diffusion with offset noise, 1 2023.
  * [56] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps.  In NeurIPS, 2022.
  * [57] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In CVPR, 2016.
  * [58] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross B. Girshick, and Kaiming He.  A large-scale study on unsupervised spatiotemporal representation learning.  In CVPR, 2021.
  * [59] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  arXiv preprint arXiv:2211.11018, 2022.
  * [60] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang, Limin Wang, and Yu Qiao.  Internvid: A large-scale video-text dataset for multimodal understanding and generation.  arXiv preprint arXiv:2307.06942, 2023.
  * [61] Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xi Yin.  Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation.  arXiv preprint arXiv:2304.08477, 2023.
  * [62] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  SDXL: improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.
  * [63] João Carreira and Andrew Zisserman.  Quo vadis, action recognition? A new model and the kinetics dataset.  In CVPR, 2017.
  * [64] Pika labs.  Accessed December 18, 2023.  [Online]. Available: https://www.pika.art/.
  * [65] Gen-2.  Accessed December 18, 2023.  [Online]. Available: https://research.runwayml.com/gen2.