# Selective quantum Zeno effect of ultracold atom-molecule scattering in
dynamic magnetic fields

Hanwei Yang  State Key Laboratory for Mesoscopic Physics and Collaborative
Innovation Center of Quantum Matter, School of Physics, Peking University,
Beijing 10087, China  Zunqi Li  These authors contributed equally to this
work. State Key Laboratory for Mesoscopic Physics and Collaborative Innovation
Center of Quantum Matter, School of Physics, Peking University, Beijing 10087,
China  Songbin Zhang  Department of Physics, Shaanxi Normal University, Xi’an
710119, People’s Republic of China  Lushuai Cao  Wuhan National Laboratory for
Optoelectronics and School of Physics, Huazhong University of Science and
Technology, Wuhan 430074, China  John Bohn  JILA, University of Colorado,
Boulder, Colorado 80309, USA  Shutao Zhang  State Key Laboratory for
Mesoscopic Physics and Collaborative Innovation Center of Quantum Matter,
School of Physics, Peking University, Beijing 10087, China  Haitan Xu
<EMAIL_ADDRESS>School of Physical Sciences, University of Science and
Technology of China, Hefei 230026, China  Shenzhen Institute for Quantum
Science and Engineering, Southern University of Science and Technology,
Shenzhen 518055, China  Gaoren Wang<EMAIL_ADDRESS>Department of
Physics, Dalian Institute of Technology, Dalian, China  Zheng Li
<EMAIL_ADDRESS>State Key Laboratory for Mesoscopic Physics and
Collaborative Innovation Center of Quantum Matter, School of Physics, Peking
University, Beijing 10087, China

###### Abstract

We demonstrated that final states of ultracold scattering between atom and
molecule can be selectively produced using dynamic magnetic fields of multiple
frequencies. The mechanism of the dynamic magnetic field control is based on a
generalized quantum Zeno effect for the selected scattering channels. In
particular, we use an atom-molecule spin flip scattering to show that the
transition to the selected final spin projection of the molecule in the
inelastic scattering can be suppressed by dynamic modulation of coupling
between the Floquet engineered initial and final states.

The ancient Zeno’s arrow paradox claims that a flying arrow seems not moving
when being instantly observed. This concept is generalized to the quantum Zeno
effect (QZE) which states that one can freeze the evolution of a quantum
system by frequent measurement. In this context, Kofman and Kurizki (KK)
proposed dynamical control of quantum mechanical decay based on continuous
modulation of the coupling to an ancillary system Kofman and Kurizki (2001),
which is proved to be equivalent to the concept of ”frequent observation” of
the system since a ”measurement” is nothing but an interaction with an
external system playing the role of apparatus Facchi and Pascazio (2002);
Facchi  _et al._ (2004), and the decay is suppressed owing to an coupling
modulation. Such effects have been broadly applied in quantum optics, quantum
computation and quantum communication Guo  _et al._ (2018); Dreisow  _et al._
(2008); Barontini  _et al._ (2015); Bretheau  _et al._ (2015); Paz-Silva  _et
al._ (2012).

Adjusting the microscopic collision via external fields has always been
ultimate target in molecular dynamics Zare (1998); Krems (2006); Tscherbul and
Krems (2015). This goal has stimulated the development of quantum control of
molecular processes Shapiro and Brumer (2012), which have achieved amazing
results in unimolecular chemical reactions Zhang  _et al._ (2021); Liu  _et
al._ (2021). Attaining control over molecular collisions with simple physical
pictures, however, has proven to be a much bigger challenge due to the
complexity of the molecular interaction in the rotational, translational and
spin degrees of freedom of the system Herschbach (2009). Recent theory and
experiments Volpi and Bohn (2002); González-Martínez and Hutson (2007);
Tscherbul  _et al._ (2009); Campbell  _et al._ (2007); Wallis and Hutson
(2009) demonstrate that inelastic collisions in an ultracold gas of molecular
mixture can be effectively tuned by applying a static magnetic field.
Furthermore, enhancing the rate of specifically chosen reaction channels
stimulated the development of quantum control schemes such as optimal control
and coherent control Shapiro and Brumer (2012); Devolder  _et al._ (2021).

Despite of studies on tuning the scattering with static magnetic field, the
effects of time dependent magnetic fields on the product state distributions
and branching ratios for the different reaction channels remain largely
unknown. For the transition between bound states, it was found that the
transition rate can be reduced by using a single frequency modulation of the
external field Zhou  _et al._ (2009). In addition, Smith et al. proposed that
a single frequency magnetic field can enhance the pairwise interactions
resonantly for the elastic scattering with initial and final states being both
unbound Smith (2015). However, for the more general case of inelastic
scattering, the simple physical picture of quantum control with an external
field of single frequency could break down due to the complexity of the final
state spectrum.

Figure 1:  Schematic diagram of the selective quantum Zeno effect in the
collision between a molecule AB and an atom C. The red and blue curves
represent the transition probabilities $G_{f}(E)$ from initial state of the
molecule to two different final states $\left|f\right>$ ($f=1,2$) after the
collision as a function of final state energy $E$, which have minima at
different energies. The spectral modulation function $F(E)$ reflects the
temporal variation of eigenenergies of the collision complex induced by
external field in the Zeno dynamics, its overlap with the scattering
probability $G_{f}(E)$ determines the cross section for transition to final
state $\left|f\right>$, i.e. $\sigma_{f}\propto\int F(E)G_{f}(E)dE$. By
choosing a proper external field, we can make the distribution of $F(E)$
coincide with one of the valleys of $G_{f}(E)$, and the cross section of the
corresponding channel is then suppressed, while other channels are left almost
intact. In this manner, we can engineer the spectral modulation function to
either keep the yields of two final states balanced (a), suppress the yield of
final state $\left|1\right>$ (red part in the lower panel of (b)) or
$\left|2\right>$ (blue part in the lower panel (c)).

In this Letter, we demonstrate a theory of selective quantum Zeno effect
(SQZE) for multi-channel inelastic scattering by a time dependent magnetic
pulse train, which is schematically illustrated in Fig. 1. Our theory permits
inverse Floquet engineering of external control fields with clear physical
picture, by which we can determine the waveform of magnetic field to
selectively suppress one of the product channels. Numerically, we incorporate
multi-frequency Floquet theory into coupled channel calculation to
characterize the interaction of the collision complex with the pulsed magnetic
field, and show with a concrete example that the cross sections of different
spin-flip channels of the scattering between $\rm{{}^{3}He}$ with
$\rm{{}^{17}O_{2}}$ at ultracold temperatures can be precisely controlled.
This work realizes final state selection in ultracold inelastic scattering,
and suggests that a broad range of collision processes are submissive to this
dynamic control method, including resonant magneto-association Zhang  _et al._
(2015); Roberts  _et al._ (2000) and multi-channel reactive collision
Tscherbul and Krems (2015).

We begin by outlining the multi-frequency Floquet quantum scattering approach
used to treat the interaction of collision complex with the magnetic pulse
train. For the collisions of an atom with oxygen molecule in its ground
electronic states ${}^{3}\Sigma_{g}^{-}$ in pulsed magnetic field, we use the
Floquet Hamiltonian for the system (in units of $\hbar$)

$\hat{H}_{\rm{F}}=-\frac{1}{2\mu R}\frac{\partial^{2}}{\partial
R^{2}}R+\frac{\hat{l}^{2}}{2\mu
R^{2}}+\hat{V}(\bm{R},\bm{r})+\hat{H}_{\rm{as}}-i\frac{\partial}{\partial
t}\,,$ (1)

where $R$ is the atom-molecule separation, $\mu$ is the reduced mass of the
collision complex, $\hat{l}$ is the orbital angular momentum of the collision,
$r$ is the internuclear distance between atoms in the diatomic molecule, and
$\hat{V}(\bm{R},\bm{r})$ is the interaction potential between atom and
molecule Krems and Dalgarno (2004). The asymptotic Hamiltonian
$\hat{H}_{\rm{as}}$ depicts the rotational motion of the isolated molecule and
interaction of its electron spin $S$ with the time-dependent magnetic field
$B(t)$ through Zeeman effect. The total wave function of the collision complex
can be expressed in a direct product basis, $\Psi=R^{-1}\sum_{\alpha
KlM_{l}}F^{lm_{l}}_{\alpha K}(R)\Phi_{\alpha K}|{lm_{l}}\rangle$, where
$m_{l}$ denotes the projection of $\hat{l}$ on the magnetic field axis,
$\Phi_{\alpha K}$ is the eigenstate of the asymptotic Floquet Hamiltonian
$\hat{H}_{\rm{asF}}=\hat{H}_{\rm{as}}-i\frac{\partial}{\partial t}$, $\alpha$
indicates different channels and $K$ is the index of Floquet eigenstates.

The magnetic pulse train $B(t)=B_{0}+\mathcal{B}(t)$ consists of a set of
oscillatory magnetic fields of different frequencies. Here $B_{0}$ is the
static part and $\mathcal{B}(t)=\sum_{n}{{a_{n}}}\cos(n\omega_{\mathrm{B}}t)$
is the pulsed field containing multiple Fourier components of frequencies
$n\omega_{\mathrm{B}}$. We expand the asymptotic Floquet eigenstates
$\Phi_{\alpha K}$ in Fourier basis as $\Phi_{\alpha
K}=\sum_{NM_{N}SM_{S}n}W_{\alpha K,NSJM_{J}n}|NSJM_{J}n\rangle$, where
$\hat{N}$ is the rotational angular momentum, $\hat{J}=\hat{N}+\hat{S}$ and
$M_{J}$ is the projection of $J$ on the magnetic field axis, $|n\rangle$
satisfies $\langle t|n\rangle=e^{in\omega_{\mathrm{B}}t}$. Since the couplings
between states of different $N$s and $J$s are weak compared to the rotational
energy, $N$, $J$ and $M_{J}$ can be considered approximately as good quantum
numbers Volpi and Bohn (2002), we use $|NSJM_{J}\rangle$ to label different
collision channels. The asymptotic Floquet Hamiltonian matrix can be expressed
as $\langle\alpha n|H_{\rm{asF}}|\beta
m\rangle=H^{n-m}_{\alpha\beta}+n\omega_{\mathrm{B}}\delta_{\alpha\beta}\delta_{nm}$,
where $H^{n-m}_{\alpha\beta}$ are Fourier components of $H_{\rm{as}}$, i.e.
$\langle\alpha|\hat{H}_{\rm{as}}|\beta\rangle=\sum_{k=-\infty}^{\infty}H_{\alpha\beta}^{k}e^{ik\omega_{B}t}$.
The energy difference between $\Phi_{\alpha K}$ and $\Phi_{\alpha,K+L}$ is
$L\omega_{\mathrm{B}}$. And the total wavefunction is calculated using multi-
frequency Floquet coupled channel (MFF-CC) formalism (see SI for details).

We now apply the MFF-CC method to study the effects of pulsed magnetic fields
on the spin-flip collision
$\mathrm{{}^{3}He}\mathrm{+^{17}O_{2}}(M_{J}=1)\to\mathrm{{}^{3}He}\mathrm{+^{17}O_{2}}(M_{J}=0,-1)$
at temperature $\sim 1\mathrm{\mu K}$ with two different channels giving final
spin states of $M_{J}=0$ and $M_{J}=-1$. A schematic diagram in Fig. 2
demonstrates the mechanism of transitions between spin and Floquet states
while the laboratory projection of total angular momentum $M_{J}+m_{l}$ is
conserved. The channel-resolved cross sections of the collision have minima at
distinct external magnetic field for the two final spin states Volpi and Bohn
(2002), and provides the possibility to suppress the scattering in the
selected channel. We drop $N,S,J$ since they remain unchanged for the initial
and final states of the collision considered in this work, and abbreviate
$|NSJM_{J}K\rangle$ as $|M_{J}K\rangle$, and focus on the inelastic collision
for the initial spin-up state $|1,0\rangle$, the two spin-flip channels are
$|1,0\rangle\to|0,K\rangle$ and $|1,0\rangle\to|-1,L\rangle$. We assume the
energies of $|M_{J}=0,0\rangle,|M_{J}=\pm 1,0\rangle$ to be $e_{0},e_{\pm 1}$.
For the magnetic field considered in this work, $e_{+1}-e_{0}$ and
$e_{0}-e_{-1}$ are equal and proportional to $B_{0}$, we can set $e_{0}=0$ and
$e_{\pm 1}=\pm hB_{0}$, where $h=1.2202\times 10^{-4}~{}\rm{K}/\rm{G}$ for the
$\rm{{}^{17}O_{2}}$ molecule in ${}^{3}\Sigma_{g}^{-}$ state. The kinetic
energy after inelastic collision is
$E_{\rm{out}}=E_{\rm{in}}+hB_{0}-K\omega_{\mathrm{B}}$ and
$E_{\rm{out}}=E_{\rm{in}}+2hB_{0}-L\omega_{\mathrm{B}}$ for the
$|1,0\rangle\to|0,K\rangle$ and $|1,0\rangle\to|-1,L\rangle$ channels,
respectively. It should be noted that the asymptotic energies of states with
$M_{J}=0,\pm 1$ in laboratory frame are $0,\pm hB(t)$, thus the total energy
before the collision is $E_{\rm{in}}+hB(t)=E_{\rm{in}}+hB_{0}+h\mathcal{B}(t)$
and it will change to $E_{\rm{out}}+0=E_{\rm{in}}+hB_{0}-K\omega_{B}$ or
$E_{\rm{out}}-hB(t)=E_{\rm{in}}+hB_{0}-h\mathcal{B}(t)-L\omega_{B}$ for the
scattering $|1,0\rangle\to|0,K\rangle$ or $|1,0\rangle\to|-1,L\rangle$, i.e.
the inelastic collision produces populations in a series of evenly spaced
energy levels with spacing of $\omega_{B}$.

Figure 2:  The mechanism of spin flipping process $|1,0\rangle\to|0,K\rangle$
(red arrows) and $|1,0\rangle\to|-1,L\rangle$ (blue arrows). The dashed lines
indicate the series of Floquet levels with energy spacing of
$\omega_{\mathrm{B}}$.

We still use the asymptotic molecular states $|1,0\rangle$, $|0,K\rangle$ and
$|-1,L\rangle$ to label distinct levels whose total energies are
$E_{\rm{in}}+hB_{0}+h\mathcal{B}(t)$, $E_{\rm{in}}+hB_{0}-K\omega_{B}$ and
$E_{\rm{in}}+hB_{0}-h\mathcal{B}(t)-L\omega_{B}$, respectively. To reveal the
underlying mechanism of the selective magneto-quantum Zeno effect, we model
the inelastic scattering as transitions from initial state $|1,0\rangle$ to a
series of final states $\\{|0,K\rangle\\}$ and $\\{|-1,L\rangle\\}$ of
$\rm{{}^{17}O_{2}}$ molecule with the effective Hamiltonian
$\hat{H}(t)=\hat{H}_{0}+\hat{V}+H_{1}(t)$ Kofman and Kurizki (2001), where

$\displaystyle\hat{H}_{0}$ $\displaystyle=$ $\displaystyle
hB_{0}|1,0\rangle\langle
1,0|+\sum_{K}(hB_{0}-K\omega_{\mathrm{B}})|0,K\rangle\langle 0,K|$ (2)
$\displaystyle+$
$\displaystyle\sum_{L}(hB_{0}-L\omega_{\mathrm{B}})|-1,L\rangle\langle-1,L|$

with $hB_{0}$, $hB_{0}-K\omega_{\mathrm{B}}$ and $hB_{0}-L\omega_{\mathrm{B}}$
being the time-independent energies of $|1,0\rangle$, $|0,K\rangle$ and
$|-1,L\rangle$. Here we neglect the constant $E_{\rm{in}}$ in the formula. And

$\displaystyle\hat{V}$ $\displaystyle=$
$\displaystyle\sum_{K}V_{0}(hB_{0}-K\omega_{\mathrm{B}})|1,0\rangle\langle
0,K|$ (3) $\displaystyle+$
$\displaystyle\sum_{L}V_{-1}(2hB_{0}-L\omega_{\mathrm{B}})|1,0\rangle\langle-1,L|+\mathrm{H.c.}$

stands for the effective coupling of the initial state of molecule
$|1,0\rangle$ with the final states $\\{|0,K\rangle\\},\\{|-1,L\rangle\\}$,
where the coupling strengths $V_{0}$ and $V_{-1}$ depends on the exit kinetic
energy $E_{\rm{out}}$. We define the transition probability functions
$G_{0}(E_{\rm{out}})\propto|V_{0}(E_{\rm{out}})|^{2}$ and
$G_{-1}(E_{\rm{out}})\propto|V_{-1}(E_{\rm{out}})|^{2}$, which should be
understood as the average of coupling and independent of time. The molecule-
field interaction

$\hat{H}_{1}(t)=h\mathcal{B}(t)|1,0\rangle\langle
1,0|-\sum_{L}h\mathcal{B}(t)|-1,L\rangle\langle-1,L|$ (4)

depicts the perturbations of molecular energies due to the time dependent
magnetic field.

Figure 3:  $\rm{{}^{3}He}+\rm{{}^{17}O_{2}}$ inelastic cross sections (in
$\AA{}^{2}$) of $M_{J}=1\to M_{J}=0$ ($\sigma_{0}$) and $M_{J}=1\to M_{J}=-1$
($\sigma_{-1}$) as a function of static magnetic field strength $B_{0}$ with
collision energy $E_{\rm{in}}=1~{}\rm{\mu K}$. The point marked by the black
circle represents the values of $\sigma_{0}$ and $\sigma_{-1}$ are equal when
$B_{0}=320~{}\rm{G}$. The inset shows the dependency of
$\sigma_{0},\sigma_{-1}$ on exit kinetic energy $E_{\rm{out}}$, where the
minima of $\sigma_{0}$ and $\sigma_{-1}$ both correspond to
$E_{\rm{out}}=0.1027~{}\rm{K}$. The dashed black line shows the
$(E_{\rm{out}})^{5/2}$ scaling of the cross section.

The wave function of the system at time $t$ can be approximated as

$\displaystyle\Psi(t)$ $\displaystyle=$ $\displaystyle
c_{1}(t)e^{-ihB_{0}t-i\int_{0}^{t}h\mathcal{B}(t^{\prime})dt^{\prime}}|1,0\rangle+$
(5) $\displaystyle\sum_{K}$ $\displaystyle
c_{0,K}(t)e^{-i(hB_{0}-K\omega_{\mathrm{B}})t}|0,K\rangle+$
$\displaystyle\sum_{L}$
$\displaystyle\\!c_{\\!-\\!1,L}(t)e^{-i(hB_{0}-L\omega_{\mathrm{B}})t+i\int_{0}^{t}h\mathcal{B}(t^{\prime})dt^{\prime}}|-1,L\rangle\,,$

with the initial condition being $c_{1}(t)=1,c_{0,K}(t)=0,c_{-1,L}(t)=0$.
Because transition to $\left|1,M\right>$ with $M\neq 0$ is of higher order,
the amplitude $c_{1,M}$ is negligible. For the spin-flip transition
$M_{J}=1\to M_{J}=0$, we introduce a modulation function
$\epsilon_{0}(t)=\exp[-i\int_{0}^{t}\delta_{1,0}(t^{\prime})dt^{\prime}]$ and
expand it in Fourier series
$\epsilon_{0}(t)=\sum_{K}\lambda_{0,K}e^{-iK\omega_{\mathrm{B}}t}$, where
$\delta_{1,0}(t)=h\mathcal{B}(t)$ is the modulated energies difference between
$|1,0\rangle$ and $\\{|0,K\rangle\\}$. Similarly we introduce
$\epsilon_{-1}(t)=\exp[-i\int_{0}^{t}\delta_{1,-1}(t^{\prime})dt^{\prime}]=\sum_{L}\lambda_{-1,L}e^{-iL\omega_{\mathrm{B}}t}$
for $M_{J}=1\to M_{J}=-1$ transition, where
$\delta_{1,-1}(t)=2h\mathcal{B}(t)$. This temporally modulated system can be
treated perturbatively Kofman and Kurizki (2001); Facchi and Pascazio (2002);
Streed  _et al._ (2006), which gives the transition rates
$\partial_{t}|c_{0,K}(t)|^{2}\propto|\lambda_{0,-K}|^{2}G_{0}(hB_{0}-K\omega_{\mathrm{B}})$
and
$\partial_{t}|c_{-1,L}(t)|^{2}\propto|\lambda_{-1,-L}|^{2}G_{-1}(2hB_{0}-L\omega_{\mathrm{B}})$
(see SI for details). The cross sections $\sigma_{0,K}$ and $\sigma_{-1,L}$
for the transitions from $|1,0\rangle$ to the final states $|0,K\rangle$ and
$|-1,L\rangle$ are proportional to
$|\lambda_{0,-K}|^{2}G_{0}(hB_{0}-K\omega_{\mathrm{B}})$ and
$|\lambda_{-1,-L}|^{2}G_{-1}(2hB_{0}-L\omega_{\mathrm{B}})$ as well. We set up
the spectral functions
$F_{0}(E)=\sum_{K}|\lambda_{0,K}|^{2}\delta(E-hB_{0}-K\omega_{\mathrm{B}})$
and
$F_{-1}(E)=\sum_{L}|\lambda_{-1,L}|^{2}\delta(E-2hB_{0}-L\omega_{\mathrm{B}})$
which account for the modulation of exit energies $E_{\rm{out}}$. Let
$\sigma_{0}=\sum_{K}\sigma_{0,K}$ and $\sigma_{-1}=\sum_{L}\sigma_{-1,L}$
denote total cross sections of the two spin-flip channels, we have
$\sigma_{0}\propto\int_{-\infty}^{+\infty}F_{0}(E)G_{0}(E)dE=\sum_{K}|\lambda_{0,K}|^{2}G_{0}(hB_{0}+K\omega_{\mathrm{B}})$
and
$\sigma_{-1}\propto\int_{-\infty}^{+\infty}F_{-1}(E)G_{-1}(E)dE=\sum_{L}|\lambda_{-1,L}|^{2}G_{-1}(2hB_{0}+L\omega_{\mathrm{B}})$.
In particular, when only the static magnetic field $B_{0}$ is applied, we
obtain $\epsilon_{0}(t)=\epsilon_{-1}(t)=1$ and $\sigma_{0}\propto
G_{0}(hB_{0})$, $\sigma_{-1}\propto G_{-1}(2hB_{0})$. Fig. 3 shows the
inelastic cross sections as a function of static magnetic field at
$E_{\rm{in}}=1~{}\rm{\mu K}$ with the potential calculated by M$\phi$ller-
Plesset perturbation theory Volpi and Bohn (2002). The sharp minima of the
cross sections arise from interference between the incident $s$ and emergent
$d$ radial wave functions in the inelastic scattering Volpi and Bohn (2002).

In order to obtain $G_{0}(E_{\rm{out}})$ and $G_{-1}(E_{\rm{out}})$, we used
series of trial magnetic pulses to calculate the Fourier coefficients
$\lambda_{0,K},\lambda_{-1,L}$. And by applying the multi-frequency Floquet
coupled channel algorithm, we calculated the inelastic cross sections
$\sigma_{0,K},\sigma_{-1,L}$ numerically. Using the relationships
$G_{0}(hB_{0}-K\omega_{\mathrm{B}})\propto\sigma_{0,K}/|\lambda_{0,-K}|^{2}$
and
$G_{-1}(2hB_{0}-L\omega_{\mathrm{B}})\propto\sigma_{-1,L}/|\lambda_{-1,-L}|^{2}$,
we retrieve the relative values of $G_{0}(hB_{0}-K\omega_{\mathrm{B}})$ and
$G_{-1}(2hB_{0}-L\omega_{\mathrm{B}})$. Fig. 4 illustrates the calculated
transition probabilities $G_{f}(E_{\mathrm{out}})$ at $E_{\rm{in}}=1~{}\rm{\mu
K}$. Within the range of the collision energy and the magnetic field
considered here, the wave inelastic scattering is dominated by the $s$ partial
wave, and the spin-flip transitions require boosting the angular momentum from
$l=0$ to $l=2$. Since the centrifugal barrier of $\sim 0.59$ K for the $d$
partial wave in the exit channel is much larger than $E_{\rm{out}}$, Wigner’s
threshold law is well reflected by the threshold behavior of transition
probabilities, that $G_{0}$ and $G_{-1}$ are proportional to
$(E_{\rm{out}})^{5/2}$ Volpi and Bohn (2002).

Figure 4:  Transition probabilities as a function of the exit kinetic energy
$E_{\rm{out}}$ for the scattering channel $M_{J}=1\to M_{J}=0$ ($G_{0}$) and
$M_{J}=1\to M_{J}=-1$ ($G_{-1}$). Notice that the values of
$G_{0}(hB_{0}),G_{-1}(2hB_{0})$ are equal to $\sigma_{0},\sigma_{-1}$
respectively when only the static field $B_{0}$ is applied. In order to select
one channel, we choose the time-varying field $B(t)=B_{0}+\mathcal{B}(t)$,
where $B_{0}=320~{}G$ and the specific form of $\mathcal{B}(t)$ is given in
the main text, to concentrate $F_{0}(E_{\rm{out}})$ or $F_{-1}(E_{\rm{out}})$
around the minimum. The inset depicts two pulses of the pulse train $B(t)$ in
time domain. a,
$\mathcal{B}_{1}(t)=\sum_{n=1}^{6}a_{n}\cos(n\omega_{\mathrm{B}}t)$ and
$\sigma_{0}/\sigma_{-1}=5.16$. b,
$\mathcal{B}_{2}(t)=\sum_{n=1}^{3}a_{n}\cos(n\omega_{\mathrm{B}}t)$ and
$\sigma_{-1}/\sigma_{0}=209.42$.

By just tuning the time dependent part of the magnetic field $\mathcal{B}(t)$
and keeping $B_{0}$ constant, we can dramatically vary the scattering cross
sections of different channels and select the final states. The selection of
different scattering channels can be easily implemented by condensing the
channel resolved spectral function $F_{0}(E_{\rm{out}})$ (or
$F_{-1}(E_{\rm{out}})$) around the valley of corresponding transition
probabilities $G_{0}(E_{\rm{out}})$ (or $G_{-1}(E_{\rm{out}})$), and the
scattering cross section of $M_{J}=1\to M_{J}=0$ or $M_{J}=1\to M_{J}=-1$ will
be effectively suppressed. In order to facilitate the comparison, we choose
$B_{0}=320~{}\rm{G}$ such that the cross sections $\sigma_{0},\sigma_{-1}$ are
equal. Taking the fundamental frequency $\omega_{\mathrm{B}}$ and amplitude
$a_{n}$ of the time varying part of magnetic field $\mathcal{B}(t)$ as
variables and $\int_{-\infty}^{+\infty}F_{f}(E)G_{f}(E)dE$ as the objective
function, we use genetic algorithm to find the optimal magnetic field to
suppress one of the scattering channels. To suppress the $M_{J}=1\to M_{J}=0$
channel, the optimized field has $\omega_{\mathrm{B}}=2\pi\times
147.46~{}{\rm{MHz}},a_{1}=184.40~{}{\rm{G}},a_{2}=-165.31~{}{\rm{G}},a_{3}=132.02~{}{\rm{G}},a_{4}=-95.21~{}{\rm{G}},a_{5}=59.59~{}{\rm{G}},a_{6}=-30.52~{}{\rm{G}}$.
The distributions of channel resolved modulation functions
$F_{0}(E_{\rm{out}})$ and $F_{-1}(E_{\rm{out}})$ are plotted in Fig. 4(a). The
channel resolved cross sections are $\sigma_{0}=9.7835\times
10^{-4}\rm{\text{\AA}^{2}},\sigma_{-1}=1.8959\times
10^{-4}\rm{\text{\AA}^{2}}$ and $\sigma_{0}$ is about five times as large as
$\sigma_{-1}$. To suppress the other channel $M_{J}=1\to M_{J}=-1$, we
obtained the optimized field with $\omega_{\mathrm{B}}=2\pi\times
297.41~{}{\rm{MHz}},a_{1}=-853.72~{}{\rm{G}},a_{2}=-595.55~{}{\rm{G}},a_{3}=-312.67~{}{\rm{G}}$,
and the distributions of modulation functions are plotted in Fig. 4(b). The
magnitude of the cross sections are completely reversed,
$\sigma_{0}=2.2945\times 10^{-4}\rm{\text{\AA}^{2}}$,
$\sigma_{-1}=4.8051\times 10^{-2}\rm{\text{\AA}^{2}}$, and $\sigma_{-1}$ is
two orders of magnitude larger than $\sigma_{0}$.

Though the collision energy $E_{\rm{in}}$ has a certain Maxwellian
distribution, the shapes of $G_{0}(E_{\rm{out}})$ and $G_{-1}(E_{\rm{out}})$
are almost independent of $E_{\rm{in}}$ for ultracold collision, our control
scheme is robust for a thermal distribution of the incident kinetic energy
(see explanation in SI). Although the time dependent magnetic field can
generate electric field, $\rm{{}^{17}O_{2}}$ has no permanent dipole moment,
the energy shift (about $10~{}\rm{kHz}$) caused by the Stark effect is much
smaller than the Zeeman splitting and Floquet level spacing, hence we only
need to consider the effect of magnetic field. The method is also applicable
to more complex pulse trains consisting of low-frequency magnetic field (about
1 kHz) that can be more easily achieved in experiment.

In conclusion, we have demonstrated a selective quantum Zeno effect that
allows flexible tuning of inelastic scattering cross sections by the pulsed
magnetic field, and can realize effective selection of scattering channels. We
have also introduced a theoretical method for solving the quantum scattering
problem in the presence of a pulsed external field based on the multi-
frequency Floquet approach. The SQZE mechanism can be directly extended to
control broad types of multichannel inelastic and reactive scattering
processes with time dependent magnetic, microwave and laser fields of complex
temporal structure.

The authors are thankful to G. Chałasiński for helpful discussions. This work
has been supported by National Natural Science Foundation of China (No.
12174009).

## References

  * Kofman and Kurizki (2001) A. G. Kofman and G. Kurizki, Phys. Rev. Lett. 87, 270405 (2001).
  * Facchi and Pascazio (2002) P. Facchi and S. Pascazio, Phys. Rev. Lett. 89, 080401 (2002).
  * Facchi  _et al._ (2004) P. Facchi, D. A. Lidar, and S. Pascazio, Phys. Rev. A 69, 032314 (2004).
  * Guo  _et al._ (2018) X. Guo, C.-L. Zou, L. Jiang, and H. X. Tang, Phys. Rev. Lett. 120, 203902 (2018).
  * Dreisow  _et al._ (2008) F. Dreisow, A. Szameit, M. Heinrich, T. Pertsch, S. Nolte, A. Tünnermann, and S. Longhi, Phys. Rev. Lett. 101, 143602 (2008).
  * Barontini  _et al._ (2015) G. Barontini, L. Hohmann, F. Haas, J. Estève, and J. Reichel, Science 349, 1317 (2015).
  * Bretheau  _et al._ (2015) L. Bretheau, P. Campagne-Ibarcq, E. Flurin, F. Mallet, and B. Huard, Science 348, 776 (2015).
  * Paz-Silva  _et al._ (2012) G. A. Paz-Silva, A. T. Rezakhani, J. M. Dominy, and D. A. Lidar, Phys. Rev. Lett. 108, 080501 (2012).
  * Zare (1998) R. N. Zare, Science 279, 1875 (1998).
  * Krems (2006) R. V. Krems, Phys. Rev. Lett. 96, 123202 (2006).
  * Tscherbul and Krems (2015) T. V. Tscherbul and R. V. Krems, Phys. Rev. Lett. 115, 023201 (2015).
  * Shapiro and Brumer (2012) M. Shapiro and P. Brumer,  _Quantum Control of Molecular Processes_ (Wiley-VCH, 2012).
  * Zhang  _et al._ (2021) Z. Zhang, L. Chen, K.-X. Yao, and C. Chin, Nature 592, 708 (2021).
  * Liu  _et al._ (2021) Y. Liu, M.-G. Hu, M. A. Nichols, D. Yang, D. Xie, H. Guo, and K.-K. Ni, Nature 593, 379 (2021).
  * Herschbach (2009) D. Herschbach, Faraday Discuss. 142, 9 (2009).
  * Volpi and Bohn (2002) A. Volpi and J. L. Bohn, Phys. Rev. A 65, 052712 (2002).
  * González-Martínez and Hutson (2007) M. L. González-Martínez and J. M. Hutson, Phys. Rev. A 75, 022702 (2007).
  * Tscherbul  _et al._ (2009) T. V. Tscherbul, Y. V. Suleimanov, V. Aquilanti, and R. V. Krems, New J. Phys. 11, 055021 (2009).
  * Campbell  _et al._ (2007) W. C. Campbell, E. Tsikata, H.-I. Lu, L. D. van Buuren, and J. M. Doyle, Phys. Rev. Lett. 98, 213001 (2007).
  * Wallis and Hutson (2009) A. O. G. Wallis and J. M. Hutson, Phys. Rev. Lett. 103, 183201 (2009).
  * Devolder  _et al._ (2021) A. Devolder, P. Brumer, and T. V. Tscherbul, Phys. Rev. Lett. 126, 153403 (2021).
  * Zhou  _et al._ (2009) L. Zhou, S. Yang, Y.-X. Liu, C. P. Sun, and F. Nori, Phys. Rev. A 80, 062109 (2009).
  * Smith (2015) D. H. Smith, Phys. Rev. Lett. 115, 193002 (2015).
  * Zhang  _et al._ (2015) R. Zhang, Y. Cheng, H. Zhai, and P. Zhang, Phys. Rev. Lett. 115, 135301 (2015).
  * Roberts  _et al._ (2000) J. L. Roberts, N. R. Claussen, S. L. Cornish, and C. E. Wieman, Phys. Rev. Lett. 85, 728 (2000).
  * Krems and Dalgarno (2004) R. V. Krems and A. Dalgarno, J. Chem. Phys. 120, 2296 (2004).
  * Streed  _et al._ (2006) E. W. Streed, J. Mun, M. Boyd, G. K. Campbell, P. Medley, W. Ketterle, and D. E. Pritchard, Phys. Rev. Lett. 97, 260402 (2006).


#  MotionBooth: Motion-Aware Customized Text-to-Video Generation

Jianzong Wu1,3, Xiangtai Li2,3 $\dagger$, Yanhong Zeng3, Jiangning Zhang4,
Qianyu Zhou5,
Yining Li3, Yunhai Tong1, Kai Chen3
1PKU 2S-Lab, NTU 3Shanghai AI Laboratory 4ZJU 5SJTU
Project Page: https://jianzongwu.github.io/projects/motionbooth
Email<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

††footnotetext: Work done when Jianzong is an intern at Shanghai AI
Laboratory. $\dagger$: Project Lead.

In this work, we present MotionBooth, an innovative framework designed for
animating customized subjects with precise control over both object and camera
movements. By leveraging a few images of a specific object, we efficiently
fine-tune a text-to-video model to capture the object’s shape and attributes
accurately. Our approach presents subject region loss and video preservation
loss to enhance the subject’s learning performance, along with a subject token
cross-attention loss to integrate the customized subject with motion control
signals. Additionally, we propose training-free techniques for managing
subject and camera motions during inference. In particular, we utilize cross-
attention map manipulation to govern subject motion and introduce a novel
latent shift module for camera movement control as well. MotionBooth excels in
preserving the appearance of subjects while simultaneously controlling the
motions in generated videos. Extensive quantitative and qualitative
evaluations demonstrate the superiority and effectiveness of our method.
Models and codes will be made publicly available.

Figure 1: Motion-aware customized video generation results of MotionBooth. Our
method animates a customized object with controllable subject and camera
motions.

##  1 Introduction

Generating videos for customized subjects, such as specific scenarios
involving a particular dog’s type or appearance, has gained research attention
[51, 25, 39]. This customized generation field originated from text-to-image
(T2I) generation methods, which learn a subject’s appearance from a few images
and generate diverse images of that subject [12, 39, 29]. Following them,
subject-driven text-to-video (T2V) generation has seen increasing interest,
which has found a wide range of applications in personal shorts or film
production [51, 25, 39, 53, 13]. Can you imagine your toy riding along the
road from a distance to the camera or your pet dog dancing on the street from
the left to the right? However, rendering such lovely imaginary videos is a
challenging task. It often involves subject learning and motion injection
while maintaining the generative capability to generate diverse scenes.
Notably, VideoBooth [25] trains an image encoder to embed the subject’s
appearance into the model, generating a short clip of the subject. However,
the generated videos often display minimal or missing motion, resembling a
"moving image." This approach underutilizes the motion diversity of pre-
trained T2V models. Another line of works [53, 57, 13] fine-tunes the
customized model on specific videos, requiring motion learning for each
specific camera or subject motion type. Their pipelines restrict the type of
motion and require fine-tuning a new adapter for each motion type, which is
inconvenient and computationally expensive.

The key lies in the conflict between subject learning and video motion
preservation. During subject learning, training on limited images of the
specific subject significantly shifts the distribution of the base T2V model,
leading to significant degradation (e.g., blurred backgrounds and static
video). Therefore, existing methods often need additional motion learning for
specific motion control. In this paper, we argue that the base T2V model
already has diverse motion prior, and the key is to preserve video capability
during subject learning and digging out the motion control during inference.

To ensure subject-driven video generation with universal and precise motion
control, we present MotionBooth, which can perform motion-aware customized
video generation. The videos generated by MotionBooth are illustrated in Fig.
1. MotionBooth can take any combination of subject, subject motion, and camera
motion as inputs and generate diverse videos, maintaining quality on par with
pre-trained T2V models.

MotionBooth learns subjects without hurting video generation capability,
enabling a training-free motion injection for subject-driven video generation.
First, during subject learning, we introduce subject region loss and video
preservation loss, which enhance both subject fidelity and video quality. In
addition, we present a subject token cross-attention loss to connect the
customized subject with motion control signals. During inference, we propose
training-free techniques to control the camera and subject motion. We directly
manipulate the cross-attention maps to control the subject motion. We also
propose a novel latent shift module to govern the camera movement. It shifts
the noised latent to move the camera pose. Through quantitative and
qualitative experiments, we demonstrate the superiority and effectiveness of
the proposed motion control methods, and they can be applied to different base
T2V models without further tuning.

Our contributions are summarized as follows: 1) We propose a unified
framework, MotionBooth, for motion-aware customized video generation. To our
knowledge, this is the first framework capable of generating diverse videos by
combining customized subjects, subject motions, and camera movements as input.
2) We propose a novel loss-augmented training architecture for subject
learning. This includes subject region loss, video preservation loss, and
subject token cross-attention loss, significantly enhancing subject fidelity
and video quality. 3) We develop innovative, training-free methods for
controlling subject and camera motions. Extensive experiments demonstrate that
MotionBooth outperforms existing state-of-the-art video generation models.

##  2 Related Work

Text-to-video generation. T2V generation leverages deep learning models to
interpret text input and generate corresponding video content. It builds upon
earlier breakthroughs in text-to-image generation [41, 38, 19, 21, 34, 45, 56,
58] but introduces more complex dynamics by incorporating motion and time [43,
20, 18, 2, 63, 55]. Recent advancements particularly leverage diffusion-based
architectures. Notable models such as ModelScopeT2V [48] and LaVie [50]
integrate temporal layers within spatial frameworks. VideoCrafter1 [6] and
VideoCrafter2 [7] address the scarcity of video data by utilizing high-quality
image datasets. Latte [33] and W.A.L.T [14] adopt Transformers as backbones
[46]. VideoPoet [28] explores generating videos autoregressively to produce
consistent long videos. Recent Sora [3] excels in generating videos with
impressive quality, stable consistency, and varied motion. Despite these
advancements, controlling video content through text alone remains
challenging, highlighting a continuing need for research into more refined
control signals.

Customized generation. Generating images and videos with customized subjects
is attracting growing interest. Most works concentrate on learning a specific
subject with a few images from the same subject [23, 8, 10, 40, 44, 36] or
specific domains [15, 16, 47]. Textual Inversion [12] proposes to train a new
word to capture the feature of an object. In contrast, DreamBooth [39] fine-
tunes the whole U-Net, resulting in a better IP preservation ability.
Following them, many works explore more challenging tasks, such as customizing
multiple objects [29, 51, 32, 5], developing common subject adapter [54, 25,
60, 11, 62], and simultaneously controlling their positions [11, 32]. However,
the customization of video models from a few images often results in
overfitting. The models fail to incorporate significant motion dynamics. A
recent work, DreamVideo [53], addresses this by learning specific motion types
from video data. Yet, this method is restricted to pre-defined motion types
and lacks the flexibility of text-driven input. In contrast, our work
introduces MotionBooth to control both the subject and camera motions without
needing pre-defined motion prototypes.

Motion-aware video generation. Recent works explore incorporating explicit
motion control in video generation. This includes camera and object motions.
To control camera motion, existing works like AnimateDiff [13], VideoComposer
[49], CameraCtrl [17], Direct-A-Video [61], and MotionCtrl [52] design
specific modules to encode the camera movement or trajectory. These models
usually rely on training on large-scale datasets [1, 9], leading to high
computational costs. In contrast, our MotionBooth framework builds a training-
free camera motion module that can be easily integrated with any T2V model,
eliminating the need for re-training. For object motion control, recent works
[59, 30, 31, 24, 61, 4, 64, 26, 22] propose effective methods to manipulate
attention values during the inference stage. Inspired by these approaches, we
connect subject text tokens to the subject position using a subject token
cross-attention loss. This allows for straightforward control over the motion
of a customized object by adjusting cross-attention values.

Figure 2: The overall pipeline of MotionBooth. We first fine-tune a T2V model
on the subject. This procedure incorporates subject region loss, video
preservation loss, and subject token cross-attention loss. During inference,
we control the camera movement with a novel latent shift module. At the same
time, we manipulate the cross-attention maps to govern the subject motion.

##  3 Method

###  3.1 Overview

Task formulation. We focus on generating motion-aware videos featured by a
customized subject. To customize video subjects, we fine-tune the T2V model on
a specific subject. This process can be accomplished with just a few
(typically 3-5) images of the same subject. During inference, the fine-tuned
model generates motion-aware videos of the subject. The motion encompasses
both camera and subject movements, which are freely defined by the user. For
camera motion, the user inputs the horizontal and vertical camera movement
ratios, denoted as $\mathbf{c}_{cam}=[c_{x},c_{y}]$. For subject motion, the
user provides a bounding box sequence
$[\mathbf{B}_{1},\mathbf{B}_{2},...,\mathbf{B}_{L}]$ to indicate the desired
positions of the subject, where $L$ represents the video length. Each bounding
box specifies the x-y coordinates of the top-left and bottom-right points for
each frame. By incorporating these conditional inputs, the model is expected
to generate videos that include a specific subject, along with predefined
camera movements and subject motions.

Overall pipeline. The overall pipeline of MotionBooth is illustrated in Fig.
2. During the training stage, MotionBooth learns the appearance of the given
subject by fine-tuning the T2V model. To prevent overfitting, we introduce
video preservation loss and subject region loss in Section 3.2. Additionally,
we propose a subject token cross-attention (STCA) loss in Fig. 4 to explicitly
connect the subject tokens with the subject’s position on cross-attention
maps, facilitating the control of subject motion. Camera and subject motion
control are performed during the inference stage. We manipulate the cross-
attention maps by amplifying the subject tokens and their corresponding
regions while suppressing other tokens in Section 3.3. This ensures that the
generated subjects appear in the desired positions. By training on the cross-
attention map, the STCA loss enhances the subjects’ motion control. For camera
movement, we introduce a novel latent shift module to shift the noised latent
directly, achieving smooth camera movement in the generated videos in Section
3.4.

###  3.2 Subject Learning

Figure 3: Case study on subject learning. “Region” indicates subject region
loss. “Video” indicates video preservation loss. The images are extracted from
generated videos.

Given a few images of a subject, previous works have demonstrated that fine-
tuning a diffusion model on these images can effectively learn the appearance
of the subject [39, 23, 8, 10, 40, 44]. However, two significant challenges
remain. First, due to the limited size of the dataset, the model quickly
overfits the input images, including their backgrounds, within a few steps.
This overfitting of the background impedes the generation of videos with
diverse scenes, a problem also noted in previous works [39, 12]. Second, fine-
tuning T2V models using images can impair the model’s inherent ability to
generate videos, leading to severe background degradation in the generated
videos. To illustrate these issues, we conducted a toy experiment. As depicted
in Fig. 3, without any modifications, the model overfits the background to the
subject image. To address this, we propose computing the diffusion
reconstruction loss solely within the subject region. However, even with this
adjustment, the background in the generated videos remains over-smoothed. This
degradation likely results from tuning a T2V model exclusively with images,
which damages the model’s original weights for video generation. To mitigate
this, we propose incorporating video data as preservation data during the
training process. Although training with video data but without subject region
loss still suffers from overfitting, our approach, MotionBooth, can generate
videos with detailed and diverse backgrounds. (xiangtai: hard to understand.)

Preliminary. T2V diffusion models learn to generate videos by reconstructing
noise in a latent space [39, 29, 51, 12]. The input video is first encoded
into a latent representation $\mathbf{z}_{0}$. Noise $\epsilon$ is added to
this latent representation, resulting in a noised latent $\mathbf{z}_{t}$,
where $t$ represents the timestamp. This process simulates the reverse process
of a fixed-length Markov Chain [38]. The diffusion model
$\mathbf{\epsilon}_{\theta}$ is trained to predict this noise. The training
loss, which is a reconstruction loss, is given by:

$\mathcal{L}=\mathbb{E}_{\mathbf{z},\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t,\mathbf{c}}\left[||\mathbf{\epsilon}-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c},t)||^{2}_{2}\right],$
(1)

where $\mathbf{c}$ is the conditional input used in classifier-free guidance
methods, which can be text or a reference image. During inference, a pure
noise $\mathbf{z}_{T}$ is gradually denoised to a clean latent
$\mathbf{z}^{\prime}_{0}$, where $T$ is the length of the Markov Chain. The
clean latent is then decoded back into RGB space to generate the video
$\mathbf{X}^{\prime}$.

Subject region loss. To address the challenge of overfitting backgrounds in
training images, we propose a subject region loss. The core idea is to
calculate the diffusion reconstruction loss exclusively within the subject
region, thereby preventing the model from learning the background.
Specifically, we first extract the subject mask for each image. This can be
done manually or through automatic methods, such as a segmentation model. In
practice, we use SAM [27] to collect all the masks. The subject region loss is
then calculated as follows:

$\mathcal{L}_{sub}=\mathbb{E}_{\mathbf{z},\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t,\mathbf{c}}\left[||(\mathbf{\epsilon}-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{i},t))\cdot\mathbf{M}||^{2}_{2}\right],$
(2)

where $\mathbf{M}$ represents the binary masks for the training images. These
masks are resized to the latent space to compute the dot product.
$\mathbf{c}_{i}$ is a fixed sentence in the format "a [V] [class name]," where
"[V]" is a rare token and "[class name]" is the class name of the subject
[39]. We have found that with the subject region loss, the trained model
effectively avoids the background overfitting problem.

Video preservation loss. Image customization datasets like DreamBooth [39] and
CustomDiffusion [29] provide excellent examples of multiple images from the
same subject. However, in the customized video generation task, directly fine-
tuning the video diffusion model on images leads to significant background
degradation. Intuitively, this image-based training process may harm the
original knowledge embedded in video diffusion models. To address this, we
introduce a video preservation loss designed to maintain video generation
knowledge by joint training with video data. Unlike the class-specific
preservation data used in previous works [39, 51], we utilize common videos
with captions denoted as $\mathbf{c}_{v}$. Our experiments in Section 4
demonstrate that common videos are more effective for subject learning and
preserving video generation capabilities. The loss function is formulated as
follows:

$\mathcal{L}_{vid}=\mathbb{E}_{\mathbf{z},\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),t,\mathbf{c}}\left[||\mathbf{\epsilon}-\mathbf{\epsilon}_{\theta}(\mathbf{z}_{t},\mathbf{c}_{v},t)||^{2}_{2}\right].$
(3)

Figure 4: Case study on subject token cross-attention maps. (b) and (c) are
visualization of cross-attention maps on tokens “[V]” and “dog”.

Subject token cross-attention loss. To control the subject’s motion, we
directly manipulate the cross-attention maps during inference. Since we
introduce a unique token, “[V]”, in the training stage and associate it with
the subject, we need to link this special token to the subject’s position
within the cross-attention maps. As illustrated in Fig. 4, fine-tuning the
model does not effectively connect the unique token to the cross-attention
maps. Therefore, we propose a Subject Token Cross-Attention (STCA) loss to
guide this process explicitly. First, we extract the cross-attention map,
$\mathbf{A}$, at the tokens “[V] [class name]”. We then apply a Binary Cross-
Entropy Loss to ensure that the corresponding attention map is larger at the
subject’s position and smaller outside this region. This process incorporates
the subject mask and can be expressed as:

$\mathcal{L}_{stca}=-\left[\mathbf{M}\log(\mathbf{A})+(1-\mathbf{M})\log(1-\mathbf{A})\right].$
(4)

During training, the overall loss function is defined as:

$\mathcal{L}=\mathcal{L}_{sub}+\lambda_{1}\mathcal{L}_{vid}+\lambda_{2}\mathcal{L}_{stca},$
(5)

where $\lambda_{1}$ and $\lambda_{2}$ are hyperparameters that control the
weights of the different loss components.

###  3.3 Subject Motion Control

We chose bounding boxes as the motion control signal for subjects because they
are easy to draw and manipulate. In contrast, providing object masks for every
frame is labor-intensive, requiring consideration of the subject’s shape
transformation between frames. In practice, we find that bounding boxes are
sufficient for precisely controlling the positions of subjects. Previous works
like GLIGEN [30] attempt to control object positions by training an extra
condition module with large-scale image data. However, these training methods
fix the models and cannot easily align with customized models fine-tuned for
specific subjects. Therefore, we adopt an alternative approach that directly
edits the cross-attention maps during inference in a training-free manner [61,
26, 4]. This cross-attention editing method is plug-and-play and can be used
with any customized model.

In cross-attention layers, the query features $\mathbf{Q}$ are extracted from
the video latent and represent the vision features. The key and value features
$\mathbf{K}$ and $\mathbf{V}$ are derived from input language tokens. The
calculation process of the edited cross-attention layer can be formulated as
follows:

$\text{EditedCrossAttn}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}+\alpha\mathbf{S}\right)\mathbf{V},$
(6)

where $d$ is the feature dimension of $\mathbf{Q}$ and serves as a
normalization term. $\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}$ is the
normalized production between $\mathbf{Q}$ and $\mathbf{K}$, representing the
attention scores between vision and language features. We manipulate the
production by adding a new term $\alpha\mathbf{S}$, where $\mathbf{S}$ has
positive values on the subject region provided in bounding boxes and large
negative values outside the desired positions. $\alpha$ is a hyperparameter to
control the editing strength. The editing matrix $\mathbf{S}$ is set as
follows:

$S_{k}[i,j]=\begin{cases}1-\frac{|\mathbf{B}_{k}|}{|\mathbf{Q}|},&\text{if
}i\in\mathbf{B}_{k}\text{ and }j\in\mathbf{P}\text{ and }t\geq\tau\\\
0,&\text{if }i\in\mathbf{B}_{k}\text{ and }j\in\mathbf{P}\text{ and }t<\tau\\\
-\infty,&\text{otherwise}\end{cases}$ (7)

where $i$, $j$, and $k$ indicate the vision token, language token, and frame
indexes, respectively. $\mathbf{P}$ represents the indexes for subject
language tokens in the text prompt. In this work, we choose “[V]” and “[class
name]” as subject tokens. The SCTA loss in Fig. 4 binds the two tokens with
cross-attention maps. $t$ is the denoising timestamp and $\tau$ is a
hyperparameter defining a timestamp threshold. Since diffusion models tend to
form the approximate object layout in earlier denoising steps and refine the
details in later steps [59], we apply stronger attention amplification in
earlier steps and no amplification in later steps. Note that attention
suppression outside the bounding box regions persists throughout the
generation. $|\mathbf{B}_{k}|$ and $|\mathbf{Q}|$ are the areas of the box and
query, respectively. Following previous works [26, 61], smaller boxes should
have larger amplifications, and we do not apply any editing on the <start> and
<end> tokens.

Figure 5: Illustration of camera movement control through shifting the noised
latent.

###  3.4 Camera Movement Control

Simply editing the cross-attention map can efficiently control the motion of
the subject. This suggests that the latent can be considered a "shrunk image,"
which maintains the same visual geographic distribution as the generated
images. For camera movement control, an intuitive approach is to directly
shift the noised latent during inference based on the camera movement signal
$\mathbf{c}_{cam}=[c_{x},c_{y}]$. The latent shift pipeline is illustrated in
Table 2. The key challenge with this idea is filling in the missing parts
caused by the latent shift (the question mark region in Step 1). To address
this issue, we propose sampling tokens from the original noised latent and
using them to fill the gap. This is based on the prior knowledge that when a
camera moves in a video, the new scene it captures is semantically close to
the previous one. For example, in a video with forest scenes, when the camera
pans left, it is highly likely to capture more trees similar to those in the
original scene. Another assumption is that in a normally angled video, a
visual element is more likely to be semantically close to elements along the
same x-axis or y-axis rather than other elements. For instance, in the
waterfall video in Fig. 5, trees are at the top and bottom, spreading
horizontally, while the waterfall spans the middle x-axis area.
Experimentally, we over that sampling tokens horizontally and vertically
provides better initialization and results in smoother video transitions.
Randomly sampling tokens degrades the generated video quality. The latent
shift process for timestamp $t$ can be formulated as follows:

$\displaystyle\mathbf{h}_{x}$
$\displaystyle=\text{SampleHorizontal}(\mathbf{z}_{t},\mathbf{B},c_{x}),$ (8)
$\displaystyle\mathbf{h}_{y}$
$\displaystyle=\text{SampleVertical}(\mathbf{z}_{t},\mathbf{B},c_{y}),$
$\displaystyle\mathbf{z}_{\text{shift}}$
$\displaystyle=\text{Crop}(\text{Shift}(\mathbf{z}_{t},c_{x},c_{y})),$
$\displaystyle\mathbf{z}_{t}$
$\displaystyle=\text{Fill}(\mathbf{z}_{\text{shift}},\mathbf{h}_{x},\mathbf{h}_{y},c_{x},c_{y}),$

where $\mathbf{h}_{x}$ and $\mathbf{h}_{y}$ are sampled tokens along the x and
y axes, respectively. $\text{Crop}(\cdot)$ removes the tokens outside the
camera view after the shift. $\mathbf{B}$ is the subject bounding box. We
filter out the tokens belonging to the subjects because they are not likely to
occur in the new scenes. In addition, to avoid a drastic change in latent in
one shift, we spread the latent shift over multiple timestamps, with each step
only shifting a small number of tokens. Note that the latent shift needs to be
applied after the subject’s approximate layout is fixed but before the video
details are completed. We set a pair of hyperparameters $\sigma_{1}$ and
$\sigma_{2}$. The latent shift only applies in the timestamp range
$[\sigma_{1},\sigma_{2}]$.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. For customization, we collect a total of 26 objects from DreamBooth
[39] and CustomDiffusion [29]. These objects include pets, plushies, toys,
cartoons, and vehicles. To evaluate camera and object motion control, we built
a dataset containing 40 text-object motion pairs and 40 text-camera motion
pairs, ensuring that the camera and object motion patterns are consistent with
the text prompts. This dataset evaluates the videos generated for each subject
in various scenarios and motions.

Implementation details. We train MotionBooth for 300 steps using the AdamW
optimizer, with a learning rate of 5e-2 and a weight decay of 1e-2. We collect
500 preservation videos from the Panda-70M [9] training set, chosen randomly.
Each batch consists of one batch for images and one for videos, with batch
sizes equal to the number of training images and 1 for images and videos,
respectively. The loss weight parameters $\lambda_{1}$ and $\lambda_{2}$ are
set to 1.0 and 0.01. We use Zeroscope and LaVie as base models. During
inference, we perform 50-step denoising using the DDIM scheduler and set the
classifier-free guidance scale to 7.5. The generated videos are 576x320x24 and
512x320x16 for Zeroscope and LaVie, respectively. The training process
finishes in around 10 minutes in a single NVIDIA A100 80G GPU. Additional
implementation details can be found in Appendix A.3.

Baselines. Since we are pioneering motion-aware customized video generation,
we compare our methods with closely related works, including DreamBooth [39],
CustomVideo [51], and DreamVideo [53]. Dreambooth customizes subjects for
text-to-image generation. We follow its practice with class preservation
images and fine-tune T2V models for generating videos. CustomVideo is a recent
video customizing method. We adopt its parameter-efficient training procedure.
DreamVideo learns motion patterns from video data. To provide such data, we
sample videos from Panda-70M, which are most relevant to the evaluation
motions. Since these methods cannot control motions during inference, we apply
our camera and object motion control technologies for a fair comparison.
Additionally, we compare our camera control method with training-based
methods, AnimateDiff [13] and CameraCtrl [17], focusing on camera motion
control without subject customization. Since AnimateDiff is trained with only
basic camera movement types and cannot take user-defined camera movement
$\mathbf{c}_{cam}=[c_{x},c_{y}]$ as input, we use the closest basic movement
type for evaluation.

Evaluation metrics. We evaluate motion-aware customized video generation from
three aspects: region subject fidelity, temporal consistency, and camera
motion fidelity. 1) To ensure the subject is well-preserved and accurately
generated in the specified motion, we introduce region CLIP similarity
(R-CLIP) and region DINO similarity metrics (R-DINO). These metrics utilize
the CLIP [37] and DINOv2 [35] models to compute the similarities between the
subject images and frame regions indicated by bounding boxes. Additionally, we
use CLIP image-text similarity (CLIP-T) to measure the similarity between
entire frames and text prompts. 2) We evaluate temporal consistency by
computing CLIP image features between each consecutive frame. 3) We use
VideoFlow [42] to predict the optical flow of the generated videos. Then, we
calculate the flow error by comparing the predicted flow with the ground-truth
camera motion provided in the evaluation dataset.

Table 1: Quantitative comparison for motion-aware customized video generation.

T2V Model | Method | R-CLIP $\uparrow$ | R-DINO $\uparrow$ | CLIP-T $\uparrow$ | T-Cons. $\uparrow$ | Flow error $\downarrow$
---|---|---|---|---|---|---
Zeroscope | DreamBooth [39] | 0.608 | 0.279 | 0.231 | 0.951 | 0.690
CustomVideo [51] | 0.657 | 0.267 | 0.245 | 0.955 | 0.516
DreamVideo [53] | 0.656 | 0.238 | 0.258 | 0.954 | 0.349
MotionBooth (Ours) | 0.667 | 0.306 | 0.258 | 0.958 | 0.252
LaVie | DreamBooth [39] | 0.696 | 0.426 | 0.238 | 0.958 | 1.156
CustomVideo [51] | 0.634 | 0.189 | 0.248 | 0.911 | 1.055
DreamVideo [53] | 0.649 | 0.216 | 0.243 | 0.925 | 0.691
MotionBooth (Ours) | 0.712 | 0.472 | 0.247 | 0.962 | 0.332

Table 2: Quantitative comparison for camera movement control.

Method | Module Weight Storage | CLIP-T $\uparrow$ | T-Cons. $\uparrow$ | Flow error $\downarrow$
---|---|---|---|---
AnimateDiff [13] | 74M | 0.245 | 0.925 | 1.683
CameraCtrl [17] | 2.5G | 0.237 | 0.939 | 0.807
MotionBooth (Zeroscope) | [No Training] | 0.252 | 0.948 | 0.190
MotionBooth (LaVie) | [No Training] | 0.241 | 0.963 | 0.296

###  4.2 Main Results

Quantitative results. We conduct quantitative comparisons with baseline models
on both motion-aware customized video generation and camera movement control.
The results for motion-aware customized video generation are shown in Table 1.
The results demonstrate that MotionBooth outperforms all baselines on both
Zeroscope and LaVie models, indicating that our proposed technologies can be
extended to different T2V models. Thanks to the training-free architecture of
subject and camera motion control methods, MotionBooth is expected to be
adaptable to more open-sourced models in the future, such as Sora [3].
Notably, DreamVideo [53] achieves the second-best scores in T-Cons. and flow
error, which aligns with our observation that incorporating video data as
auxiliary training data enhances video generation performance. On the other
hand, CustomVideo [51] shows inferior performance in R-DINO scores, indicating
a poorer ability to generate subjects in given positions. This may be
attributed to its approach of only fine-tuning the text embeddings and cross-
attention layers of the diffusion models, which is insufficient for learning
the subjects.

For camera movement control, we compare our method with two training-based
methods, AnimateDiff [13] and CameraCtrl [17]. The results are shown in Table
2. Remarkably, MotionBooth achieves superior results compared to the two
baselines with our training-free latent shift module. Specifically,
MotionBooth outperforms the recent method CameraCtrl by 0.617, 0.015, and
0.009 in flow error, CLIP-T, and T-Cons. metrics with Zeroscope, and 0.511,
0.004, and 0.024 for the LaVie model. These results demonstrate that the
latent shift method is simple yet effective.

Figure 6: Qualitative comparison of customizing objects and controlling their
motions. Figure 7: Qualitative comparison of camera motion control. Lines and
points are used to help the readers track the camera movement more easily.

Qualitative results. The qualitative comparison results for video generation
with customized objects and controlled subject motions are presented in Fig.
6. Our observations reveal that MotionBooth excels in subject motion
alignment, text prompt alignment, and overall video quality. In contrast,
DreamBooth and CustomVideo produce videos with vague backgrounds, highlighting
that generated backgrounds deteriorate when training is conducted without
video data. Additionally, CustomVideo and DreamVideo struggle to capture the
subjects’ appearances, likely because their approach tunes only part of the
diffusion model, preventing the learning process from fully converging.

We also conduct qualitative experiments focused on camera movement control,
with results shown in Fig. 7. AnimateDiff, limited to basic movements, does
not support user-defined camera directions. Although the CameraCtrl method can
accept user input, it generates videos with subpar aesthetics and objects that
exhibit flash movements. In contrast, our MotionBooth model outperforms both
the Zeroscope and Lavie models. The proposed latent method generates videos
that adhere to user-defined camera movements while maintaining time
consistency and high video quality.

###  4.3 Ablation Studies

Table 3: Ablation study for training technologies. “mask” means subject region
loss. “STCA” means subject token cross-attention loss. “video” means video
preservation loss. “w/ class video” means utilizing class-specific videos in
video preservation loss. The results are evaluated on LaVie.

Method | R-CLIP $\uparrow$ | R-DINO $\uparrow$ | CLIP-T $\uparrow$ | T-Cons. $\uparrow$ | Flow error $\downarrow$
---|---|---|---|---|---
w/o mask | 0.673 | 0.216 | 0.245 | 0.943 | 0.451
w/o STCA | 0.710 | 0.453 | 0.244 | 0.962 | 0.363
w/o video | 0.677 | 0.364 | 0.244 | 0.953 | 0.344
w/ class video | 0.677 | 0.364 | 0.244 | 0.953 | 0.345
MotionBooth | 0.712 | 0.472 | 0.247 | 0.962 | 0.332

Training technologies. We analyze the technologies proposed during the subject
learning stage. The ablation results are shown in Table 3. Clearly, without
the proposed modules, the quantitative metrics drop accordingly. These results
demonstrate that the proposed subject region loss, STCA loss, and video
preservation loss are beneficial for subject learning and generating motion-
aware customized videos. Specifically, the R-DINO metric decreases
significantly by 0.256 without the subject region loss, highlighting its core
contribution in filtering out image backgrounds during training. Additionally,
the "w/ class video" experiment, which uses class-specific videos instead of
randomly sampled common videos, yields worse results. This approach restricts
the scenes and backgrounds in class-specific videos, hindering the models’
ability to generalize effectively.

##  5 Conclusion

This paper introduces MotionBooth, a novel framework for motion-aware,
customized video generation. MotionBooth fine-tunes a T2V diffusion model to
learn specific subjects, utilizing subject region loss to focus on the subject
area. The training procedure incorporates video preservation data to prevent
background degradation. Additionally, an STCA loss is designed to connect
subject tokens with the cross-attention map. During inference, training-free
technologies are proposed to control both subject and camera motion. Extensive
experiments demonstrate the effectiveness and generalization ability of our
method. In conclusion, MotionBooth can generate vivid videos with given
subjects and controllable subject and camera motions.

Acknowledgement. This work is supported by the National Key Research and
Development Program of China (No. 2023YFC3807600).

## References

  * [1] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In ICCV, 2021.
  * [2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In CVPR, 2023.
  * [3] Tim Brooks, Bill Peebles, Connor Holmes, Yufei Guo Will DePue, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.  Sora: Creating video from text, 2024.
  * [4] Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li.  Motion-zero: Zero-shot moving object control framework for diffusion-based video generation.  arXiv preprint arXiv:2401.10150, 2024.
  * [5] Hong Chen, Xin Wang, Guanning Zeng, Yipeng Zhang, Yuwei Zhou, Feilin Han, and Wenwu Zhu.  Videodreamer: Customized multi-subject text-to-video generation with disen-mix finetuning.  arXiv preprint arXiv:2311.00990, 2023.
  * [6] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al.  Videocrafter1: Open diffusion models for high-quality video generation.  arXiv preprint arXiv:2310.19512, 2023.
  * [7] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan.  Videocrafter2: Overcoming data limitations for high-quality video diffusion models.  arXiv preprint arXiv:2401.09047, 2024.
  * [8] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.  Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-image generation.  arXiv preprint arXiv:2305.03374, 2023.
  * [9] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al.  Panda-70m: Captioning 70m videos with multiple cross-modality teachers.  arXiv preprint arXiv:2402.19479, 2024.
  * [10] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  NeurIPS, 2024.
  * [11] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.  Anydoor: Zero-shot object-level image customization.  CVPR, 2024.
  * [12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [13] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  In ICLR, 2024.
  * [14] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama.  Photorealistic video generation with diffusion models.  arXiv preprint arXiv:2312.06662, 2023.
  * [15] Yue Han, Jiangning Zhang, Junwei Zhu, Xiangtai Li, Yanhao Ge, Wei Li, Chengjie Wang, Yong Liu, Xiaoming Liu, and Ying Tai.  A generalist facex via learning unified facial representation.  arXiv preprint arXiv:2401.00551, 2023.
  * [16] Yue Han, Junwei Zhu, Keke He, Xu Chen, Yanhao Ge, Wei Li, Xiangtai Li, Jiangning Zhang, Chengjie Wang, and Yong Liu.  Face adapter for pre-trained diffusion models with fine-grained id and attribute control.  arXiv preprint arXiv:2405.12970, 2024.
  * [17] Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang.  Cameractrl: Enabling camera control for text-to-video generation.  arXiv preprint arXiv:2404.02101, 2024.
  * [18] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [19] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In NeurIPS, 2020.
  * [20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  In NeurIPS, 2022.
  * [21] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans.  simple diffusion: End-to-end diffusion for high resolution images.  In ICML, 2023.
  * [22] Teng Hu, Jiangning Zhang, Ran Yi, Yating Wang, Hongrui Huang, Jieyu Weng, Yabiao Wang, and Lizhuang Ma.  Motionmaster: Training-free camera motion transfer for video generation.  arXiv preprint arXiv:2404.15789, 2024.
  * [23] Miao Hua, Jiawei Liu, Fei Ding, Wei Liu, Jie Wu, and Qian He.  Dreamtuner: Single image is enough for subject-driven generation.  arXiv preprint arXiv:2312.13691, 2023.
  * [24] Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl.  Peekaboo: Interactive video generation via masked-diffusion.  arXiv preprint arXiv:2312.07509, 2023.
  * [25] Yuming Jiang, Tianxing Wu, Shuai Yang, Chenyang Si, Dahua Lin, Yu Qiao, Chen Change Loy, and Ziwei Liu.  Videobooth: Diffusion-based video generation with image prompts.  In CVPR, 2024.
  * [26] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu.  Dense text-to-image generation with attention modulation.  In ICCV, 2023.
  * [27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [28] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al.  Videopoet: A large language model for zero-shot video generation.  arXiv preprint arXiv:2312.14125, 2023.
  * [29] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In CVPR, 2023.
  * [30] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In CVPR, 2023.
  * [31] Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal.  Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning.  arXiv preprint arXiv:2309.15091, 2023.
  * [32] Zhiheng Liu, Yifei Zhang, Yujun Shen, Kecheng Zheng, Kai Zhu, Ruili Feng, Yu Liu, Deli Zhao, Jingren Zhou, and Yang Cao.  Cones 2: Customizable image synthesis with multiple subjects.  arXiv preprint arXiv:2305.19327, 2023.
  * [33] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao.  Latte: Latent diffusion transformer for video generation.  arXiv preprint arXiv:2401.03048, 2024.
  * [34] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In ICML, 2021.
  * [35] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  arXiv preprint arXiv:2304.07193, 2023.
  * [36] Xu Peng, Junwei Zhu, Boyuan Jiang, Ying Tai, Donghao Luo, Jiangning Zhang, Wei Lin, Taisong Jin, Chengjie Wang, and Rongrong Ji.  Portraitbooth: A versatile portrait model for fast identity-preserved personalization.  arXiv preprint arXiv:2312.06354, 2023.
  * [37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICCV, 2021.
  * [38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [39] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In CVPR, 2023.
  * [40] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  arXiv preprint arXiv:2307.06949, 2023.
  * [41] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  NeurIPS, 2022.
  * [42] Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li.  Videoflow: Exploiting temporal cues for multi-frame optical flow estimation.  In ICCV, 2023.
  * [43] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792, 2022.
  * [44] James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin.  Continual diffusion: Continual customization of text-to-image diffusion with c-lora.  arXiv preprint arXiv:2304.06027, 2023.
  * [45] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [47] Chaoyang Wang, Xiangtai Li, Lu Qi, Henghui Ding, Yunhai Tong, and Ming-Hsuan Yang.  Semflow: Binding semantic segmentation and image synthesis via rectified flow.  arXiv preprint arXiv:2405.20282, 2024.
  * [48] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.  Modelscope text-to-video technical report.  arXiv preprint arXiv:2308.06571, 2023.
  * [49] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  NeurIPS, 2024.
  * [50] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al.  Lavie: High-quality video generation with cascaded latent diffusion models.  arXiv preprint arXiv:2309.15103, 2023.
  * [51] Zhao Wang, Aoxue Li, Enze Xie, Lingting Zhu, Yong Guo, Qi Dou, and Zhenguo Li.  Customvideo: Customizing text-to-video generation with multiple subjects.  arXiv preprint arXiv:2401.09962, 2024.
  * [52] Zhouxia Wang, Ziyang Yuan, Xintao Wang, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan.  Motionctrl: A unified and flexible motion controller for video generation.  arXiv preprint arXiv:2312.03641, 2023.
  * [53] Yujie Wei, Shiwei Zhang, Zhiwu Qing, Hangjie Yuan, Zhiheng Liu, Yu Liu, Yingya Zhang, Jingren Zhou, and Hongming Shan.  Dreamvideo: Composing your dream videos with customized subject and motion.  arXiv preprint arXiv:2312.04433, 2023.
  * [54] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  In ICCV, 2023.
  * [55] Jianzong Wu, Xiangtai Li, Chenyang Si, Shangchen Zhou, Jingkang Yang, Jiangning Zhang, Yining Li, Kai Chen, Yunhai Tong, Ziwei Liu, et al.  Towards language-driven video inpainting via multimodal large language models.  CVPR, 2024.
  * [56] Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, and Dacheng Tao.  Towards open vocabulary learning: A survey.  T-PAMI, 2024.
  * [57] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In ICCV, 2023.
  * [58] Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, and Chen Change Loy.  Mosaicfusion: Diffusion models as data augmenters for large vocabulary instance segmentation.  arXiv preprint arXiv:2309.13042, 2023.
  * [59] Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou.  Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion.  In ICCV, 2023.
  * [60] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen.  Paint by example: Exemplar-based image editing with diffusion models.  In CVPR, 2023.
  * [61] Shiyuan Yang, Liang Hou, Haibin Huang, Chongyang Ma, Pengfei Wan, Di Zhang, Xiaodong Chen, and Jing Liao.  Direct-a-video: Customized video generation with user-directed camera movement and object motion.  arXiv preprint arXiv:2402.03162, 2024.
  * [62] Haoyu Zhao, Tianyi Lu, Jiaxi Gu, Xing Zhang, Zuxuan Wu, Hang Xu, and Yu-Gang Jiang.  Videoassembler: Identity-consistent video generation with reference entities using diffusion model.  arXiv preprint arXiv:2311.17338, 2023.
  * [63] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  arXiv preprint arXiv:2211.11018, 2022.
  * [64] Yupeng Zhou, Daquan Zhou, Ming-Ming Cheng, Jiashi Feng, and Qibin Hou.  Storydiffusion: Consistent self-attention for long-range image and video generation.  arXiv preprint arXiv:2405.01434, 2024.

##  Appendix A Appendix

Overview. The supplementary includes the following sections:

  * •

Section A.1. Human preference study.

  * •

Section A.2. Limitations and future work.

  * •

Section A.3. Implementation details of the experiments.

  * •

Section A.4. Ablation studies.

  * •

Section A.5. Social impacts.

  * •

Section A.6. More qualitative results.

Video Demo. We also present a video in a separate supplementary file, which
shows the results in video format.

###  A.1 Human Preference Study

Figure 8: Human preference study. Our MotionBooth achieves the best human
preference scores in all the evaluation aspects.

To evaluate our approach to understanding user preferences, we conducted a
user study experiment. We collected 30 groups of videos generated by
MotionBooth and baseline methods. We then asked 7 colleagues to select the
best videos based on the following criteria: subject motion alignment, camera
movement alignment, subject appearance alignment, and temporal consistency.
For each group of videos, the annotators selected only the best one. For the
subject appearance alignment, the annotators were provided with corresponding
subject images. As shown in Fig. 8, MotionBooth was the most preferred method
across all models and evaluation aspects, particularly in subject appearance
alignment. These results further demonstrate the effectiveness of our method.

###  A.2 Limitations and Future Work

In Fig. 9, we illustrate several failure cases of MotionBooth. One significant
limitation of MotionBooth is its struggle with generating videos involving
multiple objects. As shown in Fig. 9(a), the subject’s appearance can
sometimes merge with other objects, resulting in visually confusing outputs.
This issue might be resolved by incorporating advanced training technologies
for multiple subjects.

Figure 9: Failure cases of MotionBooth.

Another limitation is the model’s capability to depict certain motions
indicated by the text prompt. As depicted in Fig. 9(b), MotionBooth may fail
to accurately represent motions that are unlikely to be performed by the
subject. For example, it is hard to imagine a scene where a wolf plushie is
riding a bike. These failure cases highlight the need for further improvement
in the model’s subject separation and motion understanding capabilities to
enhance the realism and accuracy of the generated videos. Utilizing more
powerful T2V models may eliminate these drawbacks. Future work could focus on
refining these aspects to address the current limitations and provide more
robust performance in complex scenarios.

###  A.3 Implementation Details

Hyperparameters. For the LaVie model, we set $\alpha$ = 10.0, $\tau$ =
0.7$\mathbf{T}$, $\sigma_{1}$ = 0.8$\mathbf{T}$, and $\sigma_{2}$ =
0.6$\mathbf{T}$. For the Zeroscope model, we set $\alpha$ = 10.0, $\tau$ =
0.9$\mathbf{T}$, $\sigma_{1}$ = 0.9$\mathbf{T}$, and $\sigma_{2}$ =
0.7$\mathbf{T}$.

Figure 10: The evaluation dataset. We present one picture for each subject.

Evaluation dataset. We collect a total of 26 subjects from DreamBooth [39] and
CustomDiffusion [29]. We show one image for each subject in Fig. 10. The
subjects contain a wide variety of types, including pets, plushie toys,
cartoons, and vehicles, which can provide us with a thorough analysis of the
model’s effectiveness.

Figure 11: The application interface for user study.

User study interface. We show the application interface for human preference
study in Fig. 11. During user study, we ask the annotators to select the best
video based on the question, e.g., “Which video do you think has the best
temporal consistency?”

Pseudo-code of latent shift. To present the latent shift module more clearly,
we show the pseudo-code of the algorithm in Fig. 13. Our latent shift module
can control the camera movement in videos in a training-free manner at minimal
costs.

###  A.4 Ablation Studies

(a) Ablation study on subject motion control. Only the first frame is shown.
Experiments on Zeroscope.

(b) Ablation study on latent shift. Experiments on LaVie. A higher $\sigma$
means an earlier denoising step.

Figure 12: Ablation study on motion control hyperparameters.

Subject motion control hyperparameters. We conduct ablation studies on the
hyperparameter for subject motion control, with the results presented in Fig.
12(a). We examined the effects of varying the factor $\alpha$ of $\mathbf{S}$
and the maximum cross-attention manipulation timestamp $\tau$. The findings
indicate that increasing $\alpha$ and extending the controlling steps lead to
stronger control. With lower control strengths, the subject does not appear in
the desired position, or only part of its body aligns with the intended spot.
Conversely, when the control strengths are too high, the generated subjects
tend to appear unnaturally square in shape.

Latent shift hyperparameters. We experiment with the influence of $\sigma_{1}$
and $\sigma_{2}$ in the latent shift module. The results are shown in Fig.
12(b). The results indicate that applying latent shift in the earlier steps of
the denoising process results in incomplete camera movement, as evidenced by
the trees in the background. Conversely, shifting the latent in the later
steps degrades video quality and introduces artifacts, highlighted by the red
boxes in the last row. Empirically, setting $\sigma_{1}$ and $\sigma_{2}$ to
middle values provides optimal control over camera movement.

Table 4: Ablation study for the number of video preservation data.

# Videos | R-CLIP $\uparrow$ | R-DINO $\uparrow$ | CLIP-T $\uparrow$ | T-Cons. $\uparrow$ | Flow error $\downarrow$
---|---|---|---|---|---
100 | 0.714 | 0.488 | 0.245 | 0.964 | 0.324
300 | 0.711 | 0.486 | 0.245 | 0.962 | 0.330
500 | 0.712 | 0.472 | 0.247 | 0.962 | 0.332
700 | 0.712 | 0.473 | 0.247 | 0.963 | 0.340
900 | 0.712 | 0.480 | 0.244 | 0.963 | 0.328

Number of preservation videos. We conduct an ablation study on the number of
preservation videos. As shown in Table 4, ranging the preservation videos from
100 to 900 does not bring large changes to the quantitative scores. We
conclude that the key is to use video data to preserve the video generation
ability of the pre-trained T2V models. The number of video data can be
flexible.

###  A.5 Social Impacts

Positive societal impacts. MotionBooth allows for precise control over
customized subjects and camera movements in video generation, opening new
avenues for artists, filmmakers, and content creators to produce unique and
high-quality visual content without extensive resources or professional
equipment.

Potential negative societal impacts. The ability to generate realistic
customized videos could be misused to create deepfakes, leading to potential
disinformation campaigns, privacy violations, and reputational damage. This
risk is particularly significant in the context of political manipulation and
social media. If the underlying models are trained on biased datasets, the
generated content might reinforce harmful stereotypes or exclude certain
groups. Ensuring diversity and fairness in training data is crucial to
mitigate this risk.

Mitigation strategies. Developing and adhering to strict ethical guidelines
for the use and dissemination of video generation technologies can help
mitigate misuse. This includes implementing usage restrictions and promoting
transparency about the generated content.

###  A.6 More Qualitative Results

We show more qualitative results in Fig. 14.

Figure 13: Pseudo-code of the latent shift algorithm. Figure 14: More
qualitative results of our MotionBooth.