# Quasinormal frequencies for a black hole in a bumblebee gravity

R. Oliveiraa<EMAIL_ADDRESS>D. M. Dantasa<EMAIL_ADDRESS>C. A.
S. Almeidaa<EMAIL_ADDRESS>aUniversidade Federal do Ceará (UFC),
Departamento de Física, Campus do Pici, Caixa Postal 6030, 60455-760,
Fortaleza, Ceará, Brazil

###### Abstract

After recent observational events like the LIGO-Virgo detections of
gravitational waves and the shadow image of M87* supermassive black hole by
Event Horizon Telescope (EHT), the theoretical study of black holes was
significantly improved. Quantities as quasinormal frequencies, shadows, and
light deflection become more important to analyze black hole models. In this
context, an interesting scenario to study is a black hole in the bumblebee
gravity. The bumblebee vector field imposes a spontaneous symmetry breaking
that allows the field to acquire a vacuum expectation value that generates
Lorentz-Violating (LV) into the black hole. In order to compute the
quasinormal modes (QNMs) via the WKB method, we obtain the Reege-Wheeler’s
equation with a bell-shaped potential for this black hole. Both QNMs, the
scalar and tensorial modes, are computed for the black hole in the bumblebee
scenario. Moreover, the stability of the Schwarzschild-like solution and some
bounds to LV parameters are analyzed.

Black holes, Quasinormal modes, Lorentz-Violating.

##  I introduction

The study of black holes is improving its observational knowledge due to the
LIGO-Virgo detections of gravitational waves Ligo  and the shadow images of a
supermassive black hole M87* by the international Event Horizon Telescope
(EHT) tele . In this context, it is essential to deal with the computation of
observable quantities shadows Crispino ; Kerr-Sen-Bum , quasinormal modes
(QNMs) KerrQNM  and the deflecting of light Kerr-Sen def  in black holes and
other astronomical compact objects. Moreover, these quantities can help spot
modifications in General relativity (GR) and the standard model (SM) of
particle physics once that the parameters of modified models should correct
the observed result.

The bumblebee gravity is one of many extensions of the Standard Model
Extension (SME) where the Lorentz is spontaneously violated (LV) KS . A
Schwarzschild-like solution in the bumblebee gravity is described in the work
of Ref. Casana . Based on this model, the LV effect over the shadows of a
black hole is studied in Ref. Casana2 , whereas the influence of LV on the
black hole accretion is presented in Ref. Accretion . Moreover, a wormhole
solution in the bumblebee gravity was conceived in the works of Refs. Ovgun1 ;
Rondinelly , the radiative corrections in bumblebee electrodynamics are
studied in Euclides  and the black hole with the cosmological constant in
bumblebee gravity was presented in Ref. Maluf .

The quasinormal frequencies are one of the most frequent quantities for
testing black holes. In 1957, Regge and Wheeler computed the QNMs for the
Schwarzschild solution R-W . After, Zerilli Zerilli  extended the work of
Regge and Wheeler and described the Schrödinger-like equation for these
perturbations. The damping oscillations for the QNMs were shown by
Vishveshwara vish . The approximation method studied by Gregor Wentzel,
Hendrik Anthony Kramers, and Leon Brillouin (WKB) of third-order was
implemented in the Schwarzschild black hole by S. Iyer and C. Will Iyer .
Moreover, Konoplya has extended the order of the WKB method for higher-order
in reference Iyer , where the 6th order approximation has been shown more
accurate than the 3rd order.

In this work, we compute the Regge-Wheeler equation for the black hole in the
context of bumblebee gravity. The differences of the potentials are detailed
for the scalar and gravitational perturbations. For both field perturbations,
the LV parameter slightly modifies the result of the Schwarzschild black hole.
Moreover, the quasinormal modes are computed via the WKB method for
perturbations to 3rd and 6th orders. Furthermore, the time profile for the
scalar and gravitational perturbations are shown.

##  II QNMs of Schwarzschild-like solution in bumblebee gravity

A solution of a black hole in the context of the bumblebee gravity was
proposed in Ref. Casana . The metric of this Schwarzschild-like solution with
the Lorentz spontaneously violation is represented by Casana :

$\displaystyle
ds^{2}=\Big{(}1-\frac{2M}{r}\Big{)}dt^{2}-(1+\lambda)\left(1-\frac{2M}{r}\right)^{-1}dr^{2}-r^{2}d\theta^{2}-r^{2}\sin^{2}\theta
d\phi^{2},$ (1)

where $\lambda\ll 1$ is the LV parameter (for $\lambda=0$ the usual
Schwarzschild black hole is obtained).

In order to compute the quasinormal modes (QNM), the WKB method will be
applied. Since the solution of Eq. (1) shares some similarities with the
Schwarzschild metric, we hope to find a similar QNM slightly changed by the LV
parameter. For this end, the black hole perturbations are described by the
Schrödinger-like wave equation, the so-called Regge-Wheeler equation Iyer ;
Kono :

$\frac{d^{2}\,\psi(x)}{dx^{2}}+Q(x)\,\psi(x)=0,$ (2)

where $x$ is the tortoise coordinate. The term $Q(x)=\omega^{2}-V(x)$ depends
on the analogue potential $V(x)$ and on the frequencies $\omega$. Moreover,
the wave function $\psi$ is imposed to following the conditions:

$\psi(x)\sim C\pm e^{\mp i\,\omega\,t}\quad\text{com}\quad
x\rightarrow\pm\infty.$ (3)

Furthermore, the WKB method requires the analogue potential $V(x)$ have some
aspects. The potential should have a maximum centered at $x_{0}$, and two
returning points $x_{1}$ and $x_{2}$. The potential at limits $(x=-\infty)$
and $(x=+\infty)$ assume constant values. So, the potential should exhibit a
bell-shaped profile.

Let us use as an example the usual Schwarzschild black hole. The Schwarzschild
solution has the following potential in the original coordinate $r$ Kono :

$\mathcal{V}\left(x(r)\right)=\left(1-\frac{1}{r}\right)\left(\frac{l(l+1)}{r^{2}}+\frac{1-s^{2}}{r^{3}}\right),$
(4)

the parameter $l$ is the orbital angular momentum, and $s$ is the spin of
field perturbations.

For $s=0$ the potential for a scalar field is showed in Fig. (1), where the
bell-shaped behavior is verified. Note that other spins exhibit a similar
profile once that the only modification occurs in the term $(1-s^{2})$ and the
higher perturbation studied here is the gravitational one $(s=2)$.

Figure 1: The Regge-Wheeler potential for the scalar field in the
Schwarzschild black hole with some values of $l$.

With these conditions, Iyer and Will Iyer  found the expression of the QNM via
the WKB method as

$\frac{i\,Q_{0}}{\sqrt{2\,Q_{0}^{{}^{\prime\prime}}}}-\Lambda(n)-\Omega(n)=n+\frac{1}{2},$
(5)

where $Q_{0}$ is the value of the potential at $x_{0}$ and the primes denote
derivatives concerning $x$ coordinates. Corrections compose the terms
$\Lambda(n)$ and $\Omega(n)$ up to third order Rondinelly ; Iyer . An improved
version of WKB method was presented in Ref. Kono , where high order
corrections were studied. For the sixth order, the WKB method can be written
as Kono

$\frac{i\,Q_{0}}{\sqrt{2\,Q_{0}^{{}^{\prime\prime}}}}-\Lambda_{2}(n)-\Lambda_{3}(n)-\Lambda_{4}(n)-\Lambda_{5}(n)-\Lambda_{6}(n)=n+\frac{1}{2}.$
(6)

The Tables 1 e 2 are the results of Ref. Kono  of the WKB method (third and
sixth orders) for the scalar and gravitational perturbations in Schwarzschild
black hole. Note that the results for the 6th order are better approximated to
the numerical values than the 3rd order for the WKB method.

$l,\ n$ | numerical | 3rd order WKB | 6th order WKB
---|---|---|---
$l=1,n=0$ | $0.2929-0.0977i$ | $0.2911-0.0980i$ | $0.2929-0.0977i$
$l=2,n=1$ | $0.4639-0.2956i$ | $0.4632-0.2958i$ | $0.4638-0.2956i$
$l=2,n=2$ | $0.4305-0.5086i$ | $0.4317-0.5034i$ | $0.4304-0.5087i$
Table 1: The QNM for $s=0$, results from Ref. Kono . $l,\ n$ | numerical | 3rd order WKB | 6th order WKB
---|---|---|---
$l=2,n=0$ | $0.3737-0.0890i$ | $0.3732-0.0892i$ | $0.3736-0.0890i$
$l=2,n=1$ | $0.3467-0.2739i$ | $0.3460-0.2749i$ | $0.3463-0.2735$
$l=2,n=2$ | $0.3011-0.4783i$ | $0.3029-0.4711i$ | $0.2985-0.4776i$
$l=3,n=0$ | $0.5994-0.0927i$ | $0.5993-0.0927i$ | $0.5994-0.0927i$
$l=3,n=1$ | $0.5826-0.2813i$ | $0.5824-0.2814i$ | $0.5826-0.2813i$
$l=3,n=2$ | $0.5517-0.4791i$ | $0.5532-0.4767i$ | $0.5516-0.4790i$
$l=3,n=3$ | $0.5120-0.6903i$ | $0.5157-0.6774i$ | $0.5111-0.6905i$
Table 2: The QNM for $s=2$, results from Ref. Kono .

In this work, we compute the Regge-Wheeler equation of Eq. (2) using the
metric given in Eq. (1), for both perturbations of the scalar (spin $s=0$) and
the gravitational (spin $s=2$) fields. As a result, the quasinormal modes will
be obtained from eq. (6), and the influence of the bumblebee parameter over
frequencies will be analyzed. Also, we compare the differences in the 3rd and
6th order of the WKB method.

##  III Quasinormal modes of perturbations of scalar fields

In this section, the first case of spin $s=0$ scalar field will be described.
The massless Klein-Gordon equation in the presence of gravity leads to the
following equation of motion Gordon

$\frac{1}{\sqrt{-g}}\partial_{\mu}(g^{\mu\nu}\sqrt{-g}\,\partial_{\nu}\,\Phi)=0.$
(7)

Applying the coordinates decomposition, the variables can be separated as
follows:

$\Phi=\sum_{l=0}^{\infty}\sum_{m=-l}^{l}\frac{R(r,t)}{r}\,Y_{l\,m}(\theta,\phi),$
(8)

where $Y_{l\,m}(\theta,\phi)$ are the spherical harmonics.

By substituting this decomposition of Eq. (8) into Eq. (2) with the metric of
Eq. (1) we obtain:

$\frac{d^{2}\,\phi(x)}{dx^{2}}+(\omega^{2}-V_{s}(l,x,\lambda))\,\phi(x)=0,$
(9)

where for Eq. (9) the transformation of tortoise coordination $x$ is given by

$\frac{dx}{dr}=\frac{\sqrt{1+\lambda}}{\left(1-\frac{1}{r}\right)},\quad\Rightarrow\quad
x(r)=\sqrt{1+\lambda}\left[r+\ln(r-1)\right].$ (10)

The potential of Regge-Wheeler for the scalar field in the context of
bumblebee gravity has the formula

$V_{s}(r)=\left(1-\frac{1}{r}\right)\left(\frac{l(l+1)}{r^{2}}+\frac{\left[1+\lambda\right]^{-1}}{r^{3}}\right).$
(11)

Note that by Eq. (11), the bumblebee LV parameter $(\lambda)$ modifies the
potential of the usual Schwarzschild black hole in Eq. (4) by the term
$\left[1+\lambda\right]^{-1}$. As expected, for $\lambda=0$ the usual
potential is obtained. The LV parameter decreases the peaks of the potential,
as can be shown in Fig. (2) for $l=1$.

Figure 2: Potential for scalar field in the bumblebee black hole with some
values of $\lambda$.

By inverting $r(x)$ in Eq. (10) and obtained the $V(x)$ in Eq. (11), the terms
of WKB of 3rd order in Eq. (5) and 6th order in Eq. (6) can be computed. The
results of QNM for scalar perturbation in bumblebee gravity are shown in Table
3, for the 3rd and 6th order, respectively. It is worth highlighting that
$n\leq l$ needs to be imposed.

$l\,,\,n$ | 3rd order WKB | 6rd order WKB
---|---|---
l = 1, n = 0 | 0.291644 - 0.089100i | 0.289044 - 0.097382i
l = 2, n = 0 | 0.483357 - 0.088035i | 0.481314 - 0.096615i
l = 2, n = 1 | 0.466639 - 0.268196i | 0.461409 - 0.295206i
l = 3, n = 0 | 0.675267 - 0.087756i | 0.673701 - 0.096422i
l = 3, n = 1 | 0.662979 - 0.265386i | 0.658965 - 0.292061i
l = 3, n = 2 | 0.641291 - 0.447758i | 0.63181 - 0.495662i
l = 4, n = 0 | 0.867373 - 0.087646i | 0.866120 - 0.096344i
l = 4, n = 1 | 0.857714 - 0.264225i | 0.854492 - 0.290737i
l = 4, n = 2 | 0.839892 - 0.444019i | 0.801780 - 0.697192i
l = 4, n = 3 | 0.815917 - 0.627569i | 0.765748 - 0.913890i
l = 5, n = 0 | 1.05960 - 0.0877264i | 1.05961 - 0.096337i
l = 5, n = 1 | 1.05183 - 0.263979i | 1.050040 - 0.290154i
l = 5, n = 2 | 1.03709 - 0.442350i | 1.03150 - 0.487343i
l = 5, n = 3 | 1.01659 - 0.623493i | 1.005170 - 0.689918i
Table 3: The QNMs for scalar perturbations in bumblebee gravity via the WKB
method for $\lambda=0.1$

The QNM for the scalar field in bumblebee gravity is also denoted in Fig. (3).
Note the differences between our results, presented in Table 1 (denoted by
circles) with the results for the usual Schwarzschild solution in the same
Table 1, but denoted by triangles. From Fig. (3) we conclude that the LV
parameter slightly decreases for the real part as well as for the imaginary
part of frequencies.

Figure 3: The QNM of 6th order for spin $s=0$ with $l=2,3,4,5$. The model with
bumblebee gravity is represented by circles where $\lambda=0.1$. The
Schwarzschild black hole $(\lambda=0)$ is represented by triangles.

In the next section, we apply the same methodology for the gravitational
perturbations ($s=2$), where we conclude the results are similar to the scalar
perturbation.

##  IV Quasinormal modes of gravitational field perturbations

In order to study the gravitational perturbations, the Chandrasekhar formalism
chan  was applied. As expected, in the same way as scalar perturbations, the
Regge-Wheeler equation is similar to the Schwarzschild case one. The Regge-
Wheeler potential in the context of bumblebee gravity reads

$V_{g}(r,l,\lambda)=\left(1-\frac{1}{r}\right)\left(\frac{l(l+1)}{r^{2}}-\frac{2}{r^{2}}+\left[1+\lambda\right]^{-1}\left[\frac{2}{r^{2}}-\frac{3}{r^{3}}\right]\right).$
(12)

Note the gravitational potential in Eq. (12) differs from the gravitational
potential $(s=2)$ for Schwarzschild solution in Eq. (4) by the term
$\left[1+\lambda\right]^{-1}\left[\frac{2}{r^{2}}-\frac{3}{r^{3}}\right]$,
which for $\lambda=0$ recovers the usual result.

The same transformation $x(r)$ for the tortoise coordinate presented in Eq.
(10) and same constructions for the the QNMs of (5) and (6), leads us to the
new QNM for the gravitational field which is presented in Table 3 for
$\lambda=0.1$. Once more, we compare our results in the bumblebee gravity
(Table 4) with the Schwarzschild black hole (2) in Fig. (4), now for the
gravitational perturbations. As also verified for scalar perturbations, the LV
slightly modifies the QNMs of Schwarzchild scenario.

$l\,,\,n$ | 3rd orde WKB | 6rd orde WKB
---|---|---
l = 2, n = 0 | 0.375195 - 0.081563i | 0.372867 - 0.0891936i
l = 2, n = 1 | 0.352768 - 0.250222i | 0.345894 - 0.274473i
l = 3, n = 0 | 0.60087 - 0.0846012i | 0.599221 - 0.092926i
l = 3, n = 1 | 0.586885 - 0.25624 i | 0.582443 - 0.281986i
l = 3, n = 2 | 0.562304 - 0.433172i | 0.551488 - 0.480268i
l = 4, n = 0 | 0.810376 - 0.0857977i | 0.809084 - 0.094306i
l = 4, n = 1 | 0.799958 - 0.258791i | 0.796545 - 0.284766i
l = 4, n = 2 | 0.780779 - 0.435239i | 0.772636 - 0.480641i
l = 4, n = 3 | 0.755005 - 0.615642i | 0.739674 - 0.684973i
l = 5, n = 0 | 1.01331 - 0.0863744i | 1.01225 - 0.094966i
l = 5, n = 1 | 1.00495 - 0.260035i | 1.00218 - 0.286108i
l = 5, n = 2 | 0.989174 - 0.436100i | 0.982659 - 0.480820i
l = 5, n = 3 | 0.967365 - 0.615274i | 0.954953 - 0.681248i
l = 6, n = 0 | 1.21288 - 0.0866971i | 1.01225 - 0.0949664i
l = 6, n = 1 | 1.20589 - 0.260735i | 1.00218 - 0.286108i
Table 4: The QNMs for gravitational perturbations in bumblebee gravity via the
WKB method for $\lambda=0.1$ Figure 4: The QNMs of 6th order for spin $s=2$
with $l=2,3,4,5$. The model with bumblebee gravity is represented by circles
where $\lambda=0.1$. The Schwarzschild black hole $(\lambda=0)$ is represented
by triangles.

In order to better spot the QNM differences between the bumblebee gravity and
the Schwarzschild black hole, we plot in Fig. (5) only the variations of QNM
for the 6th order. Note that, for scalar perturbations, the corrections always
represent an increasing in the real part and a decrease in the imaginary part
of frequencies. For gravitational perturbations, the corrections for the
imaginary part can be positive or negative.

Figure 5: Variations of QNMs frequencies via the WKB method for the 6th order.
The scalar perturbation is shown in the left panel, and the gravitational
perturbation in the right panel.

Moreover, in Fig. (6) the time domain for scalar and gravitational
perturbation in the black hole with bumblebee parameter is exhibited. Note
that both perturbations exhibit damping profiles, decreasing scalar modes
slower than the gravitational modes. The angular momentum $l$ parameter faster
the decay of oscillations as verified in references Rondinelly ; kono1 ; chur
.

Figure 6: Scalar damping (left panel) and Gravitational damping (right panel)
for $\lambda=0.1$ and some values of $l$.

##  V Conclusion

In this work, we apply the WKB method to compute the quasinormal modes for a
black hole in bumblebee gravity, and the results were compared to the
Schwarzschild black hole. The scalar and gravitational perturbations were
considered. The Regge-Wheeler equation was obtained, and the correction term
responsible for the Lorentz violation was addressed. Once the modified
potential still exhibits a bell-shaped, the WKB method was applied for the 3rd
and 6th order. The LV parameter generated small variations in relation to QNMs
of Schwarzschild solution. Moreover, the time domain for both perturbations
was analyzed and represent damping profiles.

## Acknowledgments

The authors thank the Conselho Nacional de Desenvolvimento Científico e
Tecnológico (CNPq), grant no 308638/2015-8 (CASA), and Coordenação de
Aperfeiçoamento do Pessoal de Nível Superior (CAPES), for financial support.

##  VI bibliography

## References

  * (1) B. P. Abbott et al. [LIGO Scientific and Virgo], Phys. Rev. Lett. 116, no.6, 061102 (2016) doi:10.1103/PhysRevLett.116.061102
  * (2) K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. 875, no.1, L1 (2019) doi:10.3847/2041-8213/ab0ec7
  * (3) S. V. M. C. B. Xavier, P. Cunha, V.P., L. C. B. Crispino and C. A. R. Herdeiro, Int. J. Mod. Phys. D 29, no.11, 2041005 (2020) doi:10.1142/S0218271820410059
  * (4) S. K. Jha and A. Rahaman, “Bumblebee gravity with a Kerr-Sen-like solution and its Shadow,” [arXiv:2011.14916 [gr-qc]].
  * (5) K. Lin, W. L. Qian, A. B. Pavan and E. Abdalla, Mod. Phys. Lett. A 32, no.25, 1750134 (2017) doi:10.1142/S0217732317501346
  * (6) R. Uniyal, H. Nandan and P. Jetzer, Phys. Lett. B 782, 185-192 (2018) doi:10.1016/j.physletb.2018.05.006
  * (7) V. A. Kostelecký and S. Samuel, Phys. Rev. D 39, 683 (1989); Phys. Rev. Lett. 63, 224 (1989). Phys. Rev. Lett. 66, 1811 (1991).
  * (8) R. Casana, A. Cavalcante, F. P. Poulis, E. B. Santos, Phys. Rev. D 97, 104001, (2018).
  * (9) C. Ding, C. Liu, R. Casana, A. Cavalcante, Eur. Phys. J. C 80(3), 178 (2020).
  * (10) R. J. Yang, H. Gao, Y. Zheng and Q. Wu, Commun. Theor. Phys. 71, no.5, 568-572 (2019) doi:10.1088/0253-6102/71/5/568
  * (11) A. Övgün, K. Jusufi, İ. Sakallı, Phys. Rev. D 99, 024042 ,(2019).
  * (12) R. Oliveira, D. M. Dantas, V. Santos and C. A. S. Almeida, Class. Quant Grav. 36, no.10, 105013 (2019)
  * (13) R. V. Maluf, J. E. G. Silva and C. A. S. Almeida, Phys. Lett. B 749, 304-308 (2015)
  * (14) R. V. Maluf and J. C. S. Neves, Phys. Rev. D 103, no.4, 044002 (2021)
  * (15) T. Regge, J. A. Wheeler, Phys. Rev. 108, 1063 (1957).
  * (16) F. J. Zerilli, Phys. Rev. Lett. 24, 737 (1970).
  * (17) C. V. Vishveshwara, Nature 227, 936 (1970).
  * (18) S. Iyer, C. M. Will, Phys. Rev. D 35, 3621 (1987).
  * (19) R. A. Konoplya, J. Phys Stud. 8, 93 (2004).
  * (20) V. Santos, R. V. Maluf, and C. A. S. Almeida, Phys. Rev. D 93, 084047 (2016).
  * (21) S. Chandrasekhar, “The Mathematical Theory Of Black Holes,” Oxford U.P. (1985).
  * (22) M. S. Churilova, Phys. Rev. D 102, 024076 (2020).
  * (23) R. A. Konoplya, A. Zhidenko, Phys. Rev. D 88, 024054 (2013).


#  C3DAG: Controlled 3D Animal Generation using 3D pose guidance

Sandeep Mishra∗
University of Texas at Austin
Austin
<EMAIL_ADDRESS>Oindrila Saha∗
Univeristy of Massachusetts
Amherst
<EMAIL_ADDRESS>Alan C. Bovik
University of Texas at Austin
Austin
<EMAIL_ADDRESS>
###### Abstract

Recent advancements in text-to-3D generation have demonstrated the ability to
generate high quality 3D assets. However while generating animals these
methods underperform, often portraying inaccurate anatomy and geometry.
Towards ameliorating this defect, we present C3DAG, a novel pose-Controlled
text-to-3D Animal Generation framework which generates a high quality 3D
animal consistent with a given pose. We also introduce an automatic 3D shape
creator tool, that allows dynamic pose generation and modification via a web-
based tool, and that generates a 3D balloon animal using simple geometries. A
NeRF is then initialized using this 3D shape using depth-controlled SDS. In
the next stage, the pre-trained NeRF is fine-tuned using quadruped-pose-
controlled SDS. The pipeline that we have developed not only produces
geometrically and anatomically consistent results, but also renders highly
controlled 3D animals, unlike prior methods which do not allow fine-grained
pose control.

Figure 1: C3DAG : A pipeline to generate highly detailed and anatomically
accurate 3D Animals. \raisebox{-.9pt} {A}⃝ A web-based tool that lets the user
modify keypoints in 3D space and generates balloon animals using simple
geometric shapes such as spheres, cylinders, and cones with a single button
click. The figure depicts the 3D pose and skeleton of an elephant and its
generated balloon animal mesh. \raisebox{-.9pt} {B}⃝ We illustrate our method
to create a pose consistent animal by sampling 2D pose control images from a
3D pose based on camera parameters. The sampled 2D poses are used as input to
our tetrapod-pose ControlNet along with a text prompt and a NeRF rendered
image to backpropagate Score Distillation Sampling (SDS) gradients to the
learnable NeRF parameters $\theta$.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

The past decade has observed dramatic improvements in the field of computer
vision, especially on 3D generative tasks. The introduction of denoising
diffusion models and implicit representations for 3D reconstruction has
dramatically improved the quality of generative art. DreamFusion [23] showed
that knowledge of 2D image generative models can be utilized to guide
generation of 3D objects. These concepts alleviate the need for substantial
datasets of 3D ground truth, by lifting 2D images and videos into 3D shapes by
distillation. With this in mind, we focus on the controlled generation of 3D
animals guided by text and pose, with the specific goal of generating
anatomically plausible 3D animals.

3D animal generation has been previously studied, yet current methods still
suffer by the creation of problematic anatomical distortions such as multiple
heads or limbs, or displaced or missing body parts. Attempts at tackling these
problems, such as PerpNeg [2], have improved the quality of generation but
only in selective cases. Another inherent issue with distillation based
methods is the amount of time required to conduct test-time optimization.
Although it is not possible to avoid test-time optimization when generating
highly detailed 3D assets, efforts to reduce the time required for
optimization is an important consideration.

Another approach to generate 3D animals involves 3D Morphable Models (3DMMs)
which are parametric models. While this approach can generate high quality 3D
shapes, due to the time-consuming data collection process of 3D scanning, the
diversity of generated animals is quite limited. Some methods utilize 2D
images and videos to create banks of such models to expand the range of
animals they can generate, but out-of-domain generation remains an unsolved
problem with this approach. Methods such as MagicPony [31], which use images
and videos to learn parametric models, are unable to capture fine details and
end up completely omitting tails and other shape details resulting in low
quality 3D meshes.

DreamWaltz [9] demonstrates the use of SMPL [15] based initialization and
projected 2D poses to generate highly controlled 3D human-like avatars, while
usually reducing inaccuracies in the generated asset. Animals take on a
tremendous number of different shapes and body proportions, so starting from
an initialization using 3DMM based models like those generated by SMPL is not
viable, as they have limited diversity. Towards remediating these limitations,
we make the following contributions:

  * •

An automatic 3D shape creator tool that generates a naive 3D shape and pose
which can be used to initialize a NeRF.

  * •

A tetrapod-pose guided ControlNet trained on a diverse dataset including
mammals, reptiles, birds and amphibians to provide highly controlled guidance
via Score Distilation Sampling (SDS).

  * •

We combine (a) and (b) in an efficient two-stage pipeline that is able to
generate high quality and anatomically consistent 3D animal assets. Our
method, which we call C3DAG, is efficient, and produces a high quality animal
much faster than current SOTA methods.

##  2 Related Work

Generating 3D assets without specific conditions requires understanding the
diverse distributions of 3D data. There are two primary strategies: explicit
and implicit. Structured representations such as point clouds [1, 17], voxel
grids [13, 28], and mesh models [34] are categorized as explicit methods.
Implicit techniques generally rely on abstract representations, such as signed
distance functions (SDFs) [6, 7, 20], tri-planes [4], the parameters of multi-
layer perceptrons (MLPs) [8], and radiance fields [16].

The recent surge in text-to-3D asset generation research has been
significantly fueled by the availability of vast datasets of text-image pairs
and the success of text-to-image generative models. These innovations have
paved the way for the adaptation of pre-trained text-to-image models into the
3D domain, primarily those that are CLIP-guided, or are 2D diffusion-guided.
CLIP-guided methods such as [21, 10] leverage cross-modal matching models like
CLIP [24] for text-to-3D conversion, whereas diffusion-guided strategies
utilize text-to-image diffusion models [35, 5], like Imagen [26] and Stable
Diffusion [25], to generate 3D assets from textual descriptions. The latter
has been noted to deliver superior text-to-3D generation performance,
employing techniques such as Score Distillation Sampling (SDS) [23], which
refines noise in images captured from NeRFs [19], and Score-Jacobian-Chaining
[29], which aggregates image gradients into 3D asset gradients.

The field of 3D animal generation has made significant progress thanks to
groundbreaking studies that have provided new methods and insights for
modeling the complex structures and movements of animals in 3D. SMAL [36]
introduced a method to fit a parametric 3D shape model, derived from 3D scans,
to animal images using 2D keypoints and segmentation masks, with extensions to
multi-view images [37]. Subsequent efforts, such as LASSIE [32], have focused
on deriving 3D shapes directly from smaller image collections by identifying
self-supervised semantic correspondences to discover 3D parts. Despite these
advances, the limited diversity of currently available 3D assets confines most
research [11, 30, 31] to the production of 3D models that are both class-
specific and small scale.

##  3 Method

We propose C3DAG: Controlled 3D Animal Generation, which is an efficient two-
stage approach to generate high-quality 3D assets visually representative of
posed tetrapods, including birds, animals and reptiles. The first stage
initializes a NeRF to a generic balloon animal created using our automatic 3D
shape creator tool. The second stage refines the initialization by using
occlusion aware quadruped-3D-pose consistent Score Distillation Sampling (SDS)
to generate high quality 3D animal assets. Unlike DreamWaltz [9], we avoid
using 3DMM models, since they are always limited to a fixed set of animals,
thus enabling C3DAG to generate a more diverse set of fauna including
quadruped land animals, reptiles and birds. We also introduce control scale
and guidance scale annealing, ensuring that the 3D optimization is able to
fully utilize the diversity and quality of Stable Diffusion. See Supplementary
Sec. 6 for more details about the prelimanary concepts used in this article.
Hereafter we describe each aspect in detail.

###  3.1 2D Pose guided ControlNet

Following the human-pose ControlNet introduced in the original paper, we train
a ControlNet that can produce various animal images including quadrupeds,
reptiles and birds using the same set of 18 keypoints. We introduce data
augmentations such as random rotations, translations, and scaling, so that the
model is robust against various factors such as occluded poses and different
scales of animals. This is crucial in the 3D generation stage as the randomly
sampled camera views of a 3D skeleton will often include heavy occlusions. We
provide more details in the section on Implementation Details.

Figure 2: Comparison with text-to-3D generation guided by 2D diffusion. Given
the prompt as shown, we qualitatively compare with the open-source version of
Dreamfusion and HiFA using their default settings. For both of these we append
“, full body” to the prompt. It is clearly observed that both prior work
suffer from various inconsistencies producing multiple heads or limbs. Stable
Dreamfusion usually produces lower details in textures. HiFA produces high-
quality textures but almost always produces anatomically incorrect animals. We
provide more results in supplementary.

###  3.2 NeRF initialization

To conduct NeRF pre-training using depth ControlNet, DreamWaltz uses the
neutral SMPL mesh. However initialization using a single generic model is
ineffective for generating tetrapods, as they vary vastly in shape and size.
Existing parametric models such as SMAL [36] also cannot be used, since they
represent only a small set of animals that does not include any birds or
reptiles. Another parametric model called 3DFauna [12], has wider diversity,
but also fails to represent elephants, and appendages like tails in certain
animals, as evidenced in the results in their original paper and website. To
remediate these issues, we instead resort to an automatic method of generating
an initial 3D mesh given a 3D skeleton, as described in Sec. 3.3. The effect
of using this pre-training is shown in supplementary.

###  3.3 Generating 3D pose and shape

We employ a set of 18 3D keypoints and 18 bone connections to represent a
generic body pose, as visualized in Fig. 1 \raisebox{-.9pt} {A}⃝. To enable
the modification of the 3D pose we created a simple THREE.js web tool that
provides a clean UI to move the 3D keypoints around. The same tool also allows
the user to initialize the 3D shape using spherical, cylindrical and conical
meshes to form the head, limbs, tails, and nose, respectively. The parameters
of the constructive components are tunable, enabling the creation of balloon
animals of different shapes and sizes. The combined components are then re-
meshed into a single mesh, which is then used in the pre-training stage using
SDS, with depth ControlNet as the guidance.

###  3.4 3D Aware Score Distillation Sampling

Given the input 3D keypoints we project 2D poses onto the camera coordinate
system, then use the projections as control images when generating pose-based-
guidance signals from our trained Tetrapod-Pose guided ControlNet. Since the
3D pose is fixed, this makes it possible to produce 3D consistent animal
views, ensuring a high quality reconstruction. We illustrate this in Fig. 1
\raisebox{-.9pt} {B}⃝. It is important to note that C3DAG uses no 3D ground-
truth data unlike MVDream [27] or Zero-1-2-3 [14], instead purely lifting 2D
poses to 3D, guided only by training on 2D image datasets. We show that
without pose control, other 2D diffusion SDS guided methods fail to maintain
geometric and anatomic consistency (Fig. 2). However, since ControlNet is
trained using a limited number of images from datasets that contain annotated
poses, exploiting the quality and diversity learned by stable diffusion is
important. Animal pose datasets inherently consider occlusion, as only visible
keypoints are annotated. In light of this, we deployed a more generic
variation of occlusion culling than that in DreamWaltz, to control the
visibility of each individual part of the head (left eye, right eye, and nose)
depending on the view-description. We utilized randomly acquired camera
parameters to determine the view description, including: front, left-side,
back, right-side, top and bottom.

Figure 3: Comparison with parametric model based method. Given the input image
3DFauna fails to capture high-frequency details and follow the input image
(see tail), whereas our method produces a highly detailed animal given input
pose and text, which closely follows the input pose control.

##  4 Implementation details

ControlNet Training: We used annotated poses from the AwA-pose [3] and Animal
Kingdom [22] datasets to train ControlNet in a similar way as the original,
which uses stable diffusion version 1.5. AwA-pose consists of 10k annotated
images covering 35 quadruped animal classes, while Animal Kingdom provides 33k
annotated images spanning 850 species, including mammals, reptiles, birds,
amphibians, fishes and insects. From a combined set of 43k samples, we
carefully selected a subset including only mammals, reptiles, birds, and
amphibians. We also eliminated any sample having less than 30% of its
keypoints annotated. The curated dataset consists of 13k annotated samples. To
increase diversity in learning, and to improve test-time generation at any
scale and transformations, we used a combination of data augmentation
strategies consisting of random rotations, translations, and scaling while
training. The model was trained over 229k iterations with a batch size of 12,
a constant learning rate of $1e^{-5}$, on a single Nvidia RTX 6000. The model
converged after around 120k iterations and would not overfit even up to 200k
iterations, owing in part to the augmentation strategy.

3D Pose and Shape generation: We used the following 18 keypoints to represent
every quadruped: left eye, right eye, nose, neck end, 4 $\times$ thighs, 4
$\times$ knees, 4 $\times$ paws, back end, and tail end. For the upper limbs
of birds, i.e. wings, their front - thighs, knees, and paws are defined in
accordance with how their upper limbs move. We began with an initial pose of a
tiger from SMAL, and modify its keypoints using the balloon animal creator
tool. An example of this process and the tool UI is depicted in Fig.
1\raisebox{-.9pt} {A}⃝. After appropriate modification of the pose the user
can press the button to create mesh around bones. This button press invokes
calls to various functions defined to create each body part, based on their
natural appearances using simple mesh components such as ellipses, cylinders,
and cones. The combined mesh and the corresponding keypoints can be downloaded
by another button click.

Mesh depth guided NeRF initialization: The mesh downloaded in the previous
step was used to provide depth maps to the pre-trained depth guided
ControlNet, which produces the gradient loss by SDS, which in turn is used to
pre-train the NeRF. The pre-training helps achieve a reasonable initial state
for the NeRF weights, which can then be refined in the final pose-guided
training stage. The diffusion model was pre-trained for 10,000 iterations
using the Adam optimizer with a learning rate of $1e-3$ and a batch size of 1.
During training, the camera positions were randomly sampled in spherical
coordinates, where the radius, azimuthal angle, and polar angle of camera
position were sampled from [1.0, 2.0], [0, 360], and [60, 120].

Pose-guided SDS for NeRF fine-tuning: Finally, we fine-tune the NeRF using the
pre-trained ControlNet to provide 2D pose guidance to SDS. The gradients
computed using the noise residual from SDS was weighted in a similar manner as
DreamFusion, where $w(t)=\sigma_{t}^{2}$ and $t$ was annealed using
$t=t_{max}-(t_{max}-t_{min})\sqrt{\frac{iter}{total\\_iters}}$. We set
$t_{max}$ to be 0.98, $t_{min}$ to be $0.4$. Similar to the previous stage, we
trained the model over $total\\_iters=10,000$ using the same settings for the
optimizer. Using cosine annealing, we reduced the control scale from an
initial value of 1 to a final value of 0.25, while updating guidance scale as
$\omega=\omega_{init}\cdot(1+\frac{iter}{total\\_iters})$, where
$\omega_{init}=50$. These settings helped reduce the impact of ControlNet
gradually over the training process, while improving quality by gradually
increasing $\omega$ of classifier-free guidance. The camera positions were
randomly sampled as in stage 1, as were the radius, azimuthal angle, and polar
angle of the camera. The 3D avatar representation renders “latent images” in
the latent space of $\mathbb{R}^{64\times 64\times 4}$ following LatentNeRF
[18], where the “latent images” can be decoded into RGB images of
$\mathbb{R}^{512\times 512\times 4}$ using the VAE decoder of Stable Diffusion
[25].

##  5 Discussion and Conclusion

We showed our method performs consistently well (Fig 2) while Stable
DreamFusion and HiFA usually produce physically implausible or low-quality
assets. It is important to note that HiFA, which is a current SOTA model,
requires about 7 hours on a 80GB A100 GPU, while ours needs only about 20
minutes on the same setting. Fig 3 shows that our generated animal contains
much more detail than the one produced by 3DFauna [12]. Our method currently
requires some human effort to re-position 3D keypoints in our balloon animal
creator tool. It is a direction of future work to fully automate this process.
To summarize, we present C3DAG – an efficient method to generate anatomically
and geometrically consistent 3D animals. We contribute 1) a 2D Tetrapod-pose
ControlNet trained on mammals, amphibians, reptiles and birds, 2) a tool to
create 3D balloon animals automatically given 3D keypoints, and 3) a method to
pipeline 1) and 2) to create high quality 3D animals.

## References

  * Achlioptas et al. [2018] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas.  Learning representations and generative models for 3d point clouds.  In _International Conference on Machine Learning_ , pages 40–49. PMLR, 2018.
  * Armandpour et al. [2023] Mohammadreza Armandpour, Ali Sadeghian, Huangjie Zheng, Amir Sadeghian, and Mingyuan Zhou.  Re-imagine the negative prompt algorithm: Transform 2d diffusion into 3d, alleviate janus problem and beyond.  _arXiv preprint arXiv:2304.04968_ , 2023.
  * Banik et al. [2021] Prianka Banik, Lin Li, and Xishuang Dong.  A novel dataset for keypoint detection of quadruped animals from images.  _arXiv preprint arXiv:2108.13958_ , 2021.
  * Chen et al. [2023a] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su.  Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2416–2425, 2023a.
  * Chen et al. [2023b] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22246–22256, 2023b.
  * Chen and Zhang [2019] Zhiqin Chen and Hao Zhang.  Learning implicit fields for generative shape modeling.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5939–5948, 2019.
  * Cheng et al. [2023] Yen-Chi Cheng, Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang-Yan Gui.  Sdfusion: Multimodal 3d shape completion, reconstruction, and generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4456–4465, 2023.
  * Erkoç et al. [2023] Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai.  Hyperdiffusion: Generating implicit neural fields with weight-space diffusion.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 14300–14310, 2023.
  * Huang et al. [2024] Yukun Huang, Jianan Wang, Ailing Zeng, He Cao, Xianbiao Qi, Yukai Shi, Zheng-Jun Zha, and Lei Zhang.  Dreamwaltz: Make a scene with complex 3d animatable avatars.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Jain et al. [2022] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole.  Zero-shot text-guided object generation with dream fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 867–876, 2022.
  * Jakab et al. [2023] Tomas Jakab, Ruining Li, Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.  Farm3d: Learning articulated 3d animals by distilling 2d diffusion.  _arXiv preprint arXiv:2304.10535_ , 2023.
  * Li et al. [2024] Zizhang Li, Dor Litvak, Ruining Li, Yunzhi Zhang, Tomas Jakab, Christian Rupprecht, Shangzhe Wu, Andrea Vedaldi, and Jiajun Wu.  Learning the 3d fauna of the web.  _arXiv preprint arXiv:2401.02400_ , 2024.
  * Lin et al. [2023] Chieh Hubert Lin, Hsin-Ying Lee, Willi Menapace, Menglei Chai, Aliaksandr Siarohin, Ming-Hsuan Yang, and Sergey Tulyakov.  Infinicity: Infinite-scale city synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 22808–22818, 2023.
  * Liu et al. [2023] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9298–9309, 2023.
  * Loper et al. [2023] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  Smpl: A skinned multi-person linear model.  In _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , pages 851–866. 2023.
  * Lorraine et al. [2023] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, and James Lucas.  Att3d: Amortized text-to-3d object synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 17946–17956, 2023.
  * Luo and Hu [2021] Shitong Luo and Wei Hu.  Diffusion probabilistic models for 3d point cloud generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2837–2845, 2021.
  * Metzer et al. [2023] Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or.  Latent-nerf for shape-guided generation of 3d shapes and textures.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12663–12673, 2023.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Mittal et al. [2022] Paritosh Mittal, Yen-Chi Cheng, Maneesh Singh, and Shubham Tulsiani.  Autosdf: Shape priors for 3d completion, reconstruction and generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 306–315, 2022.
  * Mohammad Khalid et al. [2022] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa.  Clip-mesh: Generating textured meshes from text using pretrained image-text models.  In _SIGGRAPH Asia 2022 Conference papers_ , pages 1–8, 2022.
  * Ng et al. [2022] Xun Long Ng, Kian Eng Ong, Qichen Zheng, Yun Ni, Si Yong Yeo, and Jun Liu.  Animal kingdom: A large and diverse dataset for animal behavior understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19023–19034, 2022.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Shi et al. [2023] Yichun Shi, Peng Wang, Jianglong Ye, Long Mai, Kejie Li, and Xiao Yang.  Mvdream: Multi-view diffusion for 3d generation.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Smith and Meger [2017] Edward J Smith and David Meger.  Improved adversarial systems for 3d object generation and reconstruction.  In _Conference on Robot Learning_ , pages 87–96. PMLR, 2017.
  * Wang et al. [2023] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich.  Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12619–12629, 2023.
  * Wu et al. [2023a] Shangzhe Wu, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  Dove: Learning deformable 3d objects by watching videos.  _International Journal of Computer Vision_ , 131(10):2623–2634, 2023a.
  * Wu et al. [2023b] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.  Magicpony: Learning articulated 3d animals in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8792–8802, 2023b.
  * Yao et al. [2022] Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani.  Lassie: Learning articulated shapes from sparse image ensemble via 3d part discovery.  _Advances in Neural Information Processing Systems_ , 35:15296–15308, 2022.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.
  * Zhang et al. [2021] Song-Hai Zhang, Yuan-Chen Guo, and Qing-Wen Gu.  Sketch2model: View-aware 3d modeling from single free-hand sketches.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6012–6021, 2021.
  * Zhu et al. [2023] Junzhe Zhu, Peiye Zhuang, and Sanmi Koyejo.  Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Zuffi et al. [2017] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black.  3D menagerie: Modeling the 3D shape and pose of animals.  In _IEEE Conf. on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Zuffi et al. [2018] Silvia Zuffi, Angjoo Kanazawa, and Michael J Black.  Lions and tigers and bears: Capturing non-rigid, 3d, articulated shape from images.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3955–3963, 2018.

Supplementary Material

##  6 Preliminaries

Recent works have shown impressive results in 2D and 3D content generation
using a combination of the following:

  * •

Diffusion Model:
Generative pre-training on extensive image-text datasets using denoising
diffusion models serves as effective priors for text-to-3D generation. These
models undergo two primary processes:

Forward Process: This involves incrementally adding noise to the data $x\sim
p(x)$, transforming it towards a Gaussian distribution over $T$ steps. The
noise-augmented data at step $t$ is described by:

$z_{t}=\sqrt{\overline{\alpha}_{t}}x+\sqrt{1-\overline{\alpha}_{t}}\epsilon,\quad\epsilon\sim\mathcal{N}(0,I),$

where $\overline{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$ and
$\alpha_{t}\in(0,1)$ as part of the predefined noising schedule.

Backward Process: In this phase, the model learns to reverse the noise
addition, aiming to reconstruct the original data from its noised version. The
models learns to estimate the noise by minimizing

$L_{t}=\mathbb{E}_{x,\epsilon\sim\mathcal{N}(0,I)}\left[\|\epsilon-\epsilon_{\phi}(z_{t},t)\|^{2}\right].$

This dual process enables the diffusion model to accurately estimate and
reconstruct $x$ from noisy observations, effectively leveraging clean data
from noised counterparts.

  * •

Neural Radiance Fields (NeRF):
Widely embraced for text-to-3D generation tasks, NeRFs utilize a trainable
Multilayer Perceptron (MLP) parameterized by $\theta$ as their core 3D
representation. When rendering, a set of rays $r(k)=o+kd$ are sampled, where
$o$ represents the camera position and $d$ the direction, calculated on a per-
pixel basis. The MLP processes each ray $r(k)$, outputting the density $\tau$
and color $c$ for each sampled point. The final color $C$ of a pixel is then
determined by approximating the volume rendering integral using numerical
quadrature:

$\hat{C}(r)=\sum_{i=1}^{N_{c}}\Omega_{i}\cdot(1-\exp(-\tau_{i}\delta_{i}))c_{i},$

where $N_{c}$ is the number of points sampled along a ray,
$\Omega_{i}=\exp\left(-\sum_{j=1}^{i-1}\tau_{j}\delta_{j}\right)$ is the
accumulated transmittance, and $\delta_{i}$ is the distance between adjacent
sample points.

  * •

Score Distillation Sampling (SDS):
Introduced by DreamFusion and further utilized in various studies, SDS is a
method designed to transfer the knowledge from a pre-trained diffusion model
$\epsilon_{\phi}$ into a differentiable 3D representation, enhancing a NeRF
model’s parameters $\theta$. The rendered output $x$ from a NeRF model can be
obtained via $x=g(\theta)$, where $g$ signifies a differentiable rendering
function. The core of SDS lies in computing the gradients of the NeRF
parameters $\theta$, formulated as:

$\nabla_{\theta}L_{\text{SDS}}(\phi,x)=\mathbb{E}_{t,\epsilon}\left[w(t)\left(\epsilon_{\phi}(x_{t};y,t)-\epsilon\right)\frac{\partial
z_{t}}{\partial x}\frac{\partial x}{\partial\theta}\right],$

where $w(t)$ is a timestep-dependent weighting function and $y$ is the input
text prompt.

Figure 4: Effect of NeRF pre-training using shape generated by the automatic
3D shape creator tool. The top row shows the final generated 3D without any
initial pre-training stage. This has visible shortcomings in form of body
occlusion as well as extra volume in the foreground which does not belong to
the body of the animal. In contrast, fine-tuning after a pre-training stage
generates a clean result.

  * •

ControlNet guided SDS:
A novel approach that extends the principles of Score Distillation Sampling
(SDS) to include conditioning on additional image-based information,
facilitating enhanced control when generating 3D models. Like SDS, ControlNet
[33] leverages a pre-trained diffusion model $\epsilon_{\phi}$ but also
introduces a conditioning image $c$, which could be a skeleton/pose map, depth
map, normal map, or a combination thereof, to guide the generation process
more precisely. For a model parameterized by $\theta$, and its output $x$,
ControlNet refines the gradient calculation of the parameters $\theta$ by
incorporating $c$, as shown in:

$\nabla_{\theta}L_{\text{SDS}}(\phi,x)=\mathbb{E}_{t,\epsilon}\left[w(t)\left(\epsilon_{\phi}(x_{t};y,t,{\color[rgb]{1,0,0}c})-\epsilon\right)\frac{\partial
z_{t}}{\partial x}\frac{\partial x}{\partial\theta}\right],$

where $w(t)$ is a weighting function dependent on the timestep $t$, $y$
denotes the given text prompt, and $c$ is the conditioning image that
significantly influences the generation process. This approach benefits by
ControlNet’s innovative method of integrating additional contextual cues into
the generative framework, thereby enriching the model’s capacity to compute
detailed and accurate 3D reconstructions using complex conditioning
information.

##  7 Impact of NeRF Pre-training

Fig. 4 shows the impact of the NeRF pre-training stage using depth ControlNet
as described in Sec 3.2 using the 3D balloon animal generated by the automatic
3D shape creator tool. This shows that using the pre-training stage results in
visibly cleaner results without any body occlusions.


# EVE: Efficient zero-shot text-based Video Editing
with Depth Map Guidance and Temporal Consistency Constraints

Yutao Chen1111Yutao Chen and Xingning Dong contribute equally to this
manuscript., Xingning Dong2111Yutao Chen and Xingning Dong contribute equally
to this manuscript., Tian Gan1
Chunluan Zhou2, Ming Yang2, Qingpei Guo2222Qingpei Guo is the corresponding
author.
1Shandong University, 2Ant Group
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Motivated by the superior performance of image diffusion models, more and more
researchers strive to extend these models to the text-based video editing
task. Nevertheless, current video editing tasks mainly suffer from the dilemma
between the high fine-tuning cost and the limited generation capacity.
Compared with images, we conjecture that videos necessitate more constraints
to preserve the temporal consistency during editing. Towards this end, we
propose EVE, a robust and efficient zero-shot video editing method. Under the
guidance of depth maps and temporal consistency constraints, EVE derives
satisfactory video editing results with an affordable computational and time
cost. Moreover, recognizing the absence of a publicly available video editing
dataset for fair comparisons, we construct a new benchmark ZVE-50 dataset.
Through comprehensive experimentation, we validate that EVE could achieve a
satisfactory trade-off between performance and efficiency. We will release our
dataset and codebase to facilitate future researchers.

We present EVE, an efficient and robust zero-shot text-based video editor,
which successfully trades off editing performance and efficiency.

##  1 Introduction

Owing to powerful diffusion models [25, 10], recent years have witnessed
dramatic progress in text-based image synthesis and editing tasks, igniting
the soaring research interest in extending these methods to the video editing
field. Nevertheless, current text-based video editing methods, which
manipulate attributes or styles of videos under the guidance of the driven
text, mainly suffer from the dilemma between the considerable fine-tuning cost
and the unsatisfied generation performance.

Recent video editing methods could be roughly divided into two classes:
tuning-based methods [27, 23] and zero-shot ones [2, 15]. The former
approaches mainly rely on fine-tuning image diffusion models to derive strong
generative priors. Nevertheless, they are usually costly as the fine-tuning
step would consume substantial time and GPUs. Towards this end, zero-shot
video editing methods aim to directly edit real-world videos without time-
consuming fine-tuning. Nevertheless, the edited videos in the zero-shot manner
may suffer from the spatio-temporal distortion and inconsistency. Besides,
some zero-shot methods are built upon diffusion models fine-tuned on video
datasets, which may not be free of the high cost as the tuning-based ones.

In this paper, we attempt to achieve a trade-off between editing performance
and efficiency. Specifically, we adopt the approach of zero-shot video
editing, while improving editing performance based upon initial image
diffusion models rather than video tuning-based ones. Consequently, the
primary challenge is how to preserve and improve the temporal consistency of
edited videos.

Let’s begin by considering human editing. When dealing with images, adjusting
object appearances or attributes is relatively straightforward. However, when
it comes to videos, a comprehensive evaluation of all edited frames becomes
imperative to prevent the spatio-temporal distortion and inconsistency in
edited videos. As a result, we conjecture that videos necessitate more
temporal constraints to preserve the time consistency, whose editing process
could not be as unconstrained as images. This hypothesis also interprets
unsatisfied performance when directly extending image diffusion models to
videos, as current image editing methods seldom enforce explicit constraints.

Given this argument, different from current methods that neither explicitly
control over individual frame editing nor enforce additional constricts on
inter-frame generation, we propose two strategies to reinforce temporal
consistency constraints during zero-shot video editing: 1) Depth Map Guidance.
Depth maps locate spatial layouts and motion trajectories of moving objects,
providing robust prior cues for the given video. Therefore, we incorporate
depth maps into video editing to improve the temporal consistency. And 2)
Frame-Align Attention. We enhance the temporal encoding by forcing models to
place their attentions on both previous and current frames.

Moreover, by narrowing the gap of whether introducing depth maps into the
noise-to-image inference procedure, we design an efficient parameter
optimization strategy that directly updates target latent features without
fine-tuning the complex diffusion model. In this way, it takes about 83.1
seconds to edit a video with 8 frames on average.

Currently, there lack public video editing datasets for fair performance
comparisons. Towards this end, we construct a new ZVE-50 dataset, where each
collected video is associated with four corresponding driven text. We conduct
extensive experiments to benchmark our ZVE-50 dataset.

Our contributions are summarized in four folds:

  * •

We propose EVE, a zero-shot text-based video editor with a satisfactory trade-
off between the generation capability and efficiency.

  * •

We argue the indispensability of temporal consistency constraints in the video
editing task. Towards this end, we propose two strategies to improve the
temporal consistency, achieving robust editing performance.

  * •

We construct a new benchmark ZVE-50 dataset. To the best of our knowledge,
ZVE-50 is the first dataset for zero-shot text-based video editing, which
facilitates future researchers to perform a fair comparison.

  * •

We conduct extensive experiments on ZVE-50 dataset. Experimental results
indicate that the proposed EVE is a robust and efficient zero-shot video
editing method.

##  2 Related Work

Diffusion Models. Large-scale diffusion models [18, 3, 4] have achieved start-
of-the-art performance in image synthesis and translation. Diffusion models,
in essence, are generative probabilistic models that approximate a data
distribution $p(x)$ by gradually denoising a normally distributed variable.
Nevertheless, training a diffusion model from scratch is often expensive and
time-consuming.

Text-based Image Editing. Text-based image editing [11, 26, 1] aims to
manipulate the attributes or styles of one image with the guidance of the
driven text. Based on powerful diffusion models, researchers have proposed
various methods. E.g., DreamBooth [19] proposes a subject-driven generation
technology by fine-tuning diffusion models, while T2I-Adapters [13] provides
an efficient image editing approach with a low training cost.

Text-based Video Generation and Editing. Motivated by text-based image
editing, video editing [12, 8] has attracted increasing research interest
recently, which could be roughly divided into two categories: tuning-based
ones [5] and zero-shot ones [15]. The former approaches mainly edit video
attributes by fine-tuning powerful image diffusion models, whose training cost
is inevitably expensive. Alternatively, FateZero [15] proposes the zero-shot
video editing task, attempting to generate a text-driven video without extra
optimization on complicated generative priors. Nevertheless, FateZero suffers
from the limited video editing performance due to weak constraints on the
temporal consistency. Moreover, FateZero still heavily relies on Tune-A-Video
[23], which is a tuning-based diffusion model and is still costly and time-
consuming.

Figure 1: The SIMPLIFIED version of the proposed EVE, presenting the overall
video editing pipeline. Figure 2: The ELABORATED version of the proposed EVE,
detailing the DDIM inversion and denoising procedures.

##  3 Methodology

###  3.1 Preliminary: DMs, LDMs, and DDIMs

Diffusion Models (DMs) [20] are essentially generative probabilistic models
that approximate a data distribution $p(x)$ by gradually denoising a normally
distributed variable. Specifically, diffusion models learn to reconstruct the
reverse process of a fixed forward Markov chain
${x}_{1},{x}_{2},\cdots,{x}_{T}$, where ${T}$ is the length. The forward
Markov chain ($1\rightarrow T$) could be treated as an image-to-noise
procedure, where each Markov transition step $q({x}_{t}|{x}_{t-1})$ is usually
formulated as a Gaussian distribution ($\mathcal{N}$) with a variance schedule
${\beta}_{t}\in(0,1)$, that is:

$q({x}_{t}|{x}_{t-1})=\mathcal{N}({x}_{t};\sqrt{1-{\beta}_{t}}{x}_{t-1},{\beta}_{t}{\mathbf{I}}).$
(1)

The reverse Markov chain ($T\rightarrow 1$) could be treated as a noise-to-
image procedure, where each reverse Markov transition step
$p({x}_{t-1}|{x}_{t})$ is formulated as:

${p}_{\theta}({x}_{t-1}|{x}_{t})=\mathcal{N}({x}_{t-1};{\mu}_{\theta}({x}_{t},t),{\Sigma}_{\theta}({x}_{t},t)),$
(2)

where $\theta$ denotes learnable parameters to guarantee that the reverse
process is close to the forward one.

Empirically, current diffusion models could be interpreted as an equally
weighted sequence of denoising auto-encoders ${\epsilon}_{\theta}({x}_{t},t)$,
which is utilized to recover a denoised variant of their input ${x}_{t}$, and
${x}_{t}$ is a noisy version of the input ${x}$. The optimization objective
could be simplified as:

$\mathbb{E}_{x,\epsilon\sim\mathcal{N}(0,1),t}[{\|\epsilon-{\epsilon}_{\theta}({x}_{t},t)\|}_{2}^{2}].$
(3)

Latent Diffusion Models (LDMs) [18] are trained in the learned latent space
${z}_{t}$ rather than redundant spatial dimensionality ${x}_{t}$, aiming to
remove the noise added to latent image features ${\epsilon}_{x}$. LDMs are
generally composed of an encoder $\mathcal{E}$, a time-conditional UNet
$\mathcal{U}$, and a decoder $\mathcal{D}$, where $z=\mathcal{E}(x)$ and
$x\approx\mathcal{D}(\mathcal{E}(x))$. The optimization objective could be
formulated as:

$\mathbb{E}_{{\epsilon}_{x},\epsilon\sim\mathcal{N}(0,1),t}[{\|\epsilon-{\epsilon}_{\theta}({z}_{t},t)\|}_{2}^{2}].$
(4)

Denoising Diffusion Implicit Models (DDIMs) [11] could accelerate the sampling
from the distribution of images/videos at the denoising step. During
inference, deterministic DDIM sampling ($T\rightarrow 1$) aims to recover a
clean latent ${z}_{0}$ from a random noise ${z}_{T}$ with a noise schedule
${\alpha}_{t}$, which could be formulated as:

${z}_{t-1}=\sqrt{\frac{{\alpha}_{t-1}}{{\alpha}_{t}}}{z}_{t}+(\sqrt{1-{{\alpha}_{t-1}}}-\sqrt{\frac{1}{{\alpha}_{t}}-1})\cdot{\epsilon}_{\theta}.$
(5)

On the contrary, DDIM inversion ($1\rightarrow T$) aims to process a clean
latent ${z}_{0}$ into a noise one ${\hat{z}}_{T}$, which could be simplified
as:

${\hat{z}}_{t}=\sqrt{\frac{{\alpha}_{t}}{{\alpha}_{t-1}}}{\hat{z}}_{t-1}+(\sqrt{1-{{\alpha}_{t}}}-\sqrt{\frac{1}{{\alpha}_{t-1}}-1})\cdot{\epsilon}_{\theta}.$
(6)

Compared with conventional DMs that directly employ random noise as inputs and
attempt to map each noise vector to a specific image, we exploit DDIM
inversion to produce a $T$ steps trajectory between the clean latent ${z}_{0}$
to a Gaussian noise vector ${z}_{T}$. Then we treat ${z}_{T}$ as the start
vector of the denoising step. This configuration seems appropriate for our
video editing task, since it ensures that the generated video would be close
to the original one.

Note that we employ LDMs and DDIM inversion/denoising in zero-shot text-based
video editing. Readers can refer to [18] (LDM) and [21] (DDIMs) for more
details of formulation derivations if necessary.

###  3.2 Problem Formulation

Given a video $V$ and a prompt text $P$, zero-shot text-based video editing
aims to generate an edited video $\hat{V}$, which aligns with the description
outlined in the prompt $P$ and looks similar to the original video $V$.

###  3.3 Overall Framework

As shown in Figure 1 and 2, we present both simplified and elaborated versions
of the overall framework. The simplified version could be treated as a flow
chart that reveals the whole processing pipeline of our EVE. While the complex
one presents detailed information mainly on the iterative DDIM inversion and
denoising procedures.

As shown in Figure 1, our EVE is built upon the pre-trained latent diffusion
model (LDM), which is composed of a UNet for T-timestep DDIM inversion and
denoising. To enforce the temporal consistency of the generated video, we
introduce depth maps and exploit them to guide the editing process. Moreover,
we propose two consistency constraints to prevent edited videos from spatial
or temporal distortion.

We first present the overall pipeline of our EVE based upon Figure 2,
including the following five steps.

1\. Frozen Features Extraction. Given a video $V$, we first derive $K$ frames
from $V$, and utilize an image encoder $\mathcal{E}_{I}$ to obtain frozen
latent features ${\mathbf{Z}}_{0}=\mathcal{E}(V)$, where
${\mathbf{Z}}_{0}=\\{{z}_{0}^{i}\\}_{i=1}^{K}$. Meanwhile, we employ the MiDas
Detector [17] to generate $K$ depth maps from $V$, and utilize another visual
encoder $\mathcal{E}_{M}$ to obtain frozen depth-map features
${\mathbf{M}}=\\{{m}^{i}\\}_{i=1}^{K}$. Moreover, we utilize a text encoder
$\mathcal{E}_{p}$ to process the prompt $P$ into frozen features $p$.

2\. DDIM Inversion. Then we repeat DDIM inversion for $T$ steps to derive
Gaussian noise vectors ${\textbf{Z}}_{T}$ from video latent features
${\textbf{Z}}_{0}$. Each DDIM inversion at timestep $t$ could be formulated
as:

${\mathbf{Z}}_{t}={\rm{DDIM}_{inv}}({\mathbf{Z}}_{t-1}\ |\
{\mathbf{M}},t)\quad t=1\rightarrow T,$ (7)

where ${\rm{DDIM}_{inv}}$ denotes DDIM inversion shown in Eq. 6.

To prevent the edited video from temporal distortion and inconsistency, we
improve the image-based DDIM inversion operation by introducing depth-map
features into the down-sampling pass of the frozen UNet, which could rectify
the discrepancies among neighboring frames at each inversion step. In this
way, we ensure that the generated noise vectors ${\textbf{Z}}_{T}$ would not
severely spoil the temporal consistency.

Specifically, we repeat $T$ DDIM inversion steps to process video latent
features ${\mathbf{Z}}_{0}$ into generated noise vectors ${\mathbf{Z}}_{T}$

3\. DDIM Denoising. Afterward, we repeat DDIM denoising for $T$ steps to
obtain edited video features ${\mathbf{\hat{Z}}}_{0}$ from DDIM inverted noise
${\mathbf{\hat{Z}}}_{T}$, where ${\mathbf{\hat{Z}}}_{T}={\mathbf{Z}}_{T}$.
Each DDIM denoising at timestep $t$ could be formulated as:

${\mathbf{\hat{Z}}}_{t-1}={\rm{DDIM}_{den}}({\mathbf{\hat{Z}}}_{t}\ |\
p,{\mathbf{M},t}),\quad t=T\rightarrow 1,$ (8)

where ${\rm{DDIM}_{den}}$ denotes DDIM denoising shown in Eq. 5.

To prevent the edited video from temporal distortion and inconsistency, we
improve the image-based DDIM denoising operation from two aspects: 1) We
introduce depth-map features into the down-sampling pass of the frozen UNet as
DDIM inversion. And 2) we propose the frame-aligned attention to place
explicit temporal constraints on the edited video, which is discussed in the
following subsection.

Specifically, we repeat $T$ DDIM denoising steps to otbain edited video
features ${\mathbf{\hat{Z}}}_{0}$ from DDIM inverted noise
${\mathbf{\hat{Z}}}_{T}$

4\. Parameter Optimization. To reduce the computation cost and make the
generation process more efficient, we freeze all feature extractors (i.e.,
$\mathcal{E}_{I}$, $\mathcal{E}_{M}$, and $\mathcal{E}_{P}$) and Unets, and
only set noise vectors ${\mathbf{\hat{Z}}}_{t}$ in DDIM denoising to be
trainable. In another word, different from conventional editing methods that
update “neural networks”, we directly update “latent noise” to obtain edited
videos.

Specifically, at each timestep $t$ in DDIM denoising, except for
${\mathbf{\hat{Z}}}_{t-1}$, we also derive auxiliary vectors
${\mathbf{\hat{Z}}}_{t-1}^{{}^{\prime}}$ as:

${\mathbf{\hat{Z}}}_{t-1}^{{}^{\prime}}={\rm{DDIM}_{den}}({\mathbf{\hat{Z}}}_{t}\
|\ p,t),\quad t=T\rightarrow 1.$ (9)

Compared with ${\mathbf{\hat{Z}}}_{t-1}$ (Eq. 8),
${\mathbf{\hat{Z}}}_{t-1}^{{}^{\prime}}$ is obtained without strict depth map
constraints, which could be treated as free image editing that could unleash
the generation capacity of powerful image-based diffusion models. In brief,
${\mathbf{\hat{Z}}}_{t-1}$ sacrifices the creativity to preserve the temporal
consistency, while ${\mathbf{\hat{Z}}}_{t-1}^{{}^{\prime}}$ is just the
opposite. Therefore, we leverage the more creative
${\mathbf{\hat{Z}}}_{t-1}^{{}^{\prime}}$ and more temporal consistent
${\mathbf{\hat{Z}}}_{t-1}$, pursuing to achieve a trade-off between diversity
and quality.

The detailed DDIM denoising procedure is illustrated in Algorithm 1, including
the parameter optimization step (Lines 4-5). ${\Delta}_{x}(\mathcal{L})$
denotes updating trainable $x$ by the gradient descent procedure according to
the loss $\mathcal{L}$, and ${\rm cosin}(\cdot,\cdot)$ denotes the cosine
similarity computation.

Input: DDIM inverted noise ${\mathbf{\hat{Z}}}_{T}$, text prompt features $p$,
depth-map features $\mathbf{M}$, learning rate $\lambda$

1

Output: edited video features ${\mathbf{\hat{Z}}}_{0}$

2

3

4for  _$i\leftarrow T$ to $1$_ do

5

6 ${\mathbf{\hat{Z}}}_{t-1}={\rm{DDIM}_{den}}({\mathbf{\hat{Z}}}_{t}\ |\
p,{\mathbf{M}},t)$ ;

7

8
${\mathbf{\hat{Z}}}_{t-1}^{{}^{\prime}}={\rm{DDIM}_{den}}({\mathbf{\hat{Z}}}_{t}\
|\ p,t)$ ;

9

10 $\mathcal{L}=1-{\rm
cosin}({\mathbf{\hat{Z}}}_{t-1},\mathbf{\hat{Z}}_{t-1}^{{}^{\prime}})$

11
$\mathbf{\hat{Z}}_{t-1}=\mathbf{\hat{Z}}_{t-1}-\lambda{\Delta}_{{\hat{Z}}_{t-1}}(\mathcal{L})$

12 end for

13

Algorithm 1 DDIM Denoising Procedure.

5\. Edited Video Decoding. Ultimately, we feed the frozen visual decoder
$\mathcal{D}$ with generated latent features ${\mathbf{\hat{Z}}}_{0}$,
obtaining the edited video $\hat{V}=\mathcal{D}({\mathbf{\hat{Z}}}_{0})$.

###  3.4 Temporal Consistency Constraints

As aforementioned, we assume that videos necessitate more temporal constraints
to preserve the time consistency. Therefore, we propose two strategies to
alleviate temporal distortion and inconsistency problems.

1\. Depth Map Guidance. Depth maps record visual representations of the
distance information, revealing spatial layouts and motion trails of all
objects within a video. Therefore, depth maps could be treated as strong prior
cues to guide the video editing procedure close to the initial version.
Nevertheless, recent video editing methods seldom take advantage of depth maps
and neglect the significance of explicitly intervening in the video editing
procedure, resulting in intractable temporal distortion and inconsistency
problems. Towards this end, we introduce depth maps into the down-sampling
pass of the frozen UNet for both DDIM inversion and denoising procedures,
forcing the editing process to imitate motion trails and scene transformations
of the origin video. In this way, the stability and consistency of the edited
video would be improved.

Figure 3: The architecture of the attention block within the UNet. Note that
we propose the Frame-Align Attention to improve the temporal consistency.

2\. Frame-Align Attention. We propose the frame-align attention (FAA) to
explicitly introduce the temporal information during video editing. As
illustrated in Figure 3, a typical UNet comprises a series of “Conv-Attn”
blocks to conduct the down-sampling and up-sampling calculation. The
conventional attention block (Attn) contains a self-attention (SA) module
[22], a cross-attention (CA) module [28], and a feed-forward network (FFN).
The computation of ${\rm SA}(Q,K,V)$ and ${\rm CA}(Q,K,V)$ could be formulated
as:

$\left\\{\begin{aligned}
{\rm{SA}}:Q={W}^{Q}{\dot{z}}^{i},K={W}^{K}{\dot{z}}^{i},V={W}^{V}{\dot{z}}^{i},\\\
{\rm{CA}}:Q={W}^{Q}{\dot{z}}^{i},K={W}^{K}{p},\
\;V={W}^{V}{p},\end{aligned}\right.$ (10)

where ${W}$ denotes frozen projection matrices, ${\dot{z}}^{i}$ is the latent
features of the ${i}^{th}$ frame within the video, and ${p}$ is the latent
features of the text prompt.

Conventional self-attention modules are inherited from image diffusion models,
which encodes each frame separately and seem insufficient in preserving the
temporal consistency for video editing. Therefore, we propose the frame-align
attention (FAA) to replace $K$ and $V$ with the first frame features
${\dot{z}}^{1}$, forcing models to emphasize both previous and current frames
for better temporal encoding. The computation of ${\rm FAA}(Q,K,V)$ could be
formulated as:

${\rm{FAA}}:Q={W}^{Q}{\dot{z}}^{i},K={W}^{K}{\dot{z}}^{1},V={W}^{V}{\dot{z}}^{1}.$
(11)

No. | Model | DMG | Attn | Temporal Consistency | Prompt Consistency
---|---|---|---|---|---
OR | OA | ST | BC | AVG | OR | OA | ST | BC | AVG
Performance comparisons between the proposed EVE and FateZero.
A1 |  FateZero |  - |  STSA |  95.53 |  95.64 |  95.84 |  96.62 |  95.91 |  28.92 |  29.43 |  29.24 |  28.90 |  29.12
A2 |  EVE |  $\surd$ |  FAA |  96.41 |  96.65 |  96.43 |  96.70 |  96.54 |  30.39 |  31.01 |  31.27 |  28.94 |  30.40
Ablation study of two proposed temporal consistency constraints within EVE.
B1 |  |  - |  SA |  92.59 |  92.94 |  92.07 |  96.30 |  93.48 |  25.39 |  26.43 |  27.38 |  28.88 |  27.02
B2 |  |  - |  FAA |  95.26 |  95.20 |  95.16 |  95.68 |  95.33 |  27.77 |  29.37 |  29.11 |  29.71 |  28.99
B3 |  EVE |  $\surd$ |  SA |  94.57 |  94.02 |  94.88 |  95.45 |  94.73 |  30.58 |  31.12 |  31.16 |  28.45 |  30.33
B4 |  |  $\surd$ |  SCA |  95.74 |  96.18 |  96.13 |  96.62 |  96.17 |  30.26 |  31.01 |  31.00 |  28.91 |  30.29
B5 |  |  $\surd$ |  FAA |  96.41 |  96.65 |  96.43 |  96.70 |  96.54 |  30.39 |  31.01 |  31.27 |  28.94 |  30.40
Table 1: Performance comparisons between our EVE with FateZero, and ablation
study of two proposed temporal consistency constraints within EVE. We report
the detailed results of four video editing missions and their average
performance (underlined), where OR = Object Replacement, OA = Object Adding,
ST = Style Transfer, and BC = Background Changing. All experiments are
conducted on one A40 GPU under the same setting. “DMG” denotes with/without
the depth map guidance.

##  4 Experiments

###  4.1 Dataset Construction

Since zero-shot text-based video editing is a novel task, to the best of our
knowledge, there lacks a public dataset to perform fair performance and
efficiency comparisons. Towards this end, we construct Zero-shot Video Editing
50 (dubbed as ZVE-50) to fulfill this job.

Data Collection. We collect videos from two resources: DAVIS-2017 [14] and
stock-video-footage ***https://www.videvo.net/stock-video-footage/. DAVIS-2017
is a competition dataset for the video object segmentation task [24], while
stock-video-footage is a public website for free stock video clips and motion
graphics. After filtering out videos with similar scenes and styles to avoid
the repeatability and promote the diversity, we collect 14 short videos from
DAVIS-2017 and 36 ones from stock-video-footage, resulting in the ZVE-50
dataset.

Caption Generation. Then we feed collected videos into BLIP2 [9] to obtain the
corresponding captions. Specifically, we generate several candidate captions
and select the longest one as the ground-truth text.

Prompt Generation. Afterward, we employ GPT-4 †††https://openai.com/gpt-4 to
generate the driven text derived from video captions and our manually made
prompts. There are four types of driven text, requiring models to edit the
given video by 1) Object Replacement (OR), 2) Object Adding (OA), 3) Style
Transfer (ST), and 4) Background Changing (BC). Here we present an example of
feeding GPT-4 with the manually written prompt and ground-truth caption to
obtain the driven text:

Q (human): Here is a sentence. Please replace the object with another object
with a similar shape: “a pink lotus flower in the water with green leaves”

A (GPT-4): a pink lotus flower floating in a tranquil koi pond with lily pads

Ultimately, we manually check all videos, captions, and prompt text to ensure
the correctness of the ZVE-50 dataset.

###  4.2 Experimental Settings

Implementation Details. Zero-shot text-based video editing directly takes a
given video and outputs its edited version, which differs from previous
methods with explicit training or testing procedure. Specifically, we freeze
the pre-trained Latent Diffusion Model as our basic model, where the visual
encoder $\mathcal{E}_{I}$, the UNet, and the visual decoder $\mathcal{D}$ are
inherited from [18] with the version of v1.5. We employ MiDas [17] to derive
depth maps, and utilize frozen Resnet blocks [6] to extract depth map features
$M$. The text encoder $\mathcal{E}_{p}$ is the pre-trained CLIP text encoder
[16].

During video editing, following [23] and [15], we uniformly sample 8 frames at
the resolution of 512*512 from each video, and conduct DDIM inversion and
denoising steps 50 ($T$) times. The learning rate $\lambda$ is 0.8. It takes
about 83 seconds to edit a video on an A40 GPU.

Evaluation Metrics.

We employ two metrics, i.e., Temporal Consistency (TC) and Prompt Consistency
(PC), to thoroughly evaluate the quality of edited videos: 1) Following [5],
we first extract CLIP embedding of all frames within the edited video, and
calculate the average cosine similarity between all pairs of neighborhood
frames to derive the Temporal Consistency score. 2) Following [15], we utilize
Text-Video CLIP Score to evaluate the Prompt Consistency between the edited
video $\hat{V}$ ($K$ frames) and the driven text $p$, which could be
formulated as:

${\rm CLIP}(\hat{V},p)=\frac{1}{K}{\sum}_{k=1}^{K}{\rm CLIP}(\hat{v}^{k},p).$
(12)

Model | Tune-A-Video | FateZero | EVE
---|---|---|---
Time |  $\sim$ 30 minutes |  247.6 seconds |  83.1 seconds
Table 2: Efficiency comparisons between our EVE with tuning-based Tune-A-Video
and zero-shot FateZero. All experiments are conducted on one A40 GPU.

###  4.3 Performance and Efficiency Comparisons

As aforementioned, zero-shot text-based video editing is a novel task without
public datasets and widely-employed baselines. Thus, it is intractable to
conduct a fair performance comparison with other methods. Therefore, we
compare the video editing efficiency between our EVE with the tuning-based
Tune-A-Video [23] and the zero-shot Fatezero [15]. We also compare the zero-
shot video editing performance between our EVE and Fatezero in two
quantitative metrics.

Regarding efficiency comparisons, as illustrated in Table 2, compared with the
tunning-based Tune-A-Video that takes about 30 minutes to generate an edited
video, zero-shot video editing methods are much more efficient as they shorten
the time to less than 5 minutes. Moreover, compared with the baseline
FateZero, the proposed EVE only costs about 1/3 of the total time (83s vs.
247s) to edit a video, which is more time-efficient and user-friendly.

Regarding performance comparisons, as illustrated in Table 1 (A1 vs. A2), we
observe that our proposed EVE outperforms the baseline FateZero in all four
tasks on the constructed ZVE-50 dataset, achieving an average improvement of
+0.63$\%$ on the temporal consistency and +1.28$\%$ on the prompt consistency.
It indicates that EVE is an efficient and robust video editing method, which
improves the temporal consistency of the generated video.

###  4.4 Ablation Study

Based on the argument that video editing necessitates more temporal
constraints to preserve the time consistency, we propose two constraints to
alleviate temporal distortion and inconsistency problems. We conduct several
ablation study to verify their effectiveness on our ZVE-50 dataset.

As illustrated in Table 1, we have three observations:

1) Depth maps are strong generative priors that prevent the edited video from
temporal distortion and inconsistency. Compared with B2 (without DMG) and B5,
we witness an obvious performance decay on both temporal and prompt
consistency, indicating the indispensability of the proposed depth map
guidance strategy.

2) The proposed Frame-Align Attention (FAA) reinforce the temporal encoding to
improve the consistency of edited videos. Compared with B3 (without FAA) and
B5, methods equipped with FAA would outperform conventional ones with SA by a
large margin, especially on the metric of the temporal consistency.

3) We also compare our FAA with the Sparse-Causal Attention (SCA) mechanism
proposed by Tune-A-Video. SCA calculates attentions among current frames and
the previous neighborhood ones, which could be formulated as:

${\rm{SCA}}:Q={W}^{Q}{\dot{z}}^{i},K={W}^{K}[{\dot{z}}^{1};\dot{z}^{i-1}],V={W}^{V}[{\dot{z}}^{1};\dot{z}^{i-1}],$
(13)

where $[\cdot]$ denotes the concatenation operation. We implement SCA based
upon our backbone with the same setting. Compared with B4 (with SCA) and B5,
we outperform SCA on both temporal and prompt consistency in all four tasks,
proving the advantages of our proposed FAA strategy.

###  4.5 Visualization Results and Applications

As illustrated in Figure 6, our EVE supports four types of applications
towards zero-shot text-based video editing:

1) Object Replacement (OR). OR replaces an object with another one in the
given video. E.g., “man $\rightarrow$ woman”.

2) Object Adding (OA). OA adds a new object to the original video. E.g., “man
$\rightarrow$ man with glasses”.

3) Style Transfer (ST). ST transfers the original video into different styles.
E.g., “style $\rightarrow$ Van Gogh style”.

4) Background Changing (BC). BC changes the video background. E.g.,
“background $\rightarrow$ under stars”.

##  5 Conclusion

We present EVE, a robust and efficient zero-shot text-based video editing
method, to tackle with the dilemma between the considerable fine-tuning cost
and the unsatisfied generation performance. Motivated by the observation that
videos necessitate more constraints to preserve the time consistency, we
introduce depth maps and two temporal consistency constraints to guide the
video editing procedure. In this way, the proposed EVE achieves a satisfactory
trade-off between performance and efficiency. Moreover, we construct and
benchmark ZVE-50, a public video editing dataset that provides a fair
comparison for future researchers.

##  6 Future Work

In the future, we aim to further improve the quality of edited videos,
narrowing the performance gap between tuning-based video editing methods and
zero-shot ones. E.g., introducing the triplet attention mechanism [29] into
the attention block of Unets to promote the temporal stability; and generating
pseudo labels by recording attention maps of Unets during the DDIM inversion
procedure, which helps to build a knowledge distillation mechanism [7] in the
following denoising step.

Visualization results of the proposed EVE on four applications: Object
Replacement, Object Adding, Style Transfer, and Background Changing.

## References

  * [1] Jooyoung Choi, Yunjey Choi, Yunji Kim, Junho Kim, and Sungroh Yoon.  Custom-edit: Text-guided image editing with customized diffusion models.  arXiv preprint arXiv:2305.15779, 2023.
  * [2] Paul Couairon, Clément Rambour, Jean-Emmanuel Haugeard, and Nicolas Thome.  Videdit: Zero-shot and spatially aware text-driven video editing.  arXiv preprint arXiv:2306.08707, 2023.
  * [3] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [4] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [5] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  arXiv preprint arXiv:2302.03011, 2023.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [7] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [8] Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang.  Shape-aware text-driven layered video editing.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 14317–14326, 2023.
  * [9] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [10] Yifan Li, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen.  Diffusion models for non-autoregressive text generation: A survey.  arXiv preprint arXiv:2303.06574, 2023.
  * [11] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6038–6047, 2023.
  * [12] Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yael Pritch, Yaniv Leviathan, and Yedid Hoshen.  Dreamix: Video diffusion models are general video editors.  arXiv preprint arXiv:2302.01329, 2023.
  * [13] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [14] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  arXiv preprint arXiv:1704.00675, 2017.
  * [15] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  arXiv preprint arXiv:2303.09535, 2023.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [17] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1623–1637, 2020.
  * [18] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [19] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [20] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In International Conference on Machine Learning, pages 2256–2265. PMLR, 2015.
  * [21] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [23] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [24] Rui Yao, Guosheng Lin, Shixiong Xia, Jiaqi Zhao, and Yong Zhou.  Video object segmentation and tracking: A survey.  ACM Transactions on Intelligent Systems and Technology, 11(4):1–47, 2020.
  * [25] Chenshuang Zhang, Chaoning Zhang, Mengchun Zhang, and In So Kweon.  Text-to-image diffusion model in generative ai: A survey.  arXiv preprint arXiv:2303.07909, 2023.
  * [26] Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren.  Sine: Single image editing with text-to-image diffusion models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6027–6037, 2023.
  * [27] Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu.  Controlvideo: Adding conditional control for one shot text-to-video editing.  arXiv preprint arXiv:2305.17098, 2023.
  * [28] Lecheng Zheng, Yu Cheng, Hongxia Yang, Nan Cao, and Jingrui He.  Deep co-attention network for multi-view subspace learning.  In Proceedings of the Web Conference, pages 1528–1539, 2021.
  * [29] Haoyi Zhou, Jianxin Li, Jieqi Peng, Shuai Zhang, and Shanghang Zhang.  Triplet attention: Rethinking the similarity in transformers.  In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 2378–2388, 2021.