# Spectrally pure photon pair generation in asymmetric heterogeneously coupled
waveguides

Xiangyan Ding  School of Astronautics and National Key Laboratory of Science
and Technology on Tunable Laser, Harbin Institute of Technology, 92 West Dazhi
Street, Harbin, 150001, China  Jing Ma  School of Astronautics and National
Key Laboratory of Science and Technology on Tunable Laser, Harbin Institute of
Technology, 92 West Dazhi Street, Harbin, 150001, China  Liying Tan  School of
Astronautics and National Key Laboratory of Science and Technology on Tunable
Laser, Harbin Institute of Technology, 92 West Dazhi Street, Harbin, 150001,
China  Amr S. Helmy  The Edward S. Rogers Department of Electrical and
Computer Engineering, University of Toronto, 10 King’s College Road, Toronto,
Ontario, Canada M5S 3G4  Dongpeng Kang  School of Astronautics and National
Key Laboratory of Science and Technology on Tunable Laser, Harbin Institute of
Technology, 92 West Dazhi Street, Harbin, 150001, China  Corresponding author:
<EMAIL_ADDRESS>
###### Abstract

In this work, we develop a design methodology to generate spectrally pure
photon pairs in asymmetric heterogeneously coupled waveguides by spontaneous
parametric down-conversion. Mode coupling in a system of waveguides is used to
directly tailor the group velocity of a supermode to achieve group velocity
matching (GVM) that is otherwise not allowed by material dispersion. Design
examples based on thin film lithium niobate waveguides are provided,
demonstrating high spectral purity and temperature tunability. This approach
is a versatile strategy applicable to waveguides of different materials and
structures, allowing more versatility in single-photon source designs.

††journal: ol

Single photons are a vital resource in various applications of quantum
information processing, including quantum key distribution, optical quantum
computing, simulation and metrology, etc[1]. The underline quantum processes
that rely on the interference between different quantum states of light
require high purity and indistinguishability of these states. An ideal
approach to generate single photons is to use quantum emitters such as quantum
dots [2] or crystal vacancies [3], which emit one photon at a time. Although
tremendous progress has been made in this approach, it is still a challenge to
generate indistinguishable photons from different emitters at different
wavelength regimes[4]. Moreover, such quantum emitters often do not afford
room-temperature operation[5] and require expensive cryogenic cooling. A more
readily available approach that has been widely used in various quantum
optical experiments [6] is to use heralded single-photon sources to generate
photon pairs via nonlinear optical processes including spontaneous parametric
down-conversion (SPDC) in $\chi^{\left(2\right)}$ and spontaneous four-wave
mixing (SFWM) in $\chi^{\left(3\right)}$ nonlinear media [7, 8, 9]. In such a
case, the detection of one photon indicates the existence of its twin [10].

For heralded single photons to be in a pure quantum state, paired photons
should not be correlated in frequency or any other degree of freedom. However,
this is usually not the case, as the purity of heralded single photons
generated by SPDC are often limited by the spectral correlations between
photons in a pair due to energy conservation. A common strategy is to use
narrow-band spectral filters to remove these correlations, which, however, is
obtained at the expense of heralding efficiency and increased source
complexity. While micro-resonators which provide enhanced nonlinear
interactions can be used to remove spectral correlations[11], for typical non-
resonant waveguides, group velocity matching (GVM) [12, 13] needs to be
employed in order to directly generate uncorrelated photon pairs on-chip. For
example, material dispersion in potassium titanyl phosphate (KTP) naturally
satisfies GVM in a certain wavelength range, thus allowing the generation of
pure heralded single photons in periodically poled KTP (PPKTP) waveguides in a
given wavelength range[14, 15]. However, material dispersion in most materials
do not satisfy GVM requirements, and thus do not allow the direct generation
of pure heralded single photons.

On the other hand, dispersion engineering in integrated photonic chips
provides a powerful tool to achieve functions that are otherwise not
achievable, and has been applied in a broad range of applications such as
parametric optical processes [16] and supercontinuum generation [17], etc.
Several photonic structures including Bragg gratings [18], nonlinear photonic
nanowires [19], and asymmetric slot waveguides [20] have been adopted to
engineer the dispersion.

In this work, we develop a technique to achieve GVM by dispersion engineering
in asymmetric heterogeneously coupled waveguides. Instead of focusing on group
velocity dispersion (GVD) as in previous work [21, 22], we utilize a mode-
coupling approach to tailor the group velocities of the modes involved to
satisfy GVM in materials that are otherwise unachievable. This is a generic
technique that can be used in nonlinear waveguides of different materials to
achieve GVM that is not naturally allowed by material dispersion. As an
illustrative example, we provide structures based on asymmetric
heterogeneously coupled lithium niobate (LN)- arsenic selenide (As2Se3) that
are capable of generating spectral pure photon pairs.

The two-photon state generated by SPDC in a nonlinear waveguide is given
by[23]

$|\psi\rangle=\int d\omega_{S}\int
d\omega_{I}f(\omega_{S},\omega_{I})|\omega_{S}\rangle|\omega_{I}\rangle,$ (1)

where $\omega_{S}$ and $\omega_{I}$ are the frequencies of signal and idler
photons, respectively, and $f(\omega_{S},\omega_{I})$ is the normalized bi-
photon wave function (BFW) given by
$\alpha(\omega_{S}+\omega_{I})\phi(\omega_{S},\omega_{I})$. Here
$\alpha(\omega_{S}+\omega_{I})$ is the pump spectral amplitude and
$\phi(\omega_{S},\omega_{I})$ is the phase-matching function, usually in the
form of a sinc function. For pure heralded single photons, the BWF must be
factorable, i.e.,
$f(\omega_{S},\omega_{I})=f_{S}(\omega_{S})f_{I}(\omega_{I})$, with
$f_{S}(\omega_{S})$ and $f_{I}(\omega_{I})$ being two single-variable
functions. To achieve this directly without spectral filtering, the group
velocities of the pump and down-converted photons need to satisfy the
following condition of GVM [12]

$v_{S}\leq v_{P}\leq v_{I},\mathrm{or}\;v_{I}\leq v_{P}\leq v_{S}.$ (2)

However, this condition is not satisfied except for a handful of cases.
Typically, $v_{P}$ is significantly lower than both $v_{S}$ and $v_{I}$.

Figure 1: (a) Schematic cross-sectional view of an asymmetric heterogeneously-
coupled waveguide with a mode crossing point at $\lambda_{0}$. (b) Modal field
profiles of symmetric (upper row) and antisymmetric (lower row) supermodes at
different wavelengths of $\lambda<\lambda_{0}$, $\lambda=\lambda_{0}$, and
$\lambda>\lambda_{0}$. (c) The inverse group velocities of the symmetric
(blue) and antisymmetric (red) supermodes given by Eq. (4).

Here we consider an asymmetric heterogeneously coupled waveguide with a
structure schematically shown in Fig. 1 (a), in which two waveguides of
different materials are placed close enough such that the waveguide mode of
one couples to the other. Fig. 1 (b) gives the modal electric field profiles
of symmetric (upper row) and antisymmetric (lower row) supermodes at different
wavelengths $\lambda$, where $\lambda_{0}$ is the mode crossing wavelength.
For a coupling coefficient $\kappa$ that depends on the waveguide spacing and
index difference, the propagation constants of the coupled symmetric
($\beta_{+}$) and antisymmetric ($\beta_{-}$) supermodes are given by [21]

$\beta_{\pm}=\frac{1}{2}\left(\beta_{1}+\beta_{2}\right)\pm\sqrt{\left(\frac{\beta_{1}-\beta_{2}}{2}\right)^{2}+|\kappa|^{2}},$
(3)

where $\beta_{i}$ ($i=1,2$) is the propagation constant of an isolated mode.
It’s a function of frequency $\omega$, and by neglecting higher-order
dispersions, can be expressed as
$\beta_{i}\left(\omega\right)\approx\beta_{i}\left(\omega_{0}\right)+(\omega-\omega_{0})/v_{i}$,
with $v_{i}=d\omega/d\beta_{i}$ being the modal group velocity at the mode
crossing point $\omega_{0}$. By substituting the approximate values of
$\beta_{1}$ and $\beta_{2}$ given above into Eq. (3) and performing first-
order derivative with respect to $\omega$, the group velocities of the coupled
supermodes can be obtained as

$\frac{1}{v_{\pm}(\omega)}=\frac{1}{2}\left(\frac{1}{v_{1}}+\frac{1}{v_{2}}\right)\pm\frac{1}{2}\left(\frac{1}{v_{1}}-\frac{1}{v_{2}}\right)\widetilde{\omega}\left(\widetilde{\omega}^{2}+1\right)^{-1/2},$
(4)

where $\widetilde{\omega}=\left(\omega-\omega_{0}\right)/\delta\omega$, with
$\delta\omega$ being the characteristic bandwidth defined by

$\delta\omega=2|\kappa|\left|\frac{1}{v_{1}}-\frac{1}{v_{2}}\right|^{-1}.$ (5)

In obtaining Eq. (4), we have neglected the frequency dependence of $\kappa$,
which is small in the frequency range of interest.

This group velocity tunability offered by waveguide coupling is illustrated in
Fig. 1(c). In the extreme case of $\widetilde{\omega}\rightarrow\infty$, the
inverse of group velocities can be obtained as $1/v_{+}\rightarrow 1/v_{1}$
and $1/v_{-}\rightarrow 1/v_{2}$. In the opposite extreme of
$\widetilde{\omega}\rightarrow-\infty$, they are given by $1/v_{+}\rightarrow
1/v_{2}$ and $1/v_{-}\rightarrow 1/v_{1}$ instead. These results are
intuitively understandable, because in either case, the modal field locates
mostly in one of the waveguides, as shown in Fig. 1(b), and therefore the
modal dispersion of a coupled supermode is dominated by a that of single
waveguide. As a result, the group velocities of two isolated waveguide modes
provide the upper and lower bounds for those of the coupled supermodes.

This strategy can be applied to tailor the group velocity of one of the down-
converted photons, such that it is generated in a supermode with a center
wavelength in the coupling regime. By choosing an appropriate combination of
material and dimensions for the neighboring waveguide and let it couple with
the original one, GVM could be satisfied at the desired wavelength. It should
be emphasized that this is a versatile technique applicable to waveguides of
different materials and structures. Although Fig. 1(a) shows a vertically
coupled ridge waveguide, in reality, coupled waveguides of any geometry could
be used, depending on the available fabrication techniques associated with
materials of consideration.

Figure 2: Schematic of a vertically coupled As2Se3-LN-on-insulator waveguide
and the related modal intensity profiles, with LN, SiO2 and As2Se3 being
colored in red, gray and green, respectively. An adiabatic taper could be used
on the As2Se3 layer after the QPM section to convert the supermode of the
coupled waveguide to the fundamental mode of a LN-on-insulator waveguide.

To demonstrate the proposed technique, here we provide a design example based
on $z$-cut $y$-propagating thin film LN-on-insulator waveguides, in which the
TE polarized pump at 775 nm generates cross-polarized photon pairs centered
around 1550 nm via the type-II SPDC process using quasi-phase matching (QPM).
The LN film is bonded on SiO2 and is assumed to have a width $w_{L}=2$ µm and
thickness $h_{L}=0.48$ µm. The group velocities of pump, signal (assumed to be
TE) and idler (assumed to be TM) are calculated to be $v_{P}=1.23\times
10^{8}$ m/s, $v_{S}=1.27\times 10^{8}$ m/s and $v_{I}=1.26\times 10^{8}$ m/s.
Thus our goal is to reduce $v_{S}$ so that $v_{S}\leq v_{P}$ while keeping
$v_{I}$ unattended.

Here we choose As2Se3 [24] as the coupling waveguide material, because of its
faster index variation than LN that allows a lower group velocity in an
isolated waveguide, and a layer of lower index SiO2 is used as the spacing.
The structure and modal profiles are schematically shown in Fig. 2. The signal
photon is designed to be generated in the TE polarized antisymmetric
supermode, with its group velocity tailored by the thicknesses of As2Se3 layer
($h_{T}$) and SiO2 layer ($g$). After photon pairs are generated, an adiabatic
taper on the As2Se3 layer could be added to convert the supermode to the
fundamental mode of the isolated LN waveguide to minimize the loss occurs
during coupling with the following photonic circuit or fiber.

Fig. 3 shows the difference between the inverse group velocities of the signal
and pump photons, $1/v_{S}-1/v_{P}$, as a function of waveguide geometric
parameters $h_{T}$ and $g$. The contour line with a value of $0$ represents
$v_{S}=v_{P}$. Any combination of $h_{T}$ and $g$ that falls on or to the left
of this line ensures GVM is satisfied. Two specific designs, labeled by white
and purple marks in Fig. 3, are discussed in the following.

Figure 3: Difference between the inverse group velocities of the signal photon
and the pump photon, $1/v_{S}-1/v_{P}$, as a function of the thicknesses of
As2Se3 layer ($h_{T}$) and SiO2 layer ($g$). The contour represents
$v_{S}=v_{P}$. Figure 4: (a) The inverse group velocity of the antisymmetric
supermode of the design example with $h_{T}=193$ nm and $g=500$ nm, and (b)
inverse group velocities of the pump, signal, and idler near the phase
matching wavelength, and (c) JSI of the photon pairs. (d) The inverse group
velocity of the antisymmetric supermode of the design example with $h_{T}=184$
nm, $g=500$ nm, and (e) inverse group velocities of the pump, signal, and
idler near the phase matching wavelength, and (f) JSI of the photon pairs.
Markers in (a) and (d) represent the working wavelength.

In the first case, with $h_{T}=193$ nm and $g=500$ nm, the inverse group
velocity of the antisymmetric supermode as a function of wavelength is shown
in Fig. 4(a). Group velocities of the signal and pump photons are almost
identical at the signal wavelength of 1550 nm, i.e., $v_{S}=v_{P}$, as shown
in Fig. 4(b). The joint spectral intensity (JSI), defined by
$|f(\omega_{S},\omega_{I})|^{2}$, is shown in Fig. 4(c), where we take
Gaussian pump pulses with a duration of 500 fs, and the waveguide length of
$L=16$ mm.

To quantify the degree of spectral entanglement, Schmidt decomposition
$f\left(\omega_{S},\omega_{I}\right)=\sum_{n}\sqrt{p_{n}}u_{n}(\omega_{S})v_{n}(\omega_{I})$
is performed, where $p_{n}$ is the probability of obtaining the $n$–th state,
with $\sum_{n}p_{n}=1$. The degree of factorizability can be quantified by the
Schmidt number $K=1/\sum_{n}p_{n}^{2}$. For the JSI shown in Fig. 4 (c), the
corresponding Schmidt number is $K=1.05$, which is close to 1, proving that
the two-photon state is almost factorizable. The generation rate is calculated
to be $\eta=3.4\times 10^{-8}$ pairs/pump photon according to the theory in
[25], assuming a first order QPM grating is used. This value is a factor of 4
smaller than that of a comparative LN-on-insulator waveguide without the
As2Se3 layer, indicating the nonlinear overlap does not decrease
significantly. Note that the remaining spectral correlation due to the side
lobes of a JSI can be further removed with the adoption of Gaussian phase
matching [26].

In the second case with $h_{T}=184$ nm and $g=500$ nm, the inverse group
velocity of the antisymmetric supermode as a function of wavelength is shown
in Fig. 4(d). The signal group velocity is further decreased such that
$2/v_{P}\approx 1/v_{S}+1/v_{I}$, as shown in Fig. 4(e). With a pump pulse
duration of 3.5 ps, the JSI is circularly shaped, as shown in Fig. 4(f). The
Schmidt number in this case is $K=1.17$, and the generation rate is
$\eta=9.2\times 10^{-9}$ pairs/pump photon.

In both cases, the modal overlap is assumed to be constant within the
frequency range of interest in calculating the BWF and generation rate. This
is because mode couplings take place within a wavelength range of a few
hundred nanometers, according to Figs. 4(a) and (c), significantly larger than
the spectral bandwidths of the photon pairs. It is also in sharp contrast to
similar structures in [21, 22], where extremely high GVDs are obtained within
bandwidths of only a few nanometers.

An important feature of these waveguides is that they could be thermally tuned
such that the shape of the JSI is switched from circular to elliptical, or
vice versa. Taking the second design discussed above as an example, the phase
matching wavelength decreases with the increase of temperature, as shown in
Fig. 5(a). This leads to the condition of $v_{S}=v_{P}$, shown in Fig. 5(b),
and thus an elliptically shaped JSI similar to that of Fig. 4(c). The
temperature tuning range required ($\Delta T\approx 120\leavevmode\nobreak\
^{\circ}$C in this example) depends on the thermo-optic coefficients
($\mathrm{d}n/\mathrm{d}t$) of all materials involved as well as the initial
separation between $v_{S}$ and $v_{I}$. For a separate design based on LN-Si
hybrid platform, we find the temperature tuning range is reduced to $\sim
20\leavevmode\nobreak\ ^{\circ}$C, due to the much larger thermo-optic
coefficient of Si ($\sim 10^{-4}$ /K) than that of As2Se3 ($\sim 10^{-5}$ /K)
and a lower $v_{I}$. As such, given the versatility and hybrid nature of these
structures, which are well feasible with existing technologies, the technique
and designs proposed provide a viable route for on-chip generation of tunable
photon pairs.

Figure 5: (a) The dependence of phase matching wavelength $\lambda_{PM}$ on
temperature variation for the design with $h_{T}=184$ nm and $g=500$ nm. (b)
The corresponding temperature dependences of inverse group velocities of the
three modes.

We emphasize again that the technique presented here is not limited to any
particular material system or waveguide geometry. Furthermore, even QPM is not
necessary. One can design a waveguide which utilizes modal phase matching
(MPM) similar to that in [27], while simultaneously achieving GVM by
dispersion engineering via waveguide coupling. In addition, tailoring of group
velocity enabled by this technique can be utilized to other applications where
group velocity control is critical, such as broadband wavelength conversion
[20, 16] and polarization entangled photon generation[28].

In summary, we have developed a generic technique to generate spectral pure
photon pairs in asymmetric heterogeneously coupled waveguides. By engineering
the waveguide structure, the group velocity of a coupled supermode can be
tailored such that GVM is achieved. This allows the generation of spectral
pure photon pairs without the need for bandpass filtering. Design examples
based on vertically coupled As2Se3-LN-on-insulator waveguides are provided.
This technique is also directly applicable to the continuously-variable
regime[29] and applications involving temporal modes[30].

## Funding

National Natural Science Foundation of China (61705053); China Postdoctoral
Science Foundation (2016M600249), Heilongjiang Postdoctoral Special Funds;
Fundamental Research Funds for the Central Universities, Natural Sciences and
Engineering Research Council of Canada.

## Disclosures

The authors declare no conflicts of interest.

## References

  * [1] F. Flamini, N. Spagnolo, and F. Sciarrino, Reports on Progress in Physics 82, 016001 (2019).
  * [2] H. Utzat, W. Sun, A. E. Kaplan, F. Krieg, M. Ginterseder, B. Spokoyny, N. D. Klein, K. E. Shulenberger, C. F. Perkinson, M. V. Kovalenko _et al._ , Science 363, 1068 (2019).
  * [3] Y. Xue, H. Wang, N. Xie, Q. Yang, F. Xu, B. Shen, J.-J. Shi, D. Jiang, X. Dou, T. Yu, and B.-Q. Sun, Journal of Physical Chemistry Letters 11, 2689 (2020).
  * [4] Y. Bao, Q. Lin, R. Su, Z.-K. Zhou, J. Song, J. Li, and X.-H. Wang, Science Advances 6 (2020).
  * [5] L. Seravalli and F. Sacconi, Journal of Physics: Materials 3, 042005 (2020).
  * [6] S. Slussarenko and G. J. Pryde, Applied physics reviews 6, 041303 (2019).
  * [7] Y. Hashimoto, A. Goban, Y. Hirabayashi, Y. Kobayashi, T. Araki, and T. Aoki, Opt. Express 29, 3533 (2021).
  * [8] C. Chen, J. E. Heyes, K. Hong, M. Y. Niu, A. E. Lita, T. Gerrits, S. W. Nam, J. H. Shapiro, and F. N. C. Wong, Optics Express 27, 11626 (2019).
  * [9] J. B. Christensen, J. G. Koefoed, K. Rottwitt, and C. J. Mckinstrie, Optics Letters 43, 859 (2018).
  * [10] Y. Wang, K. D. Jöns, and Z. Sun, Applied Physics Reviews 8, 011314 (2021).
  * [11] B. M. Burridge, I. I. Faruque, J. G. Rarity, and J. Barreto, Opt. Lett. 45, 4048 (2020).
  * [12] W. P. Grice, A. B. Uren, and I. A. Walmsley, Physical Review A 64, 063815 (2001).
  * [13] B. Wei, W.-H. Cai, C. Ding, G.-W. Deng, R. Shimizu, Q. Zhou, and R.-B. Jin, Opt. Express 29, 256 (2021).
  * [14] L. Padberg, M. Santandrea, M. Rüsing, J. Brockmeier, P. Mackwitz, G. Berth, A. Zrenner, C. Eigner, and C. Silberhorn, Opt. Express 28, 24353 (2020).
  * [15] F. Kaneda, J. Oikawa, M. Yabuno, F. China, S. Miki, H. Terai, Y. Mitsumori, and K. Edamatsu, Opt. Express 28, 38993 (2020).
  * [16] E. Nitiss, B. Zabelich, O. Yakar, J. Liu, R. N. Wang, T. J. Kippenberg, and C.-S. Brès, Photon. Res. 8, 1475 (2020).
  * [17] G. F. C. Gonzalez, M. Malinowski, A. Honardoost, and S. Fathpour, Applied Optics 58 (2019).
  * [18] Z. Du, C. Xiang, T. Fu, M. Chen, S. Yang, J. E. Bowers, and H. Chen, APL Photonics 5, 101302 (2020).
  * [19] F. Xu, Z. xing Wu, and Y. qing Lu, Progress in Quantum Electronics 55, 35 (2017).
  * [20] S. Kim and M. Qi, Optics Express 24, 773 (2016).
  * [21] U. Peschel, T. Peschel, and F. Lederer, Applied physics letters 67, 2111 (1995).
  * [22] B. Mia, N. Jaidye, and S. Kim, Optics Express 27, 10426 (2019).
  * [23] W. P. Grice and I. A. Walmsley, Phys. Rev. A 56, 1627 (1997).
  * [24] L. Zhang, A. M. Agarwal, L. C. Kimerling, and J. Michel, Nanophotonics 3, 247 (2014).
  * [25] S. V. Zhukovsky, L. G. Helt, P. Abolghasem, D. Kang, J. E. Sipe, and A. S. Helmy, Journal of The Optical Society of America B-optical Physics 29, 2516 (2012).
  * [26] P. B. Dixon, J. H. Shapiro, and F. N. C. Wong, Opt. Express 21, 5879 (2013).
  * [27] R. Luo, Y. He, H. Liang, M. Li, and Q. Lin, Laser & Photonics Reviews 13, 1800288 (2019).
  * [28] D. Kang, A. Pang, Y. Zhao, and A. S. Helmy, Journal of The Optical Society of America B-optical Physics 31, 1581 (2014).
  * [29] G. Harder, T. J. Bartley, A. E. Lita, S. W. Nam, T. Gerrits, and C. Silberhorn, Phys. Rev. Lett. 116, 143601 (2016).
  * [30] M. Raymer and I. Walmsley, Physica Scripta 95, 064002 (2020).

paper


# Fairy: Fast Parallellized Instruction-Guided Video-to-Video Synthesis

Bichen Wu Ching-Yao Chuang Xiaoyan Wang Yichen Jia
Kapil Krishnakumar Tong Xiao Feng Liang Licheng Yu Peter Vajda
GenAI, Meta
Project page: https://fairy-video2video.github.io

###### Abstract

In this paper, we introduce Fairy, a minimalist yet robust adaptation of
image-editing diffusion models, enhancing them for video editing applications.
Our approach centers on the concept of anchor-based cross-frame attention, a
mechanism that implicitly propagates diffusion features across frames,
ensuring superior temporal coherence and high-fidelity synthesis. Fairy not
only addresses limitations of previous models, including memory and processing
speed. It also improves temporal consistency through a unique data
augmentation strategy. This strategy renders the model equivariant to affine
transformations in both source and target images. Remarkably efficient, Fairy
generates 120-frame 512$\times$384 videos (4-second duration at 30 FPS) in
just 14 seconds, outpacing prior works by at least 44$\times$. A comprehensive
user study, involving 1000 generated samples, confirms that our approach
delivers superior quality, decisively outperforming established methods.

Figure 1: Fairy for Instruction-Guided Video Editing. Given a video and an
instruction for editing, Fairy performs accurate edits while ensuring temporal
coherence. Remarkably efficient, 120 frames of 512$\times$384 video can be
generated in just 14 seconds. We refer readers to our supplementary material
to check the results in video format.

##  1 Introduction

The advent of generative artificial intelligence has heralded a new era of
creative potential, characterized by the ability to create or modify content
in an effortless manner. In particular, image editing has undergone
significant evolution, driven by text-to-image diffusion models pretrained on
billion-scale datasets. This surge has catalyzed a vast array of applications
in image editing and content creation.

Building on the accomplishments of image-based models, the next natural
frontier is transitioning these capabilities to the temporal dimension to
enable effortless and creative video editing. A direct strategy to leap from
image to video, is to simply process a video on a frame-by-frame basis using
an image model. Nonetheless, generative image editing is inherently high-
variance – there are countless ways to edit a given image based on the same
text prompt. As a result, it is difficult to maintain temporal coherence if
each frame is edited independently [30].

Previous and concurrent studies have proposed several ways to improve the
temporal consistency, and one promising paradigm is what we call tracking-and-
propagation: one first apply an image editing model on one or a few frames,
then tracks pixels across all frames and propagate the edit to the entire
video. Existing works [33, 24, 12, 4, 6, 14, 16, 19] track pixels mainly
through optical flow or by reconstructing videos as some canonical layered
representations. Despite some successful applications, this paradigm is not
robust, since tracking is an unsolved computer vision challenge. Existing
methods, including optical flow or layered video representation, often
struggle with videos with large motion and complex dynamics.

In this work, we introduce _Fairy_ , a versatile and efficient video-to-video
synthesis framework that generates high-quality videos with remarkable speed
(Figure 1). Our work re-examines the tracking-and-propagation paradigm under
the context of diffusion model features. In particular, we bridge cross-frame
attention with correspondence estimation, showing that it temporally tracks
and propagates intermediate features inside a diffusion model. The cross-frame
attention map can be interpreted as a similarity metric assessing the
correspondence between tokens throughout various frames, where features from
one semantic region will assign higher attention to similar semantic regions
in other frames, as shown in Figure 3. Consequently, the current feature
representations are refined and propagated through a weighted sum of similar
regions across frames via attention, effectively minimizing feature disparity
between frames, which translates to improved temporal consistency.

The analysis gives rise to our _anchor-based model_ , the central component of
Fairy. To ensure temporal consistency, we sample $K$ anchor frames from which
we extract diffusion features, and the extracted features define a set global
features to be propagated to successive frames. When generating each new
frame, we replace the self-attention layer with cross-frame attention with
respect to the cached features of anchor frames. With cross-frame attention,
the tokens in each frame take the features in anchor frames that exhibit
analogous semantic content, thereby enhancing consistency. In addition, by
sampling $K$ anchor frames instead of computing cross-attention with respect
to all frames, Fairy achieves several advantages: (1) it ensures temporal
consistency by sharing the same global features, (2) it overcomes the memory
issue due to extensive frame number, (3) it enhances processing speed through
the caching of anchor frame features, and (4) it streamlines parallel
computation, thereby facilitating remarkably rapid generation on multiple
GPUs.

Despite the improvement from anchor-based cross-frame attention, the model is
still sensitive to minor variations within the input frames, even with the
same text prompt and initial latent noise. Such small changes could stem from
natural movements within a video sequence or from affine transformations
applied to the input. The gold standard solution is to train the model with
pairs of original and edited videos, thereby accommodate it to recognize and
adapt to such variations. However, collecting such a dataset is far from
straightforward. To emulate these transformations, we employ a data
augmentation strategy. Starting with an input image and its edited
counterpart, we apply a sequence of affine transformations to both, generating
successive frames. The assumption is that the affine transformations applied
to input images should correspondingly affect the edited images. This method
of _equivariant finetuning_ leads to notable enhancements in temporal
consistency.

To verify the effectiveness of Fairy, we conduct a large-scale evaluation
consists of 1000 generated videos. Both human evaluation and quantitative
metrics confirm that our model achieves significantly better quality.
Moreover, thanks to the simplicity of the design and the parallelizable
architecture, Fairy achieves $>$44x speedup over baselines.

In short, this work makes the following contributions: (1) We adopt a series
of simple yet effective adaptions that transform an image-editing model for
video-to-video synthesis. (2) We evaluate our approach via extensive human
study with 1000 generated videos, confirming that Fairy delivers superior
quality over prior state-of-the art methods. (3) Fairy is blazing fast,
achieving $>$44x speedup over previous methods when utilizing 8-gpu parallel
generation.

##  2 Related works

Conditional video generation: Following the success of diffusion models in
text-to-image generation [25, 22, 7, 23], there has been a surge in video
generation. Based on a text-to-video model, video-to-video generation can be
achieved by conditioning the model on attributes extracted from a source
video. For example, Gen-1 [9] conditions on the estimated depth while
VideoComposer [31] integrates additional cues, such as depth, motion vectors,
sketches, among others. Building such models requires training on video
datasets, which are much more scarce than image datasets [26]. Training such
models also imposes considerable computational demands. Consequently, these
constraints confine video models to reduced resolution, shorter duration, and
smaller model size, leading to a decline in visual quality when contrasted
with contemporary image generation models. In comparison, our model is adapted
from a pretrained image-to-image model. Our finetuning only requires image
data, and the training cost (30 hours on 8 A100 GPUs) is substantially smaller
than video models.

Tracking and propagation: this paradigm involves initiating edits on a single
image, identifying pixel correspondences across the video sequence, then
propagating the edit. The key in this approach lies in tracking. Numerous
efforts [33, 24, 12] have adopted optical flow, keypoint tracking, or other
motion cues to tackle this. Another stream of efforts [19, 4, 6, 14, 16]
reconstruct the video using a multi-layer canonical representation,
associating pixels to canonical points on the representation. However, video
tracking is an unsolved computer vision challenge and often fails on complex
videos. Additionally, tracking-and-propagation does not allow edits to alter
object contours, which breaks the pixel correspondence. Instead of tracking in
pixel space, our model leverages cross-frame attention to implicitly track
corresponding regions and propagate features to reduce frame discrepancy.
Owing to the robustness and versatility of diffusion features, as also
observed in Tang et al. [27], our approach accommodates a broader spectrum of
videos and offers enhanced editing flexibility.

Image model adaptation: Many works also adapt image-to-image models to video.
For example, [15] modifies self-attention in diffusion models. [32] performs
per-video finetuning and utilizes a inversion-denoising procedure for editing.
[29, 17, 21, 10] adapt image-to-image pipelines [11, 28, 5] to edit videos, by
modifying/adding cross-frame attention modules, null-text inversion, etc. Most
of these methods can only generate video clips with a small number of frames,
while [10] leverages a nearest-neighbor field on diffusion features to
propagate key frame features to more frames. Our model improves the design of
spatial temporal attention [15, 29, 17, 21] to anchor-based cross-frame
attention, which enables generating long videos with arbitrarily many frames.
We further improves its temporal consistency by equivariant finetuning. Our
work bears resemblance to the concurrent work [10]. To edit a video, [10]
first performs a latent inversion on the original video, extract a nearest-
neighbor field, which is then used for feature propagation to generate the
target video. Our pipeline is much simpler and more efficient. We do not
require latent inversion; and the feature propagation is achieved through
attention; our architectures naturally allows parallel generation. As a
result, our model is 53 times faster than [10].

##  3 Preliminaries

#### Video-to-Video Diffusion Models

In this work, we primarily focus on instruction-guided video editing. Given an
input video with $N$ frames
${\bm{I}}=\\{I^{1},\dots,I^{N}\\}\in{\mathcal{I}}^{N}$, the goal is to edit it
into a new video
${\bm{I}}^{\prime}=\\{{I^{1}}^{\prime},\dots,{I^{N}}^{\prime}\\}\in{\mathcal{I}}^{N}$
according to an natural language instruction $c\in{\mathcal{T}}$ that
preserves the semantic of the original video. A straightforward baseline is to
adopt an image-based editting model
$f:({\mathcal{I}},{\mathcal{T}})\rightarrow{\mathcal{I}}$ to edit the video
frame by frame: ${\bm{I}}^{\prime}=\\{f(I^{1},c),\dots,f(I^{N},c)\\}$. In this
work, we build upon this line of work and improve the consistency with a
variant of cross-frame attention.

#### Self-attention and Cross-frame attention

Self-attention has played a crucial role in the diffusion networks. In a self-
attention block, features of tokens are projected into queries
${\bm{Q}}\in{\mathbb{R}}^{n\times d}$, keys ${\bm{K}}\in{\mathbb{R}}^{n\times
d}$, and values ${\bm{V}}\in{\mathbb{R}}^{n\times d}$, where the output is
defined as

$\displaystyle\mathrm{SelfAttention}({\bm{Q}},{\bm{K}},{\bm{V}})=\mathrm{softmax}\left(\frac{{\bm{Q}}{\bm{K}}^{T}}{\sqrt{d}}\right){\bm{V}}.$

The output from the softmax is commonly referred to as the _attention score_
or _attention map_. Given $N$ frames, to extend the self-attention to cross-
frame attention, one can simply concat the keys and values from all frames,
e.g., ${\bm{K}}^{\ast}=[{\bm{K}}^{1},\cdots,{\bm{K}}^{N}]$, and compute the
self-attention as $\textnormal{Self-
Attention}({\bm{Q}},{\bm{K}}^{\ast},{\bm{V}}^{\ast})$. In particular, cross-
frame attention provides temporal modeling capability by attending to other
frames and shows encouraging results in improving temporal consistency [30,
17].

##  4 Implicit Tracking via Cross-frame Attention

We first bridge cross-frame attention with correspondence estimation,
fostering a straightforward yet effective feature propagation mechanism for
video-to-video generation.

The primary objective of self-attention is to select appropriate values
${\bm{V}}$ with the attention scores determined by ${\bm{Q}}{\bm{K}}^{T}$. In
the case of cross-frame attention, given a token location $p$ in a frame, the
attention score is computed by the cosine similarity between ${\bm{Q}}_{p,:}$
and each token in ${\bm{K}}^{\ast}$, where the key values ${\bm{V}}^{\ast}$
are the features of tokens across both spatial and temporal dimension.

It is noteworthy that the mathematical formulation exhibits profound
similarities to feature propagation mechanisms. Specifically, the attention
score serves as the estimated correspondence, and the output of attention
module could be interpreted as a fused representation of warped features
derived from successive frames. We will empirically substantiate this
hypothesis through analyses of the tracking behavior inherent in the attention
score.

Figure 2: Position Accuracy $\delta^{x}$ on DAVIS. The cross-frame attention
score demonstrates significant tracking proficiency, particularly evident in
the initial and final stages of the UNet.

###  4.1 Temporal Tracking with Attention Score

In this section, we provide evidences that the attention scores in cross-frame
attention implicitly serve as a correspondence estimation across frames. In
particular, we adopt a conditional image-to-image diffusion model and examine
the attention map between two frames of a video clip. Consider ${\bm{Q}}^{t}$
and ${\bm{K}}^{t}$ as the respective query and key representations
corresponding to the frame at time $t$. To corroborate our conjecture
regarding the role of attention scores, we designate a specific query point
$p$ at time $t$ and endeavor to ascertain its corresponding coordinate $q$ at
a subsequent time $t^{\prime}$ through the expression:

$\displaystyle
q=\operatorname*{arg\,max}_{p^{\prime}}{\bm{A}}_{p,p^{\prime}},\;\textnormal{where}\;{\bm{A}}=\mathrm{softmax}(\frac{{\bm{Q}}^{t}{{\bm{K}}^{t^{\prime}}}^{T}}{\sqrt{d}}),$

where ${\bm{A}}_{p,p^{\prime}}$ denotes the element of the matrix ${\bm{A}}$
located at the row index $p$ and column index $p^{\prime}$. The correspondence
is estimated by selecting the location $p^{\prime}$ with the highest attention
score with respect to $p$. For multi-head attention, we average the attention
scores from all heads. By evaluating the tracking ability of the proposed
estimator, we can verify whether the attention scores are good correspondence
estimator for feature propagation.

###  4.2 Video Tracking Experiments: TAP-Vid

In our evaluation, we utilize the DAVIS datasets from the TAP-Vid [8, 20],
with 30 videos clips ranging from 34-104 frames. The frames are resize to
$256\times 256$ for evaluation. We measure the $<\delta^{x}$ position accuracy
proposed in TAP-Vid, which calculates the fraction of points that are within
$\delta^{x}$ pixels of their ground truth position. The dimensions of the
attention map inherently impose a constraint on the precision achievable in
point tracking. Since diffusion UNets adopts spatial downsampling, we
configure $\delta^{x}$ at the values of 16 and 32 for our experiments. We set
the number of diffusion step to $10$ with Euler ancestral sampler [13].

Figure 2 shows the position accuracy for attention scores across different
layers and diffusion step. We can see that the first and last few layers
demonstrate a strong tracking results, achieving over $60\%/70\%$ accuracy for
$\delta^{x}=16/32$. Interestingly, the results are consistent across different
diffusion step, demonstrating the strong tracking ability of cross-frame
attention. The observed degradation in accuracy at the central layers of the
UNet architecture can be attributed primarily to the reduction in the spatial
resolution of the feature maps. For instance, within the seventh layer of the
network, the feature map dimensions are constrained to $4\times 4$. Figure 3
visualizes the attention score on a target frame given a query point. We can
see that the attention map locate the corresponding position in target frame.

#### Cross-frame Attention $\approx$ Tracking and Feature Propagation

Our experimental findings disclose an unexpectedly potent tracking capability
associated with the attention score. These results robustly validate our
hypothesis: _even in the absence of explicit finetuning, cross-frame attention
implicitly executes a formidable feature propagation mechanism_. In
particular, features ${\bm{V}}^{\ast}$ from alternative frames are transmitted
to the current frame based on the correspondence determined through the
attention scores.

Figure 3: Visualization of Attention Score. The left image shows the query
point $p$ within the current frame, and the right image is the target frame.
Cross-frame attention performs accurate temporal correspondence estimation
without any finetuning.

##  5 Fairy: Fast Video-to-Video Synthesis

Building on the analyses, we present Fairy, an efficacious video-to-video
framework that leverages the inherent feature propagation of cross-frame
attention. In particular, we propose to propagate the value features from a
collection of anchor frames to a candidate frame using cross-frame attention.
The performance can be further enhanced through the proposed equivariant
finetuning method. We also demonstrate that Fairy is easily parallelizable,
facilitating fast generation of arbitrarily long videos.

Figure 4: Illustration of Attention Blocks (a) Given a set of anchor frames,
we extract and cache the attention feature ${\bm{K}}_{anc}$ and
${\bm{V}}_{anc}$. (b) Given an input frame, we perform cross-frame attention
with respect to the cached features of anchor frames.

###  5.1 Anchor-Based Model

Inspired by prior research in tracking-and-propagation, where the edits to one
or a few frames are propagated to the entire video, we sample a set of _anchor
frames_ and edit them with an image-based model
$f:({\mathcal{I}},{\mathcal{T}})\rightarrow{\mathcal{I}}$. Similarly, our
objective is to extend the edits in the anchor frames to the successive
frames, but utilizing cross-frame attention mechanisms instead of optical flow
or explicit point tracking. In particular, given a set of anchor frames
${\bm{I}}_{anc}=\\{\tilde{I}^{1},\dots,\tilde{I}^{K}\\}\subseteq{\bm{I}}=\\{I^{1},\dots,I^{N}\\}$,
we treat them as a batch and feed them to the diffusion model $f$, where the
self-attention in the model is replaced with cross-frame attention in a zero-
shot manner. Throughout the diffusion process, for each anchor frame
$\tilde{I}^{n}$, we store its key and value vectors
${{\bm{K}}}^{n,l,t},{{\bm{V}}}^{n,l,t}$ for every cross-frame attention layer
$l$ and every diffusion step $t$ in cache. Intuitively, ${\bm{V}}^{n,l,t}$
defines a set global features to be propagated to successive frames. To
simplify the notation, we will drop the subscript $l$ and $t$ in the following
sections.

Let ${\bm{K}}_{anc}=[{\bm{K}}^{1},\dots,{\bm{K}}^{K}]$ and
${\bm{V}}_{anc}=[{\bm{V}}^{1},\dots,{\bm{V}}^{K}]$ be the concatenated anchor
key and value vectors. To edit any frame $I^{t}\in{\bm{I}}$, we modify the
self-attention module to the cross-frame attention with respect to the key and
value vectors of anchor frames as follows:

$\displaystyle\mathrm{softmax}\left(\frac{{\bm{Q}}[{\bm{K}},{\bm{K}}_{anc}]^{T}}{\sqrt{d}}\right)[{\bm{V}},{\bm{V}}_{anc}],$

where ${\bm{Q}},{\bm{K}}$ and ${\bm{V}}$ are the self-attention vectors of the
input frame $I^{t}$. The idea is that the attention score generated by the
softmax facilitates cross-frame tracking by estimating the temporal
correspondence between the input frame and anchor frames. The global value
vectors then be propagated to input frame by multiplying the attention score
with ${\bm{V}}_{anc}$. By substituting the self-attention module with an
anchor-based cross-frame attention mechanism, we found that the model could
generate highly consistent video edits. In the default setting, we choose
anchor frames uniformly across the video, and we did not notice consistent
performance improvement or degradation when adopting different anchor-frame
selection strategies.

#### Fast Generation via Parallelization

Note that editing frame $I_{t}$ does not require other frames as input except
the cached features ${\bm{K}}_{anc}$ and ${\bm{V}}_{anc}$ from anchor frames.
Therefore, we can edit arbitrary long videos by splitting them into segments
and leverage multi-GPUs to parallize the generation, while the computation
remains numerically identical. As a result, our method achieves significant
speedup compared to previous works. Moreover, it delivers superior quality
outputs without succumbing to memory-related constraints. This efficiency
underscores our approach’s enhanced scalability and practicality, setting a
new benchmark for performance in the realm of video editing.

###  5.2 Equivariant Finetuning

While anchor-based attention greatly improves the quality, we still
occasionally observed temporal inconsistency. In particular, we found that for
generated contents that do not have semantic correspondence with the input,
small changes in input frames can cause significant variances in the output
frames.

Figure 5: Diverse Video Editing via Fairy. Fairy enables a wide range of video
edits with different types of subjects.

To improve the consistency, we leverage the following intuition to design a
data augmentation strategy. In particular, if an input frame $I^{t}$ differs
from $I^{t-1}$ only in the camera position, then the output frame
$\hat{I}^{t}$ and $\hat{I}^{t-1}$ should only be different in the camera
position as well. This inspires us to come up with a data augmented strategy
that can be applied to any image editing dataset to imporve the temporal
consistency. Given a pair of images, the original and the edited, denoted as
$(I,I^{\prime})$, we randomly sample an affine transformation
$g:{\mathcal{I}}\rightarrow{\mathcal{I}}$ and apply them to both images to
obtain $(g(I),g(I^{\prime}))$. We implement this using torchvision’s random
affine transformation [2], setting random rotations degrees to $<5^{\circ}$,
random translation to $[-0.05,0.05]$, random scaling factor to $[0.95,1.05]$,
and random shear degrees to $[-5^{\circ},5^{\circ}]$ on both axis. We also
apply random resized crop, scaling the original image to 288pix, and randomly
crop a square image with 256 pix. We then fine-tuned the base image-to-image
model to generate the transformed $g(I^{\prime})$ given the transformed
$g(I)$. The proposed fine-tuning process makes the model equivariant to affine
transformations, leading us to denote our approach as _equivariant
finetuning_. Empirically, we observe a notable enhancements in temporal
consistency after finetuning (Section 6.3).

##  6 Results

We implement Fairy based on an instruction-based image editing model, similar
to [5], and replace the model’s self-attention with cross-frame attention. We
set the number of anchor frames to 3. Anchor frames are uniformly selected
with equal intervals among all frames. The model can accept input with
different aspect ratios, and we rescale the input resolution with the longer
size to be 512, and keep the aspect ratio unchanged. We edit all frames of the
input video, without temporal downsampling. We distribute the computation to 8
A100 GPUs. We use the Euler Ancestral sampler with 10 diffusion steps.

For equivariant finetuning, we use the same dataset that was used to train the
image editing model, and apply the data augmentation discussed in Section 5.2.
We load the image editing model’s pretrained checkpoint, and resumed training
for 50,000 steps with a batch size of 128, costing 30 hours on 8 A100 GPUs
with 80GB memories.

###  6.1 Qualitative Evaluation

We first show qualitative results of Fairy. Since most of the PDF readers do
not render videos properly, we only show a small number of frames for each
video. We strongly recommend readers to checkout our supplementary materials
to watch the complete videos. In Figure 5, we show that our model is capable
conducting edits on different subjects. In Figure 6, we show that our model is
able to conduct different types of editing, including stylization, character
swap, local editing, and attribute editing, following textual instructions. In
Figure 9, we show that our model can transform the source character into
different target characters based on instructions. Note that our model can
adapt to different input aspect ratios without need for re-training. Our input
videos contain large motions, occlusions, and other complex dynamics. Despite
those challenges, videos generated by our model are temporally consistent and
visually appealing. We also show our model’s capabilities to generate long
videos in the supplementary material.

###  6.2 Quantitative Evaluation

Quantitatively evaluating video generative models is challenging. First, the
generation task is intrinsically high-variance – there are countless ways to
edit an video given the instruction. Second, previous works have adopted
metrics such as CLIP scores [9, 10] to evaluate the generation quality.
However, these metrics are not necessarily aligned with human perception [18].
Lastly, human evaluation is still the golden standard to judge the quality.
Yet, due to the cost of human evaluation, previous works have only conducted
small scale human evaluations ($<100$ samples).

In this paper, we conduct a large-scale user study on an evaluation set
consists of 1000 video-instruction samples. The evaluation set is divided into
two parts: first, to test a model’s robustness across different videos, we
construct the evaluation set of 50 videos $\times$ 10 instructions. And to
test a model’s robustness across different instructions, we construct a dual
evaluation set of 10 videos $\times$ 50 instructions. The videos are
accessible from ShutterStock [3]. To our best knowledge, this is the largest
evaluation in the video-to-video generation literature so far.

Figure 6: Different type of editing. Fairy is able to handle a diverse set of
instructions and perform appropriate editing.

We conduct a A/B comparison to compare our method with three previous works,
Rerender [33] (tracking and propagation), Tokenflow [2] (image model
adaptation), and Gen-1 [9] (conditional video model), which are the strongest
representative of the three paradigms for video-to-video generation today.
Results from baselines are collected from [1]. Prompts for baselines are
descriptive, e.g., ”a dog running on grass, in Van Gogh style”. We re-write
the prompt for our method as an edit instruction, e.g., ”in Van Gogh Style”.
Since Gen-1 is not open-sourced, the evaluation is done on a smaller
evaluation set of 100 videos. In each evaluation tuple, we show the input
video, the editing instruction or prompt, and the output videos generated by
Fairy and a baseline. We ask human evaluators to choose the better video in
terms of their single frame quality, temporal consistency, prompt
faithfulness, input faithfulness, and overall quality. Each comparison is
rated by 3 different annotators and the decision is determined by the majority
vote. We report the overall quality comparison in Figure 7, which demonstrates
that videos produced by Fairy are more preferable, with a win rate of 41% vs
36% against Rerender, 73% vs. 16% against TokenFlow, and 72% vs 26% against
Gen-1. More details in the supplementary material.

Figure 7: A/B Comparison with Baselines. Fairy significantly surpassed baseline models, demonstrating its effectivity.  |  Latency (sec) $\downarrow$ |  Frame-Acc $\uparrow$ |  Tem-Con $\uparrow$
---|---|---|---
TokenFlow | 744 | 0.537 | 0.973
Rerender | 608 | 0.775 | 0.972
Ours | 13.8 | 0.819 | 0.974
Table 1: We assess our method’s temporal consistency and fidelity to the
target text prompt using CLIP similarity metrics. Figure 8: Ablation Study.
Without equivariant fine-tuning and anchor-based attention, we observed
inconsistencies, particularly in the changing patterns of body and
earscostumes over time. This inconsistency is further exacerbated upon the
removal of anchor-based attention, leading to lower temporal consistency
score.

Figure 10 shows the visual comparison with the baselines. We observe that both
Tokenflow [10] and Rerender [33] do not adhere closely to the provided
instructions, resulting in evident inconsistencies. Outputs from Gen-1 often
over-modify the entire scene and do not retain the original content
effectively. In contrast, Fairy meticulously follows the instruction,
delivering high-quality, temporal consistent, and authentic generations.

Figure 9: Diverse Character Swap: Fairy possesses the capability to
interchange the individual with a diverse array of characters.  Figure 10:
Comparison with Baselines. Fairy consistently outperform baselines in terms of
consistency and instruction-faithfulness.

Lastly, we compute metrics adopted by previous works, Tem-Con and Frame-Acc
[21, 33]. Tem-Con assesses temporal consistency by calculating the cosine
similarity of CLIP feature across successive frame pairs, and Frame-Acc
measures the percentage of frames where the edited image exhibits greater CLIP
similarity to the target prompt than to the source prompt. The results in
Table 1 demonstrates that Fairy achieves the best temporal consistency and
frame-wise editing accuracy against the baselines.

#### Speed Comparison

In Table 1, we also compare the latency of different models. In particular, we
calculate the inference time of editing a 4-seconds, 30 FPS, 512p$\times$384p
video on a server with 8 A100 GPUs. The key-frame interval of Rerender is set
to 4 instead of the default 10, since the test videos contain faster motion.
This leads to improved quality for Rerender. All other parameters were
default. Due to its architecture simplicity, Fairy is already significantly
faster than baselines using 1 GPU. Using a single GPU, Fairy completes
inference in just 78 seconds, achieving 9.5$\times$ faster than TokenFlow and
7.5$\times$ faster than Rerender. When utilizing all 8 GPUs on the node, Fairy
is 53$\times$ faster than TokenFlow and 44$\times$ faster than Rerender.

###  6.3 Ablation Studies

We conduct an ablation study to verify the effectiveness of our model’s
component. We gradually remove equivariant fine-tuning and anchor-based
attention, ultimately leading to the adoption of a standard frame-by-frame
editing approach. The results are shown in Figure 8. The model becomes
sensitive to the camera motion without equivariant finetuning, rendering
inconsistency in the details. The subsequent removal of anchor-based
attention, transitioning to a frame-based model, introduces further
inconsistencies in the generated video. We compute the Tem-Con metric based on
150 videos and report in Figure 8. It confirms our observation that the
proposed methodology effectively improves the temporal consistency, lifting
the Tem-Con from 0.959 (baseline) to 0.968 (w/ anchors) to 0.974 (w/ anchor
and equivariant finetuning).

###  6.4 Limitations

The efficacy of Fairy is intrinsically tied to the underlying image-editing
model. This means that any inherent constraints of this underlying model,
e.g., face and text distortion, etc., will naturally manifest in the video
editing capacities of Fairy . In our observations, a notable side effect of
equivariant finetuning is the diminished ability to accurately render dynamic
visual effects, such as lightning or flames. The process seems to overly focus
on maintaining temporal consistency, which often results in the depiction of
lightning as static or stagnate, rather than dynamic and fluid. See the
supplementary material for visualization.

##  7 Conclusion

Fairy offers a transformative approach to video editing, building on the
strengths of image-editing diffusion models. By leveraging anchor-based cross-
frame attention and equivariant finetuning, Fairy guarantees temporal
consistency and superior video synthesis. Moreover, it tackles the memory and
processing speed constraints observed in preceding models. With the capability
to produce high-resolution videos at a blazing speed, Fairy firmly establishes
its superiority in terms of quality and efficiency, as further corroborated by
our extensive user study.

## References

  * [1] Exploring video-to-video synthesis: A comparative analysis of rerender, tokenflow, and gen-1.  https://medium.com/@lwen9595/exploring-video-to-video-synthesis-a-comparative-analysis-of-rerender-tokenflow-and-gen-1-9a63f281c4e1.
  * [2] pytorch documentation.  https://pytorch.org/vision/main/transforms.html.  Accessed: 2023-11-02.
  * Shu [2023] Stock footage video, royalty-free hd, 4k video clips, 2023.
  * Bar-Tal et al. [2022] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In _European conference on computer vision_ , pages 707–723. Springer, 2022.
  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18392–18402, 2023.
  * Chai et al. [2023] Wenhao Chai, Xun Guo, Gaoang Wang, and Yan Lu.  Stablevideo: Text-driven consistency-aware diffusion video editing.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 23040–23050, 2023.
  * Dai et al. [2023] Xiaoliang Dai, Ji Hou, Chih-Yao Ma, Sam Tsai, Jialiang Wang, Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al.  Emu: Enhancing image generation models using photogenic needles in a haystack.  _arXiv preprint arXiv:2309.15807_ , 2023.
  * Doersch et al. [2022] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar, João Carreira, Andrew Zisserman, and Yi Yang.  Tap-vid: A benchmark for tracking any point in a video.  _Advances in Neural Information Processing Systems_ , 35:13610–13626, 2022.
  * Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7346–7356, 2023.
  * Geyer et al. [2023] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  _arXiv preprint arXiv:2307.10373_ , 2023.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Jamriška et al. [2019] Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sỳkora.  Stylizing video by example.  _ACM Transactions on Graphics (TOG)_ , 38(4):1–11, 2019.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Kasten et al. [2021] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel.  Layered neural atlases for consistent video editing.  _ACM Transactions on Graphics (TOG)_ , 40(6):1–12, 2021.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint arXiv:2303.13439_ , 2023.
  * Lee et al. [2023] Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang.  Shape-aware text-driven layered video editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14317–14326, 2023.
  * Liu et al. [2023a] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  _arXiv preprint arXiv:2303.04761_ , 2023a.
  * Liu et al. [2023b] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan.  Evalcrafter: Benchmarking and evaluating large video generation models.  _arXiv preprint arXiv:2310.11440_ , 2023b.
  * Ouyang et al. [2023] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen.  Codef: Content deformation fields for temporally consistent video processing.  _arXiv preprint arXiv:2308.07926_ , 2023.
  * Pont-Tuset et al. [2017] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool.  The 2017 davis challenge on video object segmentation.  _arXiv preprint arXiv:1704.00675_ , 2017.
  * Qi et al. [2023] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  _arXiv:2303.09535_ , 2023.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022\.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * [24] Alexander S.  Disco diffusion v5.2 - warp fusion.  https://github.com/Sxela/DiscoDiffusion-Warp.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Tang et al. [2023] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan.  Emergent correspondence from image diffusion.  _arXiv preprint arXiv:2306.03881_ , 2023.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1921–1930, 2023.
  * Wang et al. [2023a] Wen Wang, kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  _arXiv preprint arXiv:2303.17599_ , 2023a.
  * Wang et al. [2023b] Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen.  Zero-shot video editing using off-the-shelf image diffusion models.  _arXiv preprint arXiv:2303.17599_ , 2023b.
  * Wang et al. [2023c] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  _arXiv preprint arXiv:2306.02018_ , 2023c.
  * Wu et al. [2023] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7623–7633, 2023.
  * Yang et al. [2023] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.  _arXiv preprint arXiv:2306.07954_ , 2023.

##  Appendix A Webpage Demo

The videos in the main paper and appendix can be viewed with our demo webpage
by opening the webpage/index.html in the supplementary material using a web
browser.

##  Appendix B Additional Results from Human Evaluation

As mentioned in Section 6.2, we conduct A/B comparison between Fairy with
baselines. We ask annotators to compare our generated video with a baseline
method’s result, and decide which one is better.Each video pair is evaluated
by three independent annotators, and the majority vote is considered as the
final rating. We ask raters to evaluate by four attributes: frame quality –
visual quality of single frame; temporal consistency – whether the frames are
coherent or flickering; prompt faithfulness – whether the output followed the
editing instruction or target prompt; input faithfulness – whether the output
video followed the contents of the original video. We reported the overall
rating in Section 6.2, Figure 10. Here, we report a more detailed comparison
along each attributes in Figure 11, 12, 13. Compared with Rerender (Figure
11), Fairy loses in terms of single frame visual quality. This is mainly due
to the limitation of the foundational image editing model, while Rerender
utilizes LoRA to enhance frame quality. Yet, Fairy significantly outperforms
Rerender in terms of temporal consistency, achieves better prompt
faithfulness, and performs similarly in terms of input faithfulness. Compared
with TokenFlow, Fairy outperforms significantly in terms of frame quality,
temporal consistency, and prompt faithfulness. They performs similarly in
terms of input faithfulness. Compared with Gen-1, Fairy significantly
outperforms in all attributes.

Figure 11: Comparison with Rerender. Figure 12: Comparison with TokenFlow.
Figure 13: Comparison with Gen-1.

In addition to the A/B comparison, in which we ask human annotators to compare
our method with a baseline, we also conduct a standalone evaluation to examine
output video’s quality. Each time we show an annotator the original video, an
editing instruction, and the result video. We then ask the annotator to rate
the output as good or bad by the same four attributes. We ask 3 annotators to
rate each video, and the decision is determined by their majority vote. We
report the success rate by each attributes in Figure 14.

Figure 14: Standalone success rate by attributes. We report Fairy’s success
rate in terms of frame quality, temporal consistency, prompt faithfulness, and
input faithfulness.

##  Appendix C More Results

###  C.1 Character Swap

Figure 15: Additional Results on Character Swap: Fairy is able to interchange
the characters for videos with arbitrary ratios.

In Figure 15, we demonstrate more results of character swap, where Fairy is
able to interchange individuals with various characters. Note that our model
can adapt to different input aspect ratios without need for re-training.

###  C.2 Stylization

Figure 16: Additional Results on Stylization: Fairy enables a wide range of
style editing.

Figure 16 demonstrates more stylization results of Fairy. In particular, our
model is able to recognize various styles, while perform high quality and
temporal consistent edit based on the stylization instructions.

###  C.3 Arbitrary Long Videos

Fairy is able to scale to arbitrary long video without memory issue due to the
proposed anchor-based attention. In Figure 17, we show that our model is able
to generate a 27 second long video with high quality, while the latency is
less than 71.89 seconds via 6 A100 GPUs. In particular, the Fairy manage to
retain decent temporal consistency even number of frames (664 frames) is way
more than the number of anchor frames (3 frames).

Figure 17: Any-length Video Editing. Fairy is able to scale to arbitrary long
video without memory issue.

###  C.4 Ablation Study

Figure 18: Additional Results on Ablation Study. We demonstrate that both
equivariant finetuning and anchor-based attention are crucial to Fairy .

Figure 18 shows more ablation results by removing equivariant finetuning and
anchor-based attention. We can see that without equivariant finetuning, the
model is sensitive to local motion and movement of the subject and therefore
degenerate the frame quality and temporal consistency. For instance, in the
first video, the tail of the cat becomes the head of the lion in some of the
frames, and the face of the cat in second video vary significantly between
frames. Without anchor-based attention, the edit of each frame is completely
independent, rendering in significant worse temporal consistency.

Figure 19: Ablation study on number of anchor frames. We found that setting
number of anchor frames to 3 yields the best results.

Figure 19 demonstrates results generated with different number of anchor
frames. When number of anchor frames equals to 1, the global features model
can leverage are too restricted, which lead to suboptimal edits. In contrast,
we observe that when the number of anchor frames is greater than 7, the
quality also gradually degrades, losing some visual details.

Figure 20: Ablation study on number of diffusion steps. We found that
diffusion steps above 5 generally yield good results.

In Figure 20, we perform ablation study on the number of diffusion steps
during generation. The model perform reasonably well when the number of
diffusion step is above 10. We therefore set the diffusion step to 10 for all
of our experiments to optimize the latency.

###  C.5 Limitations

Figure 21: Limitations of Fairy. Our model cannot accurately render dynamic
visual effects, such as lightning, flames, or rain.

Finally, Figure 21 demonstrates some limitations we point out in section 6.4.
Since the model is never trained on video data, it does learn to generate
concepts containing motion such as raining, lightning, or flames. Fairy also
inherits the limit of the image editing model, where it is not able to follow
the instructions that involve camera motion, such as zoom in or zoom out.