# Evolutionary status of selected post-AGB stars based on Gaia DR3

M. Parthasarathy  Indian Institute of Astrophysics, Bangalore 560034, India
Vanderbilt University, Department of Physics & Astronomy, Nashville, TN 37235,
USA  Marina Kounkel  Vanderbilt University, Department of Physics & Astronomy,
Nashville, TN 37235, USA  Keivan G. Stassun  Vanderbilt University, Department
of Physics & Astronomy, Nashville, TN 37235, USA

###### Abstract

The evolutionary status of 24 post-AGB stars is presented based on Gaia DR3
data. All 24 stars have parallaxes accurate to better than $3\sigma$ and have
RUWE values $<1.4$. Based on the Gaia DR3 distances the absolute luminosities
are derived. For 14 of the stars, the luminosities confirm their post-AGB
evolutionary stage. However, V1027 Cyg, which was previously classified as a
post-AGB star, is found to have a higher luminosity; thus it may be an
evolved, massive, pulsating semi-regular variable star of type G7Ia. For 9 of
the stars, the luminosities are lower than 1000 L⊙, indicating that some are
post-HB stars and some are post-RGB stars.This paper was completed more than
four months ago but there was delay in getting it published.

##  1 Introduction

Post-asymptotic giant branch (AGB) supergiants are stars that have recently
evolved off the AGB but have not reached high enough temperatures to
photoionize their circumstellar dust envelopes (Parthasarathy & Pottasch 1986,
Hrivnak et al. 1989). The evolutionary stage of post-AGB supergiants is short-
lived depending on the core mass (Schoenberner 1983). During the transition
from the tip of the AGB to the young planetary nebula phase, these stars have
spectral types that evolve from M to OB (Parthasarathy 1993a), and they mimic
the spectra of supergiants because after the termination of the AGB phase of
evolution they have a white-dwarf C–O core with a very thin extended envelope.
From an analysis of IRAS data and their multi-wavelength studies during the
past 36 years have resulted in identifying few hundred post-AGB stars (for a
list see Vickers et al. 2015). Until recently their distances were not known
therefore it was not possible to place them on the post-AGB evolutionary
tracks. With the advent of Gaia satellite now we have accurate parallaxes for
large number of post-AGB stars. It is now possible to derive their absolute
luminosities and to compare them with the results of the pst-AGB stellar
evolutionary models (Parthasarathy et al. 2020, Kamath et al. 2022,
Parthasarathy 2022, Aoki et al. 2022).

In this paper we present an analysis of Gaia DR3 data of 24 post-AGB stars.
These stars are selected because they have accurate Gaia DR3 parallaxes, RUWE
values less than 1.4 (Stassun & Torres 2021) and they are not known to be
close binaries. Section 2 describes the selection criteria and the basic Gaia
DR3 data utilized. The principal results are presented Section 3 and a brief
summary of the conclusions is provided in Section 4.

##  2 Selection of stars and their Gaia DR3 Data

The Gaia DR3 data of selected post-AGB stars (selected from the list of post-
AGB stars given in Vickers et al. 2015) is given in Table 1. Several stars
have Gaia DR3 parallaxes comparable or less than their errors in parallaxes
and several stars have RUWE values $>1.4$. We have not considered these stars.
The selection criteria that we adopted is that they need to have accurate Gaia
DR3 parallaxes ($>3\sigma$), they need to have RUWE values less than 1.4, they
are single stars (not known to be binaries) and their Gaia data has not been
analyzed so far. With this criteria we selected 24 stars. In Table 1 we also
list their $V$, $B-V$, $E(B-V)$ values and spectral types.

Table 1: Gaia EDR3 Parallaxes and RUWE Statistics for Post-AGB Stars  Star | Name | $b$ | $\pi$ | $\sigma_{\pi}$ | $d$ | $\sigma_{d}$ | $V$ | $B-V$ | $E(B-V)$ | Sp. Type | RUWE
---|---|---|---|---|---|---|---|---|---|---|---
|  | deg. | mas | mas | pc | pc | mag. | mag. | mag. |  |
IRAS 01005+7910 | … | +16.6 | 0.2414 | 0.0176 | 4142.502 | 302.0217 | 10.87 | 0.17 | 0.15 | B1.7Ibeq | 1.037
LB 3193 | … |  $-$54.9 | 0.1369 | 0.0243 | 5755.1 | 691.73 | 12.70 |  $-$0.08 | 0.03 | B5I | 0.779
IRAS 02528+4350 | … |  $-$13.3 | 2.5409 | 0.0188 | 393.561 | 2.9119 | 10.68 | 0.42 | 0.44 | A0e | 1.234
IRAS 05040+4820 | SAO 40039 | +04.8 | 0.3221 | 0.0145 | 3104.626 | 139.7612 | 9.74 | 0.41 | 0.50 | A4Ia | 1.020
IRAS 05338$-$3051 | RV Col |  $-$28.8 | 0.5225 | 0.0158 | 1913.876 | 57.8741 | 9.30 | 1.11 | 0.09 | G5I | 1.094
IRAS 05381+1012 | HD 246299 |  $-$10.6 | 1.0380 | 0.0161 | 963.39 | 14.9428 | 10.54 | 0.82 | 0.06 | G2I | 1.042
IRAS 10456$-$5712 | HD 93662 | +01.5 | 0.8997 | 0.0255 | 1111.482 | 31.5025 | 6.248 | 1.706 | 0.02 | M0III | 0.984
IRAS 11353$-$6037 | HD 306753 | +0.7 | 0.1692 | 0.0133 | 5910.165 | 464.5697 | 12.4 | 0.55 | 0.65 | B5Ie | 1.101
IRAS 11385$-$5517 | HD 101584 | +5.94 | 0.5452 | 0.0199 | 1834.189 | 66.9486 | 7.01 | 0.39 | … | F0Iae | 1.05
IRAS 11387$-$6113 | … | +0.2 | 0.1854 | 0.0180 | 5393.743 | 523.6644 | 11.980 | 0.54 | 0.03 | A3Ie | 1.179
IRAS 11531$-$6111 | … | +0.7 | 0.1582 | 0.0180 | 4710.18 | 423.35 | 14.99 | 1.26 | 1.29 | B8Iae | 0.974
HD 105262 | BD+13 2491 | +72.47 | 0.5972 | 0.0267 | 1674.481 | 74.8638 | 7.08 | 0.0 | 0.0 | A0Ia | 0.8
IRAS 13110$-$5425 | HD 114855 | +8.0 | 0.6395 | 0.0314 | 1563.722 | 76.7801 | 9.23 | 0.43 | 0.11 | F5Ia/ab | 1.060
IRAS 14072$-$5446 | CD$-$54 5573 | +6.1 | 0.2072 | 0.0143 | 4826.255 | 333.0861 | 10.52 | 0.66 | 0.69 | B7Iab | 0.849
IRAS 14488$-$5405 | CD$-$53 5736 | +4.5 | 0.2684 | 0.0173 | 3725.782 | 240.1492 | 10.94 | 0.72 | 0.73 | A0Ie | 1.327
IRAS 16206$-$5956 | CD$-$59 6142 |  $-$7.5 | 0.1815 | 0.0191 | 5020.63 | 537.73 | 9.97 | 0.34 | … | A3Iabe | 1.07
IRAS 17311$-$4924 | Hen 1428 |  $-$9.04 | 0.2378 | 0.0201 | 3820.17 | 325.42 | 10.74 | 0.40 | 0.60 | B3Ie | 1.084
IRAS 19410+3733 | HD 186438 | +7.0 | 1.0421 | 0.0196 | 959.601 | 18.0483 | 7.91 | 0.41 | 0.13 | F3Ib | 1.082
IRAS 20004+2955 | V1027 Cyg |  $-$0.4 | 0.2390 | 0.0178 | 4184.100 | 311.6192 | 8.65 | 2.2 | 1.06 | G7Ia | 0.907
IRAS 20160+2734 | AU Vel |  $-$4.5 | 0.3908 | 0.0160 | 2558.854 | 104.7637 | 10.63 | 1.66 | 1.43 | F3Ie | 1.041
IRAS 20462+3416 | LSII+34 26 |  $-$5.75 | 0.1720 | 0.0184 | 4859.78 | 456.16 | 10.9 | 0.21 | 0.38 | B1.5Ia/abe | 1.235
IRAS 20490+5934 | … | +9.9 | 2.0625 | 0.0155 | 484.848 | 3.6437 | 10.32 | 0.41 | 0.36 | A3e | 0.866
IRAS 20572+4919 | V2324 Cyg | +2.4 | 1.5768 | 0.0124 | 634.196 | 4.9873 | 11.63 | 1.09 | 0.83 | F3Ie | 0.998
IRAS 21289+5815 | … | +5.2 | 1.0109 | 0.0196 | 989.218 | 19.1796 | 13.93 | 0.75 | 0.59 | A2Ie | 1.362

##  3 Results

The Gaia DR3 parallaxes (distances) of the selected sample of stars were used
to derive their absolute luminosities. Their $V$, $B-V$ and spectral types
(Table 1) were taken from SIMBAD. In order to derive their absolute
luminosities we need to take into account the interstellar and circumstellar
reddening. Many of our stars are IRAS sources and have circumstellar dust
shells.

###  3.1 Interstellar and circumstellar reddening

Several stars are located at high galactic latitudes (Table 1) therefore their
interstellar reddening values are relatively low. We took a simple approach to
estimate interstellar plus circumstellar reddening. All our stars have
observed $B-V$ values (Table 1). The observed $B-V$ is affected by both
interstellar plus circumstellar reddening. All of our stars have MK spectral
types (Table 1). We used the calibration between MK spectral types and
corresponding intrinsic $(B-V)_{0}$ values given in Allen’s Astrophysical
Quantities 4th edition (Cox 2000) and Flower (1996). The difference between
the observed $(B-V)$ and $(B-V)_{0}$ obtained from the spectral types yields
$E(B-V)$ (Table 2).

Using the following equation we derived the absolute visual magnitudes of the
stars (Table 2):

$M_{V}=V+5-5\log(d)-3.3E(B-V).$

Figure 1: HR Diagram positions of the sample stars. Table 2: Absolute luminosities of the stars Star | $M_{V}$ | $M_{\rm bol}$ | $T_{\rm eff}$ | $\log L/L_{\odot}$
---|---|---|---|---
| mag. | mag. | K | dex
IRAS 01005+7910 |  $-$3.503 |  $-$5.083 | 21500 | 3.93
LB3193 |  $-$1.2 |  $-$2.15 | 13000 | 2.76
IRAS 02528+4350 | +1.253 | +0.843 | 9500 | 1.56
IRAS 05040+4820 |  $-$4.37 |  $-$4.50 | 8750 | 3.70
RV Col |  $-$2.407 |  $-$2.737 | 5000 | 2.99
IRAS 05381+1012 | +0.423 | +0.273 | 5200 | 1.76
IRAS 10456$-$5712 |  $-$4.296 |  $-$5.746 | 4000 | 4.19
IRAS 11353$-$6037 |  $-$3.573 |  $-$4.523 | 13600 | 3.71
IRAS 11385$-$5517 |  $-$5.33 |  $-$5.44 | 8500 | 4.07
IRAS 11387$-$6113 |  $-$3.362 |  $-$3.642 | 9000 | 3.35
IRAS 11531$-$6111 |  $-$2.632 |  $-$3.292 | 11100 | 3.21
HD 105262 |  $-$3.928 |  $-$4.328 | 9000 | 3.63
IRAS 13110$-$5425 |  $-$2.104 |  $-$2.134 | 6400 | 2.75
IRAS 14072$-$5446 |  $-$5.175 |  $-$5.455 | 9000 | 4.08
IRAS 14488$-$5405 |  $-$4.325 |  $-$4.735 | 10000 | 3.79
IRAS 16206$-$5956 |  $-$4.689 |  $-$5.099 | 11200 | 3.94
IRAS 17311$-$4924 |  $-$4.15 |  $-$5.73 | 20500 | 4.19
IRAS 19410+3733 |  $-$2.495 |  $-$2.515 | 6700 | 2.90
IRAS 20004+2955 |  $-$7.956 |  $-$8.376 | 5000 | 5.246
IRAS 20160+2734 |  $-$6.129 |  $-$6.16 | 6700 | 4.36
IRAS 20462+3416 |  $-$3.787 |  $-$5.367 | 18000 | 4.04
IRAS 20490+5934 | +0.704 | +0.604 | 9000 | 1.65
IRAS 20572+4919 |  $-$0.12 |  $-$0.13 | 7500 | 1.95
IRAS 21289+5815 | +1.578 | +1.298 | 9400 | 1.38

###  3.2 Luminosities

To calculate the absolute bolometric magnitudes $M_{\rm bol}$ we used the
bolometric corrections (BC) given in Allen’s Astrophysical Quantities (4th
edition) (Cox 2000). The derived $M_{\rm bol}$ values and $\log(L/L_{\odot})$
values are given in Table 2. For several stars in this sample $T_{\rm eff}$
values are available in the literature (based on the analysis of their
spectra). The references for $T_{\rm eff}$ values are given in the Section 3.4
notes. We have also used spectral types, $(B-V)_{0}$ and $T_{\rm eff}$
calibration given in Allen’s Astrophysical Quantities (4th edition) (Cox
2000). $T_{\rm eff}$ values are not used in deriving the absolute
luminosities. $T_{\rm eff}$ values are used only to examine the location of
the stars in the H-R diagram (Figure 1).

###  3.3 Uncertainties

The errors in parallaxes and distances are given Table 1. The errors in
$E(B-V)$ are of the order of $\pm$0.05 to $\pm$0.10. The errors in derived
$M_{V}$ values are on the average less than 0.1. The errors in derived
absolute luminosities $\log(L/L_{\odot})$ of the order 0.10 dex. The errors in
$T_{\rm eff}$ values are of the order of 500 K.

###  3.4 Notes on individual sources

####  3.4.1 IRAS 01005+7910

It is a high galactic latitude carbon-rich hot post-AGB star. Zhang & Kwok
(2011) found fullerene C60 in the Spitzer/IRS mid-IR spectrum of this star.
Iglesias-Groth et al. (2013) found absorption bands at 9577Å and 9632Å which
are due to fullerene C60+ cation. Klochkova et al. (2002) from an analysis of
high resolution optical spectrum of this star derived $T_{\rm eff}$ = 21,500 K
and found it to be carbon-rich. Thus the photosphere and circumstellar matter
of this star is carbon-rich. It appears that the star has evolved from a
carbon-rich AGB star to carbon-rich post-AGB stage and it may evolve into a
[WC] type planetary nebula (PN) (Parthasarathy 1999). Its luminosity and
$T_{\rm eff}$ (Table 2) confirms that it is a hot post-AGB star. It may soon
evolve into a young PN similar to Hen 1357 (SAO 244567) (Parthasarathy et al.
1993).

####  3.4.2 LB 3193

It is a high galactic latitude hot metal-poor star. McCausland et al. (1992)
analyzed high resolution optical spectrum of this star and they classified it
as a galactic halo hot post-AGB star. They found underabundance of metals and
severe underabundance of carbon. The star seems to have left the AGB stage of
evolution before the third dredge up. They derived $T_{\rm eff}$ = 13,000 K,
$\log g=2.2$ and [Fe/H] = $-1.0$. Quin & Lamers (1992) analyzed UV (IUE)
spectrum of this star and derived $T_{\rm eff}$ = 14,000 K and $E(B-V)=0.03$.
It is not an IRAS source hence there is no circumstellar dust envelope. From
Gaia DR3 data we derive the absolute luminosity $\log(L/L_{\odot})=2.76$. Its
luminosity is very low when compared with the luminosity of a typical post-AGB
star. It may be a post-HB star or a AGB-manque star.

####  3.4.3 IRAS 02528+4350

It is classified as a high galactic latitude post-AGB star of spectral type
A0e (Suarez et al. 2006). Fujii et al. (2002) made $BVRIJHK$ photometric
observations. From Gaia DR3 data (Tables 1 and 2) we find that its absolute
luminosity is only 36.3 L⊙. It is not a post-AGB star. It may be a post-HB
star or a high galactic latitude A-type young star with circumstellar dust
shell/disk. To further understand its evolutionary status an analysis of its
high resolution optical spectrum is needed. It may be a normal A-type dwarf.

####  3.4.4 IRAS 05040+4820 (SAO 40039)

On the basis of IRAS colors and far-IR flux distribution, high galactic
latitude and A4 Ia spectral type Parthasarathy (1993b) classified it as a
post-AGB star. Fujii et al. (2002) made $BVRIJHK$ photometry and modeled the
SED including the near and far-IR (IRAS) data and concluded that it is a low-
mass post-AGB star. Parthasarathy et al. (2005) made $BVRI$ polarization
measurements and concluded that the circumstellar dust shell is asymmetric.
Rao et al. (2011) find that the this star may be mildly He-rich. However the
$T_{\rm eff}$ used by them in the analysis of spectrum may be relatively low.
Fujii et al. (2000) from SED find $T_{\rm eff}$ = 8750 K. Klochkova et al.
(2015) presented a spectral atlas. Parthasarathy et al. (2020) obtained K-band
spectrum and find no He I and H2 lines. They find the Brackett Gamma to be a
broad absorption line. The absolute luminosity derived from the Gaia DR3 data
(Tables 1 and 2) is in good agreement with the post-AGB evolutionary track of
a low-mass star.

####  3.4.5 RV Col (IRAS F05338-3051) (CD-30 2512)

It is a high galactic latitude G5 semi-regular (SRD) variable with a period of
105.7 days. The variations in light are up to one magnitude. Arkhipova et al.
(2011) studied the photometric variability and considered it as a post-AGB
candidate. The luminosity derived from Gaia DR3 data (Tables 1 and 2) shows
that it is not a post-AGB star. It may be a post-RGB star. Detailed chemical
composition study of this star from an analysis of high resolution spectra is
needed. Large amplitude light variations also indicate that it is not a post-
AGB star. However the presence circumstellar dust shell indicates that it
suffered mass-loss in the recent past.

####  3.4.6 IRAS 05381+1012

It is a high galactic latitude G2I star. Pereira & Roig (2006) analyzed high
resolution spectrum of this star. They classified it as a post-AGB star and
estimated a distance of 2700 pc. They have derived $T_{\rm eff}$ = 5200 K,
$\log g=1.0$ and [Fe/h] = $-0.8$, and rotational velocity of 40 km/sec. We
find Gaia DR3 data (Tables 1 and 2) that its distance is 963.39 pc and
$\log(L/L_{\odot})=2.00$. Its luminosity is too low hence it is not a post-AGB
star. It may be post-RGB star. The presence of circumstellar dust shell, low
metallicity, high galactic latitude, raid rotation makes this an interesting
object for further study. Kounkel et al. (2018, 2019) derived $T_{\rm eff}$ =
5493 K, $\log g=3.26$ and rotational velocity of 50.89 km/sec and radial
velocity of 43.280 km/sec.

####  3.4.7 IRAS 10456-5712

It is a relatively bright star. In SIMBAD its spectral type is K5/M0III. There
is no spectroscopic study of this star. On the basis of IRAS colors and far-IR
flux distribution it is classified as a post-AGB star. It may have cold
circumstellar dust disk. From Gaia DR3 data (Tables 1 and 2) we derive a
distance of 1111.482 pc and $\log(L/L_{\odot})=4.194$. It may be a post-AGB
star. High resolution spectroscopic study is needed to further understand its
evolutionary status.

####  3.4.8 IRAS 11353-6037 (HD 306753)

It is a hot (B5Ie) post-AGB star (Suarez et al. 2006). The luminosity (Table
2) suggests that it is a post-AGB star. There is no detailed study of this
star.

####  3.4.9 IRAS 11385-5517 (HD 101584)

On the basis of IRAS colors, far-IR flux distribution and circumstellar dust
shell parameters Parthasarathy & Pottasch (1986) classified it as a post-AGB
F0Ia supergiant similar to HD 161796. The near and far-IR flux distribution
shows that it is having hot and cold dust circumstellar dust shell/disk.
Olofsson et al. (2021) classify it as post-RGB star. Sivarani et al. (1999)
from an analysis high resolution spectra derived $T_{\rm eff}$ = 8500 K, $\log
g=1.5$ and [Fe/H] = 0.0. They found several emission lines and P-Cygni
profiles indicating the presence of stellar wind, very extended envelope and
circumstellar disk. Diaz et al. (2007) found radial velocity variations and
they derived a period of 144 days. It shows bipolar flows and complex
circumstellar envelope/disk (Olofsson et al. 2021). Olofsson et al. (2021)
find heavy element Rydberg transition line emission using high angular
resolution ALMA observations. From the analysis Gaia DR3 data (Tables 1 and 2)
we confirm that it is a post-AGB star with $\log(L/L_{\odot})=4.072$. The Gaia
parallax yields a distance of 1834.189 pc. Its luminosity (Table 2) indicates
that the progenitor main-sequence star may be of 2 to 2.5 M⊙.

####  3.4.10 HD 105262

It was discovered by Abt (1996) to be a metal-poor post-AGB star with large
proper motion. Abt (1996) concluded that its spectrum is similar to that of
post-AGB star HR 4049. It is a very high galactic latitude B9I star. Reddy et
al. (1996) and Arentsen et al. (2019) analyzed the spectrum of this star and
find that $T_{\rm eff}$ = 8225 K, $\log g=1.4$, [Fe/H] = $-1.94$. Its Gaia DR3
data reveals it is at a distance of 1674.481 pc. Its luminosity (Table 2)
confirms that it is a post-AGB star. It is not an IRAS source. There seems to
be no circumstellar dust shell.

####  3.4.11 IRAS 13110-5425 (HD 114855) (V956 Cen)

It is a high galactic latitude F5Ia/ab star. The luminosity derived from Gaia
DR3 data (Tables 1 and 2) is significantly lower than that of a post-AGB star.
It may be a post-RGB star. There is no high resolution spectroscopic study of
this star.

####  3.4.12 IRAS 16206-5956 (SAO 243756) (CD-59 6142)

Parthasarathy et al. (2000) presented low resolution spectrum of this star and
classified it as a high galactic latitude A3Iabe post-AGB star. Gauba &
Parthasarathy (2003, 2004) find variability in the UV (IUE) spectrum. They
derived $T_{\rm eff}$ = 11,200 K, $\log g=2.3$. The luminosity derived from
Gaia DR3 data (Tables 1 and 2) confirms that it is a post-AGB star.

####  3.4.13 IRAS 17311-4924 (Hen 1428)

On the basis of IRAS colors, far-IR flux distribution and circumstellar dust
shell parameters Parthasarathy & Pottasch (1989) were the first to classify it
as a high galactic latitude hot post-AGB star. Gauba & Parthasarathy (2003,
2004), Sarkar et al. (2005) and Mello et al. (2012) studied this star and
derived $T_{\rm eff}$ = 20,500 K, $\log g=2.35$. The luminosity derived from
Gaia DR3 data (Tables 1 and 2) confirms that it is a post-AGB star.

####  3.4.14 IRAS 19410+3733 (HD 186438) (LSII +37 1)

The luminosity derived from Gaia DR3 data (Tables 1 and 2) indicates that it
is not a post-AGB star. It may be a post-RGB star.

####  3.4.15 IRAS 20004+2955 (V1027 Cyg) (HD 333385)

It is in the galactic plane. It is a semiregular variable star of spectral
type G7Ia. Arkhipova et al. (2016) studied the photometric and spectral
variability. The luminosity derived from Gaia DR3 data (Tables 1 and 2)
indicates that it is not a post-AGB star. It is most likely a massive star.

####  3.4.16 IRAS 20462+3416 (LS II +34 26)

On the basis of IRAS colors, far-IR flux distribution, and cicumstellar dust
shell parameters Parthasarathy (1993a) discovered it to be a hot (B1.5Ia)
post-AGB star. Garcia-Lario et al. (1977) analyzed high resolution optical and
UV (IUE) spectrum of this star and derived $T_{\rm eff}$ = 18000 K. The
presence of low excitation nebular emission lines was first reported by
Parthasarathy (1994) and he has concluded that it is a rapidly evolving hot
post-AGB star similar to SAO 244567 (Parthasarathy et al. 1993). The
luminosity derived from Gaia DR3 data (Tables 1 and 2) confirms that it is a
post-AGB star. Parthasarathy et al. (2020) from the K-band spectrum of this
star find the presence of H2 lines at 2.12, 2.22, and 2.247 $\mu$m. They have
also find the Br $\gamma$ emission line in the spectrum. Gledhill & Forde
(2015) made a detailed study of the 2 $\mu$m to 2.4 $\mu$m spectrum of this
star and found that Br $\gamma$ emission is spatially extended.

####  3.4.17 IRAS 20490+5934

The luminosity (Tables 1 and 2) indicates that it is not a post-AGB star.

####  3.4.18 IRAS 20572+4919 (V2324 Cyg)

The luminosity (Tables 1 and 2) indicates it is not a post-AGB star.

####  3.4.19 IRAS 21289+5815

The luminosity (Tables 1 and 2) indicates it is not a post-AGB star.

##  4 Conclusions

Based on the Gaia DR3 parallaxes of 24 selected post-AGB stars we derived
their absolute luminosities. We find 14 of them have luminosities that confirm
their post-AGB status. HD 101584 which was originally classified as a post-AGB
star (Parthasarathy & Pottasch 1986); it is indeed a post-AGB star and not a a
post-RGB star. LS II +34 26, SAO 40039, SAO 243756 and Hen 1428 are also
confirmed to be hot post-AGB stars from this study.

Nine of the stars have luminosities $\log(L/L_{\odot}$) much less than 3.5. LB
3193 which was classified as a high galactic latitude, galactic halo metal-
poor hot post-AGB star is found to be only 575 L⊙. It may be a post-HB star or
a AGB-manque star. Some of the less luminous stars may be post-HB stars and
some may be post-RGB stars. Some may be misclassified as post-AGB stars.

Finally, V1027 Cyg is found to have much higher luminosity than that of a
typical post-AGB star. V1027 Cyg seems to be a massive, very luminous
semiregular variable G7Ia supergiant.

## References

Abt,H., 1996, PASP 108, 849

Aoki,W., Matsuni, T., Parthasarathy,M., 2022, PASJ (in press)

Arentsen,A., et al., 2019, A&A 627, 138

Arkhipova, V.P., et al., 2011, AstL 37, 635

Arkhipova, V.P., et al., 2016, AstL 42, 756

Cox,A.N., 2000, Allen’s Astrophysical Quantities (4th edition)

Diaz,F., et al., 2007, IAU Symp. 240, 127

Flower,P.J., 1996, ApJ 469, 355

Fujii,T., Nakada,Y., Parthasarathy,M., 2002, A&A 385, 884

Garcia-Lario,P., Parthasarathy,M., De -Martino,D., et al., 1997, A&A 326, 110

Gauba,G., & Parthasarathy,M., 2003, A&A 407, 1007

Gauba,G., Parthasarathy,M., 2004, A&A 417, 201

Hrivnak, B.J., et al. 1989, ApJ 346, 265

Gledhill, T,M., & Forde, K.P., 2015, MNRAS 447, 1080

Iglesias-Groth,S., et al., 2013, ApJ 776, L2

Kamath,D., et al., 2022, ApJ, 927, L13

Klochkova,V.G., 2002, A&A 392, 143

Klochkova,V.G., et al., 2015, AstBu., 70, 99

Kounkel,M., et al., 2018, AJ, 156, 84

Kounkel,M., et al., 2019, AJ 157, 196

McCausland, R.J.H., et al., 1992, ApJ 394, 298

Mello, D.R.C., et al., 2012, A&A

Miller-Bertolami,M.M., 2016, A&A 588, 25

Olofsson,H., et al., 2021, A&A 651, 35

Parthasarathy,M., 1993a, ApJ 414, L109

Parthasarathy,M., 1993b, ASP Conf. Series 45, 173

Parthasarathy,M., 1994, ASP Conf.Series 60, 261

Parthasarathy,M., 1999, IAU Symp. 191, 475

Parthasarathy,M., 2022, RNAAS 6, 33

Parthasarathy,M., et al., 1993, A&A 267, L19

Parthasarathy,M., Pottasch, S.R., 1986, A&A 154, L16

Parthasarathy,M., Pottasch,S.R., 1989, A&A 225, 521

Parthasarathy, M., Jain, S.K., Sarkar, G., 2005, AJ 129, 2451

Parthasarathy,M., Matsuno,T., Aoki,W., 2020, PASJ 72, 99

Parthasarathy,M., Muthumariappan,C., Muneer,S., 2020, Ap&SS 365, 127

Parthasarathy,Vijapurkar,J., Drilling, J.S., 2000, A&AS 145, 269

Pereira,C.B., & Roig, F., 2006, A&A 452, 571

Pereira,C.B., Miranada,L.F., 2007, A&A 462, 231

Quin, D.A., & Lamers, H.J.G.L.M., 1992, A&A 260, 261

Rao,S.S., et al., 2011, ApJ 737, L7

Reddy, B.E., Parthasarathy,M., Sivarani, T., 1996, A&A 313, 191

Sarkar,G., Parthasarathy,M., Reddy,B.E., 2005, A&A 431, 1007

Schoenberner,D., 1983, ApJ 272, 708

Sivarani,T., Parthasarathy,M., Garcia-Lario, P., et al., 1999, A&AS 137, 505

Stassun,K.G., & Torres, G., 2021, ApJ 907, L33

Suarez,O., et al., 2006, A&A 458, 173

Vickers, S.B., et al., 2015, MNRAS 447, 1673

Zhang,Y., Kwok,S., 2011, ApJ 730, 126


# DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis

Zhongjie Duan1, Lizhou You2, Chengyu Wang3, Cen Chen1, Ziheng Wu3,
Weining Qian1, Jun Huang3

###### Abstract

In recent years, diffusion models have emerged as the most powerful approach
in image synthesis. However, applying these models directly to video synthesis
presents challenges, as it often leads to noticeable flickering contents.
Although recently proposed zero-shot methods can alleviate flicker to some
extent, we still struggle to generate coherent videos. In this paper, we
propose DiffSynth, a novel approach that aims to convert image synthesis
pipelines to video synthesis pipelines. DiffSynth consists of two key
components: a latent in-iteration deflickering framework and a video
deflickering algorithm. The latent in-iteration deflickering framework applies
video deflickering to the latent space of diffusion models, effectively
preventing flicker accumulation in intermediate steps. Additionally, we
propose a video deflickering algorithm, named patch blending algorithm, that
remaps objects in different frames and blends them together to enhance video
consistency. One of the notable advantages of DiffSynth is its general
applicability to various video synthesis tasks, including text-guided video
stylization, fashion video synthesis, image-guided video stylization, video
restoring, and 3D rendering. In the task of text-guided video stylization, we
make it possible to synthesize high-quality videos without cherry-picking. The
experimental results demonstrate the effectiveness of DiffSynth. All videos
can be viewed on our project page111https://anonymous456852.github.io/. Source
codes will also be
released222https://github.com/alibaba/EasyNLP/tree/master/diffusion.

## Introduction

In recent years, diffusion models have achieved remarkable success in the
field of image synthesis, surpassing Generative Adversarial Networks (GANs)
(Dhariwal and Nichol 2021). In open-source communities, Stable Diffusion
(Rombach et al. 2022) has emerged as the most popular diffusion model, and
fine-tuned models based on Stable Diffusion have achieved astonishing success
in various artistic styles. Furthermore, several research breakthroughs have
improved the capabilities of Stable Diffusion (Gal et al. 2022; Zhang and
Agrawala 2023; Hu et al. 2021; Ruiz et al. 2023; Meng et al. 2021). The
achievements of Stable Diffusion have established it as the mainstream
approach of image synthesis.

In our work, we further investigate the capabilities of diffusion models in
video synthesis. It is well-known that directly applying image synthesis to
each frame leads to significant flickering (Yu et al. 2021). To address this
problem, researchers have proposed solutions from various perspectives. One
end-to-end solution is to pre-train a new video synthesis model using video
datasets (Blattmann et al. 2023). Yet, there are no large-scale and high-
resolution video datasets available to train video diffusion models, and the
computational resources required for training are enormous. Furthermore, most
existing techniques based on Stable Diffusion cannot be directly applied to
newly trained models, making it difficult to control the generated results of
such models. Therefore, we focus on one-shot and zero-shot methods, aiming to
transfer existing image synthesis models to video synthesis with minimal or
even no training. In recent years, several methods for video synthesis have
been proposed and gained popularity. For example, Tune-A-Video (Wu et al.
2022) achieves video editing by fine-tuning the textual part of the model.
Text2Video-Zero (Khachatryan et al. 2023) restricts the content of adjacent
frames using cross-frame attention. However, these methods are difficult to
completely eliminate flicker in videos. Our aim is to design a more effective
approach to fully eliminate flicker in videos synthesized by diffusion models.

To address the challenges mentioned above, we propose a novel approach named
DiffSynth. Specifically, we design a latent in-iteration deflickering
framework, with the aim of removing flicker during the intermediate iterations
of video synthesis. In our approach, the video-level deflickering is applied
to the decoded videos in the latent space. After that, the videos are encoded
back from the latent deflickered representations. Hence, we can effectively
prevent the accumulation of flicker in denoising steps. For video
deflickering, we design a patch blending algorithm to ensure the performance.
By remapping different frames to the same frame, we can obtain the appearance
features of the same object in different frames. The remapping operator is
implemented based on patch matching (Barnes et al. 2009), which can estimate
the nearest-neighbor field between two frames. Next, we blend the results to
obtain a video with consistent contents. This algorithm can either eliminate
high-frequency flicker using a sliding window, or thoroughly remove all
flicker by blending all frames. Note that the latter has high computational
complexity. Hence, we specifically propose a low time-complexity approximation
algorithm, making our approach effective for long video synthesis.

DiffSynth is compatible with most models based on Stable Diffusion. Leveraging
the research achievements in image synthesis, we have designed pipelines for
multiple downstream tasks, including text-guided video stylization, fashion
video synthesis, image-guided video stylization, video restoring, and 3D
rendering. These video synthesis pipelines are all transferred from image
synthesis pipelines, and we provide hyperparameter templates for these
application scenarios to facilitate their usage. To validate the effectiveness
of our method, we conducted extensive experiments in two application
scenarios. Without any cherry-picking, we are able to generate coherent and
realistic videos. Unsurprisingly, DiffSynth comprehensively outperforms
existing baseline methods in quantitative metrics and user studies. We
summarize the contributions of this paper as follows:

  * •

We propose DiffSynth, a novel approach for coherent and realistic video
synthesis.

  * •

We devise a latent in-iteration deflickering framework, which applies video-
level deflickering to the latent space of diffusion models, avoiding the
accumulation of flicker during the iterative process.

  * •

Based on patch matching, we propose a patch blending algorithm that can
significantly eliminate flicker in videos synthesized by diffusion models.

  * •

We design several video synthesis pipelines utilizing DiffSynth for various
tasks and demonstrate the superiority of our method in extensive experiments.

## Related Work

### Diffusion Models

Diffusion models (Song and Ermon 2019; Nichol and Dhariwal 2021; Sohl-
Dickstein et al. 2015) are a kind of generative model that decomposes the
image synthesis process into a sequential application of denoising models.
Unlike GAN-based models (Goodfellow et al. 2020; Huang et al. 2017), diffusion
models do not rely on adversarial training and are generally easier to train.
Latent Diffusion (Rombach et al. 2022) introduces the concept of converting
images from the pixel space to the latent space, making it possible to train
the model on limited computational resources. Based on Latent Diffusion,
Stable Diffusion becomes the most popular and powerful model in research
communities. Recent studies have further enhanced the impressive capabilities
of Stable Diffusion in various ways. LoRA (Low-Rank Adaptation) (Hu et al.
2021) reduces computational resource requirements for fine-tuning by
incorporating low-rank matrices into the model. To enhance the controllability
of images generated by Stable Diffusion, ControlNet (Zhang and Agrawala 2023)
uses zero convolution to inject additional conditional information into the
model, controlling the model generating objects in specific shapes, characters
in specific poses, etc. Textual Inversion (Gal et al. 2022) can synthesize a
specific object by adding a new word to the vocabulary and training with a few
images. Dreambooth (Ruiz et al. 2023) further improves the model’s ability to
generate specific objects. Our proposed method is seamlessly compatible with
these approaches, enhancing its capabilities in video synthesis.

### Diffusion-based Video Synthesis

Inspired by the remarkable success of diffusion models, researchers endeavored
to transfer diffusion models to video synthesis tasks, such as text-to-video
synthesis, video style transfer and video editing. For example, Make-A-Video
(Singer et al. 2022) and VideoLDM (Blattmann et al. 2023) extend a text-to-
image diffusion model to a text-to-video diffusion model by incorporating
temporal blocks. Gen-1 (Esser et al. 2023) utilizes a similar temporal
architecture to transfer the video style. Besides pre-training a large-scale
video model, recent studies focus on synthesizing videos using image models.
Tune-A-Video (Wu et al. 2022) only fine-tunes several modules using input
video, enabling video editing according to the given prompts. Some zero-shot
methods, including FateZero (Qi et al. 2023), Pix2Video (Ceylan, Huang, and
Mitra 2023), and Text2Video-Zero (Khachatryan et al. 2023), have explored the
possibility of synthesizing videos without additional training. These studies
collectively demonstrate the feasibility of applying diffusion models to video
synthesis tasks. Motivated by the findings of these studies, we propose a
novel zero-shot approach, aiming to convert existing image synthesis pipelines
to video synthesis pipelines expediently.

## Methodology

In this section, we briefly review the diffusion models and then introduce our
proposed approach.

### Preliminaries

Generally, diffusion models include many different architectures (Feng et al.
2023; Saharia et al. 2022; Ramesh et al. 2022). In this paper, we focus mainly
on Stable Diffusion (Rombach et al. 2022), which is the most popular open-
source architecture. A typical Stable Diffusion model contains the following
three components:

  * •

Text Encoder. A transformer-based language model in CLIP (Radford et al.
2021), converting texts to text embeddings. The text embeddings are
subsequently used in classifier-free guidance (Ho and Salimans 2021).

  * •

U-Net (Ronneberger, Fischer, and Brox 2015). A vision model $\epsilon$ with
self-attention (Vaswani et al. 2017), cross-attention, and residual
connections (He et al. 2016). This model is trained to denoise images in the
latent space.

  * •

VAE (Kingma and Welling 2013). A model consists of an encoder $\mathcal{E}$
and a decoder $\mathcal{D}$, where the encoder converts images to latent
tensors, and the decoder reconstructs images according to latent tensors.

Both the diffusion process and its reverse process (i.e., the generation
process) are conducted in latent space. There are $T+1$ steps with
$\\{0,1,\dots,T\\}$ representing different levels of noise. In the generation
process, we start with a noise tensor $x_{T}$ sampled from a Gaussian
distribution, and then denoise it stepwise. The iterative formula of each step
$t$ is

$x_{t-1}=\sqrt{\alpha_{t-1}}\left(\frac{x_{t}-\sqrt{1-\alpha_{t}}\
\epsilon(x_{t})}{\sqrt{\alpha_{t}}}\right)+\sqrt{1-\alpha_{t-1}}\
\epsilon(x_{t}),$ (1)

where $\alpha_{t}$ is the hyperparameter describing how much noise it contains
in step $t$. After the iterative process, the latent tensor $x_{0}$ is decoded
to an image $X=\mathcal{D}(x_{0})$. In addition to text-to-image synthesis, we
can modify this process to implement other pipelines. For example, if we add
noise to an image and then denoise from an intermediate step, we obtain an
image-to-image pipeline for image editing.

To convert an image synthesis pipeline to a video synthesis pipeline, a naive
approach is to generate each frame independently. However, most models based
on Stable Diffusion are trained on text-image datasets and lack the ability to
generate coherent video frames. After each iterative step, the difference
between two adjacent frames becomes larger. The difference is accumulated
during the generation process and finally results in irreparable flicker and
inconsistency.

### Latent In-Iteration Deflickering

As we mentioned above, the generation process of diffusion models is conducted
in latent space, not pixel space. Inspired by deflickering methods designed
for videos (Lei, Xing, and Chen 2020; Lei et al. 2023), we design a framework
to apply deflickering to the latent space for better video synthesis.

For a video consisting of $n$ frames, at step $t$, we have $n$ latent tensors
$\\{x_{t}^{1},x_{t}^{2},\dots,x_{t}^{n}\\}$ corresponding to each frame. If we
directly compute $\\{x_{t-1}^{1},x_{t-1}^{2},\dots,x_{t-1}^{n}\\}$ using
formula (1), these latent tensors will become more inconsistent because each
tensor is computed independently. To visualize the latent tensors, we first
skip to the final step to calculate the estimation of
$\\{x_{0}^{1},x_{0}^{2},\dots,x_{0}^{n}\\}$:

$\hat{x}_{0}^{i}=\frac{x_{t}^{i}-\sqrt{1-\alpha_{t}}\
\epsilon(x_{t}^{i})}{\sqrt{\alpha_{t}}}.$ (2)

Then, we decode $\\{\hat{x}_{0}^{1},\hat{x}_{0}^{2},\dots,\hat{x}_{0}^{n}\\}$
to images using the decoder component of VAE:

$\hat{X}^{i}=\mathcal{D}(\hat{x}_{0}^{i}).$ (3)

Theoretically, $\\{\hat{X}^{1},\hat{X}^{2},\dots,\hat{X}^{n}\\}$ represent the
frames when we denoise $\\{x_{t}^{1},x_{t}^{2},\dots,x_{t}^{n}\\}$ along with
a straight line directed by
$\\{\epsilon(x_{t}^{1}),\epsilon(x_{t}^{2}),\dots,\epsilon(x_{t}^{n})\\}$. We
employ a video-level deflickering method $\mathcal{F}$ to make the video
coherent, i.e.,

$\\{\overline{X}^{1},\overline{X}^{2},\dots,\overline{X}^{n}\\}=\mathcal{F}\\{\hat{X}^{1},\hat{X}^{2},\dots,\hat{X}^{n}\\}.$
(4)

Next, we encode the processed frames into the latent space:

$\overline{x}_{0}^{i}=\mathcal{E}\left(\overline{X}^{i}\right).$ (5)

To synthesize the video frames
$\\{\overline{X}^{1},\overline{X}^{2},\dots,\overline{X}^{n}\\}$, the
predicted noise at step $t$ should be:

$\overline{\epsilon}(x_{t}^{i})=\frac{x_{t}^{i}-\sqrt{\alpha_{t}}\
\overline{x}_{0}^{i}}{\sqrt{1-\alpha_{t}}}.$ (6)

Thus, we obtain the reconstructed iterative formula as:

$x_{t-1}^{i}=\sqrt{\alpha_{t-1}}\ \overline{x}_{0}^{i}+\sqrt{1-\alpha_{t-1}}\
\overline{\epsilon}(x_{t}^{i}).$ (7)

The pipeline (2-7) makes it possible to apply existing deflickering methods to
latent tensors. At each denoising step, we can keep the difference between
frames controllable and avoid the accumulation of flicker.

Frame 1 | Frame 2 | Frame 3 | Frame 4 | Frame 5 | Frame 6 | Frame 7 | Frame 8
---|---|---|---|---|---|---|---
$\hat{X}^{1}$ | $\hat{X}^{2}$ | $\hat{X}^{3}$ | $\hat{X}^{4}$ | $\hat{X}^{5}$ | $\hat{X}^{6}$ | $\hat{X}^{7}$ | $\hat{X}^{8}$
| $[\hat{X}^{1\to 2}]_{1}$ |  | $[\hat{X}^{3\to 4}]_{1}$ |  | $[\hat{X}^{5\to 6}]_{1}$ |  | $[\hat{X}^{7\to 8}]_{1}$
|  | $[\hat{X}^{1\to 4}]_{2}\oplus[\hat{X}^{2\to 4}]_{1}$ |  |  | $[\hat{X}^{5\to 8}]_{2}\oplus[\hat{X}^{6\to 8}]_{1}$
|  |  |  | $[\hat{X}^{1\to 8}]_{3}\oplus[\hat{X}^{2\to 8}]_{2}\oplus[\hat{X}^{3\to 8}]_{2}\oplus[\hat{X}^{4\to 8}]_{1}$
Table 1: An example of a remapping table in the patch blending algorithm.

### Patch Blending Algorithm

Another problem to be addressed is how to design the video-level deflickering
method $\mathcal{F}$ mentioned above. In a video synthesis task without any
reference, we can only employ blind video deflickering methods (Lei, Xing, and
Chen 2020; Lei et al. 2023), thus it is difficult to keep the video fluent. In
many video synthesis tasks (e.g., style transferring and video editing), we
have the original video for reference. Hence, we mainly focus on these tasks
in this work and propose the following deflickering algorithm.

Assuming that we have obtained the synthesized frames
$\\{\hat{X}^{1},\hat{X}^{2},\dots,\hat{X}^{n}\\}$ in one denoising step, we
aim to make the video smoother with reference to the original video
$\\{X^{1},X^{2},\dots,X^{n}\\}$. For an object in frame $\hat{X}^{i}$, it may
also occur in another frame $\hat{X}^{j}$, thus we intend to remap the
corresponding areas in $\hat{X}^{j}$ to $\hat{X}^{i}$ and then blend them
together. The blended frame will show the consistent information of both
$\hat{X}^{i}$ and $\hat{X}^{j}$ if the remapping result is accurate. Although
recent studies (Kirillov et al. 2023; Han et al. 2022) have achieved
impressive success in object segmentation and tracking, it is still difficult
to achieve pixel-level accuracy. We have also considered optical flow (Teed
and Deng 2020), but we find that optical flow is typically not accurate when
the interval of two frames is large. Finally, we decide to utilize a patch
matching algorithm (Barnes et al. 2009), which is an effective algorithm to
estimate the correspondence between two frames. In this algorithm, the two
frames $X^{i}$ and $X^{j}$ are divided into some overlapping patches. We first
compute a nearest neighbor field (NNF) that represents the matched patches and
then reconstruct $\hat{X}^{i}$ using $\hat{X}^{j}$. Finally, we blend the
reconstructed frames with the synthesized frames.

As an efficient implementation, we only remap and blend in a small sliding
window. Yet, in the worst case, we need to remap and blend all frames
together, which requires $\mathcal{O}(n^{2})$ times of NNF estimation. The
high time complexity is the main pitfall that prevents this algorithm from
application. To improve computational efficiency, we propose an
$\mathcal{O}(n\log n)$ approximate algorithm.

For convenience, we use a remapping operator $[j\to i]$ to denote the
remapping process and use $\hat{X}^{j\to i}$ to denote the remapped results
from $\hat{X}^{j}$ to $\hat{X}^{i}$. Note that the remapped and blended
results could be remapped to another frame again, but multiple times of
remapping and blending may result in non-negligible errors. We use a subscript
number to denote the times of remapping and blending, i.e.,

$[\hat{X}^{j\to i}]_{0}=\hat{X}^{i},$ (8) $[\hat{X}^{j\to
i}]_{u+1}=[\hat{X}^{j\to k}]_{u}[k\to i].$ (9)

The blending process is denoted as a blending operator $\oplus$. If we
directly use the average, this operator is equivalent to “+”. Note that the
coefficient $\frac{1}{2}$ is not included in the blending operator. The final
blended result should be divided by the number of frames. We can also train a
lightweight network to implement this operator, in order to reduce the errors
in the following approximate calculation. We leave this component for future
work.

We use a remapping table to store some intermediate variables. An example of a
remapping table is presented in Table 1. First, we store each synthesized
frame in the first row. Then in the $k$-th iteration and the $i\cdot 2^{k}$-th
column, we blend the frames in the $(i\cdot 2^{k}-2^{k-1})$-th column and then
remap it to the $i\cdot 2^{k}$-th frame. For example, see the $3$rd row and
the $4$th column in Table 1. It is calculated as follows:

$\displaystyle\left([\hat{X}^{1\to 2}]_{1}\oplus\hat{X}^{2}\right)[2\to 4]$
(10) $\displaystyle=$ $\displaystyle\left([\hat{X}^{1\to 2}]_{1}[2\to
4]\right)\oplus\left(\hat{X}^{2}[2\to 4]\right)$ $\displaystyle=$
$\displaystyle[\hat{X}^{1\to 4}]_{2}\oplus[\hat{X}^{2\to 4}]_{1}$

With another array storing the “$\oplus$” sum of each column, we can calculate
the whole table within $\mathcal{O}(n)$ time complexity, and the memory
complexity is also $\mathcal{O}(n)$.

Once we obtain the remapping table, we can quickly calculate the smoothed
frame $\overline{X}^{i}=\frac{1}{n}\oplus_{j=1}^{n}\hat{X}^{j\to i}$. For
instance, when $i=6$, we have:

$\displaystyle\oplus_{j=1}^{6}\hat{X}^{j\to 6}$ (11) $\displaystyle\approx$
$\displaystyle\left([\hat{X}^{1\to 4}]_{2}\oplus[\hat{X}^{2\to
4}]_{1}\oplus[\hat{X}^{3\to 4}]_{1}\oplus\hat{X}^{4}\right)[4\to 6]$
$\displaystyle\oplus\left([\hat{X}^{5\to 6}]_{1}\oplus\hat{X}^{6}\right)$
$\displaystyle=$ $\displaystyle[\hat{X}^{1\to 6}]_{3}\oplus[\hat{X}^{2\to
6}]_{2}\oplus[\hat{X}^{3\to 6}]_{2}\oplus[\hat{X}^{4\to 6}]_{1}$
$\displaystyle\oplus[\hat{X}^{5\to 6}]_{1}\oplus\hat{X}^{6}$

Similarly, we reverse the sequence and calculate the remaining part:

$\oplus_{j=7}^{8}\hat{X}^{j\to 6}\approx[\hat{X}^{7\to
6}]_{1}\oplus[\hat{X}^{8\to 6}]_{2}$ (12)

As we mentioned above, the “$\oplus$” sum of each column has been stored in
another array, thus we can calculate the estimation of
$\oplus_{j=1}^{n}\hat{X}^{j\to i}$ within $\mathcal{O}(\log n)$ time.

Taking into account the accuracy of the estimated “$\oplus$” sum, we can
easily prove that the maximum number of remapping and blending operations is
$\mathcal{O}(\log n)$. In equations (11) and (12), it is apparent that the
frames close to the $i$-th frame exhibit lower error compared to those located
further away. Consequently, this algorithm achieves both speed and numerical
stability.

### Other Modifications

To convert an image synthesis pipeline to a video synthesis pipeline, we
further modify the following details.

  * •

Fixed noise. When we synthesize images, sampling from the same Gaussian noise
leads to the same image if we leave other settings fixed. In video synthesis,
the frames in a video are expected to be similar; thus we synthesize each
frame from the same Gaussian noise. In some downstream tasks, some information
from the input video is supposed to be retrained in the edited video, thus we
add the same Gaussian noise to each frame.

  * •

Cross-frame attention. If we want to generate an image similar to a reference
image, we can concatenate our image with the reference image and synthesize
our image in an in-painting pipeline. This is a widely used trick to control
the generated content. However, the model will draw unexpected components near
the seam line, because it tends to combine the two images into one complete
image. Essentially, the information from the reference images is passed to our
image mainly by self-attention (Vaswani et al. 2017). Thus, we change self-
attention layers to cross-frame attention layers. In the denoising process of
the $i$-th frame, we concatenate $x_{1}$, $x_{i-1}$, $x_{i}$ and $x_{i+1}$
together in the self-attention layers. If a ControlNet is used in the
pipeline, we also convert self-attention layers of ControlNet to cross-frame
attention layers.

  * •

Adaptive resolution. Today, most videos on the social network are in the shape
of rectangles, and their resolution is usually high. However, most diffusion
models are trained to synthesize images in the shape of $512\times 512$
squares. Considering the architecture of U-Net, we can easily change the shape
and increase the resolution according to the downstream tasks. In our
experiments, we surprisingly find that higher resolution sometimes leads to
more fluent video frames. The reason is that the implicit patches represent
more fine-grained information when the resolution is higher.

  * •

Deterministic sampling of VAE. In equation (5) and image-to-image pipelines,
the VAE encoder is called to encode images to latent space. In fact, the
output of VAE is a Gaussian distribution, not a deterministic tensor. To
maintain the consistency of different frames, we use deterministic sampling
instead of Gaussian sampling (i.e., setting the standard deviation to zero).

  * •

Memory-efficient attention. Cross-frame attention greatly increases the time
consumed to synthesize a video, and the memory required is up to
$\mathcal{O}(N^{2})$, where $N$ is the number of implicit patches. Existing
studies have shown that the attention mechanism could be implemented in low
memory (Rabe and Staats 2021). Therefore, we employ flash attention (Dao et
al. 2022) to implement the attention mechanism, which is capable of
significantly improving computational efficiency and reducing the memory
required.

  * •

Smoothed ControlNet annotator. When a ControlNet model is used to control the
content, an annotator is employed for processing the original frames, but we
observed that some annotators may cause flickering control signal. For
example, the OpenPose (Cao et al. 2017) annotator can cause tic of the limbs
if the frames are not clear enough. To overcome this pitfall, we use Savitzky-
Golay smoothing filters (Press and Teukolsky 1990) to smooth the coordinates
of the key points detected.

## Experiments

To demonstrate the effectiveness of our approach, we conducted the following
experiments, including text-guided video stylization and fashion video
synthesis. We compared our approach with several state-of-the-art approaches
and evaluated the synthesized videos using both quantitive metrics and user
study.

### Experimental Settings

| Text-guided | Fashion
---|---|---
| video | video
| stylization | synthesis
Frame height | 512 | 768
Frame width | 960 | 512
Denoising steps | 20 | 10
ControlNet scale | 1.0 (Depth) | 0.3 (Depth)
1.0 (SoftEdge) | 0.6 (OpenPose)
CFG scale | 7.5 | 7.5
Deflickering window size | $\infty$ | 7
Deflickering frequency | 5 | 1
Table 2: The hyperparameters in the experiments.

Text-Guided Video Stylization. In this task, we design a pipeline to transfer
the style of videos according to given prompts. The dataset, consisting of 100
high-resolution videos, is collected from a community333https://pixabay.com/.
Each video is cut into 3 to 5 seconds, including at most 150 frames. We
manually write prompts for each video444https://github.com/ECNU-
CILAB/Pixabay100. The pipeline is composed of a popular customized model in
open-source communities555https://civitai.com/models/4384/dreamshaper and two
ControlNet models. We use
Depth666https://huggingface.co/lllyasviel/control_v11f1p_sd15_depth and
SoftEdge777https://huggingface.co/lllyasviel/control_v11p_sd15_softedge to
provide structure guidance. The information in the original video is only
delivered to the pipeline by ControlNet, thus the color is completely ignored.
After the final step, we add an additional deflickering step to reduce the
flicker generated in the last four steps and set the window size to 121. We
further improve the contrast ratio and sharpen the frames slightly to improve
video quality.

Figure 1: An example of text-guided video stylization. The prompt in this
example is “an orange and white cat”.

Fashion Video Synthesis. The second task is to customize virtual fashion
models and synthesize fashion videos on e-commerce platforms. The dataset used
in this experiment is a fashion video dataset (Zablotskaia et al. 2019), which
contains hundreds of videos. There is a fashion model with fashion clothes in
each video. We randomly select 10 source videos from the dataset and fine-tune
Stable Diffusion 1.5888https://huggingface.co/runwayml/stable-diffusion-v1-5
on each video, respectively. The fine-tuned models have learned the appearance
of each fashion model; in other words, one fine-tuned diffusion model
represents a virtual fashion model. We randomly select the other 10 target
videos, then extract the pose sequence using OpenPose (Cao et al. 2017), and
let the 10 virtual fashion models imitate the pose in the other 10 target
videos. In this pipeline, we use OpenPose
ControlNet999https://huggingface.co/lllyasviel/control_v11p_sd15_openpose to
control the pose of models. We notice that the pose extracted by OpenPose
suffers from slight tic, although we use Savitzky-Golay smoothing filters
(Press and Teukolsky 1990). To further smoothen the synthesized videos, we use
Depth ControlNet to stabilize the motion. Finally, we obtain $10\times 10$
realistic synthesized videos.

Figure 2: An example of fashion video synthesis.

Hyperparameters. The hyperparameters of the above two tasks are presented in
Table 2. These parameters are tuned by human experts. The size of the
deflickering window is $\infty$, denoting that we use the fast remapping and
blending algorithm to blend all frames. The deflickering frequency denotes how
frequently we apply the deflickering algorithm. For example, when the number
of denoising steps is 20 and the deflickering frequency is 5, we apply the
deflickering algorithm at the 1st, 6th, 11th, and 16th steps. To make the
experiments reproducible, we synthesize each video with the same random seed,
without any cherry-picking.

| Content | Prompt | Content
---|---|---|---
| Consistency | Similarity | Aesthetics
| (Pixel-MSE $\downarrow$) | (CLIP Score $\uparrow$) | (Aesthetic Score $\uparrow$)
FateZero | - | 22.32 | 5.44
Pix2Video | 1203.04 | 25.29 | 5.20
Text2Video-Zero | 342.85 | 23.86 | 5.13
DiffSynth | 54.97 | 24.63 | 5.37
Table 3: Quantitative results in text-guided video stylization. | Content | Appearance | Pose
---|---|---|---
| Consistency | Similarity | Error
| (Pixel-MSE $\downarrow$) | (FID $\downarrow$) | (Pose-MSE $\downarrow$)
DreamPose | 100.44 | 75.43 | 0.63
DiffSynth | 24.13 | 63.02 | 0.48
Table 4: Quantitative results in fashion video synthesis. Approach | FateZero | Pix2Video | Text2Video-Zero | DiffSynth
---|---|---|---|---
Percentage | 24.43 | 7.65 | 12.07 | 55.85
Table 5: Percentage of videos selected as best in user study.

### Quantitive Results

We evaluate the quality of synthesized videos, with results presented in Table
3 and Table 4. In text-guided video stylization, we compare our approach with
several baseline approaches, including FateZero (Qi et al. 2023), Pix2Video
(Ceylan, Huang, and Mitra 2023), and Text2Video-Zero (Khachatryan et al.
2023). In fashion video synthesis, we compare it with DreamPose. Following
Pix2Video, we calculate Pixel-MSE to evaluate the consistency of the content.
Note that the official code of FateZero does not support the resolution except
$512\times 512$, so we cannot calculate its Pixel-MSE. From this metric
perspective, DiffSynth clearly outperforms other approaches. It shows that our
method can generate smoother videos than others. In addition, we computed
other metrics to evaluate the quality of each frame. In text-guided video
stylization, we employ CLIP score (Radford et al. 2021) to measure the
relevance between the synthesized video and the prompt, and use the aesthetic
score (Schuhmann et al. 2022) to assess the aesthetics of frames. FateZero can
only make minor changes to the frames, failing to generate videos that match
the textual description. Pix2Video is excessively focused on textual
information, resulting in incoherent frames. Text2Video-Zero performed
slightly better than the aforementioned methods but still lags behind our
approach. In fashion video synthesis, we employ the Fréchet Inception Distance
(FID) (Heusel et al. 2017) to measure the appearance similarity between
synthesized videos and source videos, and utilize Pose-MSE (Mean Squared Error
of the keypoint distances recognized by OpenPose (Cao et al. 2017)) to assess
the pose error between synthesized videos and target videos. Our method
exceeds DreamPose in all metrics. These experimental results demonstrate the
effectiveness of DiffSynth.

### User Study

Some studies (Blattmann et al. 2023; Yang et al. 2023) have pointed out that
conventional metrics are sometimes not feasible. We invite 20 participants and
conduct a user study. We ask each participant to select the best results based
on video consistancy, text-video similarity, and aesthetics. The average
results of the 100 videos in text-guided video stylization are presented in
Table 5. Unsurprisingly, most of the participants think that the videos
synthesized by DiffSynth look better than others.

Figure 3: An example of ablation study. The prompt of this example is
“cyberpunk, city, red neon light”.

### Ablation Study

To evaluate the effectiveness of our proposed deflickering algorithm (i.e.,
the patch blending algorithm), we conduct an ablation experiment. Figure 3
illustrates an example of our ablation study. For more examples, please refer
to our project page. When the deflickering algorithm is disabled, it is
evident that the video exhibits inconsistencies in various aspects, including
the brightness of the sky, the lights on the buildings, and the texts at the
central building. When the deflickering algorithm is enabled, these objects
are more aligned, resulting in more coherent videos.

## Other Applications

Besides the above two tasks, we also design several fancy pipelines in the
following scenarios. These video synthesis pipelines are all transferred from
image synthesis pipelines.

### Image-Guided Video Stylization

Figure 4: Examples of image-guided video stylization.

In the above stylization pipeline, the frames are synthesized according to
input prompts. Practically, the synthesized videos are greatly influenced by
the prompts, and carefully tuned prompts can improve the quality of videos.
Prompt engineering has become an interesting but challenging task (Witteveen
and Andrews 2022). To intuitively guide the model generating videos, we employ
a ControlNet
model101010https://huggingface.co/lllyasviel/control_v11e_sd15_shuffle to
further enable the image guiding mechanism. As shown in Figure 4, this
pipeline can transfer the style of a content video according to the style of
an image. Compared to the above pipeline, this pipeline does not require well-
written prompts. Therefore, it is easy to use for creators who are not
familiar with diffusion models.

### Video Restoring

Figure 5: An example of video restoring.

In earlier years, due to the immaturity of photography, some old videos only
have very low resolution and may have hard-to-repair noise because of repeated
compression during distribution. Restoration of these videos requires a lot of
costs. One method of restoration using deep learning is super-resolution, but
the defects such as noise in the original video are also retained in the
restored video. We decided to use our video processing method to restore these
videos. Combined with Tile
ControlNet111111https://huggingface.co/lllyasviel/control_v11f1e_sd15_tile,
the Stable Diffusion model can ignore the defects in old videos and redraw
noisy video frames. Specifically, we first increase the resolution of the
original video using the super-resolution method and then map each frame to
the latent space. After adding noise to the intermediate steps, we
subsequently use Tile ControlNet for denoising. Unlike in the other
application scenarios, we do not generate each frame from scratch because we
tend to preserve as much information as possible in the original video, with
only limited modifications to the details. Figure 5 shows an example. We can
see that our pipeline is capable of restoring the details of historic videos.

### 3D Rendering

Figure 6: Examples of 3D rendering.

In the media industry, creators have to draw maps for a 3D object to render a
video. Utilizing our approach, we can directly render a video automatically.
We first extract some necessary information from the unrendered video frames
to represent the structure of frames. There are no unique answers to the
definition of what structural information is, and we take reference from Gen-1
(Esser et al. 2023) to extract depth information. In addition, we also use the
SoftEdge information if necessary. We then design a ControlNet-based pipeline
to render the video. An example is presented in Figure 6. Our pipeline can
transform the 3D gray object to a realistic object, and it only requires
creators to provide unrendered videos and prompts. Therefore, our work has the
capacity of benefiting the media industry by video designs.

## Conclusion and Future Work

In this paper, we investigate the application of diffusion models to video
synthesis. We propose the latent in-iteration deflickering approach, making it
possible to apply existing video deflickering methods to the latent space,
thereby avoiding flicker accumulation during the iterative process. We
specifically design a deflickering algorithm based on patch matching for
diffusion models. With this algorithm, we can synthesize coherent and
realistic videos without any cherry-picking. We further show that our approach
is applicable to various application scenarios. Comprehensive experimental
results demonstrate the effectiveness of our method, outperforming previous
methods.

Yet, our work still has a few limitations. Although the $\mathcal{O}(n\log n)$
time complexity can be achieved, the efficiency of our approach can be further
improved for wider applications. For example, running on an NVIDIA A10 GPU,
our program requires approximately 1 minute to synthesize 1 frame in fashion
video synthesis. Furthermore, the blending operator can be improved to
generate details better. We leave these problems for future work.

## References

  * Barnes et al. (2009) Barnes, C.; Shechtman, E.; Finkelstein, A.; and Goldman, D. B. 2009.  PatchMatch: A randomized correspondence algorithm for structural image editing.  _ACM Trans. Graph._ , 28(3): 24.
  * Blattmann et al. (2023) Blattmann, A.; Rombach, R.; Ling, H.; Dockhorn, T.; Kim, S. W.; Fidler, S.; and Kreis, K. 2023.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 22563–22575.
  * Cao et al. (2017) Cao, Z.; Simon, T.; Wei, S.-E.; and Sheikh, Y. 2017.  Realtime multi-person 2d pose estimation using part affinity fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 7291–7299.
  * Ceylan, Huang, and Mitra (2023) Ceylan, D.; Huang, C.-H. P.; and Mitra, N. J. 2023.  Pix2video: Video editing using image diffusion.  _arXiv preprint arXiv:2303.12688_.
  * Dao et al. (2022) Dao, T.; Fu, D.; Ermon, S.; Rudra, A.; and Ré, C. 2022.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ , 35: 16344–16359.
  * Dhariwal and Nichol (2021) Dhariwal, P.; and Nichol, A. 2021.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34: 8780–8794.
  * Esser et al. (2023) Esser, P.; Chiu, J.; Atighehchian, P.; Granskog, J.; and Germanidis, A. 2023.  Structure and content-guided video synthesis with diffusion models.  _arXiv preprint arXiv:2302.03011_.
  * Feng et al. (2023) Feng, Z.; Zhang, Z.; Yu, X.; Fang, Y.; Li, L.; Chen, X.; Lu, Y.; Liu, J.; Yin, W.; Feng, S.; et al. 2023.  ERNIE-ViLG 2.0: Improving text-to-image diffusion model with knowledge-enhanced mixture-of-denoising-experts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10135–10145.
  * Gal et al. (2022) Gal, R.; Alaluf, Y.; Atzmon, Y.; Patashnik, O.; Bermano, A. H.; Chechik, G.; and Cohen-or, D. 2022.  An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.  In _The Eleventh International Conference on Learning Representations_.
  * Goodfellow et al. (2020) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2020.  Generative adversarial networks.  _Communications of the ACM_ , 63(11): 139–144.
  * Han et al. (2022) Han, S.; Huang, P.; Wang, H.; Yu, E.; Liu, D.; and Pan, X. 2022.  Mat: Motion-aware multi-object tracking.  _Neurocomputing_ , 476: 75–86.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017\.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30.
  * Ho and Salimans (2021) Ho, J.; and Salimans, T. 2021.  Classifier-Free Diffusion Guidance.  In _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_.
  * Hu et al. (2021) Hu, E. J.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2021.  LoRA: Low-Rank Adaptation of Large Language Models.  In _International Conference on Learning Representations_.
  * Huang et al. (2017) Huang, X.; Li, Y.; Poursaeed, O.; Hopcroft, J.; and Belongie, S. 2017.  Stacked generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 5077–5086.
  * Khachatryan et al. (2023) Khachatryan, L.; Movsisyan, A.; Tadevosyan, V.; Henschel, R.; Wang, Z.; Navasardyan, S.; and Shi, H. 2023.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint arXiv:2303.13439_.
  * Kingma and Welling (2013) Kingma, D. P.; and Welling, M. 2013.  Auto-Encoding Variational Bayes.  In _International Conference on Learning Representations_.
  * Kirillov et al. (2023) Kirillov, A.; Mintun, E.; Ravi, N.; Mao, H.; Rolland, C.; Gustafson, L.; Xiao, T.; Whitehead, S.; Berg, A. C.; Lo, W.-Y.; et al. 2023.  Segment anything.  _arXiv preprint arXiv:2304.02643_.
  * Lei et al. (2023) Lei, C.; Ren, X.; Zhang, Z.; and Chen, Q. 2023.  Blind Video Deflickering by Neural Filtering with a Flawed Atlas.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10439–10448.
  * Lei, Xing, and Chen (2020) Lei, C.; Xing, Y.; and Chen, Q. 2020.  Blind video temporal consistency via deep video prior.  _Advances in Neural Information Processing Systems_ , 33: 1083–1093.
  * Meng et al. (2021) Meng, C.; He, Y.; Song, Y.; Song, J.; Wu, J.; Zhu, J.-Y.; and Ermon, S. 2021.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In _International Conference on Learning Representations_.
  * Nichol and Dhariwal (2021) Nichol, A. Q.; and Dhariwal, P. 2021.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , 8162–8171. PMLR.
  * Press and Teukolsky (1990) Press, W. H.; and Teukolsky, S. A. 1990.  Savitzky-Golay smoothing filters.  _Computers in Physics_ , 4(6): 669–672.
  * Qi et al. (2023) Qi, C.; Cun, X.; Zhang, Y.; Lei, C.; Wang, X.; Shan, Y.; and Chen, Q. 2023.  Fatezero: Fusing attentions for zero-shot text-based video editing.  _arXiv preprint arXiv:2303.09535_.
  * Rabe and Staats (2021) Rabe, M. N.; and Staats, C. 2021.  Self-attention Does Not Need $O(n^{2})$ Memory.  _arXiv preprint arXiv:2112.05682_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763. PMLR.
  * Ramesh et al. (2022) Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 10684–10695.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , 234–241. Springer.
  * Ruiz et al. (2023) Ruiz, N.; Li, Y.; Jampani, V.; Pritch, Y.; Rubinstein, M.; and Aberman, K. 2023\.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 22500–22510.
  * Saharia et al. (2022) Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022\.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35: 36479–36494.
  * Schuhmann et al. (2022) Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35: 25278–25294.
  * Singer et al. (2022) Singer, U.; Polyak, A.; Hayes, T.; Yin, X.; An, J.; Zhang, S.; Hu, Q.; Yang, H.; Ashual, O.; Gafni, O.; et al. 2022.  Make-A-Video: Text-to-Video Generation without Text-Video Data.  In _The Eleventh International Conference on Learning Representations_.
  * Sohl-Dickstein et al. (2015) Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; and Ganguli, S. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , 2256–2265. PMLR.
  * Song and Ermon (2019) Song, Y.; and Ermon, S. 2019.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ , 32.
  * Teed and Deng (2020) Teed, Z.; and Deng, J. 2020.  Raft: Recurrent all-pairs field transforms for optical flow.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16_ , 402–419. Springer.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Witteveen and Andrews (2022) Witteveen, S.; and Andrews, M. 2022.  Investigating prompt engineering in diffusion models.  _arXiv preprint arXiv:2211.15462_.
  * Wu et al. (2022) Wu, J. Z.; Ge, Y.; Wang, X.; Lei, W.; Gu, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M. Z. 2022.  Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation.  _arXiv preprint arXiv:2212.11565_.
  * Yang et al. (2023) Yang, S.; Zhou, Y.; Liu, Z.; and Loy, C. C. 2023.  Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation.  _arXiv preprint arXiv:2306.07954_.
  * Yu et al. (2021) Yu, S.; Tack, J.; Mo, S.; Kim, H.; Kim, J.; Ha, J.-W.; and Shin, J. 2021.  Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks.  In _International Conference on Learning Representations_.
  * Zablotskaia et al. (2019) Zablotskaia, P.; Siarohin, A.; Zhao, B.; and Sigal, L. 2019.  Dwnet: Dense warp-based network for pose-guided human video generation.  _arXiv preprint arXiv:1910.09139_.
  * Zhang and Agrawala (2023) Zhang, L.; and Agrawala, M. 2023.  Adding conditional control to text-to-image diffusion models.  _arXiv preprint arXiv:2302.05543_.