# Dynamical origin of primordial black holes in spatially flat gauge with
inflationary $\alpha$-attractor potentials

Chitrak Sarkar<EMAIL_ADDRESS>Centre of Advanced Studies, Department
of Physics, The University of Burdwan, Burdwan 713 104, India  Arunoday Sarkar
<EMAIL_ADDRESS>Centre of Advanced Studies, Department of Physics,
The University of Burdwan, Burdwan 713 104, India  Basundhara Ghosh
<EMAIL_ADDRESS>Department of Physics, Indian Institute of Science,
Bangalore, Karnataka, 560012, India  Buddhadeb Ghosh
<EMAIL_ADDRESS>Centre of Advanced Studies, Department of Physics,
The University of Burdwan, Burdwan 713 104, India

###### Abstract

We consider primordial black hole (PBH) formations in the framework of linear
cosmological perturbations in the spatially flat gauge with the
$\alpha$-attractor potentials. Our formalism comprises the first-order
perturbations in the quantized single inflaton field, in the classical
background metric and their coupled non-linear differential equations, in the
space of the mode momentum $k$. We first obtain the slow-roll horizon-crossing
density contrasts at various momenta in the high momentum limit and their
values during the horizon re-entry in the radiation dominated era, using a
transfer function. The density contrasts for the black hole formations are
located in the regions of $k$-space where the perturbative framework breaks
down and the slow-roll feature is violated. These locations are observed to be
associated with large enhancement in the scalar power spectra and large
negative Bardeen potentials. In our formalism, we find that the same set of
$k$-space evolution equations can conform to the Planck data in the ($n_{s}$
\- $r$) plane in the $k$ $\rightarrow$ $0$ limit and create the scenario of
PBH formations in the $k$ $\rightarrow$ $\infty$ limit, when we use the
traditional (unmodified) $\alpha$-attractor $T$-model and $E$-model
potentials. We have obtained the masses ranging from $1.35\times
10^{-13}M_{\odot}$ to $2.93\times 10^{-16}M_{\odot}$ in the $k$-range
$0.43\times 10^{13}-9.25\times 10^{13}$ Mpc-1 , which are consistent with the
LISA and the BBO forecasts of PBH detections through gravitational waves.

##  I Introduction

The concept of primordial black holes (PBHs), associated with comparatively
large density fluctuations at the early stages of the evolution of the
universe, was introduced by Zel’dovich and Novikov Zel’dovich (1967) and later
theorized in details by S. W. Hawking and B. J. Carr in Hawking (1971); Carr
and Hawking (1974); Carr (1975). Subsequently, the hydrodynamics of PBH
formation Nadezhin  _et al._ (1978), accretion of matter around PBHs Novikov
_et al._ (1979) and PBH formations in the contexts of Grand Unified Theories
Khlopov and Polnarev (1980) and in a double inflation in supergravity Kawasaki
_et al._ (1998) were studied. It is conjectured that these PBHs have either
evaporated by Hawking radiation or have evolved into supermassive black holes
Kawasaki  _et al._ (2013) or remain as dark matter in the present universe
Kawasaki  _et al._ (2013); Carr and Kuhnel (2020); Villanueva-Domingo  _et
al._ (2021); Conzinu  _et al._ (2020); Moradinezhad Dizgah  _et al._ (2019);
Clesse and García-Bellido (2017); Kovetz (2017); Inomata  _et al._ (2017);
Bringmann  _et al._ (2019); Raidal  _et al._ (2019); Green and Kavanagh
(2021); Poulin  _et al._ (2017); Wong  _et al._ (2021); Escrivà and Romano
(2021); Calabrese  _et al._ (2021). The relations between PBHs and primordial
gravitational waves have been studied in Clesse and García-Bellido (2017);
Choudhury and Mazumdar (2014); Kovetz  _et al._ (2017); Nakama  _et al._
(2017); Sasaki  _et al._ (2016); Kovetz (2017); De Luca  _et al._ (2020);
Domènech  _et al._ (2021); Özsoy and Lalak (2021). Formation of PBH during a
first order phase transition in the inflationary period has also been studied
Khlopov (2010); De Luca  _et al._ (2021). Studies of the influence of PBHs on
the CMB $\mu$ and $y$ distortions Deng (2020); Tashiro and Sugiyama (2008) and
$\mu T$ Özsoy and Tasinato (2021) cross-correlations have been carried out.
Detection of signals from stochastic gravitational wave background, connected
with PBH formations, in present and future experiments, has been discussed in
Refs. Garcia-Bellido  _et al._ (2016); Braglia  _et al._ (2021).

In some of the studies of the inflationary scenarios, PBHs have been
identified as massive compact halo object with mass $\sim 0.5M_{\odot}$ in the
work by J. Yokoyama Yokoyama (1999), who has also examined the formation of
PBHs in the framework of a chaotic new inflation Yokoyama (1998). Josan and
Green Josan and Green (2010), studied constraints on the models of inflation
through the formation of PBHs, using a modified flow analysis. Harada, Yoo and
Kohri Harada  _et al._ (2013) examined the threshold of PBH formation, both
analytically and numerically. R. Arya Arya (2020) has considered the PBH
formation as a result of enhancement of power spectrum during the thermal
fluctuations in a warm inflation. Formation of PBHs in density perturbations
was studied in two-field Hybrid inflationary models Garcia-Bellido  _et al._
(1996); Chongchitnan and Efstathiou (2007), Starobinsky model including
dilaton Gundhi  _et al._ (2021), multi-field inflation models Palma  _et al._
(2020), isocurvature fluctuation and chaotic inflation models Yokoyama (1999),
inflection-point models Choudhury and Mazumdar (2014); Ballesteros and Taoso
(2018), quantum diffusion model Biagetti  _et al._ (2018), model with smoothed
density contrast in the super-horizon limit Young (2019) and with the collapse
of large amplitude metric perturbation Musco (2019) and large-density
perturbation Young and Musso (2020) upon horizon re-entry. PBH abundance in
the framework of non-perturbative stochastic inflation has been studied by F.
Kühnel and K. Freese Kuhnel and Freese (2019). Relation between the
constraints of primordial black hole abundance and those of the primordial
curvature power spectrum has also been studied Kalaja  _et al._ (2019);
Dalianis (2019). Recently, PBHs solutions have been obtained in the framework
of non-linear cosmological perturbations and non-linear effects arising at
horizon crossing Musco  _et al._ (2021).

PBH production has recently been studied, in the framework of
$\alpha$-attractor polynomial super-potentials and modulated chaotic inflaton
potentials models Dalianis  _et al._ (2019). Mahbub Mahbub (2020) utilized the
superconformal inflationary $\alpha$-attractor potentials with a high level of
fine tuning to produce an ultra-slow-roll region, where the enhancement for
curvature power spectra giving rise to massive PBHs was found at $k\sim
10^{14}$ Mpc-1. In a subsequent work Mahbub (2021), this author re-examined
the earlier work using the optimised peak theory. The ultra-slow-roll process
along with a non-Gaussian Cauchy probability distribution has been applied in
Ref. Biagetti  _et al._ (2021) to obtain large PBHs masses. The constant-rate
ultra-slow-roll-like inflation Ng and Wu (2021) has also been applied to
obtain the enhancement in the power spectra, triggered by entropy production,
resulting in PBH formation. Ref. Teimoori  _et al._ (2021) has simulated the
onset of PBHs formation by adding a term to the non-canonical
$\alpha$-attractor potential, which enhances the curvature perturbations at
some critical values of the field. The enhancement of the power spectrum by a
limited period of strongly non-geodesic motion of the inflationary trajectory
and consequent PBH production has been studied by J. Fumagalli et al.
Fumagalli  _et al._ (2020).

In almost all of the above references, PBH formation has been studied in terms
of the curvature perturbation $\mathcal{R}$ and the curvature power spectrum
$P_{\mathcal{R}}(k)$, which are usually obtained from the Mukhanov-Sasaki
equation Mukhanov (1988); Mukhanov  _et al._ (1992); Mukhanov (2005); Sasaki
(1986) in the co-moving gauge, characterised by a zero inflaton perturbation
($\delta\phi=0$). This way of analysis, obviously ignores the role of the
inflaton field ($\phi$) in the inflationary scenario of PBH formation. In the
present work, we shall take an alternative route. We shall use the spatially
flat gauge, thereby including the role of $\phi$ in the mechanism of PBH
formation. In this respect, we shall follow the formalism developed in our
previous work Sarkar  _et al._ (2021), comprising a set of linear perturbative
evolution equations which could explain the Planck-2018 data Akrami  _et al._
(2020) in low $k$ limit. We shall show, here, that the same equations can
yield PBH-like solutions in the high $k$ regime with the conventional chaotic
$T$ and $E$ model potentials without any modifications. Thus, we find that a
salient feature of the present approach is that we can explore different
regions of $k$-space evolution in the inflationary period under suitable
initial conditions for the same differential equations. We believe that this
work will open up an avenue for the dynamical origin of PBH formation in the
deep sub-horizon $k$-space. In fact, we shall highlight here the important
role played by the Bardeen potential ($\Phi_{B}$) in building up the density
contrasts and the associated PBH formations, which, to our knowledge, has not
been done so far in the literature.

The paper is organised as follows. In Sec. II.1 we briefly describe the basic
formalism of the linear perturbation theory, leading to the setting up of the
three coupled non-linear differential equations which play the central role of
our study of the PBH formations. In Sec. II.2 we write about the
$\alpha$-attractor $T$ and $E$ model potentials which have been used in the
present study. The expression of the transfer function is written and its plot
is shown in Sec. II.3. Results and discussion are presented in Sec. III.
Finally, in Sec. IV we make some concluding remarks.

##  II Formalism

###  II.1 Linear perturbations in the metric and the inflaton field

The Einstein-Hilbert action with minimal coupling between quantised inflaton
field,

$\phi(t,\vec{X})=\int\frac{d^{3}k}{(2\pi)^{3}}[\phi(k,t)\hat{a}(\vec{k})e^{i\vec{k}.\vec{x}}+\phi^{*}(k,t)\hat{a}^{\dagger}(\vec{k})e^{-i\vec{k}.\vec{x}}],$
(1)

and the background linearly-perturbed metric, in spatially flat gauge, with no
anisotropic stress,

$ds^{2}=-(1+2\Phi)dt^{2}+2a(t)\partial_{i}Bdx^{i}dt+a^{2}(t)\delta_{ij}dx^{i}dx^{j},$
(2)

is

$S=\int
d^{4}x\sqrt{-g}\left(\frac{1}{2}R-\frac{1}{2}g^{\alpha\beta}\partial_{\alpha}\phi\partial_{\beta}\phi-V(\phi)\right).$
(3)

The linear perturbation in the inflaton field is written as

$\phi(t,\vec{X})=\phi^{(0)}(t)+\delta\phi(t,\vec{X}).$ (4)

The perturbation can be translated, through the energy-momentum tensor to that
in the density as,

$\rho(t,\vec{X})=\rho^{(0)}(t)+\delta\rho(t,\vec{X}),$ (5)

where,

$\rho^{(0)}(t)=\frac{{\dot{\phi}}^{(0)^{2}}}{2}+V(\phi^{(0)})$ (6)

and

$\delta\rho(t,\vec{X})=\frac{dV(\phi^{(0)})}{d\phi^{(0)}}\delta\phi+\dot{\phi}^{(0)}\delta\dot{\phi}-\Phi{\dot{\phi}}^{(0)^{2}}.$
(7)

(Note: The last term in Eq.(7) was neglected in Ref. Sarkar  _et al._ (2021)
as $\Phi{\dot{\phi}}^{(0)^{2}}$ is small in slow-roll approximation. In the
present paper, we retain this term as we expect the metric perturbation $\Phi$
to play a major role in the PBH formation.) Using the solutions in Baumann
(2011) of the unperturbed Einstein’s equations,

$H^{2}=\frac{\rho^{(0)}}{3},$ (8)
$\dot{H}+H^{2}=-\frac{1}{6}(\rho^{(0)}+3p^{(0)})$ (9)

and the perturbed Einstein’s equations,

$3H^{2}\Phi+\frac{k^{2}}{a^{2}}\left(-aHB\right)=-\frac{\delta\rho}{2},$ (10)
$H\Phi=-\frac{1}{2}\delta q,$ (11)
$H\dot{\Phi}+(3H^{2}+2\dot{H})\Phi=\frac{\delta p}{2},$ (12)
$(\partial_{t}+3H)\frac{B}{a}+\frac{\Phi}{a^{2}}=0$ (13)

and the Bardeen potentials Bardeen (1980)

$\Phi_{B}=\Phi-\frac{d}{dt}\left[a^{2}\left(-\frac{B}{a}\right)\right],$ (14)
$\Psi_{B}=a^{2}H\left(-\frac{B}{a}\right)$ (15)

we obtain a relation between $\Phi$ and $\Phi_{B}$ as,

$\Phi=\Phi_{B}+\partial_{t}\left(\frac{\Phi_{B}}{H}\right).$ (16)

In Eqs. (11) and (12) $\delta q$ and $\delta p$ are the magnitudes of the
momentum perturbation and pressure perturbation, respectively. Eqs. (7) and
(16) show that the density perturbation $\delta\rho$ contains $\phi^{(0)}$,
$\delta\phi$ as well as the Bardeen potential $\Phi_{B}$.

Using the slow roll dynamical horizon crossing condition, $k=aH$, we go from
the space of the cosmic time $t$ to that of the mode momentum $k$ and set up
three nonlinear coupled differential equations in the $k$-space and solve for
the quantities, $\phi^{(0)}(k)$, $\delta\phi(k)$ and $\Phi_{B}(k)$. The
equations, similar to those in Sarkar  _et al._ (2021), are,

$\delta\phi(k^{2}\phi^{\prime\prime}+k^{2}G_{1}\phi^{\prime
2}+4k\phi^{\prime}+6G_{1})+\delta\phi^{\prime}(-2k^{3}G_{1}\phi^{\prime
2})=0,$ (17)
$\delta\phi(1+12G_{1}^{2}+6G_{2})+\delta\phi^{\prime}(4k+k^{2}G_{1}\phi^{\prime})+k^{2}\delta\phi^{\prime\prime}\\\
+\Phi_{B}(-k\phi^{\prime}+12G_{1}+k^{2}G_{1}\phi^{\prime
2}+k^{3}G_{1}\phi^{\prime}\phi^{\prime\prime}-12kG_{1}^{2}\phi^{\prime}\\\
+k^{3}G_{2}\phi^{\prime
3})+\Phi^{\prime}_{B}(-2k^{2}\phi^{\prime}+12kG_{1}+k^{3}G_{1}\phi^{\prime
2})+\Phi^{\prime\prime}_{B}(-k^{3}\phi^{\prime})\\\ =0$ (18)

and

$\displaystyle\Phi^{\prime\prime}_{B}\left(-\frac{k^{4}}{3}\phi^{\prime
2}\right)+\Phi^{\prime}_{B}\left(-\frac{k^{3}}{3}\phi^{\prime
2}+\frac{k^{4}}{3}G_{1}\phi^{\prime 3}-k^{3}\phi^{\prime
2}-\frac{2k^{4}}{3}\phi^{\prime}\phi^{\prime\prime}+\frac{2k}{3}+2k^{3}\phi^{\prime
2}\right)$ $\displaystyle+\Phi_{B}\left(-\frac{2k^{2}}{3}\phi^{\prime
2}-\frac{2k^{3}}{3}\phi^{\prime}\phi^{\prime\prime}+k^{3}G_{1}\phi^{\prime
3}+\frac{k^{4}}{3}G_{2}\phi^{\prime 4}+k^{4}G_{1}\phi^{\prime
2}\phi^{\prime\prime}-2+2k^{2}\phi^{\prime 2}-2k^{3}G_{1}\phi^{\prime
3}\right)$
$\displaystyle+\delta\phi^{\prime\prime}\left(\frac{k^{3}}{3}\phi^{\prime}\right)+\delta\phi^{\prime}\left(2kG_{1}+\frac{2k^{2}}{3}\phi^{\prime}+\frac{k^{3}}{3}\phi^{\prime\prime}\right)+\delta\phi\left(2kG_{2}\phi^{\prime}+2k^{2}\phi^{\prime\prime}+2k^{2}G_{1}\phi^{\prime
2}+2k\phi^{\prime}\right)=0,$ (19)

where,
$G_{n}=\frac{\partial^{n}}{\partial\phi^{(0)^{n}}}\ln\sqrt{V(\phi^{(0)})}$,
$n=1,2$.

In the slow-roll approximation, the density contrast in $k$-space, which is
useful for the study of the PBH formation, is given by,

$\displaystyle\frac{\delta\rho(k)}{\rho^{(0)}(k)}=\frac{\delta
V(k)+V^{(0)}(k)\left(\frac{k^{2}}{3}\phi^{\prime}\delta\phi^{\prime}+\Phi_{B}\left(-\frac{k^{2}}{3}\phi^{\prime
2}+\frac{k^{3}}{3}G_{1}\phi^{\prime
3}\right)+\Phi^{\prime}_{B}\left(-\frac{k^{3}}{3}\phi^{\prime
2}\right)\right)}{V^{(0)}(k)}$
$\displaystyle=2G_{1}\delta\phi+\frac{k^{2}}{3}\phi^{\prime}\delta\phi^{\prime}+\Phi_{B}\left(-\frac{k^{2}}{3}\phi^{\prime
2}+\frac{k^{3}}{3}G_{1}\phi^{\prime
3}\right)+\Phi^{\prime}_{B}\left(-\frac{k^{3}}{3}\phi^{\prime 2}\right)$ (20)

where, $\delta V(k)\equiv 2G_{1}V^{(0)}(k)\delta\phi(k)$
and $V^{(0)}(k)\equiv V(\phi^{(0)}(k))$. In first line of Eq. (20), we have
used the Fourier-space version of the slow-roll approximation and, therefore,
following Eq. (6), we have written, $\rho^{(0)}(k)\approx V^{(0)}(k)$. We will
show that, both $\delta\phi(k)$ and $\Phi_{B}(k)$ in Eq. (20) will play a
significant role in making
$\frac{\delta\rho(k)}{\rho^{(0)}(k)}>\delta_{c}(\approx 0.41)$, i.e. in the
formation of PBHs in the early universe, $\delta_{c}$ being the density
contrast (see Sec. III for details).

###  II.2 The $\alpha$-attractor $E$-model and $T$-model potentials

We use here, the $\alpha$-attractor potentials Carrasco  _et al._ (2015);
Kallosh  _et al._ (2013):
(I) the $T$-model potential

$V(\phi)=V_{0}\tanh^{n}\frac{\phi}{\sqrt{6\alpha}},$ (21)

(II) the $E$-model potential

$V(\phi)=V_{0}(1-e^{-\sqrt{\frac{2}{3\alpha}}\phi})^{n}$ (22)

where $\alpha$ is the inverse curvature of $SU(1,1)/U(1)$ Kähler manifold
Kallosh  _et al._ (2013).

###  II.3 Transfer Function

We have related the quantities at horizon crossing during inflation to those
at horizon re-entry in the radiation-dominated era using the transfer function
given in Ref. Musco  _et al._ (2021):

$T(k,\eta)=3\frac{\sin{(k\eta}/\sqrt{3})-(k\eta/\sqrt{3})\cos{(k\eta/\sqrt{3}})}{(k\eta/\sqrt{3})^{3}},$
(23)

where $\eta$ is the conformal time.

Figure 1: Transfer function for radiation dominated period at a very small
conformal time $\eta=10^{-12}$. It has a very small oscillating behaviour
around $k\sim 10^{13}$ Mpc-1, where $k\eta>1$, although it dies down in the
high $k$ limit, $k\rightarrow\infty$. The momentum scale reflects the range of
sub-horizon momenta which will be studied in this paper.

##  III Results and Discussion

At the outset, let us give an overall perspective of the evolution of the
modes during the inflationary period. As shown and stated in Ref. Sarkar  _et
al._ (2021), the higher $k$ modes undergo less number of e-folds and thus,
while the Hubble sphere shrinks, they exit the horizon at later times. As a
consequence, the very high $k$ modes remain in the deep subhorizon region
during the major part of the inflationary period. When the Hubble sphere
starts expanding after the end of inflation, the high $k$ modes re-enter the
horizon first in small positive conformal times. The smaller $k$ modes re-
enter the horizon at later conformal times.

Our interest, here, is mainly in the high $k$ regions where, it will be shown
that, the density contrast shoots up, at a number of momentum values,
signalling the breakdown of the perturbative framework and creating a
condition favourable for the formation of PBHs, when the modes re-enter the
horizon. Interestingly, this condition is achieved in our study quite
naturally by solving the $k$-space evolution equations and not by artificially
preparing the inflaton potential to meet this goal.

Figure 2: Unperturbed part $V^{(0)}(k)$ of the chaotic $\alpha$-attractor
$T$-model potential at high $k$ limit. Figure 3: Perturbation $\delta V(k)$ of
chaotic $\alpha$ attractor $T$-model potential in high $k$ limit. The
perturbation in the potential, $\delta V(k)\gg V^{(0)}(k)$ (see Figure 2). It
blows up as $k\rightarrow\infty$, signifying the breakdown of the slow roll in
the linear perturbation formalism and creating a situation, favourable for the
PBH formation. Figure 4: Unperturbed inflaton field $\phi^{(0)}(k)$ for
chaotic $\alpha$ attractor $T$-model potential at high $k$ limit. Figure 5:
Perturbation $\delta\phi(k)$ for chaotic $\alpha$ attractor $T$-model
potential in high $k$ limit. The perturbation in the inflaton field,
$\delta\phi(k)\gg\phi^{(0)}(k)$ (see Figure 4). It blows up as
$k\rightarrow\infty$, signifying the breakdown of the slow-roll in the linear
perturbation formalism and creating a situation, favourable for the PBH
formation. Figure 6: Density contrast
$\left(\frac{\delta\rho}{\rho}\right)_{\mathrm{rad}}\left[=\left(\frac{\delta\rho}{\rho}\right)_{\mathrm{inf}}\times
T(k,\eta=10^{-12})\right]$ vs. $k$, in the radiation-dominated era, in a high
momentum range ($0.43\times 10^{13}$ Mpc-1 to $9.25\times 10^{13}$ Mpc-1),
where signatures of PBHs have been predicted Green and Kavanagh (2021). Here,
$\left(\frac{\delta\rho}{\rho}\right)_{\mathrm{inf}}$ is the density contrast
during inflation. Enhancements are observed at $17$ values of $k$. These
enhancements suggest presence of PBHs of different masses (see Table 1).
Figure 7: The Bardeen potential
$(\Phi_{B})_{\mathrm{rad}}[=(\Phi_{B})_{\mathrm{inf}}\times
T(k,\eta=10^{-12})]$ vs. $k$ in the radiation-dominated era in the same
momentum range as in Figure 6. Here, $(\Phi_{B})_{\mathrm{inf}}$ is the
Bardeen potential during inflation. Large negative values of the potentials
are observed at the $k$-values where the enhancements of the density contrast
occur (see Figure 6). This demonstrates the crucial role played by the Bardeen
potential in the formation of the PBHs. Figure 8: The scalar power spectrum
$\Delta_{s}(k)$ vs. $k$ in the same high momentum range as in Figures 2 \- 5.
Extremely high values of the power spectrum in this range indicate very high
correlations among the mode momenta belonging to the quantized inflaton field,
which is possible in the case of black hole formations.

Figure 9: Comparison of the uses of the $\alpha$-attractor chaotic $T$-model
potential (upper figure) and a power law potential, $\phi^{2}$ (lower figure)
in calculating the density contrast
$\left(\frac{\delta\rho}{\rho}\right)_{\mathrm{inf}}$, during inflation, in
the high momentum regime. While the density contrast has very large positive
values in the former case, it shows large negative values in the latter case.
The unphysical nature of the density contrast in high momentum regime, in the
case of the power law potential, clearly indicates its unsuitability in the
formation of PBHs. Same inference can be drawn during radiation dominated era
also.

In Figures 2 through 9, $n=2$, $\alpha=1$ and $V_{0}=1$ are three values of
parameters of the $\alpha$-attractor $T$-model potential (Eq. (21)). These
values belong to the range of parameters which has been shown to be
efficacious in fitting the Planck data in the ($n_{s}$ \- $r$) plane Sarkar
_et al._ (2021); Akrami  _et al._ (2020).

In Figure 2, we show the $k$-space behaviour of the unperturbed chaotic
$\alpha$-attractor $T$-model potential in the high $k$ limit, during
inflation. In Figure 3, we demonstrate the corresponding behaviour of the
perturbation in the potential, where it is shown that $\delta V(k)$ becomes
very large in the high $k$ limit, signaling breakdown of perturbation in this
limit. It may be noted that this does not happen in the low $k$ limit, where
$\delta V(k)$ remains very small Sarkar  _et al._ (2021). In Figure 4, we have
plotted the unperturbed inflaton field for the chaotic $\alpha$-attractor
$T$-model potential in the high $k$ limit and in Figure 5, the corresponding
perturbation, $\delta\phi(k)$, in the same limit. Here also we observe the
breakdown of perturbation at high $k$ values.

In Figures 6 and and 7 we plotted the density contrast and the Bardeen
potential, respectively, in the radiation-dominated era. These quantities have
been obtained by multiplying their corresponding values during inflation with
the transfer function Eq. (23). The solutions of the Eqs. (17) - (19) during
inflation have been obtained by considering $\phi^{(0)}(k=10^{8})=6.21$,
$\delta\phi(k=10^{8})=\Phi_{B}(k=10^{8})=10^{-14}$,
$\phi^{\prime(0)}(k=10^{8})=\delta\phi^{\prime}(k=10^{8})=\Phi^{\prime}_{B}(k=10^{8})=0$,
as initial conditions.

Looking at Figure 6, we observe that the values of the density contrasts at
all the peaks are above the range of threshold values viz.,
$0.41\lesssim\delta_{c}\leqslant\frac{2}{3}$, given in the current literature
Musco (2019). Thus, the peaks satisfy the primary criterion for the PBH
formations. It may be noted here that, these peaks are the results (see the
upper figure in Figure 9) of self-consistent calculation of $k$-space
evolution equations (Eqs. (17) -(19)), smoothened by the transfer function,
shown in Figure 1.

Figure 7 highlights the fact that, the large peaks in density contrast
correspond to large negative peaks in Bardeen potential at the same value of
$k\equiv k_{\mathrm{peak}}$. This result reflects the strong interplay between
the quantum-inflaton-fluctuation-induced density perturbation and the metric
perturbation i.e., the Bardeen potential, in the PBH formation.

Figure 8 shows that, the scalar power spectrum, which is a measure of two-
point correlations among the fluctuations, increases very rapidly at the $k$
values of the PBH formations. Such $k$-space behaviour of the scalar power
spectrum signifies the breakdown of the perturbative framework, as well as a
very high quantum correlation, which may be favourable for the PBH formations.

Figure 9 illustrates the fact that the requirement of large positive
inflationary density contrasts at high $k$ values for the PBH formations, is
satisfied only by the $\alpha$-attractor $T$-model potential and not by the
power law type $\phi^{2}$ potential, for example. We have also found in Sarkar
_et al._ (2021) that, such type of potential does constitute an
experimentally-favourable model for inflation at low $k$ limit within some
specified range of parameters. Therefore, the $\alpha$-attractor potential in
its pristine form has the capability of explaining the inflationary paradigm
at the low $k$ limit and the PBH formation at the high $k$ limit,
simultaneously.

Although the calculations from Figure 2 through Figure 9 are with the chaotic
$\alpha$-attractor $T$ model potential, the results will be similar for the
corresponding $E$ model potential. Therefore, we are not displaying those
results here.

To further endorse the identification of the peaks with the PBH formation
events, in Table 1 we have shown our calculations regarding some properties of
these PBHs. The momentum-dependent PBH masses in the unit of the solar
mass($M_{\odot}$) have been presented in column $4$. The calculation of the
masses is based on the formula given in Ref. Teimoori  _et al._ (2021). Table
1 suggests that, massive PBHs are formed at those values of $k$, where the
density contrast shows a large positive peak and the Bardeen potential shows a
large negative value, which reflects the efficacy of the Bardeen potential in
the PBH formations. Thus it is right to say that the Bardeen potential acts as
a source of intense gravitational field, where the tiny quantum nuggets are
amplified into massive PBHs.

The mass-dependent evaporation times and the Hawking temperatures Hawking
(1975) are presented in columns $5$ and $6$ respectively. The calculations of
these quantities are based on the formulae given in Ref. Dalianis (2019). The
evaporation time scale of the PBHs in our calculations is found in the range
$10^{25}-10^{33}$s, which is very large in comparison to the age of the
universe ($\sim 10^{17}$s). Also, the Hawking temperatures ($\sim
10^{-5}-10^{-8}$ GeV) are very small, showing that the possibility of
extinction of such PBHs by Hawking radiation is negligibly small. Thus, these
PBHs may exist in the present universe, thereby making themselves potential
candidates for the dark matter.

Table 1: PBH properties corresponding to the peaks in Figure 6. These
properties are consistent with the LISA and BBO observations Green and
Kavanagh (2021).

Peak No. $k_{\mathrm{peak}}$ (in Mpc-1)
$(\frac{\delta\rho}{\rho})_{\mathrm{rad}}|_{k=k_{\mathrm{peak}}}$
$M_{\mathrm{PBH}}$ (in $M_{\odot}$) $t_{\mathrm{eva}}$ (in sec.) $T_{H}$ (in
GeV) 1 0.43 $\times$ 1013 8.77 1.35 $\times$ 10-13 7.72 $\times$ 1033 3.72
$\times$ 10-8 2 1.05 $\times$ 1013 3.49 2.27 $\times$ 10-14 3.67 $\times$ 1031
2.21 $\times$ 10-7 3 1.60 $\times$ 1013 2.29 9.79 $\times$ 10-15 2.95 $\times$
1030 5.13 $\times$ 10-7 4 2.16 $\times$ 1013 1.70 5.37 $\times$ 10-15 4.86
$\times$ 1029 9.36 $\times$ 10-7 5 2.71 $\times$ 1013 1.37 3.41 $\times$ 10-15
1.24 $\times$ 1029 1.47 $\times$ 10-6 6 3.25 $\times$ 1013 1.14 2.37 $\times$
10-15 4.18 $\times$ 1028 2.12 $\times$ 10-6 7 3.79 $\times$ 1013 0.97 1.74
$\times$ 10-15 1.65 $\times$ 1028 2.89 $\times$ 10-6 8 4.30 $\times$ 1013 0.84
1.35 $\times$ 10-15 7.72 $\times$ 1027 3.72 $\times$ 10-6 9 4.80 $\times$ 1013
0.79 1.09 $\times$ 10-15 4.06 $\times$ 1027 4.61 $\times$ 10-6 10 5.44
$\times$ 1013 0.71 8.47 $\times$ 10-16 1.91 $\times$ 1027 5.93 $\times$ 10-6
11 5.97 $\times$ 1013 0.64 7.03 $\times$ 10-16 1.09 $\times$ 1027 7.15
$\times$ 10-6 12 6.50 $\times$ 1013 0.58 5.93 $\times$ 10-16 6.54 $\times$
1026 8.48 $\times$ 10-6 13 7.07 $\times$ 1013 0.52 5.01 $\times$ 10-16 3.95
$\times$ 1026 1.00 $\times$ 10-5 14 7.60 $\times$ 1013 0.49 4.34 $\times$
10-16 2.57 $\times$ 1026 1.15 $\times$ 10-5 15 8.16 $\times$ 1013 0.44 3.76
$\times$ 10-16 1.66 $\times$ 1026 1.33 $\times$ 10-5 16 8.70 $\times$ 1013
0.42 3.31 $\times$ 10-16 1.14 $\times$ 1026 1.51 $\times$ 10-5 17 9.25
$\times$ 1013 0.41 2.93 $\times$ 10-16 7.89 $\times$ 1025 1.72 $\times$ 10-5

##  IV Conclusions

In conclusion, we have examined in this paper the possibility of the PBH
formation in the inflationary perturbation theory in the spatially flat gauge
with the $\alpha$-attractor inflaton potentials. The implications of the
present study may be summarized in the following points.
(i) Despite the various means of enhancing the power spectrum, that has come
up in the literature, such as the modification and the fine-tuning of the
potential Choudhury and Mazumdar (2014); Ballesteros and Taoso (2018); Mahbub
(2020), introduction of the two-field Garcia-Bellido  _et al._ (1996);
Chongchitnan and Efstathiou (2007) and multi-field Palma  _et al._ (2020)
scenarios (although the Planck data Akrami  _et al._ (2020) support the
single-field inflation) etc., we have shown, in the present work, that we can
obtain PBH solutions in the natural way of $k$-space evolution, if we
incorporate the interaction of the classically-perturbed gravitational field
with the quantum fluctuations of the inflaton field and solve the resulting
non-linear coupled differential equation. The PBH formations occur whenever
the large fluctuations in the inflaton field meet the large negative
potentials in the background gravitational field in some regions of the
$k$-space.
(ii) One of the striking results, here, is that the Bardeen potential
$\Phi_{B}(k)$ in the spatially flat gauge manifests as a driving force for
accumulating mass around the inflaton perturbation, which leads to dynamical
PBH formation in radiation dominated era when large sub-horizon modes re-enter
the Hubble horizon in small conformal time.
(iii) The range of $k$-space viz., $0.43\times 10^{13}-9.25\times 10^{13}$
Mpc-1 for the PBH formation where density contrast exceeds the critical
density (see Table 1 and Figure 6) and Bardeen potential becomes largely
negative (see Figure 7), is naturally found through self-consistent solutions
of the Eqs. (17) - (19), and it is checked that no other ranges of $k$-values
meet this requirement. Therefore, we claim that, this is an important outcome
of the mutual interaction among $\phi^{(0)}(k)$, $\delta\phi(k)$ and
$\Phi_{B}(k)$ in the background of $\alpha$-attractor potential in the
framework of linear cosmological perturbation theory with the spatially flat
gauge.
(iv) In our formalism, we consider that, the PBHs are the result of large
over-density in the inflaton field under strong gravitational environment and
there are no other ingredients which can influence PBH formation. For example,
we are neither taking into account the PBH formation from dark matter collapse
or considering PBHs as dark matter - we are simply suspecting from the large
evaporation times and the small Hawking temperatures that our PBHs could be
dark matter. Therefore some other prospects regarding PBHs like abundance and
mass fraction should be considered as outside the scope of the present paper.
Our sole aim, here, is to show, how a single formalism can explain both
inflation and PBH formation from single, unmodified $\alpha$-attractor-
inflaton perturbation only. Thus, some related future works under this scheme
could be inclusion of dark matter and dark energy (or quintessence) in our
formalism, which may tighten the $k$-range or may constrain more stringently
the PBH properties, useful for upcoming experiments.

## Acknowledgements

The present work has been carried out using some of the facilities provided by
the University Grants Commission to the Center of Advanced Studies under the
CAS-II program. CS and AS acknowledge the government of West Bengal for
granting them the Swami Vivekananda fellowship. Bansundhara Ghosh acknowledges
the Department of Science and Technology for providing her the DST-Inspire
Faculty Fellowship.

## References

  * Zel’dovich (1967) Y. B. . N. Zel’dovich, I. D., Soviet Astron. AJ (Engl. Transl. ), 10, 602 (1967).
  * Hawking (1971) S. Hawking, Mon. Not. Roy. Astron. Soc. 152, 75 (1971).
  * Carr and Hawking (1974) B. J. Carr and S. W. Hawking, Mon. Not. Roy. Astron. Soc. 168, 399 (1974).
  * Carr (1975) B. J. Carr, Astrophys. J. 201, 1 (1975).
  * Nadezhin  _et al._ (1978) D. K. Nadezhin, I. D. Novikov, and A. G. Polnarev, Soviet Ast. 22, 129 (1978).
  * Novikov  _et al._ (1979) I. D. Novikov, A. Polnarev, A. A. Starobinskii, and I. B. Zeldovich, A & A 80, 104 (1979).
  * Khlopov and Polnarev (1980) M. Y. Khlopov and A. G. Polnarev, Phys. Lett. B 97, 383 (1980).
  * Kawasaki  _et al._ (1998) M. Kawasaki, N. Sugiyama, and T. Yanagida, Phys. Rev. D 57, 6050 (1998), arXiv:hep-ph/9710259 .
  * Kawasaki  _et al._ (2013) M. Kawasaki, N. Kitajima, and T. T. Yanagida, Phys. Rev. D 87, 063519 (2013), arXiv:1207.2550 [hep-ph] .
  * Carr and Kuhnel (2020) B. Carr and F. Kuhnel, Ann. Rev. Nucl. Part. Sci. 70, 355 (2020), arXiv:2006.02838 [astro-ph.CO] .
  * Villanueva-Domingo  _et al._ (2021) P. Villanueva-Domingo, O. Mena, and S. Palomares-Ruiz, Front. Astron. Space Sci. 8, 87 (2021), arXiv:2103.12087 [astro-ph.CO] .
  * Conzinu  _et al._ (2020) P. Conzinu, M. Gasperini, and G. Marozzi, JCAP 08, 031 (2020), arXiv:2004.08111 [gr-qc] .
  * Moradinezhad Dizgah  _et al._ (2019) A. Moradinezhad Dizgah, G. Franciolini, and A. Riotto, JCAP 11, 001 (2019), arXiv:1906.08978 [astro-ph.CO] .
  * Clesse and García-Bellido (2017) S. Clesse and J. García-Bellido, Phys. Dark Univ. 15, 142 (2017), arXiv:1603.05234 [astro-ph.CO] .
  * Kovetz (2017) E. D. Kovetz, Phys. Rev. Lett. 119, 131301 (2017), arXiv:1705.09182 [astro-ph.CO] .
  * Inomata  _et al._ (2017) K. Inomata, M. Kawasaki, K. Mukaida, Y. Tada, and T. T. Yanagida, Phys. Rev. D 96, 043504 (2017), arXiv:1701.02544 [astro-ph.CO] .
  * Bringmann  _et al._ (2019) T. Bringmann, P. F. Depta, V. Domcke, and K. Schmidt-Hoberg, Phys. Rev. D 99, 063532 (2019), arXiv:1808.05910 [astro-ph.CO] .
  * Raidal  _et al._ (2019) M. Raidal, C. Spethmann, V. Vaskonen, and H. Veermäe, JCAP 02, 018 (2019), arXiv:1812.01930 [astro-ph.CO] .
  * Green and Kavanagh (2021) A. M. Green and B. J. Kavanagh, J. Phys. G 48, 043001 (2021), arXiv:2007.10722 [astro-ph.CO] .
  * Poulin  _et al._ (2017) V. Poulin, P. D. Serpico, F. Calore, S. Clesse, and K. Kohri, Phys. Rev. D 96, 083524 (2017), arXiv:1707.04206 [astro-ph.CO] .
  * Wong  _et al._ (2021) K. W. K. Wong, G. Franciolini, V. De Luca, V. Baibhav, E. Berti, P. Pani, and A. Riotto, Phys. Rev. D 103, 023026 (2021), arXiv:2011.01865 [gr-qc] .
  * Escrivà and Romano (2021) A. Escrivà and A. E. Romano, JCAP 05, 066 (2021), arXiv:2103.03867 [gr-qc] .
  * Calabrese  _et al._ (2021) R. Calabrese, D. F. G. Fiorillo, G. Miele, S. Morisi, and A. Palazzo, (2021), arXiv:2106.02492 [hep-ph] .
  * Choudhury and Mazumdar (2014) S. Choudhury and A. Mazumdar, Phys. Lett. B 733, 270 (2014), arXiv:1307.5119 [astro-ph.CO] .
  * Kovetz  _et al._ (2017) E. D. Kovetz, I. Cholis, P. C. Breysse, and M. Kamionkowski, Phys. Rev. D 95, 103010 (2017), arXiv:1611.01157 [astro-ph.CO] .
  * Nakama  _et al._ (2017) T. Nakama, J. Silk, and M. Kamionkowski, Phys. Rev. D 95, 043511 (2017), arXiv:1612.06264 [astro-ph.CO] .
  * Sasaki  _et al._ (2016) M. Sasaki, T. Suyama, T. Tanaka, and S. Yokoyama, Phys. Rev. Lett. 117, 061101 (2016), [Erratum: Phys.Rev.Lett. 121, 059901 (2018)], arXiv:1603.08338 [astro-ph.CO] .
  * De Luca  _et al._ (2020) V. De Luca, G. Franciolini, P. Pani, and A. Riotto, JCAP 06, 044 (2020), arXiv:2005.05641 [astro-ph.CO] .
  * Domènech  _et al._ (2021) G. Domènech, V. Takhistov, and M. Sasaki, Phys. Lett. B 823, 136722 (2021), arXiv:2105.06816 [astro-ph.CO] .
  * Özsoy and Lalak (2021) O. Özsoy and Z. Lalak, JCAP 01, 040 (2021), arXiv:2008.07549 [astro-ph.CO] .
  * Khlopov (2010) M. Y. Khlopov, Res. Astron. Astrophys. 10, 495 (2010), arXiv:0801.0116 [astro-ph] .
  * De Luca  _et al._ (2021) V. De Luca, G. Franciolini, and A. Riotto, (2021), arXiv:2110.04229 [hep-ph] .
  * Deng (2020) H. Deng, JCAP 05, 037 (2020), arXiv:2003.02485 [astro-ph.CO] .
  * Tashiro and Sugiyama (2008) H. Tashiro and N. Sugiyama, Phys. Rev. D 78, 023004 (2008), arXiv:0801.3172 [astro-ph] .
  * Özsoy and Tasinato (2021) O. Özsoy and G. Tasinato, Phys. Rev. D 104, 043526 (2021), arXiv:2104.12792 [astro-ph.CO] .
  * Garcia-Bellido  _et al._ (2016) J. Garcia-Bellido, M. Peloso, and C. Unal, JCAP 12, 031 (2016), arXiv:1610.03763 [astro-ph.CO] .
  * Braglia  _et al._ (2021) M. Braglia, X. Chen, and D. K. Hazra, JCAP 03, 005 (2021), arXiv:2012.05821 [astro-ph.CO] .
  * Yokoyama (1999) J. Yokoyama, Prog. Theor. Phys. Suppl. 136, 338 (1999).
  * Yokoyama (1998) J. Yokoyama, Phys. Rev. D 58, 083510 (1998), arXiv:astro-ph/9802357 .
  * Josan and Green (2010) A. S. Josan and A. M. Green, Phys. Rev. D 82, 047303 (2010), arXiv:1004.5347 [hep-ph] .
  * Harada  _et al._ (2013) T. Harada, C.-M. Yoo, and K. Kohri, Phys. Rev. D 88, 084051 (2013), [Erratum: Phys.Rev.D 89, 029903 (2014)], arXiv:1309.4201 [astro-ph.CO] .
  * Arya (2020) R. Arya, JCAP 09, 042 (2020), arXiv:1910.05238 [astro-ph.CO] .
  * Garcia-Bellido  _et al._ (1996) J. Garcia-Bellido, A. D. Linde, and D. Wands, Phys. Rev. D 54, 6040 (1996), arXiv:astro-ph/9605094 .
  * Chongchitnan and Efstathiou (2007) S. Chongchitnan and G. Efstathiou, JCAP 01, 011 (2007), arXiv:astro-ph/0611818 .
  * Gundhi  _et al._ (2021) A. Gundhi, S. V. Ketov, and C. F. Steinwachs, Phys. Rev. D 103, 083518 (2021), arXiv:2011.05999 [hep-th] .
  * Palma  _et al._ (2020) G. A. Palma, S. Sypsas, and C. Zenteno, Phys. Rev. Lett. 125, 121301 (2020), arXiv:2004.06106 [astro-ph.CO] .
  * Ballesteros and Taoso (2018) G. Ballesteros and M. Taoso, Phys. Rev. D 97, 023501 (2018), arXiv:1709.05565 [hep-ph] .
  * Biagetti  _et al._ (2018) M. Biagetti, G. Franciolini, A. Kehagias, and A. Riotto, JCAP 07, 032 (2018), arXiv:1804.07124 [astro-ph.CO] .
  * Young (2019) S. Young, Int. J. Mod. Phys. D 29, 2030002 (2019), arXiv:1905.01230 [astro-ph.CO] .
  * Musco (2019) I. Musco, Phys. Rev. D 100, 123524 (2019), arXiv:1809.02127 [gr-qc] .
  * Young and Musso (2020) S. Young and M. Musso, JCAP 11, 022 (2020), arXiv:2001.06469 [astro-ph.CO] .
  * Kuhnel and Freese (2019) F. Kuhnel and K. Freese, Eur. Phys. J. C 79, 954 (2019), arXiv:1906.02744 [gr-qc] .
  * Kalaja  _et al._ (2019) A. Kalaja, N. Bellomo, N. Bartolo, D. Bertacca, S. Matarrese, I. Musco, A. Raccanelli, and L. Verde, JCAP 10, 031 (2019), arXiv:1908.03596 [astro-ph.CO] .
  * Dalianis (2019) I. Dalianis, JCAP 08, 032 (2019), arXiv:1812.09807 [astro-ph.CO] .
  * Musco  _et al._ (2021) I. Musco, V. De Luca, G. Franciolini, and A. Riotto, Phys. Rev. D 103, 063538 (2021), arXiv:2011.03014 [astro-ph.CO] .
  * Dalianis  _et al._ (2019) I. Dalianis, A. Kehagias, and G. Tringas, JCAP 01, 037 (2019), arXiv:1805.09483 [astro-ph.CO] .
  * Mahbub (2020) R. Mahbub, Phys. Rev. D 101, 023533 (2020), arXiv:1910.10602 [astro-ph.CO] .
  * Mahbub (2021) R. Mahbub, Phys. Rev. D 104, 043506 (2021), arXiv:2103.15957 [astro-ph.CO] .
  * Biagetti  _et al._ (2021) M. Biagetti, V. De Luca, G. Franciolini, A. Kehagias, and A. Riotto, Phys. Lett. B 820, 136602 (2021), arXiv:2105.07810 [astro-ph.CO] .
  * Ng and Wu (2021) K.-W. Ng and Y.-P. Wu, JHEP 11, 076 (2021), arXiv:2102.05620 [astro-ph.CO] .
  * Teimoori  _et al._ (2021) Z. Teimoori, K. Rezazadeh, M. A. Rasheed, and K. Karami,  (2021), 10.1088/1475-7516/2021/10/018, arXiv:2107.07620 [astro-ph.CO] .
  * Fumagalli  _et al._ (2020) J. Fumagalli, S. Renaux-Petel, J. W. Ronayne, and L. T. Witkowski, (2020), arXiv:2004.08369 [hep-th] .
  * Mukhanov (1988) V. F. Mukhanov, Zhurnal Eksperimentalnoi i Teoreticheskoi Fiziki 94, 1 (1988).
  * Mukhanov  _et al._ (1992) V. Mukhanov, H. Feldman, and R. Brandenberger, Physics Reports 215, 203 (1992).
  * Mukhanov (2005) V. Mukhanov,  _Physical Foundations of Cosmology_ (Cambridge University Press, 2005).
  * Sasaki (1986) M. Sasaki, Progress of Theoretical Physics 76, 1036 (1986), https://academic.oup.com/ptp/article-pdf/76/5/1036/5152623/76-5-1036.pdf .
  * Sarkar  _et al._ (2021) A. Sarkar, C. Sarkar, and B. Ghosh, JCAP 11, 029 (2021), arXiv:2106.02920 [gr-qc] .
  * Akrami  _et al._ (2020) Y. Akrami _et al._ (Planck), Astron. Astrophys. 641, A10 (2020), arXiv:1807.06211 [astro-ph.CO] .
  * Baumann (2011) D. Baumann, in  _Theoretical Advanced Study Institute in Elementary Particle Physics: Physics of the Large and the Small_ (2011) pp. 523–686, arXiv:0907.5424 [hep-th] .
  * Bardeen (1980) J. M. Bardeen, Phys. Rev. D 22, 1882 (1980).
  * Carrasco  _et al._ (2015) J. J. M. Carrasco, R. Kallosh, and A. Linde, Phys. Rev. D 92, 063519 (2015), arXiv:1506.00936 [hep-th] .
  * Kallosh  _et al._ (2013) R. Kallosh, A. Linde, and D. Roest, JHEP 11, 198 (2013), arXiv:1311.0472 [hep-th] .
  * Hawking (1975) S. W. Hawking, Communications in Mathematical Physics 43, 199 (1975).


# ConCLVD: Controllable Chinese Landscape Video Generation via Diffusion Model

Ding-Ming Liu  Department of Artificial Intelligence
Xiamen UniversityP. R. China ,  Shao-Wei Li  Department of Artificial
Intelligence
Xiamen UniversityP. R. China ,  Ruo-Yan Zhou  Department of Computer Science
Xiamen UniversityP. R. China ,  Li-Li Liang  Department of Computer Science
Xiamen UniversityP. R. China ,  Yong-Guan Hong  Department of Artificial
Intelligence
Xiamen UniversityP. R. China ,  Fei Chao  Department of Artificial
Intelligence
Xiamen UniversityP. R. China and  Rongrong Ji  Department of Artificial
Intelligence
Xiamen UniversityP. R. China

###### Abstract.

Chinese landscape painting is a gem of Chinese cultural and artistic heritage
that showcases the splendor of nature through the deep observations and
imaginations of its painters. Limited by traditional techniques, these
artworks were confined to static imagery in ancient times, leaving the
dynamism of landscapes and the subtleties of artistic sentiment to the
viewer’s imagination. Recently, emerging text-to-video (T2V) diffusion methods
have shown significant promise in video generation, providing hope for the
creation of dynamic Chinese landscape paintings. However, challenges such as
the lack of specific datasets, the intricacy of artistic styles, and the
creation of extensive, high-quality videos pose difficulties for these models
in generating Chinese landscape painting videos. In this paper, we propose
CLV-HD (Chinese Landscape Video-High Definition), a novel T2V dataset for
Chinese landscape painting videos, and ConCLVD (Controllable Chinese Landscape
Video Diffusion), a T2V model that utilizes Stable Diffusion. Specifically, we
present a motion module featuring a dual attention mechanism to capture the
dynamic transformations of landscape imageries, alongside a noise adapter to
leverage unsupervised contrastive learning in the latent space. Following the
generation of keyframes, we employ optical flow for frame interpolation to
enhance video smoothness. Our method not only retains the essence of the
landscape painting imageries but also achieves dynamic transitions,
significantly advancing the field of artistic video generation. The source
code and dataset will be released soon.

Artistic Video Generation, Chinese Landscape Painting Video Dataset, Motion
Module Mechanism and Contrastive Learning of Noise

††copyright: none

Figure 1. Chinese Landscape Video Generation.

.

##  1\. Introduction

Chinese landscape painting is a vital component of China’s cultural legacy. It
stands out for blending philosophical depth with aesthetic appeal, employing
distinctive light and shadow techniques, and mastering sophisticated ink
methods. However, as modern technology evolves and people’s aesthetic
preferences become more diverse and modernized, the inheritance of this
traditional art form faces challenges. Recently, advancements (ChangMM2022, ;
Xue2020EndtoEndCL, ; Lin2018TransformAS, ; Wang2019ChinaStyleAM, ;
Wang2023CCLAPCC, ; LiLZWXWZP0J23, ) in deep learning technology offer new
possibilities for the modern inheritance of this ancient art form through
innovative image generation of Chinese landscape paintings.

However, the charm of landscape painting lies not only in its static beauty
but also in the dynamic imagination and profound sense of the picture it
inspires. By utilizing the dynamic medium of video, we integrate the
traditional charm of Chinese landscape painting with the innovative power of
modern technology, infusing new life into the art form and enabling it to
exhibit a richer and more distinct layering of beauty in motion. This medium
not only attracts young audiences and promotes the dissemination and
advancement of art, but also opens new avenues of expression and inspiration,
offering valuable perspectives and tools for cultural heritage, education, and
artistic innovation. Therefore, developing high-quality dynamic Chinese
landscape painting video generation is particularly important.

Although Text-to-Video (T2V) (An2023LatentShiftLD, ; Singer2022MakeAVideoTG, ;
Wang2023VideoFactorySA, ; Xing2023MakeYourVideoCV, ; Zhou2022MagicVideoEV, ;
Deng2023EfficiencyoptimizedVD, ; Deng2023MVDiffusionMV, ) is widely studied in
various fields, it still falls short in capturing the essence of Chinese
landscape painting videos. First, the absence of specific datasets for
landscape painting styles hampers the accurate depiction of their unique style
and artistic traits. Second, there is a disparity in balancing the local and
global coherence in videos, which is crucial for depicting landscapes’ natural
transitions. Lastly, current methods fall short in capturing the similarities
and differences between frames, which affects the fluidity and authenticity of
the videos, ultimately impacting their artistic appeal.

To tackle the specific challenges of generating Chinese landscape painting
videos, notably the scarcity of datasets, we develop CLV-HD (Chinese Landscape
Video-High Definition), a dedicated text-to-video dataset with around 1,300
curated entries. This dataset aims to fill the technological void in emulating
Chinese landscape painting’s unique aesthetics, offering a vast resource to
enhance deep learning models’ ability to replicate this art form’s distinct
style. CLV-HD encompasses a range from traditional to modern styles and
details the videos’ dynamic elements, supporting our tech exploration and
providing new tools for the art’s modern evolution and innovation.

Technologically, drawing inspiration from AnimateDiff (Guo2023AnimateDiffAY,
), we merge a bespoke motion module with frozen Stable Diffusion, blending
Versatile Attention (Vaswani2017AttentionIA, ; Guo2023AnimateDiffAY, ) and
Sparse-Causal Attention (Child2019GeneratingLS, ; Hertz2022PrompttoPromptIE, ;
Ma2023FollowYP, ) for nuanced temporal data analysis. Versatile Attention
assesses the entire time series for global patterns, while Sparse-Causal
Attention hones in on local causality, enhancing future event predictions.
This synergy improves global and local trend analysis accuracy. To ensure the
similarity of adjacent frames and the dissimilarity of distant frames, we
employ a noise contrastive learning approach in latent space, using a noise
adapter to construct contrastive learning samples based on the predicted noise
output from the de-noising U-Net. In addition, we introduce a frame
interpolation strategy based on optical flow projection for smoother frame
transitions. Experiments indicate that our ConCLVD framework is capable of
producing high-quality, varied, and highly coherent videos that embody the
style of Chinese landscape painting.

Overall, the main contributions of this paper are as follows:

  * •

We innovatively address a new task of generating videos in the style of
Chinese landscape painting from text by designing our framework ConCLVD, which
combines a motion module based on Stable Diffusion and a contrastive learning
strategy in latent space, realizing cost-effective and efficient artistic
video generation.

  * •

We have created a new high-quality dataset named CLV-HD, which contains about
1,300 videos of Chinese landscape paintings, with each video accompanied by
detailed text descriptions.

  * •

We propose a new interpolation method using sparse optical flow projection, a
plug-and-play approach that doesn’t require training, which effectively
enhances video smoothness by filling in intermediate frames accurately and
efficiently.

  * •

Our model requires lower computational power, allowing us to offer a more
affordable implementation and experience for more people interested in
generating Chinese landscape painting videos.

##  2\. Related Work

###  2.1. Text-to-Video Diffusion Model

Despite significant progress in Text-to-Image (T2I) generation technology,
Text-to-Video (T2V) generation technology remains relatively lagging. This lag
is mainly due to the lack of large-scale and high-quality text-video pairing
datasets, as well as the high-dimensional modeling complexity of video data.
Early T2V research (Mittal2016SyncDRAWAV, ; Pan2017ToCW, ;
Marwah2017AttentiveSV, ; Gupta2018ImagineTS, ; Liu2019CrossModalDL, )
primarily focused on generating simple video content, but as time progressed,
researchers began exploring more advanced techniques to address the challenges
of T2V generation.

Recently, several studies (Geyer2023TokenFlowCD, ; Qi2023FateZeroFA, ;
Wu2023LAMPLA, ) have shifted towards using the knowledge of pre-trained T2I
models, simplifying the construction of T2V models by performing the diffusion
process in the latent space. In particular, T2V diffusion models have adopted
a spatio-temporal separable architecture, inheriting the spatial operations of
pre-trained T2I models and reducing the complexity of constructing intricate
models. Models such as AnimateDiff (Guo2023AnimateDiffAY, ) have enhanced the
capability of generating dynamic videos by introducing innovative motion
modeling modules, and new control mechanisms have been introduced to increase
the application flexibility and quality of generation. The development of
these diffusion models marks significant progress in T2V generation
technology. Although they still require extensive training and face
flexibility limitations in the application, they offer valuable perspectives
and methods for addressing core challenges in T2V generation, propelling
further exploration and innovation in the field.

###  2.2. Chinese Landscape Generation

GAN-based methods for generating images of Chinese landscape paintings have
been extensively studied (Xue2020EndtoEndCL, ; Lin2018TransformAS, ;
Wang2019ChinaStyleAM, ; Yuan2022LearningTG, ; Lin2018TransformAS, ;
Zhou2019AnIA, ). For example, Xue et al. (Xue2020EndtoEndCL, ) uses GANs to
generate landscape paintings in two stages: initially generating basic outline
sketches of landscape paintings from random noise; then transforming these
sketches into final paintings through edge-to-painting conversion. Polaca
(Yuan2022LearningTG, ) is a poetry-oriented landscape painting generation
model, transforming poetic texts into landscape images using GANs, generating
matching calligraphy, and finally merging both to form complete art pieces. In
recent years, diffusion models have achieved commendable results in generating
images of Chinese landscape paintings. CCLAP (Wang2023CCLAPCC, ) uses texts
and reference images as conditions, employing latent diffusion models to
generate Chinese landscape paintings with controllable content and style.
However, to date, the generation of Chinese landscape painting videos has
received limited attention. To our knowledge, we are the first to utilize
diffusion models for generating videos of Chinese landscape paintings.
Furthermore, unlike previous work where images serve as the primary guiding
condition, our focus is on capturing the transformation of scenery,
transitions of light and shadow, and other dynamic elements in landscape
paintings. Hence we frame the task as a T2V generation task.

##  3\. Method

###  3.1. Preliminary

Stable Diffusion, based on the Latent Diffusion Model (LDM)
(latentdiffusionmodel, ), is a type of diffusion model that enhances image
generation. The process starts by encoding an input image $x_{0}$ into a
latent space $z_{0}=E(x_{0})$ using an encoder $E$. This latent representation
$z_{0}$ is then modified through a series of steps according to the equation:

(1)
$q(z_{t}|z_{t-1})=\mathcal{N}(z_{t};\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I),$

where $z_{t}$ represents the latent state at step $t$, and $\beta_{t}$ are
hyperparameters controlling the noise added at each step. The process can be
summarized as:

(2)
$z_{t}=\sqrt{\overline{\alpha_{t}}}z_{0}+\sqrt{1-\overline{\alpha_{t}}}\epsilon,\epsilon\sim\mathcal{N}(0,I),$

where $\overline{\alpha_{t}}=\prod_{i=1}^{t}\alpha_{i}$ and
$\alpha_{t}=1-\beta_{t}$.

Stable Diffusion adopts the vanilla training objective as proposed in DDPM
(dhariwal2021diffusion, ), which is expressed as:

(3)
$\mathcal{L}=\mathbb{E}_{E(x_{0}),y,\epsilon\sim\mathcal{N}(0,I),t}\left[\|\epsilon-\epsilon_{\theta}(z_{t},t,\tau_{\theta}(y))\|^{2}\right],$

where $y$ is the text description and $\tau_{\theta}(\cdot)$ is a text encoder
based on the CLIP (CLIP, ) ViT-L/14 model. The architecture features a UNet
(unet, ) with downscaling and upscaling blocks, incorporating 2D
convolutional, self-attention, and cross-attention layers.

DDIM (Song2020DenoisingDI, ) is proposed to accelerate the sampling process by
optimizing the noise reduction steps. It transforms a random noise vector
$z_{T}$ into a coherent latent representation $z_{0}$ through a defined
sequence of timesteps $t:T\rightarrow 1$, which is defined as:

(4)
$z_{t-1}=\sqrt{\alpha_{t-1}}\frac{z_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(z_{t},t,\tau_{\theta}(y))}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t-1}}\epsilon_{\theta}(z_{t},t,\tau_{\theta}(y)),$

where $\alpha_{t}$ is a parameter for noise scheduling,
$\epsilon_{\theta}(z_{t},t,\tau_{\theta}(y))$ is the predicted noise within
the networks’ latent space.

AnimateDiff (Guo2023AnimateDiffAY, ) extends the Stable Diffusion model for
T2V tasks by integrating a motion modeling module for video data handling. It
progresses from processing four-dimensional image batches to handling five-
dimensional video tensors $(batch\times channels\times frames\times
height\times width)$. The transformation adapts each 2D convolution and
attention layer from the base image model into pseudo-3D layers that focus
solely on spatial aspects. This adaptation involves reorganizing the frame
dimension to merge with the batch dimension, which permits independent frame
analysis. Subsequently, feature maps undergo a transformation to a
$(batch\times height\times width)\times frames\times channels$ configuration.
This step sets the stage for the motion module, aimed at ensuring consistent
motion and content stability across frames.

Figure 2. Overview of ConCLVD. Left: the architecture. ConCLVD integrates a
trainable motion module based on a frozen Stable Diffusion and introduces a
noise adapter to accommodate contrast learning of noise in latent space.
Right: the inference framework. The video is generated by the denoising U-Net
integrated with the motion module and enhanced by sparse optical flow
projection (SOFP) interpolation technology.

Motion modules are embedded throughout the U-shaped network, using vanilla
temporal transformers with self-attention blocks operating along the temporal
axis. The training objective is to minimize the following loss function:

(5)
$\mathcal{L}=\mathbb{E}_{E(x_{0}^{1:N}),y,\epsilon\sim\mathcal{N}(0,I),t}\left[\|\epsilon-\epsilon_{\theta}(z_{t}^{1:N},t,\tau_{\theta}(y))\|_{2}^{2}\right],$

where $x_{0}^{1:N}$ represents a sample of video data, and $z_{t}^{1:N}$ is
the latent code obtained by adding noise to the initial latent code
$z_{0}^{1:N}$ at time step $t$. During training, the pre-trained weights of
the base T2I model are frozen to maintain consistency in the feature space.

###  3.2. CLV-HD: Text-to-Chinese Landscape Painting Video Dataset

Diverse text-video datasets are essential for developing high-quality T2V
generation models. However, there is a notable shortage of such datasets for
Chinese landscape painting videos, limiting the fusion of traditional Chinese
art with modern technology. The lack of comprehensive datasets with textual
annotations hampers the creation of high-quality Chinese landscape painting
videos.

Our work fills this notable gap by introducing CLV-HD, a groundbreaking text-
to-Chinese landscape painting video dataset, now publicly available. It
includes 1,300 text-video pairs, sourced from open domains in high-definition.
We collected 210 high-resolution clips from Chinese water ink animations and
over 400 original videos from YouTube. To manage complex scene transitions, we
meticulously segmented these into 1,300 single-scene clips, enhancing training
utility. Acknowledging the videos’ intricate details, we manually annotate
each text to accurately match the video content.

Our CLV-HD dataset effectively addresses the scarcity of specialized datasets
for generating Chinese landscape painting videos from textual descriptions.
This pioneering effort fosters the convergence of Chinese traditional art and
artificial intelligence, catalyzing new avenues for scholarly inquiry and
practical application.

###  3.3. Controllable Chinese Landscape Video Generation

Our model integrates two key components: a denoising U-Net and a Noise-
Adapter. The denoising U-Net uses pre-trained Stable Diffusion weights for
enhanced image layer processing. To capture motion, a motion module layer
follows each image layer, extending the T2I functionality to T2V and enabling
dynamic motion synthesis. Our training incorporates unsupervised contrastive
learning through the Noise-Adapter, which refines the noise output from the
U-Net into contrastive learning samples, enhancing noise representation in the
latent space. The structure of the motion module and the contrastive learning
strategy are detailed in Section 3.4.

During the training process, we keep the image layers of the denoising U-Net
frozen and only update the motion module layers and the Noise-Adapter. During
the inference phase, the model creates coherent landscape painting animations
through an iterative denoising process. The video’s length and smoothness are
then further refined using a projection frame interpolation technique based on
sparse optical flow, as detailed in Section 3.5. This procedure not only
amplifies the visual appeal of the animation but also enhances the video’s
coherence and aesthetic value, providing a novel method for dynamically
presenting landscape paintings.

###  3.4. Motion Module and Training

####  3.4.1. Motion Module Design

Figure 3. Design of Motion Module. The motion module is inserted following
each image layer of the pre-trained SD to process video data.

In our motion module, the core design principle is to capture dynamic changes
between frames through effective information exchange. We utilize standard
temporal transformers as the core architecture, to facilitate efficient
information exchange between frames to capture the dynamic information that
evolves within video sequences. To achieve this, we integrate two key
attention mechanisms: Versatile Attention (Vaswani2017AttentionIA, ;
Guo2023AnimateDiffAY, ) and Sparse-Causal Attention (Child2019GeneratingLS, ;
Hertz2022PrompttoPromptIE, ; Ma2023FollowYP, ) , which are specifically
optimized for time series data and analyze the temporal dimension by
processing feature tensors in a particular shape configuration. The structure
of the motion module is demonstrated in Figure. 3.

The motion module is inserted between the image layers of the pre-trained
Stable Diffusion. When a batch of data passes through our motion module, the
video data undergoes reshaping and dimension transformation in Project_In
module, converting the input tensor from from
$[batch,channels,frames,height,width]$ to $[(batch\times height\times
width)\times frames\times{inner\\_dim}]$. Through this approach, the model can
meticulously analyze interactions between individual frames and capture local
and global features using the designed attention mechanism. Finally, the
Project_Out module reverts the processed data back to its original dimensions
and adds it to the initial input as the output employing a residual
architecture.

The Versatile Attention (Vaswani2017AttentionIA, ; Guo2023AnimateDiffAY, )
adapts the traditional self-attention framework for time-series data. The
mechanism computes relationships between different time points, aiding the
model in understanding video dynamics.

The Sparse-Causal Attention (Child2019GeneratingLS, ;
Hertz2022PrompttoPromptIE, ; Ma2023FollowYP, ) mechanism focuses on simulating
the causal relationships within the frame sequence, paying attention only to
the previous frame preceding the current frame, thereby maintaining temporal
causality and allowing the model to effectively understand and predict
variations in the sequence without introducing future information.

The mathematical expressions for both attention mechanisms can be unified as
follows:

(6)
$z=\text{Attention}(Q,K,V)=\text{Softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot
V,$

where $Q=W^{Q}z_{v_{i}}$, $K=W^{K}z_{v_{j}}$, $V=W^{V}z_{v_{j}}$, with
$z_{v_{i}}$ and $z_{v_{j}}$ being the processed latent representations of the
$i^{th}$ and $j^{th}$ frames, respectively. $W^{Q}$, $W^{K}$, and $W^{V}$ are
learnable matrices that project the input into query, key, and value, and $d$
is the dimensionality of the key and query features. For Sparse-Causal
Attention, a masking mechanism is applied to include only information from the
previous frame that has a causal influence on the current frame.
The Versatile Attention mask is defined as:

(7) $\left(\mathrm{M}_{\text{Versatile }}\right)_{ij}=1.$

The Sparse-Causal Attention mask is defined as:

(8) $\left(\mathrm{M}_{\text{Sparse-Causal
}}\right)_{ij}=\left\\{\begin{array}[]{ll}1&\text{ if }j=i-1\\\ 0&\text{
otherwise }\end{array}\right..$

Specifically, when we implement the Versatile Attention mechanism, we derive
the query feature from frame $z_{v_{i}}$, and the key and value features from
the first frame $z_{v_{1}}$ to the last frame $z_{v_{T}}$. When we implement
the Sparse-Causal Attention mechanism, we derive the query feature from frame
$z_{v_{i}}$, and the key and value features from the preceding frame
$z_{v_{i-1}}$ (See Figure. 4 for a detailed visual depiction).

Figure 4. Detailed explanation of the attention mechanism. The above
represents Versatile Attention, where each frame is related to every other
frame; the below represents Sparse-Causal Attention, where each frame only
focuses on its previous frame.

Through the combination of these two attention mechanisms, our motion modeling
can not only capture the temporal dimensional features within video data but
also ensure the consistency of content and smoothness of motion.

####  3.4.2. Unsupervised Contrastive Learning of Noise Corresponding to Video
Frames in Latent Space

We introduce a contrastive learning strategy to deepen the model’s grasp of
dynamic transitions between video frames. We propose a unique approach for
contrastive learning that leverages the UNet’s distinct noise predictions for
each frame. Specifically, we process the predicted noise vectors through a
noise adapter, a method inspired by SimCLR(simCLR, ) and SeCo(SeCo, ), to
generate samples from these vectors. The essence of this strategy is to treat
consecutive frames as positive pairs and frames where the frame number
difference exceeds a threshold as negative pairs, using their respective noise
vectors to create targeted training samples. It’s important to mention that
our exploration of different network designs for the noise adapter has led us
to conclude that a basic MLP suffices. We leave the search for a better noise
adapter to future works. Specifically, for the predicted noise
$\epsilon_{\theta}^{i}$ of the $i^{th}$ frame output by the model, we assume
that it passes through a noise adapter to obtain the corresponding contrastive
learning sample $\hat{\epsilon}_{\theta}^{i}$. We calculate the similarity at
step $t$ using the cosine similarity equation:

(9)
${r_{t}}^{(i,j)}=\frac{\hat{\epsilon_{\theta}}^{i}\cdot\hat{\epsilon_{\theta}}^{j}}{\|\hat{\epsilon_{\theta}}^{i}\|\|\hat{\epsilon_{\theta}}^{j}\|},$

where $\hat{\epsilon_{\theta}}^{i}$ and $\hat{\epsilon_{\theta}}^{j}$ are two
vectors output by noise adapter for the $i^{th}$ frame and $j^{th}$ frame.
Specifically, we calculate the contrastive learning loss for the noise vectors
corresponding to all frames except the last one as anchors and calculate the
average. For the noise vector corresponding to the $i^{th}$ frame, we have the
following definition:

(10)
${l_{t}}^{(i)}=-\log\frac{\exp({r_{t}}^{(i,i+1)}/\tau)}{\exp({r_{t}}^{(i,i+1)}/\tau)+\sum_{k=1}^{N}1_{\\{|k-i|>m\\}}\exp({r_{t}}^{(i,k)}/\tau)},$

where $m$ represents the frame sequence threshold for negative samples, and
$\tau$ denotes the temperature parameter for contrastive learning. The overall
contrastive learning loss can be defined as:

(11) $\mathcal{L}_{con}=\frac{1}{N-1}\sum_{k=1}^{N-1}{l_{t}}^{(k)},$

where $N$ is the total number of frames in the video. Through this contrastive
learning of noise in the latent space, the model is encouraged to learn rich
and robust temporal sequence feature representations without explicit
annotations.

####  3.4.3. Training Objectives

In this section, we detail the training objectives of ConCLVD. We sample
videos to obtain a sequence of frames $x_{0}^{1:N}$, and encode each frame
into the latent space $z_{0}^{1:N}$ using a pre-trained Variational
Autoencoder initially. Next, the latent codes are noised using the defined
forward diffusion schedule:
$\text{z}_{t}^{1:N}=\sqrt{\bar{\alpha}_{t}}z_{0}^{1:N}+\sqrt{1-\bar{\alpha}_{t}}\epsilon$.
During the training process, our network with the motion module receives noisy
latent codes and corresponding textual prompts as input, predicting the noise
intensity added to the latent codes. This process uses the L2 loss for
calculation, and the loss for the motion module is defined as:

(12)
$\mathcal{L}_{diff}=\mathbb{E}_{\mathbb{E}(x_{0}^{1:N}),y,\epsilon\sim\mathcal{N}(0,I),t}\left[\|\epsilon-\epsilon_{\theta}(z_{t}^{1:N},t,\tau_{\theta}(y))\|_{2}^{2}\right].$

Note that during optimization, the pre-trained weights of the base Stable
Diffusion are frozen. Only the motion module and noise adapter are trainable.
Combining the diffusion model with the contrastive learning strategy, the
overall objective function of the ConCLVD can be formulated as follows:

(13)
$\mathcal{L}=\lambda_{diff}\cdot\mathcal{L}_{diff}+\lambda_{con}\cdot\mathcal{L}_{con}.$

The hyperparameters $\lambda_{diff}$ and $\lambda_{con}$ are used to balance
different losses and achieve improved performance.

###  3.5. Frame Interpolation Based on Sparse Optical Flow Projection

Figure 5. Illustration of of SOFP Interpolation. $V_{(t\rightarrow T)}$ is the
optical flow from $I_{t}$ to $I_{T}$ and $V_{(t\rightarrow t+1)}$ is the
optical flow from $I_{t}$ to $I_{(t+1)}$. The projection result of
$V_{(t\rightarrow T)}$ onto $V_{(t\rightarrow t+1)}$ is used to insert a new
frame $I_{(t+0.5)}$ between two consecutive frames $I_{t}$ and $I_{(t+1)}$.

In the field of video processing, frame interpolation techniques based on
optical flow (Park2021AsymmetricBM, ; Huang2020RealTimeIF, ; zhong2023clearer,
) are commonly used methods aimed at improving the visual experience. We
propose a frame interpolation strategy that employs optical flow projection.
This approach uses information from two adjacent frames along with the last
frame of the video to interpolate additional frames between the two adjacent
ones, aiming to achieve smooth transitions and enhance the continuity of the
video playback.

Optical flow is defined as the motion vector of pixel points between two
frames. For two images, the optical flow vector indicates the new position of
each pixel in the first image within the second frame. Given a pair of
consecutive frames $I_{t}$ and $I_{t+1}$, we calculate the optical flow
$V_{t\rightarrow t+1}$ between the two frames. Similarly, we calculate the
optical flow $V_{t\rightarrow T}$ between frame $I_{t}$ and frame $I_{T}$.

In our research, we employ the advanced optical flow estimation algorithm RAFT
(RAFT, ) to precisely calculate the motion distance between pixels. Given a
set of image triplets $\\{I_{t},I_{T},I_{t+1}\\}$, our primary task is to
estimate two sets of optical flows: one from $I_{t}$ to $I_{T}$, denoted as
$V_{t\rightarrow T}$, and the other from $I_{t}$ to $I_{t+1}$, denoted as
$V_{t\rightarrow t+1}$. For each pixel point $(x,y)$ in the image, we project
the motion vector $V_{t\rightarrow T}(x,y)$ from $I_{t}$ to $I_{T}$ onto the
motion vector $V_{t\rightarrow t+1}(x,y)$ from $I_{t}$ to $I_{t+1}$. Thus, we
define the distance mapping $D_{t}(x,y)$ as the ratio between these two
vectors:

(14) $D_{t}(x,y)=\frac{\|V_{t\rightarrow
T}(x,y)\|\cos\theta}{\|V_{t\rightarrow t+1}(x,y)\|},$

where $\theta$ is the angle between these two vectors. The projection results
are used for interpolation between frame $I_{t}$ and frame $I_{t+1}$.

The entire method of ConCLVD is summarized in Alogorithm 1.

1

Input:

– $z_{0}^{1:N}$: Latent code from source video

– $\tau_{\theta}(y)$: CLIP-encoded text prompt for the input video

2

Hyperparameters:

– $\tau$: Temperature parameter used in contrastive learning

– $m$: Threshold for negative sample

– $\lambda_{diff}$, $\lambda_{con}$: Balance parameters for the loss function

– $T$: Total number of sampling times

– $\alpha_{t}$: Parameters controlling the noise added

3

4

  * $\triangleright$

Training:

5for  _$t\sim\text{Uniform}(\\{1,\ldots,T\\})$_ do

6

7
$z_{t}^{1:N}=\text{forward}(z_{0}^{1:N},\epsilon^{1:N},t)\hfill\blacktriangleright
Eq.~{}\eqref{eq_process}$ $\epsilon_{\theta^{\prime}}^{1:N}\leftarrow
d_{frozen}(z_{t}^{1:N},t,\tau_{\theta}(y))$***$\theta^{\prime}$ indicates that
the results are obtained from a model with frozen parameters.

8 $\Delta\text{calculate }M_{\text{Versatile}}\hfill\blacktriangleright
Eq.~{}\eqref{m1}$

9 $\Delta\text{calculate }M_{\text{Sparse-Causal}}\hfill\blacktriangleright
Eq.~{}\eqref{m2}$

10 $\epsilon_{\theta}^{1:N}\leftarrow
f(\epsilon_{\theta^{\prime}}^{1:N},M_{\text{Versatile}},M_{\text{Sparse-
Causal}})\hfill$

11

12
$\mathcal{L}_{diff}=\text{Diffusion\\_loss}(\epsilon^{1:N},\epsilon_{\theta}^{1:N})\hfill\blacktriangleright
Eq.~{}\eqref{l1}$

13 for  _all $i\in\\{1,\ldots,N\\}$_ do

14 $\hat{\epsilon}_{\theta}^{i}=g(\epsilon_{\theta}^{i})$

15

16 end for

17 for  _all $i\in\\{1,\ldots,N-1\\}$_ do

18 Calculate  ${l_{t}}^{(i)}\hfill\blacktriangleright Eq.~{}\eqref{cl}$

19

20 end for

21
$\mathcal{L}_{con}=\text{Con\\_loss}({l_{t}}^{(1)},{l_{t}}^{(2)},\ldots,{l_{t}}^{(N-1)})\hfill\blacktriangleright
Eq.~{}\eqref{conloss}$

22
$\mathcal{L}=\lambda_{diff}\cdot\mathcal{L}_{diff}+\lambda_{con}\cdot\mathcal{L}_{con}$

23 update network $f$ and noise-adapter $g$ to minimize $\mathcal{L}$

24 end for

25

26

  * $\triangleright$

Inference:

for  _$t=T$ to $1$_ do

27 $\epsilon_{\theta^{\prime}}^{1:N}\leftarrow
d_{frozen}(z_{t}^{1:N},t,\tau_{\theta}(y))$††footnotemark:

28 $\Delta\text{calculate }M_{\text{Versatile}}\hfill\blacktriangleright
Eq.~{}\eqref{m1}$

29 $\Delta\text{calculate }M_{\text{Sparse-Causal}}\hfill\blacktriangleright
Eq.~{}\eqref{m2}$

30 $\epsilon_{\theta}^{1:N}\leftarrow
f(\epsilon_{\theta^{\prime}}^{1:N},M_{\text{Versatile}},M_{\text{Sparse-
Causal}})\hfill$

31
$z_{t-1}^{1:N}=\text{DDIM\\_sample}(z_{t}^{1:N},\epsilon_{\theta}^{1:N})\hfill\blacktriangleright
Eq.~{}\eqref{ddim}$

32 end for

$z_{0}^{1:2N-1}=\text{SOFP\\_Interpolation}(z_{0}^{1:N})\hfill$

Algorithm 1 ConCLVD’s main learning algorithm

##  4\. Experimentation

###  4.1. Experimental Setup

All our experiments are conducted on a single NVIDIA GeForce RTX 3090 with
24GB of VRAM. We perform all computational tasks on a Linux-based system
running Python 3.10.13, compiled with the GCC 11.2.0 compiler. To ensure
efficient execution of parallel computing tasks, we employ the NVIDIA CUDA
12.0 platform.

####  4.1.1. Training Settings

We directly use the trained Stable Diffusion v1.5 as the base model. We train
the motion module and noise adapter using the proposed CLV-HD dataset. The
video clips in the dataset are first sampled at a stride of 4, then resized
and center-cropped to a resolution of $256\times 256$. The frame length is set
to 8, the batch size to 1, the learning rate to $1\times 10^{-5}$, and the
total number of training steps to 40k. In the contrastive learning of noise,
we set the threshold for negative samples to 4, and the temperature parameter
$\tau$ to 0.07. We use a linear beta schedule as in AnimateDiff
(Guo2023AnimateDiffAY, ), where $\beta_{\text{start}}=0.00085$ and
$\beta_{\text{end}}=0.012$. For loss function, we set $\lambda_{diff}$ as 1
and $\lambda_{con}$ as 0.07.

####  4.1.2. Inference Settings

At inference, we use DDIM (Song2020DenoisingDI, ) sampler in our experiments.
We only use our frame interpolation module during inference. Our frame
interpolation strategy expands an N-frame video to 2N-1 frames by inserting
new frames between the original adjacent frames. Both the number of sampling
steps and the scale for text guidance are selectable for users. When
conducting qualitative and quantitative comparisons, we use DDIM with 25
sampling steps, and the scale for text guidance is 8.

###  4.2. Results

In this paper, we present several qualitative results from our ConCLVD model,
displaying only four frames per animation due to space limitations. We
encourage readers to visit our website for higher-quality visuals. Our method
skillfully combines brushwork beauty with the dynamic play of light and shadow
typical of Chinese landscape painting into the T2V model. It uses the “five
shades of ink”111The “five shades of ink” technique is a traditional Chinese
painting method that uses varying depths of ink to create a spectrum of
shades. technique, evident in the second row’s right-side images, where
varying ink depths create nuanced color changes, beautifully transitioning
across elements like mountains and water. Additionally, our model employs the
“combined colored ink”222The “combined colored ink” technique refers to a
method in Chinese painting where different colored inks are blended together
to achieve harmonious and unified effects. technique to blend multiple colors
harmoniously, as seen in the third row, offering a synchronized and
aesthetically cohesive landscape portrayal. In addition, we find our method
effectively differentiates main subjects from their surroundings. For example,
the animation in the first row’s right-side showcases raindrops and a house
moving at distinct speeds and blurs, creating vivid scenes. Results affirm
that ConCLVD adeptly transfers Chinese landscape painting’s brushwork,
techniques, and style to digital media, enriching modern digital content with
traditional artistic values.

Figure 6. Main Results. Our ConCLVD creates high-quality videos in the style
of Chinese landscape painting.

###  4.3. Baseline Comparisons

Figure 7. Qualitative comparisons with baseline methods. Best viewed with
zoom-in. Figure 8. User preference studies. ConCLVD outperforms all baselines
from overall aspects.

We evaluate our method against six prominent baselines: AnimateLCM
(wang2024animatelcm, ), which accelerates personalized diffusion models for
high-fidelity animation via consistency decoupling; Gen-2 (runwaymlGen2Runway,
), a multimodal AI system for novel video creation from text; Text2Video-Zero
(khachatryan2023text2videozero, ), enhancing video consistency with frame-
inter attention for zero-shot text-based video generation; Pika 1.0 (pikaPika,
), specializing in Text-to-3D special effects with parallel sampling for
cinema-grade outputs; AnimateDiff-lightning (lin2024animatedifflightning, ),
focusing on rapid video production through progressive adversarial diffusion
distillation; and ModelScope (modelscope, ), which leverages a 3D-Unet
architecture to iteratively denoise Gaussian noise into video content.

####  4.3.1. Qualitative Results

Figure 7 presents our qualitative results, comparing our ConCLVD model with
other generation models. While Gen-2 (runwaymlGen2Runway, ) produces videos
reflecting the text concept, its style deviates from Chinese landscape
painting, evident in unrealistic details like the fisherman’s attire.
Similarly, AnimateLCM (wang2024animatelcm, ) and AnimateDifflightning
(lin2024animatedifflightning, ) introduce modern elements and an overly
detailed depiction, misaligning with the traditional painting style.
Text2Video-Zero (khachatryan2023text2videozero, ) aligns closer to the
landscape style but suffers from instability and inconsistency in temporal
context, like fluctuating pine tree shapes. Pika 1.0 (pikaPika, ) achieves
consistency but lacks dynamic variation, and ModelScope (modelscope, ) fails
in vivid and dynamic depiction, rendering water waves as mere lines. In
contrast, our ConCLVD excels by incorporating raindrops, mist, and boats
moving at varied speeds, capturing the dynamic essence of ink wash painting
and harmoniously blending movement with stillness, accurately reflecting
landscape painting aesthetics. Further comparisons are available in our
supplementary materials.

####  4.3.2. Quantitative Results

Our ConCLVD is assessed against baselines using both automatic metrics and
user evaluations. For quantitative indicators, we leverage CLIP-temp
(esser2023structure, ) for temporal consistency by computing cosine
similarities between consecutive frames and CLIP-text (Qi2023FateZeroFA, ) for
text alignment by averaging CLIP scores between video frames and prompts. We
further evaluate the training computational requirements of ConCLVD with
baselines.

Table 1 presents a detailed comparison of ConCLVD with baselines. Quantitative
metric analysis shows that ConCLVD performs better in almost all metrics
compared to baselines, albeit with slightly lower text alignment than Gen-2,
which we believe is due to the frozen Stable Diffusion parameters. However,
our strategy enables efficient training on an RTX 3090, balancing technical
performance with training practicability.

####  4.3.3. Cost-effective with High Efficacy

The “GPUs for Training” column in Table 1 compares the training computing
power requirements for ConCLVD and baselines. Pika 1.0 and Gen-2, commercial
products that are not open source, are presumed to have much higher
computational requirements than ConCLVD based on the estimation of their
parameters. According to the article by Tim Dettmers (dettmers2023bestgpus, )
on selecting GPUs, given ModelScope’s total parameter count of approximately
1.7 billion and a minimum hardware requirement of at least 16GB of GPU memory,
it is inferred that suitable GPUs for this model is NVIDIA’s RTX 3090.
Text2Video-Zero is a model that requires no training, but it falls short
regarding smooth inter-frame transitions. AnimateLCM’s training uses eight
A800 GPUs and AnimateDiff-lightning’s training uses 64 A100 GPUs, whereas our
ConCLVD only needs a single RTX 3090. ModelScope’s training requirements are
similar to ours, but the videos it generates are noticeably inferior in terms
of fidelity to the landscape painting style, and the portrayal of the
elements. ConCLVD allows users to train their own Chinese landscape painting
video generation model with minimal cost.

Table 1. Quantitative results with baselines.Text and Temp denote CLIP-text and CLIP-temp respectively. ADL stands for AnimateDiff-lightning. * represents inferred result. Method | Metric | GPUs for Training
---|---|---
| Text$\uparrow$ | Temp$\uparrow$ |
Pika 1.0 (pikaPika, ) | 24.564 | 0.972 | details not disclosed
T2V-Zero (khachatryan2023text2videozero, ) | 27.633 | 0.980 | no training required
AnimateLCM (wang2024animatelcm, ) | 29.987 | 0.970 | 8 A800
Gen-2 (runwaymlGen2Runway, ) | 30.609 | 0.978 | details not disclosed
ADL (lin2024animatedifflightning, ) | 30.421 | 0.962 | 64 A100
ModelScope (modelscope, ) | 29.562 | 0.961 | RTX 3090*
ConCLVD | 30.518 | 0.983 | RTX3090

####  4.3.4. User Study

100 participants, including creators, enthusiasts, and aesthetes of Chinese
landscape painting, were surveyed to evaluate our generated videos for Text
Alignment, Temporal Consistency, adherence to Chinese landscape painting
style, and overall quality. Due to the lack of uniform quantitative indicators
for style fidelity, these participants’ insights are crucial for our research.
Figure 8 shows our ConCLVD model significantly outshining all baseline methods
in the user study. Although ConCLVD’s CLIP-text score doesn’t top Gen-2’s, its
strong stylistic fidelity to Chinese landscape painting likely influences
respondents’ preferences towards its text alignment. This highlights how
subjective style preferences can profoundly affect perceptions, indicating
that a distinctive style may have a more lasting impact.

###  4.4. Ablation Studies

Figure 9. Ablation study on Sparse-Casual Attention and Contrastive Learning of Noise. Table 2. Quantitative results of Ablation study. Model | Clip-Text | Clip-Temp |
---|---|---|---
without SC-Attn | 28.862 | 0.973 |
without CLofNoise | 29.864 | 0.964 |
ConCLVD | 30.526 | 0.981 |

We primarily conducted ablation studies on the proposed Sparse-Causal
Attention (SC-Attn) within the motion module’s residual architecture, and the
Contrastive Learning of Noise for Video Frames (CLofNoise) , individually
removing each design to assess its impact. The findings, illustrated in Figure
9, reveal that omitting SC-Attn leads to discontinuous frame transitions while
maintaining content consistency. Conversely, excluding CLofNoise maintains
content uniformity and smooth transitions but lacks in capturing dynamic
changes, such as petal movement in the video.

The quantitative results, detailed in Table 2, highlight the significance of
both SC-Attn and CLofNoise in enhancing textual accuracy and temporal
coherence. These qualitative and quantitative assessments confirm the vital
roles of our model’s components in achieving high-quality outputs.
Supplementary materials further underscore the contributions of these key
designs.

##  5\. Conclusion

In this paper, we proposed a text-to-video conversion network, termed ConCLVD
(Controllable Chinese Landscape Video Diffusion), focusing on generating
videos of Chinese landscape painting style. Utilizing the proposed CLV-HD
(Chinese Landscape Video-High Definition) dataset, motion module, contrastive
learning of noise in latent space, and a frame interpolation strategy based on
optical flow projection, our framework can effectively capture the dynamic
beauty of landscape painting imagery and create a poetic and picturesque
essence while maintaining low computational power requirements. Extensive
experiments demonstrated the effectiveness of our method in visual quality,
ink style prominence, and coherence, showcasing its potential in artistic
video generation. We believe that our framework provides an opportunity to
promote the preservation, development, and promotion of Chinese landscape
painting. We also hope that our work can lead to inspiration for possible
future works, bringing new energy to traditional art and culture using modern
technology.

## References

  * [1] Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jiebo Luo, and Xiaoyue Yin.  Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation.  ArXiv, 2023.
  * [2] Xiang Chang, Fei Chao, Changjing Shang, and Qiang Shen.  Sundial-gan: A cascade generative adversarial networks framework for deciphering oracle bone inscriptions.  In Proceedings of the 30th ACM International Conference on Multimedia, 2022.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, 2020.
  * [4] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever.  Generating long sequences with sparse transformers.  ArXiv, 2019.
  * [5] Zijun Deng, Xiangteng He, and Yuxin Peng.  Efficiency-optimized video diffusion models.  Proceedings of the 31st ACM International Conference on Multimedia, 2023.
  * [6] Zijun Deng, Xiangteng He, Yuxin Peng, Xiongwei Zhu, and Lele Cheng.  Mv-diffusion: Motion-aware video diffusion model.  Proceedings of the 31st ACM International Conference on Multimedia, 2023.
  * [7] Tim Dettmers.  The best gpus for deep learning in 2023 — an in-depth analysis, 2023\.
  * [8] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis, 2021.
  * [9] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models, 2023\.
  * [10] Institute for Intelligent Computing.  Text-to-video-synthesis model in open domain — modelscope.cn.  https://modelscope.cn/models/iic/text-to-video-synthesis/summary, 2024.
  * [11] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  ArXiv, 2023.
  * [12] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Y. Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  ArXiv, 2023.
  * [13] Tanmay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi.  Imagine this! scripts to compositions to videos.  In European Conference on Computer Vision, 2018.
  * [14] Amir Hertz, Ron Mokady, Jay M. Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  ArXiv, 2022.
  * [15] Jun ho Park, Chulwoo Lee, and Chang-Su Kim.  Asymmetric bilateral motion estimation for video frame interpolation.  2021\.
  * [16] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou.  Real-time intermediate flow estimation for video frame interpolation.  In European Conference on Computer Vision, 2020.
  * [17] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, 2023.
  * [18] Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefeng Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji.  Autodiffusion: Training-free optimization of time steps and architectures for automated diffusion model acceleration.  In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, 2023.
  * [19] Daoyu Lin, Yang Wang, Guangluan Xu, Jun Yu Li, and Kun Fu.  Transform a simple sketch to a chinese painting by a multiscale deep neural network.  Algorithms, 2018.
  * [20] Shanchuan Lin and Xiao Yang.  Animatediff-lightning: Cross-model diffusion distillation, 2024.
  * [21] Yue Liu, Xin Wang, Yitian Yuan, and Wenwu Zhu.  Cross-modal dual learning for sentence-to-video generation.  Proceedings of the 27th ACM International Conference on Multimedia, 2019.
  * [22] Yue Ma, Yin-Yin He, Xiaodong Cun, Xintao Wang, Ying Shan, Xiu Li, and Qifeng Chen.  Follow your pose: Pose-guided text-to-video generation using pose-free videos.  In AAAI Conference on Artificial Intelligence, 2023.
  * [23] Tanya Marwah, Gaurav Mittal, and Vineeth N. Balasubramanian.  Attentive semantic video generation using captions.  2017 IEEE International Conference on Computer Vision (ICCV), 2017\.
  * [24] Gaurav Mittal, Tanya Marwah, and Vineeth N. Balasubramanian.  Sync-draw: Automatic video generation using deep recurrent attentive architectures.  Proceedings of the 25th ACM International Conference on Multimedia, 2016.
  * [25] Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei.  To create what you tell: Generating videos from captions.  Proceedings of the 25th ACM International Conference on Multimedia, 2017.
  * [26] Pika labs.  Pika — pika.art.  https://pika.art/, 2024.  [Accessed 25-03-2024].
  * [27] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  2023 IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
  * [28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, 2021.
  * [29] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  pages 10674–10685, 2021.
  * [30] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015 - 18th International Conference Munich, Germany, October 5 - 9, 2015, Proceedings, Part III, 2015.
  * [31] RunwayML Research Team.  Gen-2 by Runway — research.runwayml.com.  https://research.runwayml.com/gen2, 2024.  [Accessed 25-03-2024].
  * [32] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.  Make-a-video: Text-to-video generation without text-video data.  2023\.
  * [33] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
  * [34] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In European Conference on Computer Vision, 2020.
  * [35] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Neural Information Processing Systems, 2017.
  * [36] Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, and Hongsheng Li.  Animatelcm: Accelerating the animation of personalized diffusion models and adapters with decoupled consistency learning, 2024.
  * [37] Wenjing Wang, Huan Yang, Zixi Tuo, Huiguo He, Junchen Zhu, Jianlong Fu, and Jiaying Liu.  Videofactory: Swap attention in spatiotemporal diffusions for text-to-video generation.  ArXiv, 2023.
  * [38] Yuan Wang, W. Zhang, and Peng Chen.  Chinastyle: A mask-aware generative adversarial network for chinese traditional image translation.  SIGGRAPH Asia 2019 Technical Briefs, 2019.
  * [39] Zhong Ling Wang, Jie Zhang, Zhilong Ji, Jinfeng Bai, and S. Shan.  Cclap: Controllable chinese landscape painting generation via latent diffusion model.  2023 IEEE International Conference on Multimedia and Expo (ICME), 2023.
  * [40] Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, and Xiangyu Zhang.  Lamp: Learn a motion pattern for few-shot-based video generation.  ArXiv, 2023.
  * [41] Jinbo Xing, Menghan Xia, Yuxin Liu, Yuechen Zhang, Yong Zhang, Yin-Yin He, Hanyuan Liu, Haoxin Chen, Xiaodong Cun, Xintao Wang, Ying Shan, and Tien-Tsin Wong.  Make-your-video: Customized video generation using textual and structural guidance.  IEEE Transactions on Visualization and Computer Graphics, 2023.
  * [42] Alice Xue.  End-to-end chinese landscape painting creation using generative adversarial networks.  2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 2020.
  * [43] Ting Yao, Yiheng Zhang, Zhaofan Qiu, Yingwei Pan, and Tao Mei.  Seco: Exploring sequence supervision for unsupervised representation learning.  In AAAI Conference on Artificial Intelligence, 2020.
  * [44] Shaozu Yuan, Aijun Dai, Zhiling Yan, Ruixue Liu, Meng Chen, Baoyang Chen, Zhijie Qiu, and Xiaodong He.  Learning to generate poetic chinese landscape painting with calligraphy.  2022\.
  * [45] Zhihang Zhong, Gurunandan Krishnan, Xiao Sun, Yu Qiao, Sizhuo Ma, and Jian Wang.  Clearer frames, anytime: Resolving velocity ambiguity in video frame interpolation, 2023.
  * [46] Aven Le Zhou, Qiu-Feng Wang, Kaizhu Huang, and Cheng-Hung Lo.  An interactive and generative approach for chinese shanshui painting document.  2019 International Conference on Document Analysis and Recognition (ICDAR), 2019.
  * [47] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  ArXiv, 2022.