# An Interactive Decision Support System for Analyzing Time Related
Restrictions in Renaturation and Redevelopment Planning Projects

Y. Annanias10000-0003-2259-1254  C. Meinecke10000-0002-5637-9975 and D.
Wiegreffe10000-0002-5356-6732
1Leipzig University  Image and Signal Processing Group  Germany

###### Abstract

The operation of open-cast lignite mines is a large intervention in nature,
making the areas uninhabitable even after closing the mines without
renaturation processes. Renaturation of these large areas requires a regional
planning process which is tied to many conditions and restrictions, such as
environmental protection laws. The related information is available only as
unstructured text in a variety of documents. Associated temporal aspects and
the geographical borders to these textual information have to be linked
manually so far. This process is highly time-consuming, error-prone, and
tedious. Therefore, the knowledge of experts is often used, but this does not
necessarily include all the relevant information. In this paper, we present a
system to support the experts in decision-making of urban planning,
renaturation, and redevelopment projects. The system allows to plan new
projects, while considering spatial and temporal restrictions extracted from
text documents. With this, our presented system can also be used to verify
compliance with certain legal regulations, such as nature conservation laws.

<ccs2012> <concept> <concept_id>10010405.10010497.10010498</concept_id>
<concept_desc>Applied computing Document searching</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10010405.10010497.10010500</concept_id> <concept_desc>Applied
computing Document management</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10003120.10003145.10003151</concept_id> <concept_desc>Human-
centered computing Visualization systems and tools</concept_desc>
<concept_significance>500</concept_significance> </concept> <concept>
<concept_id>10003120.10003145.10003147.10010887</concept_id>
<concept_desc>Human-centered computing Geographic visualization</concept_desc>
<concept_significance>500</concept_significance> </concept> </ccs2012>

[500]Applied computing Document searching [500]Applied computing Document
management [500]Human-centered computing Visualization systems and tools
[500]Human-centered computing Geographic visualization

##  1 Introduction

Regional planning processes are complex and often require a large amount of
data of different types. This is especially true for This is particularly
applicable to the renaturation of larger areas , but also and also holds true
for urban planning and redevelopment. The data generated by the expert reports
must then be linked to additional data such as sensor data or time points in
order to be able to draw conclusions. For example, there are restrictions that
can lead to delays because certain activities are not permitted by law at
certain times. These include, for example, periods where tree pruning is not
allowed, or various bird nesting periods. During bird nesting periods, any
activity that would disturb the animals is prohibited. Therefore, knowing the
specific restrictions, checking them, and being able to communicate them to
others, such as project managers and the public, is essential for project
planning.

In a previous work in the context of regional planning processes, we developed
an interactive decision support system that works with large amounts of
geographical data and takes care of the representation and analysis of
overlapping areas [YA_IDS]. The system also supports the analysis and linkage
of weather-related restrictions, such as the amount of precipitation per area
and weather warnings [annanias2022interactive].

Here, we extend this work to the aspect of text documents that have temporal
restrictions, such as breeding times, and show their influence on the planning
of the execution of construction processes. For this purpose, we digitally
process the text documents and list the corresponding restrictions. Then, we
support the search for corresponding documents with such restrictions by a
timeline visualization. In addition, new projects can be planned that keep
track of and display existing restrictions. For example, a new bike trail
bicycle path can be planned by drawing a path or a new area on the map. Figure
1 provides an overview of the system and the extensions.

Our project partner is the Lausitzer und Mitteldeutsche
Bergbauverwaltungsgesellschaft mbh (LMBV), which is mainly concerned with the
renaturation of open-cast mining areas in the eastern part of Germany.
Landscape redesign planning and implementation plays a major role in this
process, which involves the management of large amounts of georeferenced text
data. Part of the renaturation is the recultivation of the flora and fauna in
the affected areas, but also the settlement of residential areas and the
provision of industrial areas.

Our contributions are summarized as follows:

  * •

Support in searching for temporal restrictions in text documents (timeline
view)

  * •

Support in the planning of new projects while taking into account such
restrictions (polygon drawing)

  * •

Support in monitoring existing projects and checking for compliance with the
restrictions (brushing and linkage of views)

This introduction is followed by a description of the data and the necessary
preprocessing steps that will be used to fulfill the tasks and the resulting
requirements for the system described afterwards. This overview is followed by
related works with a focus on geospatial and temporal visualization systems.
Next, we describe our implemented solution and give an example use case.
Finally, limitations are considered, and a conclusion is drawn.

\begin{overpic}[width=433.62pt]{figures/mainTool.png} \put(2.0,46.4){{{a)}}}
\put(44.0,46.4){{{b)}}} \put(84.0,46.4){{{c)}}} \put(2.0,22.4){{{d)}}}
\end{overpic} Figure 1:  The main decision support system with multiple views.
(a) and (b) are two linked maps showing the same section. In (b), polygons of
the category “active dismantling” are displayed, whereby one was selected
(orange border). By brushing, this polygon is also displayed in (a). In (a), a
new project is planned by drawing a new polygon. (c) is the information panel,
which lists important documents and their contents, as well as a list of all
polygons that overlap with the selected one. (d) is a timeline visualization
that shows how many documents are available in the displayed area.

##  2 Data & Data Preprocessing

For the implementation of the project, the LMBV provided us with a test data
set. This test data set consists mainly of more than 60 gigabytes of text
documents with georeferences, including reports on the general condition of
the areas and the current forest stand, legal opinions and planning
requirements, soil surveys, and listings of animal and plant species typically
found in the region. The georeferences are provided as point and area
descriptions through polygon data.

###  2.1 Information Extraction

These documents are prepared by the experts of the LMBV or were created by
external experts on behalf of the LMBV in advance of planning processes and
contain, among other things, legal restrictions and other limitations that
must be adhered to. In order to be able to find restrictions regarding bird
breeding times without having to read all the documents completely, digital
processing of the documents is necessary. In a previous work, Schröder et al.
[schroder2021mining] process processed these data using optical character
recognition (OCR) and a subsequent classification of the extracted sentences
using an active learning approach. The classification divides the extracted
restrictions into “Prohibitions” (to refrain from an action in general) and
“Requirements” (to refrain from an action under certain conditions).

This approach can be easily adapted by means of suitable keywords and a new
training, to search for restrictions concerning the topic of breeding times.
Accordingly, this type of restriction falls into the category of
“Requirements” because only certain actions are prohibited in a certain period
of time. In this way, we obtain all documents and, respectively, all sentences
from the documents that match the topic.

However, some of these restrictions are valid only within a specified time
frame. This information is most often contained directly in the extracted
sentences. Using a rule-based named entity recognition (NER), the time data
was also extracted and stored with reference to the sentence. The rules were
based on patterns such as date specifications (DD.MM.YYYY), mention of a month
surrounded by numbers (February 2023), or other typical terms for dates (year
2021), as well as a variety of combinations thereof. Since not all sentences
contain a direct reference to the appropriate time period, the sentences had
to be manually annotated with a time reference.

All documents are linked to the areas where these restrictions apply, which
means that all documents are geographically referenced. Therefore, all areas
are available as additional polygons, which can be displayed on a map using a
geographic information system (GIS).

###  2.2 Data Linkage

In order to quickly find the data and keep the response times of the database
short, the data must be efficiently linked and stored. In our previous work,
we used a Neo4j graph database for this, where polygons and documents are each
stored as nodes and linked to each other via an edge. Additionally, the
polygons are intersected with each other, and if an overlap exists, both
polygon nodes are connected via an edge. This is useful because the
restrictions of the documents of both polygons apply equally in the
overlapping area. In addition, in the case of a new area, the overlaps can be
used to determine what kind of restrictions are expected.

Similarly, each restriction class is represented by a node, so for our current
problem, the node “Breeding Times” was added (see Figure 2). Each extracted
and classified sentence that refers to a breeding time is created as an edge
between the document in which it is located and the breeding time node. Such
an edge contains the associated sentence, as well as the start and end time
for the time constraint, as attributes. Thus, a query via with the node
Breeding Times immediately returns all areas for which such a restriction
exists; at the same time, the result set can also be filtered for the by
geographical location. Then, the result can also be sorted and interpreted
accordingly via the time attributes at the edges of the sentences.

Figure 2:  A section of the graph data model showing two polygon nodes, each
with a document node, and two restriction type nodes. The upper document node
contains two sentences regarding breeding times, the attributes such as the
corresponding sentence as well as the time points of this restriction are
stored at the corresponding edge.

##  3 Tasks & Requirements

###  3.1 Task

A typical domain-specific task is to determine what temporal constraints exist
in a specific area under study and to check whether they are valid for a given
point in time. Currently, restrictions have to be manually searched in
documents, which is tedious and time consuming. In some cases, the experts’
experiences and knowledge are also used, which can mean that some restrictions
are not taken into account. In particular, experts who are new to the company
cannot draw on this knowledge.

When planning new projects, it is also interesting to include existing
projects within the same area. Because the restrictions that apply to one area
automatically also apply in the overlapping section of the other area. In this
way, problems that can occur or that are to be expected due to legal
restrictions can be examined in advance.

###  3.2 Requirements

In consultation with our experts, it is necessary that the resulting system
satisfies the following requirements to support domain-specific tasks and
facilitate planning processes:

  * R1

Time restrictions in documents must be found efficiently and related documents
should be listed.

  * R2

The creation of new projects, by drawing polygon areas or paths, should be
supported.

  * R3

For new projects, it must be possible to check whether existing documents
contain restrictions that are also valid for the new project.

##  4 Related Work

Our system is related to geographic information systems (GIS) used by experts
to analyze map-based data and support decisions regarding environmental
problems. For this, we combine geospatial and temporal visualization methods
to support decision-making in renaturation processes and urban planning.

Another related area is the visual analysis of legal documents
[resck2022legalvis, lettieri2017legal].

###  4.1 Geographic Information System

Geographic information systems (GIS) can provide many ways to measure,
interpret, visualize, generalize, and interpolate collected samples
[Goodchild:2007]. Specifically, spatial interpolation techniques are used to
complete those spatial locations where measurements are not available or
missing. GIS have penetrated a variety of fields supporting experts in
decision-making and are successfully used to create continuous surfaces in,
e.g., meteorology, climatology, criminology, and water management. They can
combine textual information from the database with spatial data to overcome
the difficulties of many spatial decision-making processes. The main challenge
of GIS has been building a database from unstructured data sets, e.g., reports
or media in a variety of data formats, which are now vastly available.

A recent toolbox of commercial ARCGIS software called LocateXT [LocateXT] has
enabled connecting a large number of unstructured data sets in different
formats (e.g., pdf, ppt, doc) with GIS. The specific properties of this
toolbox include extracting latitude and longitude data from the document,
creating a layer based on those, tagging specific properties (i.e., mosque,
parks, etc.), data scraping, queries, and more. Howell et al. [Howell:2019]
used the toolbox for the parsing, processing, and visualization of scientific
articles. The tool is not applicable to our project, as it provides point
coordinates, and we need area information. Furthermore, we already have
location information from our collaboration partner. However, the toolbox does
not include advanced interaction mechanisms, such as brushing or highlighting.
Finally, we provide visualization and interaction facilities that are missing
from the toolbox.

###  4.2 Geovisualization

Next to GIS, other Geovisualization tools were used for different aspects of
environmental data to support decision-making. An example is visualization in
remedial programs, to document, represent, and communicate three-dimensional
environmental data to both professionals and the general public
[ling2014environmental]. Li et al. [li2017visual] proposed a visualization
tool for the analysis and decision-making of flood risk in cultural heritage
using a GIS to show flood peak flow. Lei et al. present a geovisualization
tool to explore the results of climate simulation in combination with
uncertain population growth and water availability based on historical trends
in the Niger River basin [lei2015interactive]. Lukasczyk et al.
[kolditz2014webgl] use three-dimensional geovisualization to show energy
consumption to support decision-making. Eighausen et al.
[eligehausen2013blendgis] visualize spatial ecology-related data created with
QuantumGIS in a tool to increase social acceptance in public participation
processes for public-funded ecosystem restoration measures and research
programs. In contrast to these works, we focus on communicating legal
restrictions regarding the environment in renaturation and urban planning
projects.

###  4.3 Visualization of Spatial-Temporal Data

Andrienko et al. [Andrienko2003] provide a list of visualization-based
techniques that allow For renaturation and redevelopment planning projects,
the visualization of geospatial and temporal data is important. For the
exploratory analysis of spatio-temporal data. , several visualization
techniques were proposed, an overview is given by Andrienko et al.
[Andrienko2003]. GeoTemCo [janicke2013geotemco] allows geospatial and temporal
comparison of multiple data sets using a dynamic delaunay triangulation to
merge glyphs on the map to avoid visual clutter and was previously applied to
data from biodiversity, climatology, and geophysics [janicke2014utilizing].
Schlegel et al. [schlegel2013determining] analyze spatio-temporal patterns of
hydrological parameters relevant for flooding.

Outside of environmental science, systems have been proposed for the analysis
of spatial-temporal data for crime analysis [roth2010user] and the exploration
and discovery of web information [dork2008visgets]. Several works propose
spatial-temporal visualizations for social media, news and topic analysis
[li2018visual, sheidin2017time, he2016spatiotemporal, zhang2016visual,
kraft2013less]. Hao et al. visualize customer feedback using sentiment
analysis and tag maps [hao2013visual], while Reckziegel and Jänicke propose a
visualization method for time varying time-varying predominance tag maps
[reckziegel2019time]. In contrast to these works, our scope lies on legal
restrictions in the text documents and not on other aspects such as topics or
frequent words.

##  5 Visualization and Interaction

For the design of the visualization system, we follow the Visual Information
Seeking Mantra [shneiderman2003eyes] and the Guidelines for Using Multiple
Views in Information Visualization [wang2000guidelines], as this was also
applied in the underlying main system. Additionally, we refer to the taxonomy
or mechanisms for concurrent tool linking (multiple displays showing data) of
Andrienko and Andrienko [Andrienko2006]. According to their taxonomy, several
displays (maps) should show the information for a selected part of the data
with important interaction methods for coordination like such as highlighting,
focusing, or zooming, and filtering. Brushing can also play an important role
in this context. They further specify that a change to one display must result
in a change to all other linked displays.

###  5.1 Tool Overview

Considering the aforementioned design guidelines, the main system we presented
in [YA_IDS] uses several views (multiple maps and an information panel)
arranged in a grid, where all views are linked to each other. At first, only
one map is displayed, which shows the relevant section of a certain region,
adjustable by zooming and panning with a panel on the right that lists the
associated text data. If needed, other maps can be added showing the same or a
different section, and, if necessary, other data sets, allowing comparison of
different data sets with as little disturbing overlap as possible. To maintain
focus, all maps are linked together. In this way, the displayed sections of a
map can be linked so that zooming and panning have the same effect on all
maps. In addition, the area displayed on one map can be used as a geographic
location filter for data selection in other maps and views. Furthermore,
brushing is used to display selected elements in the other views accordingly.

In this work, we added the temporal aspect through a timeline visualization
showing temporal restrictions in documents. Further, we include the
possibility to create new projects by drawing polygon areas, while keeping
track of existing legal restrictions. In the same way, additional maps can be
attached, and a timeline view can be added, which is responsible for filtering
documents according to certain restrictions, categories, and points in time
(as shown in Figure 1). All views are arranged according to a grid and
resized. However, the timeline always takes up occupies the entire bottom row
to have as much space as possible for the visualization.

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=433.62pt]{figures/draw.png} \put(1.0,39.0){{{a)}}}
\put(51.0,39.0){{{b)}}} \end{overpic} Figure 3:  (a) A polygon is drawn
directly onto the map, constructing it from individual points. (b) When the
path is closed, the polygon can be saved in a suitable category.

###  5.2 Map

The polygons are divided into different categories depending on the associated
documents. Polygons of a category can be displayed independently of each other
on the map. Each category is assigned a color, and the polygons are displayed
slightly transparent so that the underlying section of the map remains
visible. To prevent too much color distortion, the map is only displayed in
white and gray tones. The restriction classes were also used as classes, so
that all polygons can be directly displayed that are linked to a document that
has restrictions regarding breeding times, for example.

For planning new projects, our extension offers the possibility to draw new
polygons directly on the map (Figure 3), which supports R2. To do this, the
planning expert places one point after the other on the map, which are
automatically connected with an edge. In this way, the entire polygon can be
defined through the corner points. Once the area has been delimited, the
expert selects a category to which the new polygon should be added. Then, the
new polygon is automatically processed, intersected with the other polygons,
and saved in the database. Finally, through the linkage of the views, the
information panel on the right lists the other polygons with which the newly
created polygon intersects and which documents or restrictions apply to them.
Moreover, if the polygon is selected, it is also displayed with an orange
border on all other maps, making it clear where it is located in the shown
area through brushing (R3).

\begin{overpic}[trim=0.0pt 0.0pt 71.13188pt
0.0pt,clip,width=433.62pt]{figures/timeline-months.png}
\put(-4.0,26.0){{{a)}}}
\end{overpic}\begin{overpic}[width=433.62pt]{figures/timeline-days.png}
\put(-4.0,26.0){{{b)}}}
\end{overpic}\begin{overpic}[width=433.62pt]{figures/timeline-final.png}
\put(-4.0,26.0){{{c)}}} \end{overpic}

Figure 4:  The timeline visualization with different level-of-details and
applied filters. (a) shows several months in between 2020 and 2023. (b) shows
a part of the same interval on a daily-level, revealing that almost all of the
documents for March 2022 are related to the 4th March. (c) shows the same
interval as (a) when using the “Requirement” environmental protection as
mandatory.

###  5.3 Timeline

The timeline view in Figure 4 shows the number of documents associated with a
specific date through green bars. On the x-axis, the time range is displayed,
and the width of a bar changes based on the provided space and the current
level of detail. On the y-axis, the number of documents is mapped on a linear
scale between 0 and the highest number in the data set. Based on the selected
level of detail, the dates in the documents are counted for each bar. The
visualization allows for the selection of a specific bar and brushing of a
time interval. This allows to explore all documents associated to the
corresponding time range in an information panel and to view the associated
polygons on a map.

It is also possible to filter the timeline on the basis of the current
viewport of a map view through the linkage of the views. This allows a domain
expert to focus on one area of interest when planning new projects.
Furthermore, the timeline supports zooming into a specific time range, thus
changing the current level of detail. Based on the zoom level, the bars in the
timeline represent a decade, year, quarter, months, or a day. Figure 4a-b
shows examples for months and days. This enables a domain expert to focus on
high-level patterns, such as yearly or monthly restrictions, or to look into a
specific day of interest. It is also possible to filter the timeline based on
a time range, restrictions, or to combine these filters with the viewport
filter. The filtering of specific restrictions eases the decision-making
processes by showing only the relevant time data for a specific domain
question, and thus supports R1. Figure 4c shows a filtered version of the
timeline in Figure 4a for the restriction environmental protection.

##  6 Use Case

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=433.62pt]{figures/usecasePath.png} \put(-4.0,64.0){{{a)}}}
\end{overpic}\begin{overpic}[trim=2.00749pt 2.00749pt 2.00749pt
0.0pt,clip,width=433.62pt]{figures/usecaseInfopanel.png}
\put(-4.0,97.0){{{b)}}} \end{overpic}

Figure 5:  (a) A new project, a bicycle path, is drawn over the map and then
saved in the database. (b) Then other projects are listed in the information
panel that overlap with the newly created one. Here, a nature reserve is
listed, which is subject to various restrictions, including two from the
category of breeding times.  \begin{overpic}[trim=0.0pt 0.0pt 0.0pt
0.0pt,clip,width=433.62pt]{figures/usecase2.png} \put(2.0,52.9){{{a)}}}
\put(78.0,52.9){{{b)}}} \put(2.0,24.7){{{c)}}} \end{overpic} Figure 6:  (c) In
the timeline view, the documents are filtered according to the displayed
region in the map and the Requirement “breeding times”. The time range from
March to September was selected. (a) Through brushing, the associated projects
of the selected range are displayed on the map. (b) They are additionally
listed with further information in the information panel. A selected project
states that certain noisy activities are apparently not permitted in the
associated wildlife park during the specified time.

Refer to the previously described task, where a new project is to be planned
along existing ones. In this use case , This use case is about the planning of
a bicycle path is to be planned. Along the planned routein the region around
Markkleeberg, which is south of Leipzig. The area is also known as the
“Leipziger Neuseenland” (New Lakeland) and was created by the renaturation of
several open-cast mining areas. As a result, there are already other projects
existing projects along the planned route that are subject to certain
restrictions. If their area overlaps with that of the planned path, these
would also apply to the new project. To find out which existing restrictions
could be relevant and which are already known, an expert simply draws the bike
bicycle path to be planned on the map in our GIS tool (R2, R3, Figure 5a).

Once the path has been planned, the expert selects a category in which the
path should be saved, and then our tool calculates all overlaps with other
areas in this area. Due to the structure in the database, it only takes a few
seconds until the result is displayed in the information panel on the right.
All results listed there are areas from other projects that intersect the
bicycle path. From these areas, the already known restrictions to the
associated projects are listed (R3).

For example, there are restrictions on nature conservation, for instance, some
areas may not be entered or damaged, and building waste may not be stored
there. However, in particular, there are also restrictions on the subject of
breeding times (R1, Figure 5b). The expert now knows that in some months the
construction project may only take place under special conditions or, in some
cases, may not take place at all, as noise would disturb the animals, and
construction vehicles are thus also not allowed to be active there. With this
knowledge, the expert can now better time the path and the aspects necessary
for it.

Once planned, the project is not finished yet. Further planning is necessary,
and projects need to be monitored for compliance with certain regulations,
including temporal restrictions (R1). For this purpose, existing projects can
be displayed (see Figure 6). To do this, an expert centers the map in a
specific region. Then, the expert displays the timeline and filters the
projects listed there according to the centered region and subsequently to the
breeding times category (Figure 6c). In this way, the tool filters all
existing projects accordingly and shows which projects and especially when
these projects are subject to time restrictions on breeding times. The example
shows a project that has to observe breeding times from March to September
(Figure 6b). As a consequence, large construction vehicles may not be used in
the area, for example.

This and also other projects that match the filter are displayed on the map to
get a better overview of exactly which areas are involved (Figure 6a). With
this information, the expert can now plan the exact resources of these
projects, so that construction machines and vehicles can be used elsewhere, as
they are not allowed here at this time anyway. The planning is thus more
precise and efficient. However, the expert can also check whether these
projects adhere to the restrictions and initiate appropriate corrective
actions.

##  7 Limitations & Future Work

The presented system can support decision-making in urban planning,
renaturation, and redevelopment projects, but there are still some limitations
that can be addressed in the future. Sentences regarding breeding times can be
found well, but some of them lack time specifications. Often these are not
given directly in the sentence but can be obtained in the context of the
paragraph, or they can also result from the named bird species, as these
always breed at certain times, and thus the time specifications are not
explicitly given. In all these cases, the extracted data had to be manually
linked to their temporal aspects. A major simplification of this work can be
achieved by adapting the information extraction to this and examining the
context for exactly such temporal aspects. Furthermore, the including of
uncertainties based on the extraction process and the correction, or labeling
of entries could be helpful for the domain experts.

So far, we have only referred to bird breeding times. However, the tool could
also be extended to other categories, such as plant pruning times, flowering
times, and pollen flight times. Currently, we only focus on a subset of
documents in the Lausitzer und Mitteldeutsche region, by combining different
data sources and using data from different regions, the system could help
other communities in regional planning.

##  8 Conclusion

We present a system to support decision-making in urban planning,
renaturation, and redevelopment projects. Legal restrictions and other
limitations were extracted from georeferenced documents and linked to their
temporal facet. This means that it is no longer necessary to manually sift
through vast quantities of text documents in search of these restrictions. The
tool allows to create new polygons to plan new projects, such as bike
trailsbicycle path, while considering spatial and temporal restrictions
(“Prohibitions” and “Requirements”), like bird breeding or tree pruning
periods.

For this, a map is linked to a timeline visualization. By using this timeline,
existing projects can be filtered and found according to the temporal
component. This makes it easier to plan when and where resources are needed
for specific projects. With this, our presented system can also be used to
verify compliance with certain legal and nature conservation regulations. In
summary, the use cases show that the tasks posed at the beginning can be
accomplished with our system. In addition, existing knowledge no longer has to
rely solely on the memory of certain experts, but this knowledge has been made
explicit and discoverable for all, making the system useful for our project
partner, the LMBV.

## Acknowledgment

This research was supported by the Development Bank of Saxony (SAB) under
Grant 100400221. We thank our project partner the Lausitzer und Mitteldeutsche
Bergbauverwaltungsgesellschaft mbh (LMBV) for their support and for providing
their data.


# ConditionVideo: Training-Free Condition-Guided Text-to-Video Generation

Bo Peng1,2 Xinyuan Chen1 Yaohui Wang1 Chaochao Lu1 Yu Qiao1

###### Abstract

Recent works have successfully extended large-scale text-to-image models to
the video domain, producing promising results but at a high computational cost
and requiring a large amount of video data. In this work, we introduce
ConditionVideo, a training-free approach to text-to-video generation based on
the provided condition, video, and input text, by leveraging the power of off-
the-shelf text-to-image generation methods (_e.g.,_ Stable Diffusion).
ConditionVideo generates realistic dynamic videos from random noise or given
scene videos. Our method explicitly disentangles the motion representation
into condition-guided and scenery motion components. To this end, the
ConditionVideo model is designed with a UNet branch and a control branch. To
improve temporal coherence, we introduce sparse bi-directional spatial-
temporal attention (sBiST-Attn). The 3D control network extends the
conventional 2D controlnet model, aiming to strengthen conditional generation
accuracy by additionally leveraging the bi-directional frames in the temporal
domain. Our method exhibits superior performance in terms of frame
consistency, clip score, and conditional accuracy, outperforming compared
methods. See the project website at
https://pengbo807.github.io/conditionvideo-website/.

Figure 1: Our training-free method generates videos conditioned on different
inputs. In (a), the illustration showcases the process of generation using
provided scene videos and pose information, with the background wave
exhibiting a convincingly lifelike motion. (b), (c), and (d) are generated
based on condition only, which are pose, depth, and segmentation,
respectively.

##  1 Introduction

Diffusion-based models (Song, Meng, and Ermon 2021; Song et al. 2021; Ho,
Jain, and Abbeel 2020; Sohl-Dickstein et al. 2015) demonstrates impressive
results in large-scale text-to-image (T2I) generation (Ramesh et al. 2022;
Saharia et al. 2022; Gafni et al. 2022; Rombach et al. 2022). Much of the
existing research proposes to utilize image generation models for video
generation. Recent works (Singer et al. 2023; Blattmann et al. 2023; Hong et
al. 2022) attempt to inflate the success of the image generation model to
video generation by introducing temporal modules. While these methods reuse
image generation models, they still require a massive amount of video data and
training with significant amounts of computing power. Tune-A-Video (Wu et al.
2022b) extends Stable Diffusion (Rombach et al. 2022) with additional
attention and a temporal module for video editing by tuning one given video.
It significantly decreases the training workload, although an optimization
process is still necessary. Text2Video (Khachatryan et al. 2023) proposes
training-free generation, however, the generated video fails to simulate
natural background dynamics. Consequently, the question arises: How can we
effectively utilize image generation models without any optimization process
and embed controlling information as well as modeling dynamic backgrounds for
video synthesis?

In this work, we propose ConditionVideo, a training-free conditional-guided
video generation method that utilizes off-the-shelf text-to-image generation
models to generate realistic videos without any fine-tuning. Specifically,
aiming at generating dynamic videos, our model disentangles the representation
of motion in videos into two distinct components: conditional-guided motion
and scenery motion, enabling the generation of realistic and temporally
consistent frames. By leveraging this disentanglement, we propose a pipeline
that consists of a UNet branch and a control branch, with two separate noise
vectors utilized in the sampling process. Each noise vector represents
conditional-guided motion and scenery motion, respectively. To further enforce
temporal consistency, we introduce sparse bi-directional spatial-temporal
attention (sBiST-Attn) and a 3D control branch that leverages bi-directional
adjacent frames in the temporal dimension to enhance conditional accuracy.
These components strengthen our model’s ability to generate high-quality
conditional-guided videos. Our ConditionVideo method outperforms the baseline
methods in terms of frame consistency, conditional accuracy, and clip score.

Our key contributions are as follows. (1) We propose ConditionVideo, a
training-free video generation method that leverages off-the-shelf text-to-
image generation models to generate conditional-guided videos with realistic
dynamic backgrounds. (2) Our method disentangles motion representation into
conditional-guided and scenery motion components via a pipeline that includes
a U-Net branch and a conditional-control branch. (3) We introduce sparse bi-
directional spatial-temporal attention (sBiST-Attn) and a 3D conditional-
control branch to improve conditional accuracy and temporal consistency,
generating high-quality videos that outperform compared methods.

##  2 Related work

###  2.1 Diffusion Models

Image diffusion models have achieved significant success in the field of
generation (Ho, Jain, and Abbeel 2020; Song, Meng, and Ermon 2021; Song et al.
2021), surpassing numerous generative models that were once considered state-
of-the-art (Dhariwal and Nichol 2021; Kingma et al. 2021). With the assistance
of large language models (Radford et al. 2021; Raffel et al. 2020), current
research can generate videos from text, contributing to the prosperous of
image generation (Ramesh et al. 2022; Rombach et al. 2022).

Recent works in video generation (Esser et al. 2023; Ho et al. 2022b; Wu et
al. 2022b, 2021, a; Hong et al. 2022; Wang et al. 2023b, c) aim to emulate the
success of image diffusion models. Video Diffusion Models (Ho et al. 2022b)
extends the UNet (Ronneberger, Fischer, and Brox 2015) to 3D and incorporates
factorized spacetime attention (Bertasius, Wang, and Torresani 2021). Imagen
Video (Saharia et al. 2022) scales this process up and achieves superior
resolution. However, both approaches involve training from scratch, which is
both costly and time-consuming.

Alternative methods explore leveraging pre-trained text-to-image models. Make-
A-Video (Singer et al. 2023) facilitates text-to-video generation through an
expanded unCLIP framework. Tune-A-Video (Wu et al. 2022b) employs a one-shot
tuning pipeline to generate edited videos from input guided by text. However,
these techniques still necessitate an optimization process.

Compared to these video generation methods, our training-free method can yield
high-quality results more efficiently and effectively.

###  2.2 Conditioning Generation

While GAN-based methods have made considerable progress in conditional
generation (Mirza and Osindero 2014; Wang et al. 2018; Chan et al. 2019; Wang
et al. 2019; Liu et al. 2019; Siarohin et al. 2019; Zhou et al. 2022; WANG et
al. 2020; Wang et al. 2020, 2022), research on the conditional generation of
diffusion models is limited. For the diffusion model-based methods, T2I
Adapter (Mou et al. 2023) and ControlNet (Zhang and Agrawala 2023) aim to
enhance controllability through the use of extra annotations. T2IAdapter (Mou
et al. 2023) proposes aligning internal knowledge with external control
signals to facilitate image generation. On the other hand, ControlNet
duplicates and fixes the original weight of the large pre-trained T2I model.
Utilizing the cloned weight, ControlNet trains a conditional branch for task-
specific image control.

For diffusion-based conditional video generation, recent works have centered
on text-driven video editing (Molad et al. 2023; Esser et al. 2023; Ceylan,
Huang, and Mitra 2023; Liu et al. 2023a; Wang et al. 2023a; Qi et al. 2023).
These methods prioritize similarity with the input video rather than creating
new content. In contrast, our method uses dynamic features from the input
reference video for more creative generation and can add extra foreground
movements.

Figure 2: Illustration of our proposed training-free pipeline. (Left) Our
framework consists of a UNet branch and a 3D control branch. The UNet branch
receives either the inverted reference video $z_{T}^{INV}$ or image-level
noise $\epsilon_{b}$ for background generation. The 3D control branch receives
an encoded condition for foreground generation. The text description is fed
into both branches. (Right) Illustration of our basic spatial-temporal block.
We employ our proposed sBiST-Attn module into the basic block between the 3D
convolution block and the cross-attention block. The detail of sBiST-Attn
module is shown in Fig. 3

Concurrent works like Follow-Your-Pose (Ma et al. 2023) and Text2Video-Zero
(Khachatryan et al. 2023) generate videos based on given conditions. However,
these methods still require training or have difficulty in generating
realistic background movements (_e.g.,_ the flow of the waves in Fig. 1 (a).
The dynamic version on our website can better show the advantages of our
method.). Moreover, we propose additional techniques to improve time and
conditioning consistency and introduce dynamic scene referencing, a novel
approach in this field.

##  3 Preliminaries

##### Stable Diffusion.

Stable Diffusion employs an autoencoder (Van Den Oord, Vinyals et al. 2017) to
preprocess images. An image $x$ in RGB space is encoded into a latent form by
encoder $\mathcal{E}$ and then decoded back to RGB space by decoder
$\mathcal{D}$. The diffusion process operates with the encoded latent
$z=\mathcal{E}(x)$.

For the diffusion forward process, Gaussian noise is iteratively added to
latent $z_{0}$ over $T$ iterations (Ho, Jain, and Abbeel 2020):

$\begin{split}q\left(z_{t}\mid
z_{t-1}\right)&=\mathcal{N}\left(z_{t};\sqrt{1-\beta_{t}}z_{t-1},\beta_{t}I\right),\\\
t&=1,2,\ldots,T,\end{split}$ (1)

where $q\left(z_{t}\mid z_{t-1}\right)$ denotes the conditional density
function and $\beta$ is given.

The backward process is accomplished by a well-trained Stable Diffusion model
that incrementally denoises the latent variable $\hat{z_{0}}$ from the noise
$z_{T}$. Typically, the T2I diffusion model leverages a UNet architecture,
with text conditions being integrated as supplementary information. The
trained diffusion model can also conduct a deterministic forward process,
which can be restored back to the original $z_{0}$. This deterministic forward
process is referred to as DDIM inversion (Song, Meng, and Ermon 2021; Dhariwal
and Nichol 2021). We will refer to $z_{T}$ as the noisy latent code and
$z_{0}$ as the original latent in the subsequent section. Unless otherwise
specified, the frames and videos discussed henceforth refer to those in latent
space.

##### ControlNet.

ControlNet (Zhang and Agrawala 2023) enhances pre-trained large-scale
diffusion models by introducing extra input conditions. These inputs are
processed by a specially designed conditioning control branch, which
originates from a clone of the encoding and middle blocks of the T2I diffusion
model and is subsequently trained on task-specific datasets. The output from
this control branch is added to the skip connections and the middle block of
the T2I model’s UNet architecture.

##  4 Methods

ConditionVideo leverages guided annotation, denoted as $Condtion$, and
optional reference scenery, denoted as $Video$, to generate realistic videos.
We start by introducing our training-free pipeline in Sec. 4, followed by our
method for modeling motion in Sec. 4.2. In Sec. 4.3, we present our sparse bi-
directional spatial-temporal attention (sBiST-Attn) mechanism. Finally, a
detailed explanation of our proposed 3D control branch is provided in Sec.
4.4.

###  4.1 Training-Free Sampling Pipeline

Fig. 2 depicts our proposed training-free sampling pipeline. Inheriting the
autoencoder $\mathcal{D}(\mathcal{E}(\cdot))$ from the pre-trained image
diffusion model (Sec. 3), we conduct video transformation between RGB space
and latent space frame by frame. Our ConditionVideo model contains two
branches: a UNet branch and a 3D control branch. A text description is fed
into both branches. Depending on the user’s preference for customized or
random background, the UNet branch accepts either the inverted code
$z_{T}^{INV}$ of the reference background video or the random noise
$\epsilon_{b}$. The condition is fed into the 3D control branch after being
added with random noise $\epsilon_{c}$. We will further describe this
disentanglement input mechanism and random noise $\epsilon_{b}$,
$\epsilon_{c}$ in Sec. 4.2.

Our branch uses the original weight of ControlNet (Zhang and Agrawala 2023).
As illustrated on the right side of Fig. 2, we modify the basic spatial-
temporal blocks of these two branches from the conditional T2I model by
transforming 2D convolution into 3D with 1$\times$3$\times$3 kernel and
replacing the self-attention module with our proposed sBiST-Attn module (Sec.
4.3). We keep other input-output mechanisms the same as before.

###  4.2 Strategy for Motion Representation

##### Disentanglement for Latent Motion Representation

In conventional diffusion models for generation (_e.g.,_ ControlNet), the
noise vector $\epsilon$ is sampled from an i.i.d. Gaussian distribution
$\epsilon\sim\mathcal{N}(0,I)$ and then shared by both the control branch and
UNet branch. However, if we follow the original mechanism and let the inverse
background video’s latent code to shared by two branches, we observe that the
background generation results will be blurred (Experiments are shown in the
Appx B.). This is because using the same latent to generate both the
foreground and the background presumes that the foreground character has a
strong relationship with the background. Motivated by this observation, we
explicitly disentangle the video motion presentation into two components: the
motion of the background and the motion of the foreground. The background
motion is generated by the UNet branch whose latent code is presented as
background noise $\epsilon_{b}\sim\mathcal{N}(0,I)$. The foreground motion is
represented by the given conditional annotations while the appearance
representation of the foreground is generated from the noise
$\epsilon_{c}\sim\mathcal{N}(0,I)$.

##### Strategy for Temporal Consistency Motion Representation

To attain temporal consistency across consecutively generated frames, We
investigated selected noise patterns that facilitate the creation of cohesive
videos. Consistency in foreground generation can be established by ensuring
that the control branch produces accurate conditional controls. Consequently,
we propose utilizing our control branch input for this purpose:
$C_{cond}=\epsilon_{c}+\mathcal{E}_{c}(Condition),\epsilon_{c_{i}}\in\epsilon_{c},\epsilon_{c_{i}}\sim\mathcal{N}(0,I)\subseteq\mathbb{R}^{H\times
W\times C},\forall i,j=1,...,F,\quad\epsilon_{ci}=\epsilon_{cj},$ where $H$,
$W$, and $C$ denote the height, width, and channel of the latent $z_{t}$, $F$
represents the total frame number, $C_{cond}$ denotes the encoded conditional
vector which will be fed into the control branch and $\mathcal{E}_{c}$ denotes
the conditional encoder. Additionally, it’s important to observe that
$\epsilon_{c_{i}}$ corresponds to a single frame of noise derived from the
video-level noise denoted as $\epsilon_{c}$. The same relationship applies to
$\epsilon_{b_{i}}$ and $\epsilon_{b}$ as well.

When generating backgrounds, there are two approaches we could take. The first
is to create the background using background noise $\epsilon_{b}$:
$\epsilon_{b_{i}}\in\epsilon_{b},~{}~{}\epsilon_{b_{i}}\sim\mathcal{N}(0,I)\subseteq\mathbb{R}^{H\times
W\times C}\\\ \epsilon_{bi}=\epsilon_{bj},\quad\forall i,j=1,...,F.$ The
second approach is to generate the background from an inverted latent code,
$z_{T}^{INV}$, of the reference scenery video. Notably, we observed that the
dynamic motion correlation present in the original video is retained when it
undergoes DDIM inversion. So we utilize this latent motion correlation to
generate background videos. Our ConditionVideo method is more user-friendly
and cost-efficient compared to techniques that require motion training.

Algorithm 1 Sampling Algorithm

Input: $Condition$, $Text$, $Video$(Optional)
Parameter: $T$
Output: $\hat{X}_{0}$:generated video

1: if $Video$ is not None then

2: $z_{0}^{Video}\leftarrow\mathcal{E}(Video)\quad$ //encode video

3:
$z_{T}^{INV}\leftarrow\text{DDIM\\_Inversion}(z_{0}^{Video},T,\text{UNetBranch})$

4: $z_{T}\leftarrow z_{T}^{INV}\quad$ //customize background

5: else

6: $z_{T}\leftarrow\epsilon_{b},\quad$ //random background

7: end if

8: $C_{cond}\leftarrow\epsilon_{c}+\mathcal{E}_{c}(Condition)\quad$ //encode
condition

9: $C_{text}\leftarrow\mathcal{E}_{t}(Text)$ //encode input prompt

10: for $t=T...1$ do

11: $c_{t}\leftarrow\text{ConrtolBranch}(C_{cond},t,C_{text})$

12:
$\hat{z}_{t-1}\leftarrow\text{DDIM\\_Backward}(z_{t},t,C_{text},c_{t},$$\text{UNetBranch})$

13: end for

14: $\hat{X}_{0}\leftarrow\mathcal{D}(\hat{z}_{0})$

15: return $\hat{X}_{0}$

During the sampling process, in the first forward step $t=T$, we feed the
background latent code $z_{T}^{INV}$ or $\epsilon_{b}$ into the UNet branch
and the condition $C_{cond}$ into our 3D control branch. Then, during the
subsequent reverse steps $t=T-1,..,0$, we feed the denoised latent $z_{t}$
into the UNet branch while still using $C_{cond}$ for 3D control branch input.
The details of the sampling algorithm are shown in Alg. 1

###  4.3 Sparse Bi-directional Spatial-Temporal Attention (sBiST-Attn)

Figure 3: Illustration of ConditionVideo’s sparse bi-directional attention.
The purple blocks signify the frame we’ve selected for concatenation, which
can be computed for key and value. The pink block represents the current block
from which we’ll calculate the query. The blue blocks correspond to the other
frames within the video sequence. Latent features of frame $z_{t}^{i}$, bi-
directional frames $z_{t}^{3j+1},~{}j=0,...,\lfloor(F-1)/3\rfloor$ are
projected to query $Q$, key $K$ and value $V$. Then the attention-weighted sum
is computed based on key, query, and value. Notice that the parameters are the
same as the ones in the self-attention module of the pre-trained image model.

Taking into account both temporal coherence and computational complexity, we
propose a sparse bi-directional spatial-temporal attention (sBiST-Attn)
mechanism, as depicted in Fig. 3. For video latent $z_{t}^{i},~{}i=1,...,F$,
the attention matrix is computed between frame $z_{t}^{i}$ and its bi-
directional frames, sampled with a gap of 3. This interval was chosen after
weighing frame consistency and computational cost (see Appx C.1). For each
$z_{t}^{i}$ in $z_{t}$, we derive the query feature from its frame
$z_{t}^{i}$. The key and value features are derived from the bi-directional
frames $z_{t}^{3j+1},~{}j=0,...,\lfloor(F-1)/3\rfloor$. Mathematically, our
sBiST-Attn can be expressed as:

$\left\\{\begin{aligned}
&\mathrm{Attention}(Q,K,V)=\mathrm{Softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)\cdot
V\\\ &Q=W^{Q}z_{t}^{i},K=W^{K}z_{t}^{[3j+1]},V=W^{V}z_{t}^{[3j+1]},\\\
&j=0,1,\ldots,\lfloor(F-1)/3\rfloor\end{aligned}\right.$ (2)

where [·] denotes the concatenation operation, and $W^{Q},W^{K},W^{V}$ are the
weighted matrices that are identical to those used in the self-attention
layers of the image generation model.

###  4.4 3D Control Branch

Frame-wise conditional guidance is generally effective, but there may be
instances when the network doesn’t correctly interpret the guide, resulting in
an inconsistent conditional output. Given the continuous nature of condition
movements, ConditionVideo propose enhancing conditional alignment by
referencing neighboring frames. If a frame isn’t properly aligned due to weak
control, other correctly aligned frames can provide more substantial
conditional alignment information. In light of this, we design our control
branch to operate temporally, where we choose to replace the self-attention
module with the sBiST-Attn module and inflate 2D convolution to 3D. The
replacing attention module can consider both previous and subsequent frames,
thereby bolstering our control effectiveness.

##  5 Experiments

###  5.1 Implementation Details

We implement our model based on the pre-trained weights of ControlNet (Zhang
and Agrawala 2023) and Stable Diffusion (Rombach et al. 2022) 1.5. We generate
24 frames with a resolution of 512 × 512 pixels for each video. During
inference, we use the same sampling setting as Tune-A-Video (Wu et al. 2022b).

###  5.2 Main results

In Fig. 1, we display the success of our training-free video generation
technique. The generated results from ConditionVideo, depicted in Fig. 1 (a),
imitate moving scenery videos and show realistic waves as well as generate the
correct character movement based on posture. Notably, the style of the
backgrounds is distinct from the original guiding videos, while the motion of
the backgrounds remains constant. Furthermore, our model can generate
consistent backgrounds when sampling $\epsilon_{b}$ from Gaussian noise based
on conditional information, as shown in Fig.1 (b),(c),(d). These videos
showcase high temporal consistency and rich graphical content.

###  5.3 Comparison

#### Compared Methods

We compare our method with Tune-A-Video (Wu et al. 2022b), ControlNet (Zhang
and Agrawala 2023), and Text2Video-Zero (Khachatryan et al. 2023). For Tune-A-
Video, we first fine-tune the model on the video from which the condition was
extracted, and then sample from the corresponding noise latent code of the
condition video.

#### Qualitative Comparison

Our visual comparison conditioning on pose, canny, and depth information is
presented in Fig. 4, 5, and 6. Tune-A-Video struggles to align well with our
given condition and text description. ControlNet demonstrates improvement in
condition-alignment accuracy but suffers from a lack of temporal consistency.
Despite the capability of Text2Video to produce videos of exceptional quality,
there are still some minor imperfections that we have identified and indicated
using a red circle in the figure. Our model surpasses all others, showcasing
outstanding condition-alignment quality and frame consistency.

Figure 4: Qualitative comparison condition on the pose. “The Cowboy, on a
rugged mountain range, Western painting style”. Our results outperform in both
temporal consistency and pose accuracy, while others have difficulty in
maintaining either one or both of the qualities. Figure 5: Qualitative
comparison condition on canny. “A man is runnin”. Tune-A-Video fails in canny-
alignment. ControlNet generates low temporal consistent frames. Although
Text2Video outperforms the first two methods, it generates parts of the legs
that do not correspond to the real structure of the human body and the shoes
are not the same color. Figure 6: Qualitative comparison condition on depth.
“ice coffee”. All three methods used for comparison have the problem of
changing the appearance of the object when the viewpoint is switched, and only
our method ensures the consistency of the appearance before and after.

#### Quantitative Comparison

Method | FC(%) | CS | PA (%)
---|---|---|---
Tune-A-Video | 95.84 | 30.74 | 26.13
ControlNet | 94.22 | 32.97 | 79.51
Text2Video-Zero | 98.82 | 32.84 | 78.50
Ours | 99.02 | 33.03 | 83.12
Table 1: Quantitative comparisons condition on pose. FC,CS,PA represent frame consistency, clip score and pose-accuracy, respectively Method | Condition | FC(%) | CS
---|---|---|---
Tune-A-Video | - | 95.84 | 30.74
ControlNet | Canny | 90.53 | 29.65
Text2Video-Zero | Canny | 97.44 | 28.76
Ours | Canny | 97.64 | 29.76
ControlNet | Depth | 90.63 | 30.16
Text2Video-Zero | Depth | 97.46 | 29.38
Ours | Depth | 97.65 | 30.54
ControlNet | Segment | 91.87 | 31.85
Ours | Segment | 98.13 | 32.09
Table 2: Quantitative comparisons condition on canny, depth and segment.

We evaluate all the methods using three metrics: frame consistency (Esser et
al. 2023; Wang et al. 2023a; Radford et al. 2021), clip score (Ho et al.
2022a; Hessel et al. 2021; Park et al. 2021), and pose accuracy (Ma et al.
2023). As other conditions are hard to evaluate, we use pose accuracy for
conditional consistency only. The results on different conditions are shown in
Tab. 1 and 2. We achieve the highest frame consistency, and clip score in all
conditions, indicating that our method exhibits the best text alignment. We
also have the best pose-video alignment among the other three techniques of
conditioning on the pose.

The conditions are randomly generated from a group of 120 different videos.
For more information please see Appx D.2.

###  5.4 Ablation Study

Figure 7: Ablations of each component, generated from image-level noise. “The astronaut, in a spacewalk, sci-fi digital art style”. 1st row displays the generation result without pose conditioning. 2nd and 3rd rows show the results after replacing our sBiST-Attn with self-Attn and SC-Attn (Wu et al. 2022b). 4th row presents the result with the 2D condition-control branch. Method | FC(%) | Time
---|---|---
w/o Temp-Attn | 94.22 | 31s
S-C Attn | 98.77 | 43s
sBiST-Attn | 99.02 | 1m30s
Full-Attn | 99.03 | 3m37s
Table 3: Ablations on temporal module. Time represents the duration required
to generate a 24-frame video with a size of 512x512.

We conduct an ablation study on the pose condition, temporal module, and 3D
control branch. Our qualitative results are visualized in Fig. 7. In this
study, we alter each component for comparison while keeping all other settings
the same.

##### Ablation on Pose Condition

We evaluate performance with and without using pose, as shown in Fig. 7.
Without pose conditioning, the video is fixed as an image, while the use of
pose control allows for the generation of videos with certain temporal
semantic information.

##### Ablation on Temporal Module

Training-free video generation heavily relies on effective spatial-temporal
modeling. In addition to comparing with a self-attention mechanism without
temporal, we conduct an ablation study on three different spatial-temporal
mechanisms. First, we remove our sBiST-attention mechanism and replaced it
with Sparse-Causal attention (Wu et al. 2022b). Then, we compare our bi-
directional attention mechanism with a dense attention mechanism (Wang et al.
2023a) which attends to all frames for key and value.

The results are presented in Tab. 3. A comparison of temporal and non-temporal
attention underlines the importance of temporal modeling for generating time-
consistent videos. By comparing our method with Sparse Causal attention, we
demonstrate the effectiveness of ConditionVideo’s sBiST attention module,
proving that incorporating information from bi-directional frames improves
performance compared to using only previous frames. Furthermore, we observe
almost no difference in frame consistency between our method and dense
attention, despite the latter requiring more than double our generation
duration.

##### Ablations on 3D control branch

Method | FC(%) | CS (%) | PA (%)
---|---|---|---
2D control | 99.03 | 33.11 | 81.26
3D control | 99.02 | 33.03 | 83.12
Table 4: Ablation on 3D control branch. FC, CS, PA represent frame
consistency, clip score, and pose-accuracy, respectively.

We compare our 3D control branch with a 2D version that processes conditions
frame-by-frame. For the 2D branch, we utilize the original ControlNet
conditional branch. Both control branches are evaluated in terms of frame
consistency, clip score, and pose accuracy. Results in Tab. 4 show that our 3D
control branch outperforms the 2D control branch in pose accuracy while
maintaining similar frame consistency and clip scores. This proves that
additional consideration of bi-directional frames enhances pose control.

##  6 Discussion and Conclusion

In this paper, we propose ConditionVideo, a training-free method for
generating videos with reasonable motion. We introduce a method that generates
motion representation conditioned on background video and conditional
information. Our method additionally strengthens frame consistency and
condition alignment through our sBiST-Attn mechanism and 3D control branch.
Experimental results demonstrate that our method can generate high-quality
videos, opening new avenues for research in video generation and AI-based
content creation.

While the condition-based and enhanced temporal attention blocks contribute to
enhancing the temporal coherence of the video, we have observed that using
sparse conditions, such as pose information, can still lead to videos with
noticeable flickering. To address this issue, a potential solution would
involve incorporating more densely sampled control inputs and additional
temporal-related structures.

## References

  * Bertasius, Wang, and Torresani (2021) Bertasius, G.; Wang, H.; and Torresani, L. 2021.  Is space-time attention all you need for video understanding?  In _ICML_ , volume 2.
  * Blattmann et al. (2023) Blattmann, A.; Rombach, R.; Ling, H.; Dockhorn, T.; Kim, S. W.; Fidler, S.; and Kreis, K. 2023.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Ceylan, Huang, and Mitra (2023) Ceylan, D.; Huang, C.-H. P.; and Mitra, N. J. 2023.  Pix2video: Video editing using image diffusion.  _arXiv preprint arXiv:2303.12688_.
  * Chan et al. (2019) Chan, C.; Ginosar, S.; Zhou, T.; and Efros, A. A. 2019.  Everybody dance now.  In _Proceedings of the IEEE/CVF international conference on computer vision_.
  * Dhariwal and Nichol (2021) Dhariwal, P.; and Nichol, A. 2021.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34.
  * Esser et al. (2023) Esser, P.; Chiu, J.; Atighehchian, P.; Granskog, J.; and Germanidis, A. 2023.  Structure and content-guided video synthesis with diffusion models.  _arXiv preprint arXiv:2302.03011_.
  * Gafni et al. (2022) Gafni, O.; Polyak, A.; Ashual, O.; Sheynin, S.; Parikh, D.; and Taigman, Y. 2022.  Make-a-scene: Scene-based text-to-image generation with human priors.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV_. Springer.
  * Hessel et al. (2021) Hessel, J.; Holtzman, A.; Forbes, M.; Le Bras, R.; and Choi, Y. 2021.  CLIPScore: A Reference-free Evaluation Metric for Image Captioning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Ho et al. (2022a) Ho, J.; Chan, W.; Saharia, C.; Whang, J.; Gao, R.; Gritsenko, A.; Kingma, D. P.; Poole, B.; Norouzi, M.; Fleet, D. J.; et al. 2022a.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_.
  * Ho, Jain, and Abbeel (2020) Ho, J.; Jain, A.; and Abbeel, P. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ , 33.
  * Ho and Salimans (2022) Ho, J.; and Salimans, T. 2022.  Classifier-free diffusion guidance.
  * Ho et al. (2022b) Ho, J.; Salimans, T.; Gritsenko, A.; Chan, W.; Norouzi, M.; and Fleet, D. J. 2022b.  Video diffusion models.
  * Hong et al. (2022) Hong, W.; Ding, M.; Zheng, W.; Liu, X.; and Tang, J. 2022.  CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers.
  * Khachatryan et al. (2023) Khachatryan, L.; Movsisyan, A.; Tadevosyan, V.; Henschel, R.; Wang, Z.; Navasardyan, S.; and Shi, H. 2023.  Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators.  _arXiv preprint arXiv:2303.13439_.
  * Kingma et al. (2021) Kingma, D.; Salimans, T.; Poole, B.; and Ho, J. 2021.  Variational diffusion models.  _Advances in neural information processing systems_ , 34.
  * Liu et al. (2023a) Liu, S.; Zhang, Y.; Li, W.; Lin, Z.; and Jia, J. 2023a.  Video-P2P: Video Editing with Cross-attention Control.  _arXiv preprint arXiv:2303.04761_.
  * Liu et al. (2019) Liu, W.; Piao, Z.; Min, J.; Luo, W.; Ma, L.; and Gao, S. 2019.  Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_.
  * Liu et al. (2023b) Liu, Y.; Han, T.; Ma, S.; Zhang, J.; Yang, Y.; Tian, J.; He, H.; Li, A.; He, M.; Liu, Z.; et al. 2023b.  Summary of chatgpt/gpt-4 research and perspective towards the future of large language models.  _arXiv preprint arXiv:2304.01852_.
  * Ma et al. (2023) Ma, Y.; He, Y.; Cun, X.; Wang, X.; Shan, Y.; Li, X.; and Chen, Q. 2023.  Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videos.  _arXiv preprint arXiv:2304.01186_.
  * Mirza and Osindero (2014) Mirza, M.; and Osindero, S. 2014.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_.
  * MMPose Contributors (2020) MMPose Contributors. 2020.  OpenMMLab Pose Estimation Toolbox and Benchmark.
  * Molad et al. (2023) Molad, E.; Horwitz, E.; Valevski, D.; Acha, A. R.; Matias, Y.; Pritch, Y.; Leviathan, Y.; and Hoshen, Y. 2023.  Dreamix: Video diffusion models are general video editors.  _arXiv preprint arXiv:2302.01329_.
  * Mou et al. (2023) Mou, C.; Wang, X.; Xie, L.; Zhang, J.; Qi, Z.; Shan, Y.; and Qie, X. 2023.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_.
  * Nicki Skafte Detlefsen et al. (2022) Nicki Skafte Detlefsen; Jiri Borovec; Justus Schock; Ananya Harsh; Teddy Koker; Luca Di Liello; Daniel Stancl; Changsheng Quan; Maxim Grechkin; and William Falcon. 2022.  TorchMetrics - Measuring Reproducibility in PyTorch.
  * Park et al. (2021) Park, D. H.; Azadi, S.; Liu, X.; Darrell, T.; and Rohrbach, A. 2021.  Benchmark for compositional text-to-image synthesis.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Qi et al. (2023) Qi, C.; Cun, X.; Zhang, Y.; Lei, C.; Wang, X.; Shan, Y.; and Chen, Q. 2023.  Fatezero: Fusing attentions for zero-shot text-based video editing.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_. PMLR.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1).
  * Ramesh et al. (2022) Ramesh, A.; Dhariwal, P.; Nichol, A.; Chu, C.; and Chen, M. 2022.  Hierarchical text-conditional image generation with clip latents.
  * Rombach et al. (2022) Rombach, R.; Blattmann, A.; Lorenz, D.; Esser, P.; and Ommer, B. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_.
  * Ronneberger, Fischer, and Brox (2015) Ronneberger, O.; Fischer, P.; and Brox, T. 2015.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_. Springer.
  * Saharia et al. (2022) Saharia, C.; Chan, W.; Saxena, S.; Li, L.; Whang, J.; Denton, E. L.; Ghasemipour, K.; Gontijo Lopes, R.; Karagol Ayan, B.; Salimans, T.; et al. 2022.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35.
  * Siarohin et al. (2019) Siarohin, A.; Lathuilière, S.; Tulyakov, S.; Ricci, E.; and Sebe, N. 2019.  First order motion model for image animation.  _Advances in Neural Information Processing Systems_ , 32.
  * Singer et al. (2023) Singer, U.; Polyak, A.; Hayes, T.; Yin, X.; An, J.; Zhang, S.; Hu, Q.; Yang, H.; Ashual, O.; Gafni, O.; Parikh, D.; Gupta, S.; and Taigman, Y. 2023.  Make-A-Video: Text-to-Video Generation without Text-Video Data.  In _The Eleventh International Conference on Learning Representations_.
  * Sohl-Dickstein et al. (2015) Sohl-Dickstein, J.; Weiss, E.; Maheswaranathan, N.; and Ganguli, S. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_. PMLR.
  * Song, Meng, and Ermon (2020) Song, J.; Meng, C.; and Ermon, S. 2020.  Denoising Diffusion Implicit Models.  _CoRR_ , abs/2010.02502.
  * Song, Meng, and Ermon (2021) Song, J.; Meng, C.; and Ermon, S. 2021.  Denoising Diffusion Implicit Models.  In _International Conference on Learning Representations_.
  * Song et al. (2021) Song, Y.; Sohl-Dickstein, J.; Kingma, D. P.; Kumar, A.; Ermon, S.; and Poole, B. 2021.  Score-Based Generative Modeling through Stochastic Differential Equations.  In _International Conference on Learning Representations_.
  * Sun et al. (2019) Sun, K.; Xiao, B.; Liu, D.; and Wang, J. 2019.  Deep high-resolution representation learning for human pose estimation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_.
  * Van Den Oord, Vinyals et al. (2017) Van Den Oord, A.; Vinyals, O.; et al. 2017.  Neural discrete representation learning.  _Advances in neural information processing systems_.
  * Wang et al. (2019) Wang, T.-C.; Liu, M.-Y.; Tao, A.; Liu, G.; Kautz, J.; and Catanzaro, B. 2019.  Few-shot Video-to-Video Synthesis.  In _Advances in Neural Information Processing Systems (NeurIPS)_.
  * Wang et al. (2018) Wang, T.-C.; Liu, M.-Y.; Zhu, J.-Y.; Liu, G.; Tao, A.; Kautz, J.; and Catanzaro, B. 2018.  Video-to-Video Synthesis.  In _Conference on Neural Information Processing Systems (NeurIPS)_.
  * Wang et al. (2023a) Wang, W.; Xie, K.; Liu, Z.; Chen, H.; Cao, Y.; Wang, X.; and Shen, C. 2023a.  Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models.  _arXiv preprint arXiv:2303.17599_.
  * Wang et al. (2020) Wang, Y.; Bilinski, P.; Bremond, F.; and Dantcheva, A. 2020.  G3AN: Disentangling Appearance and Motion for Video Generation.  In _CVPR_.
  * WANG et al. (2020) WANG, Y.; Bilinski, P.; Bremond, F.; and Dantcheva, A. 2020.  ImaGINator: Conditional Spatio-Temporal GAN for Video Generation.  In _WACV_.
  * Wang et al. (2023b) Wang, Y.; Chen, X.; Ma, X.; Zhou, S.; Huang, Z.; Wang, Y.; Yang, C.; He, Y.; Yu, J.; Yang, P.; et al. 2023b.  LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models.  _arXiv preprint arXiv:2309.15103_.
  * Wang et al. (2023c) Wang, Y.; Ma, X.; Chen, X.; Dantcheva, A.; Dai, B.; and Qiao, Y. 2023c.  LEO: Generative Latent Image Animator for Human Video Synthesis.  _arXiv preprint arXiv:2305.03989_.
  * Wang et al. (2022) Wang, Y.; Yang, D.; Bremond, F.; and Dantcheva, A. 2022.  Latent Image Animator: Learning to Animate Images via Latent Space Navigation.  In _ICLR_.
  * Wu et al. (2021) Wu, C.; Huang, L.; Zhang, Q.; Li, B.; Ji, L.; Yang, F.; Sapiro, G.; and Duan, N. 2021.  Godiva: Generating open-domain videos from natural descriptions.  _arXiv preprint arXiv:2104.14806_.
  * Wu et al. (2022a) Wu, C.; Liang, J.; Ji, L.; Yang, F.; Fang, Y.; Jiang, D.; and Duan, N. 2022a.  Nüwa: Visual synthesis pre-training for neural visual world creation.  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XVI_. Springer.
  * Wu et al. (2022b) Wu, J. Z.; Ge, Y.; Wang, X.; Lei, W.; Gu, Y.; Hsu, W.; Shan, Y.; Qie, X.; and Shou, M. Z. 2022b.  Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation.  _arXiv preprint arXiv:2212.11565_.
  * Zhang and Agrawala (2023) Zhang, L.; and Agrawala, M. 2023.  Adding conditional control to text-to-image diffusion models.  _arXiv preprint arXiv:2302.05543_.
  * Zhou et al. (2022) Zhou, X.; Yin, M.; Chen, X.; Sun, L.; Gao, C.; and Li, Q. 2022.  Cross attention based style distribution for controllable person image synthesis.  In _European Conference on Computer Vision_ , 161–178. Springer.

## Appendix

In our appendix, we initially present an expanded set of generated results,
along with more of the comparative content. Subsequently, we exhibit results
to demonstrate the effectiveness of disentanglement in latent motion
representation. Finally, we provide implementation details of our experiments,
including model architecture, sampling strategy, and evaluation details.

##  Appendix A More Results

We’ve included more generated video results at
https://pengbo807.github.io/conditionvideo-website/. By leveraging the large
text-to-image diffusion model, our method can generate videos without any
fine-tuning. Videos are created from condition annotation, various prompts,
and optional reference scenery videos. When the reference scenery video is not
provided, the background can be generated from random noise. Results show our
model can generate dynamic and realistic videos by giving different prompts.
In addition, our model can generate customized backgrounds of the video when
given reference scenery videos. The multi-people half-body results are also
demonstrated on the website, where we successfully generate three dancing
Spiderman with the prompt “three Spiderman at night”.

##  Appendix B Disentanglement for Latent Motion Representation

We observe that the generated video’s foreground and background would be mixed
if we feed the same latent noise into both the UNet-branch and control branch.
Therefore, we propose to use disentanglement on latent input, which uses
different inputs for two branches. As shown in Fig. 8, the method without
disentanglement of latent motion representation tends to produce visuals where
the foreground merges with the background. In contrast, our method
successfully creates a distinct foreground.

Figure 8: The effectiveness of disentanglement for latent motion
representation. Prompt: “Fishers in the sea”.

##  Appendix C More Ablation and Comparison Results

###  C.1 more qualitative results on temporal attention

We conduct experimentation with various gap values, as depicted in Figure 9.
Additionally, we present a broader range of outcomes concerning diverse
attention mechanisms in Table 5, encompassing SC-Attn (Wu et al. 2022b), Full-
Attn (Wang et al. 2023a), and Cross-Frame Attn from Text2Video-Zero
(Khachatryan et al. 2023). Following an assessment of temporal and frame
consistency, we determine the optimal sBiST-attention gap to be 3.

Figure 9: Ablation study of sBiST-Attn gap Method | FC(%) | Time
---|---|---
SC-Attn | 98.77 | 43s
Cross-Frame-Attn | 98.86 | 34s
Full-Attn(gap1) | 99.03 | 3m37s
sBiST-Attn(gap2) | 99.01 | 1m51s
sBiST-Attn(gap3) | 99.02 | 1m30s
sBiST-Attn(gap4) | 99.00 | 1m13s
sBiST-Attn(gap5) | 99.00 | 1m7s
sBiST-Attn(gap6) | 98.99 | 1m
sBiST-Attn(gap9) | 98.99 | 54m
sBiST-Attn(gap12) | 98.90 | 47s
Table 5: Ablation on temporal module

###  C.2 More quantitative results

Here, we provide more ablation study results focusing on the individual
components of our model. Initially, our 3D control branch was replaced with a
2D control branch derived from ControlNet (abbr. w/o 3D control).
Subsequently, test the result after replacing our sBiST-Attn with Sparse-
Causal Attention (SC-Attn) (abbr. w/o BiAttn). Fig.10 and 11 demonstrate the
importance of each model component. When any of the components are replaced or
removed, it leads to a noticeable decrease in performance. Note that we also
find significant pose accuracy improvement with our sBiST-Attn against SC-
Attn, as shown in Fig.10(a).

Figs.13,14,12 show more comparison results of our model with Text2Video-Zero
condition on pose, canny and depth. From the many previous comparisons, it is
clear that our model has a significant advantage over Tune-A-Video and
ControlNet, so we omit the comparisons between the two models here. Guided by
the more ambiguous conditions, our model is more adept at reasoning about the
most plausible outlook from the bidirectional information.

Figure 10: More ablation results on pose: SC-Attn vs Ours Figure 11: More
ablation results on pose: 2D Control vs Ours Figure 12: More comparison
results on pose Figure 13: More comparison results on canny Figure 14: More
comparison results on canny

##  Appendix D Experiment Details

###  D.1 Model Architecture and Sampling Strategy

The architecture of our model is shown in Tab. 7. Inspired by ControlNet
(Zhang and Agrawala 2023), our model consists of two branches, namely the UNet
branch and the control branch. The control branch contains 1 conditional
encoder, 12 downsample blocks, and 1 middle block. The downsample blocks and
the middle block share the same structure as the UNet branch’s corresponding
block. The conditional encoder consists of 4 convolutional layers, which
convert $512\times 512$ conditions to $64\times 64$ feature maps. The output
of each block of the control branch is fed into the corresponding upsampler
after being processed by a convolution layer. For each block within our model,
we inflate the 2D convolution to a 3D convolution whose kernel size is from
$3\times 3$ to $1\times 3\times 3$. Additionally, all original self-attention
is replaced with our proposed sBiST-attention.The convolution layers of the
condition encoder are also inflated to 3D. It’s worth noting that we retain
the original Encoder and Decoder, identical to those used in the large image
generation model, thereby processing videos frame by frame.

For customized background video generation, we first use the DDIM inversion
(Song, Meng, and Ermon 2020) to obtain the latent noise of the reference scene
video. For the random background generation, we sample random Gaussian noise.
After obtaining latent noise, we perform DDIM sampling along with classifier-
free guidance (Ho and Salimans 2022). We set the guidance scale to 12.5 and
the number of time steps for DDIM sampling to 50. All the hyper-parameters of
our method are shown in Tab.6.

hyper-parameters | value
---|---
DDIM inference step | 50
classifier guidance scale | 12.5
sBiST-Attn-gap | 3
Table 6: Hyper-parameters UNet Branch | Feature Map Size | Control Branch | Feature Map Size
---|---|---|---
Encoder | (64,64) | Conditional Encoder | (24,64,64)
DownsampleBlock1 ($\times 3$) | (24,64,64) | DownsampleBlock1 ($\times 3$) | (24,64,64)
DownsampleBlock2 ($\times 3$) | (24,32,32) | DownsampleBlock2$\times 3$ | (24,32,32)
DownsampleBlock3 ($\times 3$) | (24,16,16) | DownsampleBlock3 ($\times 3$) | (24,16,16)
DownsampleBlock4 ($\times 3$) | (24,8,8) | DownsampleBlock4 ($\times 3$) | (24,8,8)
MiddleBlock | (24,8,8) | MiddleBlock | (24,8,8)
UpsampleBlock4 ($\times 3$) | (24,8,8) | convolution | (24,64,64)
UpsampleBlock3 ($\times 3$) | (24,16,16) | convolution | (24,64,64)
UpsampleBlock2 ($\times 3$) | (24,32,32) | convolution | (24,64,64)
UpsampleBlock1 ($\times 3$) | (24,64,64) | convolution | (24,64,64)
Decoder | (64,64) |  |
Table 7: Architecture of our model

###  D.2 Evaluation Details

In our evaluation process, we carried out speed analysis experiments on a
single NVIDIA A100 GPU equipped with 80GB of memory. For a fair comparison, We
use the same random seed while sampling with different models. We produce 24
frames for each video, each with a resolution of 512 × 512 pixels, and the
videos are created at a frame rate of 10 frames per second (fps). Throughout
our comparative analyses with previous methods and the ablation study, each
experiment was conducted with 100 reference conditions and 100 prompts. The
reference conditions were extracted from a dancing video extracted from an
online entertainment platform with 4800 frames. we use a frame rate of 2 and
sample 48 frames continuously from the video. Then we get 100 videos with 24
frames each. We present the details of the prompt generation in the following
subsection.

We assess all the methods using three measurement criteria: ”frame
consistency” (Esser et al. 2023; Wang et al. 2023a; Radford et al. 2021),
”clip score” (Ho et al. 2022a; Hessel et al. 2021; Park et al. 2021), and
”pose accuracy” (Ma et al. 2023). Next, we describe in detail how these three
types of metrics are implemented.

##### Frame consistency

Following the approaches of (Esser et al. 2023) and (Wang et al. 2023a), we
calculate the consistency of the generated video. Specifically, we first
calculate each frame’s CLIP image embedding (Radford et al. 2021). Then, we
compute the cosine similarity of two consecutive frames’ CLIP image embeddings
and take the average similarity of all frame pairs as the final score.

##### Clip Score

We follow the approach in (Ho et al. 2022a) to compute the clip score (Hessel
et al. 2021; Park et al. 2021). The clip score is computed on each frame and
then averaged. We use the TorchMetrics (Nicki Skafte Detlefsen et al. 2022)
package for computing clip score.

##### Pose Accuracy

Similar to (Ma et al. 2023), we use HRNet (Sun et al. 2019) to compute our
video-pose alignment quality. Specifically, we employ HRNet to predict the
skeleton of the generated video, as well as extract the ground truth skeleton
from the video where the original guiding pose heatmap was obtained. We
normalize the extracted skeletons, and then compute the distance between each
keypoint in the pose between the generated skeleton and the ground truth
skeleton. We set a threshold of 0.05. If the distance is smaller than the
threshold, we regard it as accurate. We compute the average accuracy across
all key point skeletons to get the final score. As Tune-A-Video cannot do pose
control, we consider the pose of the inverted video as the ground truth poses
for computation. We use HRNet (Sun et al. 2019), available in the MMPose
repository (MMPose Contributors 2020) as our pose detector.

###  D.3 Prompts

We employed ChatGPT (Liu et al. 2023b) to generate 100 random prompts and
produce 100 arbitrary prompts with different scenery, styles, and characters
such as Ironman, robot, and spaceman. We then employed these prompts to create
video clips. For the pose accuracy evaluation, we requested chatGPT to
generate prompts exclusively containing human characters, because the utilized
pose recognition model (HRNet (Sun et al. 2019)) would dramatically decrease
its recognizing accuracy when given non-human objects (_i.e.,_ robots).
Specifically, we additionally required chatGPT to give the text that would
allow the HRNet to easily recognize the generated video. We used the prompt
“give me 100 prompts to generate video, preferably in the form of [description
of human character], in/on [background description]’, such as The Batman, in
the sea’, The Iron man, in the forest, Makoto Shinkai style’, oil painting of
a beautiful girl avatar style’. You should make sure a pose detector like
HRNet can detect the human pose of the generated video”. For frame consistency
and clip score evaluations, we removed the constraints for human-like
characters only, allowing us to assess model performances in a wider range of
scenarios.

We use the following prompts to evaluate the proposed method.

#### List of 100 Prompts for Frame Consistency and Clip Score Evaluation

  1. 1.

a man in a suit and tie

  2. 2.

a man standing on top of a cliff

  3. 3.

man on hill watching a meteor, cartoon artwork

  4. 4.

The Samurai, on a mountaintop, traditional ink painting style

  5. 5.

The Space Explorer, in an interstellar spaceship, futuristic anime style

  6. 6.

The Pirate, on a sandy beach, digital painting style

  7. 7.

The Cyborg, in a neon-lit alleyway, cyberpunk anime style

  8. 8.

The Cowboy, on a dusty plain, oil painting style

  9. 9.

The Ninja, in a bamboo forest, watercolor painting style

  10. 10.

The Robot, on a post-apocalyptic city street, sci-fi comic book style

  11. 11.

The Knight, in a medieval castle, oil painting style

  12. 12.

The Alien, on a distant planet, surrealistic painting style

  13. 13.

The Samurai, in a cherry blossom garden, sumi-e painting style

  14. 14.

The Detective, in a noir cityscape, graphic novel style

  15. 15.

The Mermaid, in an underwater kingdom, fantasy painting style

  16. 16.

The Robot, in a high-tech laboratory, digital painting style

  17. 17.

The Pirate, on a ship deck, traditional painting style

  18. 18.

The Samurai, in a misty forest, manga-style

  19. 19.

The Steampunk Inventor, in a clockwork laboratory, Victorian-era painting
style

  20. 20.

The Ghost, in a haunted house, horror comic book style

  21. 21.

The Knight, on a battlefield, historical painting style

  22. 22.

The Alien, in a futuristic city, sci-fi painting style

  23. 23.

The Vampire, in a gothic castle, dark painting style

  24. 24.

The Samurai, in a snowy landscape, impressionist painting style

  25. 25.

The Cyborg, in a virtual reality world, digital art style

  26. 26.

The Cowboy, on a rugged mountain range, Western painting style

  27. 27.

The Robot, in a futuristic cityscape, sci-fi painting style

  28. 28.

The Detective, in a crime scene investigation, graphic novel style

  29. 29.

The Mermaid, in a coral reef, watercolor painting style

  30. 30.

The Pirate, on a deserted island, tropical painting style

  31. 31.

The Knight, in a mythical forest, medieval painting style

  32. 32.

The Alien, in a mysterious underground base, surrealistic painting style

  33. 33.

The Samurai, in a tranquil garden, traditional painting style

  34. 34.

The Cyborg, in a post-apocalyptic wasteland, digital art style

  35. 35.

The Cowboy, in a small-town saloon, Western painting style

  36. 36.

The Robot, in a cyberpunk megacity, sci-fi painting style

  37. 37.

The Detective, in a neon-lit alleyway, graphic novel style

  38. 38.

The Mermaid, in a shipwreck, fantasy painting style

  39. 39.

The Pirate, in a bustling port city, oil painting style

  40. 40.

The Knight, in a royal castle, historical painting style

  41. 41.

The Alien, in a deep space nebula, surrealistic painting style

  42. 42.

The Samurai, in a cherry blossom forest, sumi-e painting style

  43. 43.

The Cyborg, in a virtual reality cyberworld, digital art style

  44. 44.

The Cowboy, in a desert ghost town, Western painting style

  45. 45.

The Robot, in a futuristic laboratory, sci-fi painting style

  46. 46.

The Detective, in a city rooftop chase, graphic novel style

  47. 47.

The Mermaid, in a moonlit lagoon, fantasy painting style

  48. 48.

The Pirate, in a naval battle, traditional painting style

  49. 49.

The Knight, in a medieval joust, historical painting style

  50. 50.

The Alien, in an abandoned spaceship, surrealistic painting style

  51. 51.

The Samurai, in a bamboo forest at night, sumi-e painting style

  52. 52.

The Explorer, in a dense jungle, documentary style

  53. 53.

The Assassin, in a city street at night, noir painting style

  54. 54.

The Monster, in a dark cave, horror movie style

  55. 55.

The Superhero, in a city skyline, comic book style

  56. 56.

The Samurai, in a foggy bamboo forest, manga-style

  57. 57.

The Dancer, on a stage, performing arts style

  58. 58.

The Alien, in a laboratory, sci-fi digital art style

  59. 59.

The Detective, in a murder investigation, crime drama style

  60. 60.

The Pirate, on a treasure island, adventure painting style

  61. 61.

The Knight, in a medieval siege, historical painting style

  62. 62.

The Samurai, in a peaceful pond, sumi-e painting style

  63. 63.

The Robot, in a factory, sci-fi painting style

  64. 64.

The Cowboy, on a rodeo, Western painting style

  65. 65.

The Witch, in a haunted forest, dark painting style

  66. 66.

The Astronaut, in a spacewalk, sci-fi digital art style

  67. 67.

The Superhero, in a city rooftop, comic book style

  68. 68.

The Samurai, in a fiery battlefield, manga-style

  69. 69.

The Cyborg, in a futuristic city, sci-fi painting style

  70. 70.

The Monster, in a graveyard, horror movie style

  71. 71.

The Pirate, on a ship deck in a storm, traditional painting style

  72. 72.

The Knight, in a medieval banquet, historical painting style

  73. 73.

The Samurai, in a cherry blossom tree, sumi-e painting style

  74. 74.

The Robot, in a futuristic transport system, sci-fi digital art style

  75. 75.

The Cowboy, on a horseback ride, Western painting style

  76. 76.

The Detective, in a chase sequence, crime drama style

  77. 77.

The Witch, in a spell-casting ritual, dark painting style

  78. 78.

The Astronaut, in a lunar colony, sci-fi digital art style

  79. 79.

The Superhero, in a city alleyway, comic book style

  80. 80.

The Samurai, in a sacred temple, manga-style

  81. 81.

The Cyborg, in a post-human society, sci-fi painting style

  82. 82.

The Monster, in a creepy mansion, horror movie style

  83. 83.

The Pirate, on a deserted beach, adventure painting style

  84. 84.

The Knight, in a medieval battle, historical painting style

  85. 85.

The Samurai, in a peaceful waterfall, sumi-e painting style

  86. 86.

The Robot, in a futuristic military base, sci-fi digital art style

  87. 87.

The Cowboy, on a cattle drive, Western painting style

  88. 88.

The Detective, in a spy thriller, crime drama style

  89. 89.

The Witch, in a dark forest at night, dark painting style

  90. 90.

The Astronaut, in a spaceship launch, sci-fi digital art style

  91. 91.

The Superhero, in a city skyscraper, comic book style

  92. 92.

The Samurai, in a snow-covered landscape, manga-style

  93. 93.

The Cyborg, in a robotic dystopia, sci-fi painting style

  94. 94.

The Monster, in a creepy attic, horror movie style

  95. 95.

The Pirate, on a high seas adventure, traditional painting style

  96. 96.

The Knight, in a royal coronation, historical painting style

  97. 97.

The Samurai, in a dragon fight, sumi-e painting style

  98. 98.

The Robot, in a futuristic cityscape at night, sci-fi digital art style

  99. 99.

The Cowboy, in a Western shootout, Western painting style

  100. 100.

The Detective, in a crime thriller, crime drama style

#### List of 100 Prompts for Pose Accuracy Evaluation

  1. 1.

The adventurous explorer, in a dense jungle

  2. 2.

The daring race car driver, on a winding race track

  3. 3.

The graceful ice skater, on a glittering ice rink

  4. 4.

The skilled skateboarder, on a sunny skate park

  5. 5.

The mysterious magician, in a dark and dusty magic shop

  6. 6.

The brave firefighter, in a raging inferno

  7. 7.

The daring surfer, on a towering ocean wave

  8. 8.

The energetic breakdancer, in a busy city street

  9. 9.

The elegant ballroom dancer, in a grand ballroom

  10. 10.

The playful street artist, in a vibrant graffiti alleyway

  11. 11.

The graceful figure skater, on a frozen lake

  12. 12.

The bold parkour enthusiast, in an urban park

  13. 13.

The mysterious ninja, in a misty mountain landscape

  14. 14.

The daring rock climber, on a steep rock face

  15. 15.

The passionate flamenco dancer, in a bustling city square

  16. 16.

The skilled pole dancer, in a dimly lit club

  17. 17.

The determined marathon runner, on a scenic country road

  18. 18.

The elegant equestrian, on a sprawling ranch

  19. 19.

The adventurous hot air balloonist, in a sky full of clouds

  20. 20.

The talented street performer, on a crowded pedestrian walkway

  21. 21.

The daring skydiver, in a clear blue sky

  22. 22.

The graceful gymnast, in a grand gymnastics arena

  23. 23.

The skilled fencer, in a historic castle courtyard

  24. 24.

The mystical sorceress, in a dimly lit forest

  25. 25.

The daring mountain climber, on a towering mountain peak

  26. 26.

The playful acrobat, in a colorful circus tent

  27. 27.

The noble knight, in a medieval castle

  28. 28.

The mysterious detective, in a dark alleyway

  29. 29.

The glamorous fashion model, on a bustling city street

  30. 30.

The fierce warrior, on a desolate mountaintop

  31. 31.

The charming street musician, on a busy street corner

  32. 32.

The adventurous mountaineer, on a snow-covered peak

  33. 33.

The energetic zumba dancer, in a vibrant dance studio

  34. 34.

The daring parkour enthusiast, on a futuristic city rooftop

  35. 35.

The skilled trapeze artist, in a circus high wire act

  36. 36.

The determined triathlete, on a grueling triathlon course

  37. 37.

The graceful synchronized swimmer, in a luxurious pool

  38. 38.

The playful breakdancer, in an urban street art scene

  39. 39.

The bold cliff diver, on a rocky ocean cliff

  40. 40.

The talented mime artist, in a quiet park

  41. 41.

The daring rodeo cowboy, in a rodeo arena

  42. 42.

The adventurous kayaker, on a fast-flowing river

  43. 43.

The skilled BMX rider, on a dirt bike track

  44. 44.

The passionate tango dancer, in a grand dance hall

  45. 45.

The mischievous pirate, on a stormy sea

  46. 46.

The bold bungee jumper, on a tall bridge

  47. 47.

The daring ski jumper, on a snowy mountain slope

  48. 48.

The glamorous Hollywood actor, on a red carpet event

  49. 49.

The mystical mermaid, in a deep ocean abyss

  50. 50.

The playful gymnastics coach, in a grand gymnasium

  51. 51.

The mysterious vampire, in a dimly lit castle

  52. 52.

The determined kickboxer, in a bustling gym

  53. 53.

The adventurous surfer, on a secluded beach

  54. 54.

The playful street dancer, in a crowded urban street

  55. 55.

The skilled juggler, in a busy street fair

  56. 56.

The daring base jumper, on a towering cliff

  57. 57.

The noble samurai, in a peacstage eful cherry blossom garden

  58. 58.

The passionate belly dancer, in a colorful Arabian bazaar

  59. 59.

The daring wing walker, on a biplane soaring high above the clouds

  60. 60.

The mystical fairy, in a magical forest

  61. 61.

The talented magician, on a grand stage

  62. 62.

The bold snowboarder, on a snowy mountain slope

  63. 63.

The playful roller skater, in a bustling skate park

  64. 64.

The daring ice climber, on a frozen waterfall

  65. 65.

The graceful pole vaulter, on a busy track and field

  66. 66.

The adventurous kayaker, in a breathtaking canyon

  67. 67.

The skilled martial artist, in a quiet temple

  68. 68.

The elegant ballet dancer, in a grand theater

  69. 69.

The daring free runner, in an urban obstacle course

  70. 70.

The graceful rhythmic gymnast, in a grand arena

  71. 71.

The adventurous dirt bike rider, on a rough terrain

  72. 72.

The bold street racer, on a busy city street

  73. 73.

The talented ice sculptor, in a snowy park

  74. 74.

The playful hula hooper, in a colorful park

  75. 75.

The daring mountain biker, on a rocky mountain trail

  76. 76.

The noble gladiator, in a grand coliseum

  77. 77.

The mysterious ghost, in a dark haunted mansion

  78. 78.

The graceful figure skater, on a frozen river

  79. 79.

The adventurous rock climber, on a steep rock face

  80. 80.

The daring cliff diver, on a tall ocean cliff

  81. 81.

The passionate salsa dancer, in a vibrant dance hall

  82. 82.

The bold wingsuit flyer, soaring through a majestic canyon

  83. 83.

The talented street magician, in a crowded street market

  84. 84.

The playful parkour athlete, on a busy city rooftop

  85. 85.

The determined long distance runner, on a scenic trail

  86. 86.

The skilled contortionist, in a grand circus tent

  87. 87.

The bold rodeo bull rider, in a rodeo arena

  88. 88.

The adventurous sky glider, in a sky full of clouds

  89. 89.

The graceful equestrian vaulting performer, on a sprawling farm

  90. 90.

The daring sky surfer, on a windy day

  91. 91.

The talented ice skater, on a frozen pond

  92. 92.

The passionate flamenco dancer, in a vibrant city square

  93. 93.

The mysterious street magician, in a dark alleyway

  94. 94.

The skilled cross-country skier, on a snowy mountain trail

  95. 95.

The daring motocross rider, on a dirt track

  96. 96.

The playful street performer, on a busy pedestrian walkway

  97. 97.

The noble horseback rider, on a majestic countryside

  98. 98.

The daring snowmobiler, on a snowy terrain

  99. 99.

The adventurous white water rafter, on a fast-flowing river

  100. 100.

The passionate tap dancer, on a grand theater stage