# Repairing Activity Start Times to Improve Business Process Simulation

David Chapela-Campa
University of Tartu
<EMAIL_ADDRESS>Marlon Dumas
University of Tartu
<EMAIL_ADDRESS>
## I. Introduction

Business Process Simulation (BPS) is a technique for quantitative analysis of
business processes [10]. BPS is commonly used to estimate the impact of
business process changes prior to their implementation [1, 7, 14, 15, 31].
Specifically, BPS enables managers and analysts to address “what-if”
questions, such as “what would be the cycle time of the process if the rate of
arrival of new traces doubles?” or “what if 10% of the workforce becomes
unavailable for an extended time period?”.

The starting point of BPS is a business process model – e.g. in the Business
Process Model and Notation (BPMN)111https://www.bpmn.org/ – enhanced with
simulation parameters [21], such as the distribution of processing times of
each activity in the process (a BPS model). The usefulness of BPS directly
depends on the quality of the BPS model used as input [20]. Specifically, one
expects that a simulation of a BPS model leads to a collection of simulated
traces (herein a _simulated log_), which resembles the execution traces
observed in the real executions of the process. A BPS model unable to mimic
the observed behavior would be unsuitable for analyzing the process in
different scenarios. Traditionally, BPS models were manually created by
modeling experts. This task is time-consuming and error-prone [19]. To tackle
this shortcoming, several studies [17, 22] have advocated for the use of
process mining techniques to automatically discover BPS models from business
process event logs recorded by (process-aware) information systems [5, 6]. In
this context, an _event log_ is a set of events recording the execution of
activity instances pertaining to a process. Each event consists of an
identifier of the process instance (trace) it belongs to, the label of the
executed activity, the stage of the execution it records – e.g. start or end
–, the resource who performed the activity, and possibly some additional
attributes. Given this input, automated BPS model discovery methods learn a
BPS model that, when simulated, leads to a simulated log that replicates the
real process, as recorded in the original log.

Previous studies have highlighted that the applicability of process mining
techniques is often hampered by data quality issues [4, 2, 11, 24, 25]. One
recurrent issue is the lack of timestamps, and particularly, the fact that
event logs encountered in practice often do not have start timestamps, but
only end timestamps [25]. In other scenarios, start timestamps may be present,
but they do not reflect the actual time when the corresponding resource
started actively working on the activity. For example, an employee might start
working on an activity by preparing the required materials, but the system
does not record the activity start until the employee first interacts with the
system. This recurrent data quality issue has a direct impact on methods for
automated BPS model discovery, since these methods rely on both start and end
timestamps to compute the distribution of processing times of the activities
in the process. For example, if the start time does not precisely record when
the resources started working on the activity instances, two scenarios can
occur. In the first scenario, the discovered BPS model simulates the sojourn
time – i.e. the amount of time since the activity instance is ready to be
executed until it finishes – only considering the (wrongly) recorded
processing time. However, this leads to unrealistically short cycle times, as
the waiting time and non-recorded processing time are not being considered. In
the second scenario, the discovered BPS model includes waiting time previous
to the processing of the activity instance, in which the resource is
considered to be free, while she/he is already working on the activity.
Nevertheless, this can lead to a wrong allocation of the resources, as the
resource was actually blocked while working in the preparation of the
activity.

To tackle this problem, we propose to identify, for each activity instance in
an event log, the portion of its waiting time that corresponds to non-recorded
processing time – i.e. the resource was actually working on the activity –,
and repair the start time to include it as processing time. To do this, our
proposal combines the information of the enablement time of each activity
instance with the availability information of its resource in order to compute
the earliest starting point. The reasoning behind this is that, as far as
simulation is concerned, an activity instance may start once it is enabled and
the corresponding resource is available. Accordingly, we propose to consider
as non-recorded processing time the interval from the earliest starting point
until the recorded start of the activity instance.

The paper reports on an empirical evaluation involving eight real-life event
logs containing both start and end timestamps. The evaluation aims at
determining if a BPS model discovered from an event log with adjusted start
times leads to simulated logs that closely replicate the temporal dynamics
recorded in the original log, relative to a BPS model discovered directly from
the original log.

The remainder of the article is structured as follows. Sec. II gives an
overview of prior research related to start time estimation. Sec. III presents
the proposed approach to estimate activity start times. Finally, Sec. IV
discusses the empirical evaluation, and Sec. V draws conclusions and sketches
future work directions.

## II. Related Work

The importance of timestamp quality in event logs, as well as different
techniques to repair timestamp-related problems, have been previously
researched [9, 8]. Dixit et al. [9] study the order of events that are
recorded at the same time –i.e. sharing timestamp value –, and propose a
method to identify their correct order. Addressing the same problem, Conforti
et al. [8] propose a technique to repair the timestamp information of the
events being recorded at the same instant. Their proposal first detects their
correct order, and then adjust their timestamp relying on a multimodal
distribution of the duration of process activities. Nevertheless, these
techniques do not address the problem stated in Sec. I so far, as they do not
focus on repairing the start times by including as processing time the waiting
time in which the resource is actually working on the activity.

No previous research has been done to repair the start times of an event log
by including, as processing time, the waiting time in which the resource is
actually working in the next activity instance. Nonetheless, some works have
been presented focusing on the estimation of start times in event logs with
missing data, which is related to the repair of the start times of an event
log that we propose in this paper. One of the first works was presented by
Wombacher et al. [30, 29]. The authors estimate the start time of the activity
instances in an event log with only end times, in order to model the duration
of the activities in the process. They propose to first estimate, for each
activity instance, the instant in which its resource became available as the
end time of the previous activity instance processed by the same resource.
Then, they compute the instant at which the activity instance became available
as the end time of the previous activity instance in the control flow.
Finally, they estimate the start time of each activity instance as the latest
of their resource availability and enablement time (i.e. its earliest starting
point).

Senderovich et al. in [23] develop on the enablement time heuristic presented
by Wombacher et al. [30, 29] extending it to consider parallel relations using
a given process model. They focus on the discovery of resource queues in event
logs with partial information. When the information about the activity
instances’ enablement is missing in the event log, they propose to estimate it
as the end time of the previous non-parallel activity instance. Zabka et al.
[32] have also used one of the heuristics presented by Wombacher et al. [30,
29] to estimate missing start times. They aim to calculate the workforce of a
process by estimating the duration of its executed activities. To do this,
they rely on the resource availability, and estimate the durations of an
activity instance as the difference between its end time and the end time of
the last event performed by the same resource. Then, they consider the mode of
the estimated durations of an activity as the typical processing time of that
activity. To finally set the start times of each activity instance, they
define a threshold w.r.t. the typical processing time. If the calculated
duration of an activity instance does not exceed the defined threshold, they
use the estimated duration to calculate the start time w.r.t. the recorded end
time. If the estimated duration exceeds the threshold, they estimate the start
time using the typical processing time of that activity. Fracca et al. [12]
also developed on the heuristics presented by Wombacher et al. [30, 29] to
obtain the earliest estimation for the start time of each activity instance in
an event log. Then, taking the recorded end time of each activity instance as
the latest estimation –i.e. zero duration –, they perform an optimization
process (using business process simulation) to find the point in time between
these two instants that better estimates the recorded start of the activity
instances.

In this paper, we rely on similar heuristics as the presented in these
techniques. However, they specialize on their use to estimate the start times
in event logs with only end times, and their evaluation (if any) is performed
by comparing the estimation with the actual recorded start. On the other hand,
we propose to use the heuristics to identify the waiting time in which the
resources are actually working on the activity, and increase the recorded
processing time accordingly.

Finally, another work related to start time estimation was presented by
Gonçalves et al. [13]. The authors work with event logs with two types of
events: events that correspond to human work (i.e. performed by human
resources), and events that correspond to a system (i.e. automatically
registered). They aim to estimate the duration of the activities performed by
human resources, taking as anchors the system-recorded events. For this, they
group the human-recorded events in between two consecutive system-recorded
events, and propose three frameworks to perform the estimation: i) consider
the human-recorded events as the end times of those activities and, for each
one, set its start time as its previous event; ii) consider the recorded
events as the start times of those activities and, for each one, set its end
time as its following event; and iii) split the time between the two system-
recorded events equally for all the human-performed activities inside it. This
proposal, similar to the previously commented ones, specializes on estimating
the start times of activities performed by human resources, but it requires
system-recorded events to use them as anchors. In contrast, in the present
paper, we do not require the existence of such events.

## III. Approach

Table 1: Running example: fraction of an activity instance log with 10 activity instances storing the trace, the executed activity, the start and end times, and the resource. trace ($\varphi$) | activity ($\alpha$) | start time ($\tau_{s}$) | end time ($\tau_{e}$) | resource ($\rho$)
---|---|---|---|---
$\vdots$
23 | Register Order | 2021-03-07 12:59:21 | 2021-03-07 13:05:37 | Fry
24 | Register Order | 2021-03-07 13:06:53 | 2021-03-07 13:12:11 | Fry
23 | Prepare Package | 2021-03-07 13:11:07 | 2021-03-07 14:17:29 | Leela
23 | Prepare Invoice | 2021-03-07 13:15:21 | 2021-03-07 14:21:56 | Bender
24 | Prepare Invoice | 2021-03-07 14:23:12 | 2021-03-07 15:43:01 | Bender
23 | Deliver Package | 2021-03-07 14:30:00 | 2021-03-07 16:34:10 | Leela
25 | Prepare Package | 2021-03-08 10:02:32 | 2021-03-08 10:31:00 | Zoidberg
24 | Prepare Package | 2021-03-08 10:35:53 | 2021-03-08 11:11:05 | Zoidberg
24 | Deliver Package | 2021-03-08 11:15:07 | 2021-03-08 14:37:06 | Leela
25 | Prepare Invoice | 2021-03-08 14:40:23 | 2021-03-08 14:57:48 | Leela
$\vdots$

We consider a business process that involves a set of activities $A$. We
denote each of these activities with $\alpha$. An activity instance
$\varepsilon=(\varphi,\alpha,\tau_{s},\tau_{e},\rho)$ denotes an execution of
the activity $\alpha$, where $\varphi$ identifies the process trace (i.e. the
execution of the process) to which this event belongs to, $\tau_{s}$ and
$\tau_{e}$ denote, respectively, the instants in time in which this activity
instance started and ended, and $\rho$ identifies the resource that performed
the event. Accordingly, we write $\varphi(\varepsilon_{i})$,
$\alpha(\varepsilon_{i})$, $\tau_{s}(\varepsilon_{i})$,
$\tau_{e}(\varepsilon_{i})$, and $\rho(\varepsilon_{i})$ to denote,
respectively, the process trace, the activity, the start time, the end time,
and the resource associated with the activity instance $\varepsilon_{i}$. An
activity instance log $L$ is a collection of activity instances recording the
information of the execution of a set of traces of a business process. Table 1
shown an example of 10 activity instances from an activity instance log.

It must be noted that usually the process execution information is recorded in
event logs, which record the information of an activity instance as
independent events, one representing its start, and another one representing
its end. Furthermore, there are cases in which event logs also store events
recording other stages of the activity instances’ lifecycle, e.g. schedule. In
this paper, we work with the concept of activity instance log to group the
information of each activity instance – e.g. its start and end events – in a
single element. The transformation from a typical event log to its
corresponding activity instance log is straightforward, and it has been
already described in other research [18]. Nevertheless, we will use both
terms: event log and activity instance log, to refer to the recorded process
information used as starting point; and event and activity instance, to refer
to the recording of an activity execution.

The starting point of our approach is an event log $L$. We propose to identify
the instant in time in which the resource started working on the activity by
combining the availability of its resource and the enablement time of the
executed activity (developing on the ideas presented by Wombacher et al.
[29]). The result is an enhanced event log $L^{\prime}$ with the repaired
start times, such that the waiting time of each activity instance that
corresponds to non-recorded processing time – i.e. the resource was actually
working on the activity –, is included as processing time.

### i. Resource availability

We assume that when a resource ends an activity and becomes available, she/he
will start working in the next one. This reasoning is also used by Wombacher
et al. [29], and later by Fracca et al. [12], to estimate the duration of the
activities in event logs with no start times, and by Zabka et al. [32] to
calculate activity durations in order to check if the workforce of a process
has decreased or increased after the pandemic. Obviously, we recognize that
the resource can use some time to rest before starting a new activity, or
after finishing one. Nevertheless, this time is an inherent part of the
resource performance, and should be taken into account as processing time.

To develop on this idea, we assume that the resources are dedicated only to
the project under consideration, and that there is no multitasking (i.e. the
resources only work in one activity at a time). Otherwise, the information
related to the resources’ dedication to other projects, and their performance
regarding multitasking, should be known. Following this reasoning, we define
the resource availability time (_rat_) of an activity instance as the earliest
time in which its resource became available (i.e. the end time of the previous
activity instance performed by the same resource). To calculate this value, we
build a timeline for each resource by adding all the registered end times of
the activities she/he performed. Figure 1 shows an example of this timeline
for a resource in the activity instance log depicted in Table 1. Accordingly,
the resource availability time (_rat_) of an activity instance is defined as
follows:

######  Definition 1 (Resource Availability Time).

Given an activity instance log $L$, the resource availability time of an
activity instance $\varepsilon=(\varphi,\alpha,\tau_{s},\tau_{e},\rho)$, such
that $\varepsilon\in L$, is defined as
$rat(\varepsilon)=max(\\{\tau_{e}(\varepsilon_{i})\mid\varepsilon_{i}\in
L\wedge\rho(\varepsilon_{i})=\rho(\varepsilon)\wedge\tau_{e}(\varepsilon_{i})<\tau_{e}(\varepsilon)\\})$,
i.e., the largest end time of those activity instances of $L$, processed by
the same resource as $\varepsilon$, and being previous to its end.

Figure 1: Individual timeline for the resource “Leela” corresponding to the
example in Table 1, where each gray vertical line represents the end time of
each activity instance performed by this resource.

### ii. Enablement time

As explained before, we assume that when a resource ends an activity and
becomes available, she/he will start working in the next one. However, there
might be cases in which the resource became available at one point, but could
not start the next activity because other activities of the same process trace
needed to finish before. An example of this can be seen in Table 1, in which
the activity Deliver Package performed by Leela on 2021-03-08 14:37:06 could
not start when she finished her previous activity (2021-03-07 16:34:10),
because the package of that delivery was not prepared until 2021-03-08
11:11:05.

To avoid this problem, and following with the reasoning of Wombacher et al.
[29], and later Fracca et al. [12], we also consider the enablement time of
the executed activity. The enablement time ($ent$) of an activity instance is
the instant in time in which it became available for processing. For example,
in the event log from Table 1, an order has to be registered first, then the
package and the invoice are prepared (in parallel) and, after that, the
package can be delivered. In this case, the delivery of the package cannot be
performed until both the invoice and the package are prepared and, thus, its
activity is not enabled for processing until that time.

A first approach to estimate the enablement time of an activity instance is to
use the end time of the chronologically preceding activity instance (as
proposed in [29, 12]), assuming that the process follows a sequential order
where each activity enables the following one. Nevertheless, this observation
does not hold if there are concurrency relations between the activities of the
process – a common situation in real-life processes. A concurrency relation
between two activities $A$ and $B$ means that these activities can be executed
in any order, i.e. sometimes $A$ following $B$ and other times $B$ following
$A$, thus no one enables the other222We note that other notions of concurrency
have been proposed in the field of process mining. An in-depth treatment of
concurrency notions in process mining is provided in [3]. In this paper, we
adopt a basic notion of interleaved concurrency between pairs of activities as
defined above. The proposed approach is, however, modular and could be adapted
to exploit more sophisticated notions of concurrency.. For example, in Table
1, activities Prepare Package and Prepare invoice are executed in any order
(and sometimes even in parallel), as no one is required to start the other.

Senderovich et al. [23] proposed to use a process model to identify the
preceding activity(ies) of each activity, i.e. the activity(ies) enabling it.
However, this method requires a process model as input (of the discovery of
one), and requires handling deviations from the process to identify the
preceding activity when a trace does not follow a path represented in the
model. To remove this dependency, we propose to invert the analysis and use a
concurrency oracle that indicate which activities are concurrent between them
and, thus, not enabling each other. In this way, the enablement time of an
activity instance can be computed as the end time of the closest previous
activity not being concurrent to it. Accordingly, the enablement time of an
activity instance can be defined as follows:

Figure 2: Timeline of trace 24 corresponding to the example in Table 1, where
each gray vertical line represents the end time of each activity instance, and
each dotted arrow connects an activity instance (source) with an activity
instance it enables (target).

######  Definition 2 (Enablement Time).

Given an activity instance log $L$, and a concurrency oracle that states if
two activities $\alpha_{i},\alpha_{j}\in L$ have
($\alpha_{i}\parallel\alpha_{j}$) or do not have
($\alpha_{i}\nparallel\alpha_{j}$) a concurrent relation, the enablement time
of an activity instance $\varepsilon=(\varphi,\alpha,\tau_{s},\tau_{e},\rho)$,
such that $\varepsilon\in L$, is defined as
$ent(\varepsilon)=max(\\{\tau_{e}(\varepsilon_{i})\mid\varepsilon_{i}\in
L\wedge\varphi(\varepsilon_{i})=\varphi(\varepsilon)\wedge\tau_{e}(\varepsilon_{i})<\tau_{e}(\varepsilon)\wedge\alpha(\varepsilon_{i})\nparallel\alpha(\varepsilon)\\})$,
i.e., the largest end time of those activity instances of $L$, belonging to
the same process trace, being previous to $\varepsilon$’s end time, and which
activity does not have a concurrent relation with $\varepsilon$’s activity.

We propose to use the concurrency oracle of the Heuristics Miner [28]
algorithm to discover the concurrency relations between the process’
activities. This method computes a degree of confidence for each observed
relation between two activities that it is a concurrent or directly-follows
relation, based on the percentage of occurrences in each order. Then, based on
a set of defined thresholds, it retrieves the concurrent relations of the
process. It must be noted that our proposal to compute the enablement time is
not dependent on the concurrency oracle implementation. The concurrency
relations could be alternatively extracted with other concurrency discovery
algorithm, or from a process model, if available.

Figure 2 shows the timeline for one process trace, depicting the enablement
relations between the activities. As can be seen, both Prepare Invoice and
Prepare Package, which have a concurrency relation between them, are enabled
by Register Order. Finally, Deliver Package is enabled by Prepare Package, its
previous activity not being concurrent to it.

### iii. Start time repair

In order to include the non-recorded processing time into each activity
instance processing time, we propose to set its start time to the earliest
point in time in which the activity instance is enabled, and its resource is
available for processing. Figure 3 shows an example of this adjustment for an
activity instance of the event log in Table 1. Accordingly, the repair of an
activity instance’s start time can be defined as follows:

######  Definition 3 (Start Time Repair).

Given an activity instance log $L$, the repaired start time of an activity
instance $\varepsilon=(\varphi,\alpha,\tau_{s},\tau_{e},\rho)$, such that
$\varepsilon\in L$, is defined as
$\tau^{\prime}_{s}(\varepsilon)=max(rat(\varepsilon),ent(\varepsilon))$, i.e.,
the largest of its enablement time and its resource availability time.

There may exist cases in which an activity instance does not have a resource
assigned. In those cases, our technique assigns to the activity instance a
resource pool with maximum capacity, meaning this that the activity can be
executed as soon as it is enabled. Thus, its repaired start time will be
estimated as its enablement time.

Figure 3: Timeline denoting the end time, resource availability, and
enablement time for the activity instance “Deliver Package”, of the process
trace 24, corresponding to the example in Table 1. As well as its duration
based on the estimated start time.

In order to exclude the case of automatic activities which are executed at the
same time as others, e.g. control automatic activities performed after the end
of a subprocess, we offer the possibility to the user to indicate those
resources that are "bot resources" and execute the activities instantly. In
the same way, we offer the possibility to identify activities that are always
executed automatically. Accordingly, we let the start time of those cases as
their end time.

Developing on the idea presented by Zabka et al. [32], we offer a threshold to
control outlier estimations. For this, we first estimate the earliest starting
point of all activity instances. Then, we calculate their most typical
repaired duration (from the earliest starting point to the recorded end time).
Finally, given a threshold $\eta$, we establish an upper bound such that all
activity instances with a repaired duration over $\eta$ times their typical
repaired duration gets their earliest starting point re-estimated, so that
their repaired duration is $\eta$ times their typical repaired duration.

## IV. Evaluation

This section reports on an experimental evaluation of the proposed approach to
evaluate the impact of the proposed start time repair in the simulation
results.

Our hypothesis is that our repaired start times will lead to simulated logs
that replicate the temporal dynamics recorded in the original log better than
a BPS model discovered directly from the original log. The reasoning behind
this is that the simulation does not take into account exogenous waiting
times, e.g. waiting for the client to complete an activity or answer a
question. Thus, with the premise of an activity starting once the resource is
available and the activity enabled, we include that information in the
processing times (as an indirect temporal cost for the activities of the
process), and the temporal similarity is expected to be better. In this way,
our technique generates user utilization that are reflective of the
availability of the resource to perform the activities, calculating better the
tempo-dynamics of the process.

It must be noted that our hypothesis of an activity starting when the resource
is available works better under the assumption that the resources are
internal, even better with resources dedicated only to the project under
study. If the resource is external —e.g. the client, or a consultant from
another company—, there is a possibility that an activity instance did not
start even when the activity was enabled and the resource available, because
he/she was busy with activities from other projects. However, to prevent this
from happening, the information of the resources’ participation in other
projects would have to be present in the event log.

### i. Datasets

In order to enhance the ecological validity of the empirical evaluation, we
selected a set of real-life event logs with balanced complexity
characteristics:

Table 2: Characteristics of the event logs used in the evaluation: 4 real-life logs of low complexity and 4 real-life logs of high complexity. The multitasking column denotes the percentage of the resources’ working time in which they are working in more than one activity. | Type | Complexity | Traces |  | Activity
---
instances
Variants | Activities | Resources | Multitasking
ACR | REAL | Low | 954 | 4,962 | 99 | 16 | 559 | 20%
BNK | REAL | Low | 70,512 | 415,261 | 198 | 8 | 9 | 99%
MP | REAL | Low | 225 | 4,503 | 217 | 24 | 46 | 20%
CALL | REAL | Low | 3,885 | 7,548 | 354 | 6 | 48 | 52%
GOV | REAL | High | 23,506 | 110,659 | 1,003 | 131 | 30 | 95%
INS | REAL | High | 1,182 | 23,141 | 1,097 | 9 | 136 | 49%
BPIC12W | REAL | High | 8,616 | 59,302 | 2,115 | 6 | 57 | 9%
BPIC17W | REAL | High | 30,276 | 240,854 | 6,356 | 8 | 148 | 3%

  * •

Four real-life event logs of low complexity, with a number of variants ranging
from 99 to 354. These logs are from an academic credentials’ management
process (ACR), a bank process (BNK), an industrial production line (MP), and a
call center process (CALL).

  * •

Four real-life event logs of high complexity, with a number of variants
ranging from 1,000 to more than 6,000. One of these event logs records the
execution of an approval application system of a governmental agency (GOV).
Another one corresponds to an insurance process (INS). We also used the event
log from the Business Process Intelligence Challenge of 2012 [26], recording a
loan application process from a Dutch financial institution. We preprocessed
this event log retaining only the events corresponding to activities performed
by human resources (i.e. only activity instances that have a duration).
Finally, we also used the updated version of the BPIC 2012 event log, of the
Business Process Intelligence Challenge of 2017 [27]. We preprocessed this
event log by following the recommendations reported by the winning teams
participating in the BPIC 2017
(https://www.win.tue.nl/bpi/doku.php?id=2017:challenge).

The main characteristics of the event logs used in this experimentation can be
seen in Table 2.

Table 3: Different configurations of the technique proposed in this paper, where $\overline{x}$ denotes the statistic used to calculate the most typical duration, and $\eta$ the outlier threshold. | MED | MED-5 | MED-2 | MOD | MOD-5 | MOD-2
---|---|---|---|---|---|---
$\overline{x}$ | Median | Median | Median | Mode | Mode | Mode
$\eta$ | ✗ | 500% | 200% | ✗ | 500% | 200%

### ii. Experimental setup

To evaluate the impact of outlier estimations – e.g. due to the resource not
working for a long period –, we used three different configurations of our
proposal: i) with no outlier threshold, meaning that the repaired start times
are the final ones; ii) with an outlier threshold of 2 (low threshold),
meaning that all activity instances with a repaired duration exceeding 2 times
their most typical repaired duration get their start time re-repaired to that
duration; and iii) with an outlier threshold of 5 (high threshold), so the re-
repaired activity instances are those with a repaired duration over 5 times
their most typical repaired duration. For each configuration, we used both the
median and the mode to compute the most typical repaired duration, in order to
check if the results depend on the chosen statistics. Table 3 shows the
configurations of all the evaluated techniques.

For each dataset, we run each of the configuration to repair the start times
of each event log, obtaining six different repaired event logs (one per
configuration). Then, we run SIMOD [5] to automatically discover a simulation
process with each of the six repaired event logs, as well as with the original
event log. With this simulation process, we simulated five event logs with the
same number of traces than the original one. Finally, we measured the distance
of each simulated log w.r.t. the original one (for each proposal, the result
is the median of its 5 simulations).

To validate if the repaired start times improve the simulation results, we
compared the distance of the simulated logs w.r.t. the original one – the
lower the distance is, the more similar the simulated log is w.r.t. the
original log. In previous work, such comparisons are made along two
dimensions: the control-flow and the temporal dimensions [5, 6]. Given that a
start time repair does not alter the control-flow of the process, we hereby
focus on the temporal dimension. Specifically, we compared each simulated log
against the corresponding original log in two ways: i) by comparing the
distributions of their timestamps (both start and end timestamps); and ii) by
comparing the distribution of cycle times of their traces.333The cycle time of
a trace (a.k.a. the trace duration) is the difference between the largest end
timestamp and the smallest start timestamp observed in the trace. To compare
two distributions, we use the Wasserstein Distance – Earth Movers’ Distance
(EMD) –, which measures the amount of movements that have to be applied to one
discretized distribution to obtain the other, penalizing the movements by
their distance.

The EMD is defined over a pair of discretized distributions (i.e. histograms).
To discretize the distribution of timestamps, we extract all the start and end
timestamps in the log, and we group them by date-hour. For example, all
timestamps that fall between 9 February 2022 at 10:00:00 and the same day at
10:59:59 are placed in one group (bin). On the other hand, the cycle times are
discretized as follows. First, the cycle times of the original log are grouped
into 100 equidistant bins. Let $W$ be the width between these bins. Each
simulated log derived from this original log is then discretized into bins of
width $W$. Having discretized the timestamps and the cycle times of the
original log and its simulated counterparts, we then calculate their EMD and
use this measure to compare the simulation performed with the original log,
with each of the six variants of the proposed technique. Importantly, the
magnitude of the EMD depends on the dataset (e.g. larger datasets will
naturally give rise to larger EMD values). Accordingly, we use the EMD to make
intra-dataset comparisons only (i.e. to measure the relative performance of
multiple techniques within the same dataset).

It must be noted that the similarity of the simulated logs using the original
event log have an advantage, as their BPS models have been discovered by
training with the event logs then used for the similarity evaluation. On the
other hand, the BPS models of the other simulated logs have been trained with
the repaired event logs.

### iii. Results

Table 4: EMD of the timestamps for the simulated logs w.r.t. the original log (the lower, the more similar the simulated logs are w.r.t. the original log). The gray cells mark, for each dataset, the best result. | ACR | BNK | MP | CALL | GOV | INS | BPIC12W | BPIC17W
---|---|---|---|---|---|---|---|---
ORIG | 456.32 | 871.77 | 94.16 | 1,338.05 | 2,519.65 | 9,466.29 | 2,170.63 | 5,823.61
MED | 2,793.60 | 9,476.72 | 4,703.49 | 80.77 | 6,658.81 | 3,709.33 | 3,835.93 | 925.40
MED-5 | 2,770.54 | 408.45 | 1,988.31 | 73.08 | 6,330.92 | 9,242.26 | 166.80 | 5,830.43
MED-2 | 2,821.47 | 1,897.83 | 964.11 | 112.83 | 6,574.81 | 9,182.91 | 2,558.82 | 2,085.34
MOD | 130.02 | 3,108.90 | 2,238.73 | 93.88 | 6,656.59 | 8,495.60 | 686.85 | 789.32
MOD-5 | 372.14 | 2,672.34 | 2,276.79 | 75.06 | 6,644.68 | 8,801.83 | 2,383.89 | 2,094.49
MOD-2 | 2,989.33 | 1,612.38 | 1,529.58 | 112.90 | 6,686.00 | 8,362.68 | 2,332.21 | 2,247.24
$0$$500$$1{,}000$$1{,}500$TimelineNumber of eventsORIGS-ORIGS-MODE Figure 4:
Histograms of the start-end timestamps for the BPIC17W dataset. ORIG
corresponds to the timestamps of the original event log. S-ORIG corresponds to
the timestamps of the event log simulated with the original one. S-MODE
corresponds to the timestamps of the event log simulated with the repaired
event log (no outlier threshold and using the mode as the statistic).

Table 4 depicts the EMD value corresponding to the start-end timestamps. As
can be seen, the repaired start times improve the EMD values of the simulated
event logs (compared to simulating using the original event log) in six
datasets (all datasets except MP and GOV). All the configurations of our
proposal improve the simulation with the original log in CALL, INS, and
BPIC17W datasets, except for the MED-5 configuration in BPIC17W, which gets
slightly worse results (0.1% worse). In ACR and BPIC12W, only two of our
approach configurations improve the results. The impact of the multitasking
can be seen in the BNK (99%) where only one of our approach configurations
overcome the original log, and in GOV (95%) where the best result is obtained
by the simulation with the original log. Finally, in the MP dataset, the best
results are also obtained by the original log. The meaning of a better EMD
value can be better interpreted in Figure 4, which depicts the start-end
timestamp distributions for the BPIC17W process (binned by hour). As we can
see in Figure 4, using the repaired event log to discover the simulation model
leads to simulations reproducing better the distribution of events than using
the original event log.

Table 5 shows the EMD values for the cycle times distribution in the event
logs simulated with each technique. Regarding this dimension, the
configuration of our approach without outlier detection and using the mode as
statistic (MOD) obtains the best results in all datasets but two (BNK and MP).
This shows that discovering the simulation model with the repaired start times
not only allows the simulation to mimic better the original behavior in terms
of event distribution, but also in therms of cycle times.

Table 5: EMD cycle times for the simulated logs (mean of 5 simulations). The gray cells mark, for each dataset, the best result out of the repair techniques. | ACR | BNK | MP | CALL | GOV | INS | BPIC12W | BPIC17W
---|---|---|---|---|---|---|---|---
ORIG | 10.31 | 2.33 | 15.55 | 18.19 | 14.32 | 10.80 | 9.98 | 3.95
MED | 8.20 | 5.17 | 286.27 | 3.84 | 14.18 | 29.91 | 95.29 | 3.93
MED-5 | 9.71 | 5.18 | 33.26 | 3.88 | 14.36 | 11.32 | 9.47 | 3.96
MED-2 | 10.50 | 5.18 | 18.48 | 3.89 | 14.36 | 11.33 | 10.06 | 3.96
MOD | 7.58 | 5.17 | 129.49 | 3.38 | 14.18 | 9.78 | 7.51 | 3.93
MOD-5 | 9.80 | 5.18 | 67.27 | 3.91 | 14.34 | 11.35 | 10.06 | 3.96
MOD-2 | 10.52 | 5.18 | 21.52 | 3.89 | 14.35 | 11.33 | 10.06 | 3.96

In summary, regarding the different configurations of our proposal, there is
no configuration overcoming the rest in all datasets. At an individual level,
the configuration without outlier threshold and using the mode as statistic
(MOD) overcomes the simulation with the original log in five of the eight
datasets in terms of the timestamp distribution, and obtains the best result
in six of the eight datasets in terms of cycle time. At a generic level, in
order to improve simulation results, we recommend using different
configurations, as the nature of the process can affect the results. For
example, a process with highly variable activity durations and waiting times
can benefit from the use of the median as a statistic, as the mode can return
an unreliable value if all repaired durations have a low frequency. Also, a
low outlier threshold value reduces excessively the variability in the
estimations and, as we can see in the results, this ends in less accurate
simulations – the two configurations with an outlier threshold of 2 (MED-2 and
MOD-2) did not get the best result in any dataset –, although they still
overcome the simulation with the original log in some cases.

### iv. Threats to validity

The evaluation reported above is potentially affected by the following threats
to the validity.

First, regarding internal validity, the experiments rely only on a dozen
events logs. The results could be different for other datasets. To mitigate
this threat, we selected logs with different sizes and characteristics and
from different domains.

Second, regarding construct validity, we used two measures of goodness based
on discretized distributions. The results could be different if we employed
other measures, e.g. measures of similarity between time series based on
dynamic time warping, which provides an alternative framework for capturing
rhythms in time series.

Finally, regarding ecological validity, the evaluation compares the simulation
results against the original log. While this allows us to measure how well the
simulation models replicate the as-is process, it does not allow us to assess
the goodness of the simulation models in a what-if setting, i.e., predicting
the performance of the process after a change.

## V. Conclusion and Future Work

In this paper, we presented an approach to identify the waiting time in an
event log that corresponds to non-recorded processing time, and include it as
processing time by repairing the start times of the activity instances. The
approach is designed to repair the start timestamps in event logs in order to
capture all the processing time associated to each activity instance, with the
final aim of improving the quality of automatically discovered simulation
models.

The proposal combines an approach to estimate the enablement time of activity
instances in the presence of concurrency relations, and an approach to
estimate the resource availability times. In a nutshell, the underpinning
assumption is that, at least for the purposes of BPS, an activity instance can
be considered as started once it is enabled, and the resource who performed
the activity is available.

We conducted an evaluation to analyze the impact that the start time repair
has on the accuracy of automatically discovered BPS models. To this end, we
discovered a BPS model using both the original event log and the event log
with repaired start times. For the majority of real-life datasets, the BPS
models discovered from the event logs with repaired start times exhibited
higher accuracy than the BPS models discovered from the original logs. This
observation suggests that the start times repaired by our proposal are more
suitable for BPS model discovery than the start times recorded during the
execution of a process in an enterprise system (subject to threats to validity
discussed above).

A limitation of our proposal is that it requires that there is sufficient
information to estimate the enablement time of each activity instance. This is
not possible for the first activity instance in a trace, since there is no
previous activity instance (in the same trace) that can be used as an anchor
to estimate the enablement time. We foresee that this limitation could be
addressed by applying a technique to estimate the time when each process trace
arrives [16]. Indeed, the trace arrival time could be used as an anchor to
estimate the enablement time of the first activity(ies) in the trace.

Another limitation related to the evaluation is that the BPS model
automatically discovered with the original event log does not model the
waiting times previous to each activity. This could hinder its performance and
increase the difference w.r.t. the BPS models discovered with the repaired
start times. For this reason, we plan to compare the improvement of our
approach with a waiting time aware BPS model.

Furthermore, as previously acknowledged, our proposal assumes that there is no
multitasking (i.e. resources working in more than one activity instance at a
time). The evaluation suggests that the performance of the proposed approach
degrades when there is a high level of multitasking. Addressing this
shortcoming is another avenue for future work.

## acknowledgements

This research is funded by the European Research Council (PIX Project).

## References

  * [1] Santiago Aguirre, Carlos Parra, and Jorge Alvarado.  Combination of process mining and simulation techniques for business process redesign: A methodological approach.  In Philippe Cudré-Mauroux, Paolo Ceravolo, and Dragan Gasevic, editors, Proceedings of the International Symposium on Data-Driven Process Discovery and Analysis (SIMPDA 2012), volume 162 of LNBIP, pages 24–43. Springer, 2012.
  * [2] Robert Andrews and Moe Thandar Wynn.  Shelf time analysis in CTP insurance claims processing.  In U Kang, Ee-Peng Lim, Jeffrey Xu Yu, and Yang-Sae Moon, editors, Trends and Applications in Knowledge Discovery and Data Mining (PAKDD 2017 Workshops), volume 10526 of LNCS, pages 151–162. Springer, 2017.
  * [3] Abel Armas-Cervantes, Marlon Dumas, Marcello La Rosa, and Abderrahmane Maaradji.  Local concurrency detection in business process event logs.  ACM Trans. Internet Techn., 19(1):16:1–16:23, 2019.
  * [4] Jagadeesh Chandra J. C. Bose, R. S. Mans, and Wil M. P. van der Aalst.  Wanna improve process mining results?  In IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2013, Singapore, 16-19 April, 2013, pages 127–134. IEEE, 2013\.
  * [5] Manuel Camargo, Marlon Dumas, and Oscar González.  Automated discovery of business process simulation models from event logs.  Decis. Support Syst., 134:113284, 2020.
  * [6] Manuel Camargo, Marlon Dumas, and Oscar González Rojas.  Discovering generative models from event logs: data-driven simulation vs deep learning.  PeerJ Comput. Sci., 7:e577, 2021.
  * [7] Minsu Cho, Minseok Song, and Sooyoung Yoo.  A systematic methodology for outpatient process analysis based on process mining.  In Chun Ouyang and Jae-Yoon Jung, editors, Proceedings of the 2nd Asia Pacific Business Process Management Conference, (AP-BPM 2014), volume 181 of LNBIP, pages 31–42. Springer, 2014.
  * [8] Raffaele Conforti, Marcello La Rosa, Arthur H. M. ter Hofstede, and Adriano Augusto.  Automatic repair of same-timestamp errors in business process event logs.  In Dirk Fahland, Chiara Ghidini, Jörg Becker, and Marlon Dumas, editors, Proceedings of the 18th International Conference on Business Process Management (BPM 2020), volume 12168 of LNCS, pages 327–345. Springer, 2020.
  * [9] Prabhakar M. Dixit, Suriadi Suriadi, Robert Andrews, Moe Thandar Wynn, Arthur H. M. ter Hofstede, Joos C. A. M. Buijs, and Wil M. P. van der Aalst.  Detection and interactive repair of event ordering imperfection in process logs.  In John Krogstie and Hajo A. Reijers, editors, Advanced Information Systems Engineering - 30th International Conference, CAiSE 2018, Tallinn, Estonia, June 11-15, 2018, Proceedings, volume 10816 of Lecture Notes in Computer Science, pages 274–290. Springer, 2018.
  * [10] Marlon Dumas, Marcello La Rosa, Jan Mendling, and Hajo A. Reijers.  Fundamentals of Business Process Management, Second Edition.  Springer, 2018.
  * [11] Dominik Andreas Fischer, Kanika Goel, Robert Andrews, Christopher G. J. van Dun, Moe Thandar Wynn, and Maximilian Röglinger.  Enhancing event log quality: Detecting and quantifying timestamp imperfections.  In Dirk Fahland, Chiara Ghidini, Jörg Becker, and Marlon Dumas, editors, Proceedings of the 18th International Conference on Business Process Management (BPM 2020), volume 12168 of LNCS, pages 309–326. Springer, 2020.
  * [12] Claudia Fracca, Massimiliano de Leoni, Fabio Asnicar, and Alessandro Turco.  Estimating activity start timestamps in the presence of waiting times via process simulation.  In Xavier Franch, Geert Poels, Frederik Gailly, and Monique Snoeck, editors, Advanced Information Systems Engineering - 34th International Conference, CAiSE 2022, Leuven, Belgium, June 6-10, 2022, Proceedings, volume 13295 of Lecture Notes in Computer Science, pages 287–303. Springer, 2022.
  * [13] Rodrigo M. T. Gonçalves, Rui Jorge Almeida, João M. C. Sousa, and Remco M. Dijkman.  Estimation and characterization of activity duration in business processes.  In João Paulo Carvalho, Marie-Jeanne Lesot, Uzay Kaymak, Susana M. Vieira, Bernadette Bouchon-Meunier, and Ronald R. Yager, editors, Proceedings of the 16th International Conference on Information Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU 2016), volume 611 of CCIS, pages 729–740. Springer, 2016.
  * [14] Owen A. Johnson, Thamer Ba Dhafari, Angelina Kurniati, Frank Fox, and Eric Rojas.  The clearpath method for care pathway process mining and simulation.  In Florian Daniel, Quan Z. Sheng, and Hamid Motahari, editors, Workshops of the 16th International Conference on Business Process Management (BPM 2018), volume 342 of LNBIP, pages 239–250. Springer, 2018.
  * [15] Ronny Mans, Hajo A. Reijers, Daniel Wismeijer, and Michiel van Genuchten.  A process-oriented methodology for evaluating the impact of IT: A proposal and an application in healthcare.  Inf. Syst., 38(8):1097–1115, 2013.
  * [16] Niels Martin, Benoît Depaire, and An Caris.  Using event logs to model interarrival times in business process simulation.  In Manfred Reichert and Hajo A. Reijers, editors, Workshops of the 13th International Conference on Business Process Management (BPM 2015), volume 256 of LNBIP, pages 255–267. Springer, 2015.
  * [17] Niels Martin, Benoît Depaire, and An Caris.  The use of process mining in business process simulation model construction - structuring the field.  Bus. Inf. Syst. Eng., 58(1):73–87, 2016.
  * [18] Niels Martin, Luise Pufahl, and Felix Mannhardt.  Detection of batch activities from event logs.  Inf. Syst., 95:101642, 2021.
  * [19] Laura Maruster and N. R. T. P. van Beest.  Redesigning business processes: a methodology based on simulation and process mining techniques.  Knowl. Inf. Syst., 21(3):267–297, 2009.
  * [20] Francesca Meneghello, Claudia Fracca, Massimiliano de Leoni, Fabio Asnicar, and Alessandro Turco.  A framework to improve the accuracy of process simulation models.  In Renata S. S. Guizzardi, Jolita Ralyté, and Xavier Franch, editors, Research Challenges in Information Science - 16th International Conference, RCIS 2022, Barcelona, Spain, May 17-20, 2022, Proceedings, volume 446 of Lecture Notes in Business Information Processing, pages 142–158. Springer, 2022.
  * [21] Kristina Rosenthal, Benjamin Ternes, and Stefan Strecker.  Business process simulation on procedural graphical process models.  Bus. Inf. Syst. Eng., 63(5):569–602, 2021.
  * [22] Anne Rozinat, R. S. Mans, Minseok Song, and Wil M. P. van der Aalst.  Discovering simulation models.  Inf. Syst., 34(3):305–327, 2009.
  * [23] Arik Senderovich, Sander J. J. Leemans, Shahar Harel, Avigdor Gal, Avishai Mandelbaum, and Wil M. P. van der Aalst.  Discovering queues from event logs with varying levels of information.  In Manfred Reichert and Hajo A. Reijers, editors, Business Process Management Workshops - BPM 2015, 13th International Workshops, Innsbruck, Austria, August 31 - September 3, 2015, Revised Papers, volume 256 of Lecture Notes in Business Information Processing, pages 154–166. Springer, 2015.
  * [24] Suriadi Suriadi, Robert Andrews, Arthur H. M. ter Hofstede, and Moe Thandar Wynn.  Event log imperfection patterns for process mining: Towards a systematic approach to cleaning event logs.  Inf. Syst., 64:132–150, 2017.
  * [25] Suriadi Suriadi, Chun Ouyang, Wil M. P. van der Aalst, and Arthur H. M. ter Hofstede.  Event interval analysis: Why do processes take time?  Decis. Support Syst., 79:77–98, 2015.
  * [26] Boudewijn van Dongen.  Bpi challenge 2012, Apr 2012.
  * [27] Boudewijn van Dongen.  Bpi challenge 2017, Feb 2017.
  * [28] A. J. M. M. Weijters and Joel Tiago S. Ribeiro.  Flexible heuristics miner (FHM).  In Proceedings of the IEEE Symposium on Computational Intelligence and Data Mining (CIDM 2011), pages 310–317. IEEE, 2011.
  * [29] Andreas Wombacher and Maria-Eugenia Iacob.  Start time and duration distribution estimation in semi-structured processes.  In Sung Y. Shin and José Carlos Maldonado, editors, Proceedings of the 28th Annual ACM Symposium on Applied Computing, SAC ’13, Coimbra, Portugal, March 18-22, 2013, pages 1403–1409. ACM, 2013.
  * [30] Andreas Wombacher, Maria-Eugenia Iacob, and Martin Haitsma.  Towards a performance estimate in semi-structured processes.  In Kwei-Jay Lin, Christian Huemer, M. Brian Blake, and Boualem Benatallah, editors, 2011 IEEE International Conference on Service-Oriented Computing and Applications, SOCA 2011, Irvine, CA, USA, December 12-14, 2011, pages 1–5. IEEE Computer Society, 2011.
  * [31] Moe Thandar Wynn, Marlon Dumas, Colin J. Fidge, Arthur H. M. ter Hofstede, and Wil M. P. van der Aalst.  Business process simulation for operational decision support.  In Arthur H. M. ter Hofstede, Boualem Benatallah, and Hye-Young Paik, editors, Workshops of the 5th International Conference on Business Process Management (BPM 2007), volume 4928 of LNCS, pages 66–77. Springer, 2007.
  * [32] Wolf-Dietrich Zabka, Peter Blank, and Rafael Accorsi.  Has the pandemic impacted my workforce’s productivity? applying effort mining to identify productivity shifts during COVID-19 lockdown.  In Industry Forum of the 19th International Conference on Business Process Management (BPM 2021), 2021.


# Generative Rendering: Controllable 4D-Guided Video Generation with 2D
Diffusion Models

Shengqu Cai1,2,△ Duygu Ceylan2∗ Matheus Gadelha2∗ Chun-Hao Paul Huang2
Tuanfeng Yang Wang2 Gordon Wetzstein1
1Stanford University 2Adobe Research

###### Abstract

Traditional 3D content creation tools empower users to bring their imagination
to life by giving them direct control over a scene’s geometry, appearance,
motion, and camera path. Creating computer-generated videos, however, is a
tedious manual process, which can be automated by emerging text-to-video
diffusion models. Despite great promise, video diffusion models are difficult
to control, hindering a user to apply their own creativity rather than
amplifying it. To address this challenge, we present a novel approach that
combines the controllability of dynamic 3D meshes with the expressivity and
editability of emerging diffusion models. For this purpose, our approach takes
an animated, low-fidelity rendered mesh as input and injects the ground truth
correspondence information obtained from the dynamic mesh into various stages
of a pre-trained text-to-image generation model to output high-quality and
temporally consistent frames. We demonstrate our approach on various examples
where motion can be obtained by animating rigged assets or changing the camera
path. Project page: primecai.github.io/generative_rendering.

Figure 1:  Generative Rendering is a novel diffusion-based approach to render
3D animated but untextured scenes (left) directly to a stylized animation
(right), with styles specified by text prompts. Our approach offers new levels
of user control to image generation models.

††$\triangle$: Part of this work was done during an internship at Adobe
Research.††* : Equal contribution.

##  1 Introduction

Artists, designers, architects, and other creators rely on traditional 3D
content creation tools to render computer-generated videos. Unfortunately,
existing 3D workflows are laborious, time consuming, and require expertise.
Emerging generative artificial intelligence tools, such as text-to-image (T2I)
and text-to-video (T2V) models, solve these issues by automating many of the
manual steps of traditional workflows. Video generation, however, is difficult
to control in that it is not easily possible to specify scene layout and
motion in a temporally consistent manner.

Recent approaches have attempted to control diffusion models. For example,
ControlNet [41] uses a pre-trained T2I diffusion model and finetunes an
adapter network that is conditioned on depth, pose, or edge images to control
the layout. This strategy is successful for generating individual frames, but
results in flicker for video generation. Other approaches aim at learning the
complex types of motions encountered in natural videos directly [12, 32, 20,
42, 35, 39, 40, 3, 9, 11]. While successful in generating smooth motions,
these approaches are not easily controllable. Finally, video-to-video
diffusion models [6, 10, 19, 37] enable video editing and stylization, but
they require a high-fidelity video as input, which is not always available.

In this paper, we aim to combine the power of 3D workflows with T2I models for
generating 4D-guided stylized animations. Specifically, we leverage 3D tools
to rapidly prototype proxy geometry and motion of a scene (e.g., camera paths,
physically based simulation, or character animation) while utilizing the T2I
generation model as a _renderer_ to output the final stylized animations (see
Fig. 1). At the core of our method is the ability to leverage the ground truth
4D spatio-temporal correspondences that can be obtained from an animated 3D
scene to guide the image generation. First, we leverage the correspondence
information to effectively perform noise initialization which is critical for
temporal consistency. Many recent video-to-video stylization methods resort to
inverting the source video to obtain consistent noise initialization [6, 10].
However, this is not possible in our setup since we do not have an initial
source video and inversion does not work well on untextured renderings of the
3D scene. Instead, we propose to utilize the canonical representation (i.e.,
UV space) of the 3D scene to initialize random noise which is then projected
to each frame of the animation. Next, we enrich the self-attention layers of
the image diffusion model which are known to contain spatial appearance
information vital for obtaining consistent renderings [6, 37]. Specifically,
we propagate both the input and output of self-attention layers across the
frames via known correspondences to enforce more consistent results. Finally,
similar to previous work [41], we use the depth cues rendered from the 3D
scene in conjunction with control models to provide structure guidance. In
summary, our framework treats the pre-trained T2I diffusion model as a multi-
frame renderer that is able to capture the correspondences while maintaining
high-fidelity and consistent generations.

We evaluate our method on a variety of scenes and demonstrate its advantages
with comparisons to state-of-the-art methods. In summary, our contributions
include:

  * •

We present a novel framework for 4D-guided animation synthesis utilizing the
pre-trained T2I generation models as a multi-frame renderer.

  * •

We enhance the self-attention layers of the image generation model by
performing correspondence-aware blending of both input and output features to
enforce consistent appearance synthesis.

  * •

We introduce a UV-space noise initialization mechanism. Combined with the
correspondence-aware attention mechanism, this enables better consistency
across frame generations.

Figure 2:  Overview of our pipeline. Our system takes as input a set of UV and
depths maps rendered from an animated 3D scene. We use a depth-conditioned
ControlNet [41] to generate corresponding frames while using the UV
correspondences to preserve the consistency. We initialize the noise in the UV
space of each object which we then render into each image (Sec. 3.3). For each
diffusion step, we first use extended attention for a set of keyframes and
extract their pre- and post-attention features (Sec. 3.1). The post-attention
features are projected to the UV space and unified. Finally, all frames are
generated using a weighted combination of the outputs of the extended
attention with the pre-attention features of the keyframe, and the UV-composed
post-attention features from the keyframes (Sec. 3.2).

##  2 Related Work

In this section, we discuss the methods most closely related to ours. A
comprehensive review of diffusion models for visual computing can be found in
[25].

#### Video Generation

T2V diffusion models extend T2I models by generating dynamic scenes. The
majority of works in this area focus on extending pre-trained T2I diffusion
models and finetuning on video data [9, 3, 14, 12]. A typical method of
lifting image-based models to videos is to add a time-aware module. This can
include 3D convolutions [3], or pseudo-3D convolutions factorized over spatial
and temporal dimensions to reduce computational costs [11, 9, 7, 32, 7].
Recently, [11] has shown very compelling video generation results by training
a motion module with such temporal layers on top a frozen StableDiffusion
model [31]. All of these approaches, however, lack fine-grained structural
control of the video generation process in a temporally stable manner.
Moreover, training a high-quality T2V model is computationally demanding and
challenging, in part due to the limited sizes of available captioned high-
quality video datasets. This is part of the reason why existing T2V models are
limited to outputting only a few seconds of animation. For these reasons, we
build on T2I diffusion models and directly leverage ground truth
correspondences of the input meshes to generate highly expressive and
temporally consistent animations of arbitrary length.

#### Video Editing and Stylization

Another line of work focuses on video editing and stylization. Given an input
video, video editing/stylization aims to change the input video according to a
user-defined prompt, while keeping other components harmonious. Earlier work,
such as [2], decomposes a video to atlases and performs editing directly on
these atlases. This approach obtains perfect consistency, but it is only
applicable to certain types of scenes and the decomposition is very slow to
compute. Exemplar works based on diffusion models [6, 7, 37, 19, 27] along
this path generally rely heavily on extracting temporal information from the
input video, e.g., via DDIM inversion [16], then inflating the self-attention
modules to attend to multiple frames, and propagating the edited contents
across different frames. These techniques typically overfit to video inputs
and are not directly applicable to our setting. For this reason, we adopt
these baselines ensuring a fair comparison to their core methods, and
demonstrate that our approach outperforms them. In addition, as the temporal
prior is either extracted from the input video or learned from video training
data, these works generally lack 3D awareness and may not generate 3D-aware
animations. A recent attempt designed a 4D representation that employs a
canonical field and deformation field to each frame, such that one can easily
edit the video by changing contents on the canonical field which propagates
naturally to the other frames [23]. However, this representation still has a
relatively weak generalization ability and tends to only work well on very
short video clips.

In summary, while related, none of these video editing approaches operate in
the same setting as our approach, which takes a low-fidelity, untextured
rendering of a scene as input to generate a video.

#### 3D Generation

This class of methods is able to achieve near-perfect 3D consistency by using
underlying 3D representations. One line of work [15, 8, 4] focuses on scene
generation using diffusion models. These works typically predict or estimate
the scene geometry on the fly using monocular predictors such as MiDaS [29]
and use that information to constitute individual generations into a scene.
This approach is applicable only in specific domains such as indoor scenes or
nature scenes, with limited motion control ability due to depth–scale
ambiguity. The closest form of 3D generation to our task is mesh texturing.
Given an explicit representation such as a mesh, prior works [30, 36, 5]
demonstrate convincing asset mesh texturization using the powerful inpainting
function of Stable Diffusion [31]. Other works [24, 26] use Score Distillation
[26] to directly optimize a neural radiance field [22], but they suffer from
quality degradation potentially due to heavy classifier-free guidance [13].
Follow-up works [21] attempt to combine the strength of score distillation and
mesh texturization. Since these methods generate a static texture for a given
mesh, they are not able to capture appearance interactions (e.g., reflections
or shadows) in case of scene-level content and motions.

##  3 Method

The goal of generative rendering is to transfer creator-defined scene-level
proxy meshes (without appearance) and motions directly to a convincing
animation. Given a 3D scene composed of animated meshes, we render guiding
channels that consist of depth maps
$\bm{\mathcal{D}}=[{\bm{D}}^{1},...,{\bm{D}}^{N}]$ and UV coordinate maps
$\bm{\mathcal{UV}}=[{\bm{UV}}^{1},...,{\bm{UV}}^{N}]$ where $N$ is the length
of the sequence and each UV map contains the 2D texture coordinates as well as
an object ID in its RGB channels. Using a depth-conditioned 2D image
generation model such as StableDiffusion [31] with depth ControlNet [41], we
aim to transfer the sequence $({\bm{D}}^{i},{\bm{UV}}^{i})$ into a set of
generated images that depict a stylized animation. To achieve this goal, we
perform feature blending over the input and output features of the self-
attention modules of the diffusion model (Sec. 3.2) and also perform UV-guided
noise initialization (Sec. 3.3) to improve the temporal consistency. We
provide an overview of our framework in Fig. 2 and our overall algorithm in
Alg. 1. Next, we provide some background on the self-attention features of a
diffusion model before discussing the different parts of our method in detail.

###  3.1 Self-attention feature injection

Recent work has demonstrated that self-attention features encode spatial self-
similarity [34] and they are effective in terms of maintaining temporal
consistency when they are used to perform cross-frame attention [6, 37]. A
typical approach is to extend the diffusion network to process multiple frames
jointly where they can attend to each other’s features. Furthermore, it is
possible to manipulate both the input and output features of a self-attention
module to further enforce consistency. We denote these operations as _pre-
attention_ and _post-attention_ feature injections respectively and provide a
description in the following.

#### Pre-attention Feature Injection

An attention module is defined as:

${\bm{F}}=\texttt{Attn}\\!\left({{\bm{Q}};{\bm{K}};{\bm{V}}}\right)=\texttt{Softmax}\\!\left(\frac{{\bm{Q}}{\bm{K}}^{\top}}{\sqrt{d}}\right)\cdot{{\bm{V}}},$
(1)

where ${\bm{Q}}$, ${\bm{K}}$, and ${\bm{V}}$ denote the query, key, and value
features respectively. In case of a self-attention module, ${\bm{Q}}$,
${\bm{K}}$ and ${\bm{V}}$ are computed by projecting the same features
${\bm{f}}^{(i,l)}$, via the corresponding attention matrices, i.e.:

${\bm{Q}}^{(i,l)}=\mathbf{W}^{Q}{\bm{f}}^{(i,l)}\\!\\!,\,{\bm{K}}^{(i,l)}=\mathbf{W}^{K}{\bm{f}}^{(i,l)}\\!\\!,\,{\bm{V}}^{(i,l)}=\mathbf{W}^{V}{\bm{f}}^{(i,l)}$
(2)

where $i$ denotes the frame index and $l$ denotes the diffusion step and
$\mathbf{W}^{Q/K/V}$ are weight matrices for query, key, and values,
respectively. In other words, for each query feature an attention weight is
computed based on the similarity between the query and each key feature. Then,
the value features are combined based on this attention weight.

One can manipulate the input features provided to the self-attention module,
specifically the key and value features, which we denote as _pre-attention
feature injection_. As discussed by [10], in case of generating $N$ frames,
one naive option is to perform extended attention where the features
${\bm{f}}^{(i,l)}$ of all the frames are concatenated and utilized as key and
value pairs:

$\begin{split}{\bm{K}}^{(i,l)}&=\mathbf{W}^{K}[{\bm{f}}^{(1,l)},...,{\bm{f}}^{(N,l)}],\\\
{\bm{V}}^{(i,l)}&=\mathbf{W}^{V}[{\bm{f}}^{(1,l)},...,{\bm{f}}^{(N,l)}].\end{split}$
(3)

Extended attention is a synchronized operation, where all frames attend to
features of each other resulting in a more consistent generation. However,
generating multiple frames together incurs a high computational cost, which in
practice limits the number of frames that can be processed in a sequence. For
this reason, we utilize extended attention only for a subset of keyframes and
propose mechanisms to propagate the keyframe features to the rest of the
sequence, as discussed in the next section.

#### Post-attention Feature Injection

While it is possible to manipulate the input of the self-attention layer,
another option is to directly manipulate the output, which we denote as _post-
attention feature injection_. In particular, given known correspondences
between two frames $i$ and $j$, one can re-project ${\bm{F}}^{(i,l)}$, the
output of the self-attention module of frame $i$ at diffusion step $l$, to
obtain ${\bm{F}}^{(j,l)}$:

${\bm{F}}^{(j,l)}=\pi_{i,j}({\bm{F}}^{(i,l)}),$ (4)

where $\pi_{i,j}$ denotes the re-projection operation, which is enabled by the
ground truth pixel-level correspondences between frames given by our UV maps.

While post-attention feature injection is effective in achieving consistency,
it tends to create blurry results and also avoids generating of certain
interactions between objects (e.g., shadows, reflections) since it overwrites
the output the self attention. Hence, as described next, our method combines
pre- and post-attention feature injection in an effective way to generate
consistent but also high-quality frames.

###  3.2 UV-space feature injection

Given that pre- and post-attention features of the self-attention module can
be manipulated to enhance consistency, our main goal is to perform this
manipulation guided by the ground truth correspondences we have from the 3D
scene. Specifically, we utilize the canonical UV space to bring the features
from the different frames into correspondence which we denote as _UV-space
feature blending_. Given a set of feature maps for each frame $i$, we project
them to a canonical UV space. In particular, for each texel in the UV space,
we find its closest point correspondence in each frame. Having such
correspondences from multiple frames, a straightforward approach is to take
the (weighted) average to blend the features [10]. However, we find this
consistently leads to sub-optimal blurry results. Therefore, to combat over-
smoothing of the features, we additionally perform blending sequentially and
fill a certain texel with the features of its corresponding pixel in a frame
only if it has not been filled before. The final unified texture is then the
mean of the inpainted texture and the average texture. After features from all
frames are blended, we obtain a UV-space feature map which we denote as
${\bm{T}}^{(i,l)}$. Once such a unified feature map is obtained, we can
project it back to each frame to obtain $\bar{f}^{(i,l)}$.

Given the UV-space feature blending mechanism, we next describe how we utilize
it during the image generation process. For each diffusion step, we first
sample a random batch of $m$ keyframes and perform extended attention on these
keyframes, extracting both their pre-attention features
${\bm{f}}_{KF}^{(i,l)}$ and post-attention features ${\bm{F}}_{KF}^{(i,l)}$.
We further composite a UV space feature map, ${\bm{\bar{T}}}_{KF}^{l}$, by
blending ${\bm{F}}_{KF}^{(i,l)}$ as described previously. We then perform the
diffusion step on all the frames (in case of keyframes, the diffusion step is
simply repeated) in a sequential manner. For each frame, we compose the pre-
attention features by concatenating the pre-attention features of the
keyframes,
$\mathbf{f}_{KF}^{l}=[{\bm{f}}_{KF}^{(1,l)},...,{\bm{f}}_{KF}^{(m,l)}]$, with
the pre-attention features of the current frame ${\bm{f}}^{(i,l)}$ resulting
in:

$\begin{split}{\bm{K}}^{(i,l)}&=\mathbf{W}^{K}[{\bm{f}}^{(i,l)},\mathbf{f}_{KF}^{l}],\\\
{\bm{V}}^{(i,l)}&=\mathbf{W}^{V}[{\bm{f}}^{(i,l)},\mathbf{f}_{KF}^{l}].\end{split}$
(5)

After executing the self-attention module, we blend the output, i.e., post-
attention features of the current frame ${\bm{\hat{F}}}^{(i,l)}$, with the
projection of the blended post-attention features to the current frame
${\bm{\bar{F}}}^{(i,l)}$. Hence, the updated output of the module becomes:

${\bm{F}}_{out}^{(i,l)}=\alpha\cdot{\bm{\bar{F}}}^{(i,l)}+(1-\alpha)\cdot{\bm{\hat{F}}}^{(i,l)}$
(6)

where ${\bm{\hat{F}}}^{(i,l)}$ is the output of the inflated self-attention
block computed using keys and values in Eq. 5.

###  3.3 UV noise initialization

Prior video editing works [6, 10] generally rely on inversion methods, such as
DDIM inversion [16], to invert each source frame to the noise space which are
then used as initialization for editing. Since source frames in the input
video are coherent, this inversion strategy yields relatively coherent noise
patterns that in turn helps with consistent editing. However, in our case,
since we have an untextured scene, inverting rendered frames that lack
appearance does not lead to coherent noisy latents. Another option is to keep
the noise fixed across all frames. However, this results in severe texture-
sticking issues as we demonstrate in the supplementary material. We seek an
alternative to capture temporal correlations in initial noise patterns.
Inspired by previous work [18, 9], we leverage the fact that we have a
canonical UV space by initializing the noise in this space by creating a
Gaussian noise texture. We then project this noise to each frame by utilizing
the frame–UV correspondences. As we illustrate in our evaluations, this
significantly boosts the coherency of the generated outputs.

###  3.4 Latent normalization

We notice a color flickering issue, where the overall color distribution and
contrast change between frames, even if the motion is limited. Therefore, we
apply an AdaIN operation on the predicted latents to make every frame’s
overall distribution similar to the first frame. The simple AdaIN operation
solves this issue for some samples, but we find that calculating the
distribution over all pixels is very sensitive to the motions. Therefore,
inspired by [1], we calculate the statistics only on the background pixels,
which are easy to separate in our setup.

Algorithm 1 Generative Rendering

Input: UV maps $\bm{\mathcal{UV}}=[{\bm{UV}}^{1},...,{\bm{UV}}^{N}]$; depth
maps $\bm{\mathcal{D}}=[{\bm{D}}^{1},...,{\bm{D}}^{N}]$; text prompt
${\bm{c}}$; diffusion model $\Phi$; projection from UV to image
$\pi_{\bm{\mathcal{UV}}}$; $T$ diffusion steps; $k$ keyframes

$z_{\bm{\mathcal{UV}}}\sim\mathcal{N}(\bm{0},\mathbf{I})\leavevmode\nobreak\ $
// Sample noise in UV space
$\mathbf{z}=\mathbf{z}_{T}=\pi_{\bm{\mathcal{UV}}}(z_{\bm{\mathcal{UV}}})$
For $t=T,\dots,1$ do

$KF=\text{shuffle}(\\{i_{1},\dots,i_{k}\\})\;\;$ // Sample keyframes

${\bm{f}}_{KF},{\bm{F}}_{KF}\leftarrow\Phi(\mathbf{z}_{KF},\bm{\mathcal{D}}_{KF},c)$
// Grab features from $\Phi$

${\bm{\bar{T}}}_{KF}=\text{blend}(\pi_{\bm{\mathcal{UV}}}^{-1}({\bm{F}}_{KF}))$
// Blends features in UV space

$\mathbf{z}_{t-1}=[\;]$

For $z^{i}$ in $\mathbf{z}$ do

${\bm{\bar{F}}}^{i}=\pi_{UV^{i}}({\bm{\bar{T}}}_{KF})$

$z^{i}_{t-1}=\Phi(z^{i},\bm{\mathcal{D}}_{KF},c,{\bm{f}}_{KF},{\bm{\bar{F}}}^{i})$

$\mathbf{z}_{t-1}\text{.append}(z^{i}_{t-1})$

$\mathbf{z}=\mathbf{z}_{t-1}$

Output: $\mathbf{z}$

Figure 3:  Qualitative comparisons. We adapt the feature blending mechanisms
proposed by Pix2Video [6] and TokenFlow [10] to our setting. While Pix2Video
produces reasonable results, it lacks a mechanism to maintain frame-level
consistency. TokenFlow uses a token propagation & interpolation mechanism
between nearby keyframes, but this results in blurred outputs in our setting.
Figure 4:  Camera and object rotation. Our algorithm supports camera and
object rotations for static scenarios.  Figure 5:  Qualitative results. Our
method generalizes to a diverse set of input meshes, styles, and prompts.
Figure 6:  Ablation study. We disabled several key building blocks and observe
clear artifacts or inconsistencies in all ablations.

##  4 Experiments

#### Datasets

We evaluate our method on scenes with various complexity demonstrating three
main types of animation sequences: 1) Camera rotations: given a single static
object we create a camera path with 360-rotation around the object; 2)
Physical simulation: we prepare a motion sequence by utilizing physically
based simulation, e.g., bouncing balls where we use rigid body dynamics for
collision handling; 3) Character animation: we create scenes with rigged and
animated characters obtained from Mixamo††https://www.mixamo.com/ and render
them with various dance motions.

#### Baselines

To our knowledge, no published work directly focuses on the same task our
method does. Therefore, we adapt two recent methods to our setting as
baselines. 1) we use the pre-attention feature propagation mechanism of
Pix2Video [6] (i.e., each frame attends to an anchor and previous frame)
together with depth conditioning, and 2) we apply the extended attention
mechanism and token propagation from TokenFlow [10] using nearest neighbor
matching. We note that the original TokenFlow method computes correspondences
by matching features obtained via inverting original frames. Since this is not
meaningful in our case, we instead provide the ground truth correspondences we
obtain from 3D. Both Pix2Video and TokenFlow rely on inversion for noise
initialization. Again, this is not meaningful in our setting, so we instead
use the same noise to initialize each frame.

#### Evaluation metrics

Following prior video editing evaluation protocols [3, 10, 6, 7, 37], we
report two metrics based on CLIP [28]. To measure _frame consistency_ , we
compute CLIP image embeddings for each frame of the output video and compute
the average cosine similarity between all pairs of consecutive video frames.
To measure _prompt fidelity_ , we compute the mean CLIP embedding similarity
score between each frame of the output video and the corresponding input
prompt.

#### Implementation details

We use the StableDiffusion [31] v1.5 model and a depth-conditioned ControlNet
[41] as our image generator. We generate all results at 512 × 512 resolution
using Karras scheduler [17] with 50 denoising steps.

###  4.1 Results

#### Qualitative results

Qualitative comparisons to the baselines are shown in Fig. 3. While Pix2Video
[6] produces high-quality frames it shows limited spatio-temporal consistency,
because it purely relies on the cross-frame attention mechanism to attend to
corresponding features. TokenFlow [10] can produce smoother results, but we
find that it consistently produces blurry textures. This is in part due to the
lack of DDIM inversion [16], which provides a decent level of feature
similarities across frames. In our setting, this DDIM inversion is not
meaningful and skipping the self-attention module to directly blend the
features of keyframes means averaging less similar features, resulting in
blurry appearance. Without accurate DDIM inversion and correct nearest
neighbor matching, the blending oversmoothes the features and lead to blurry
outputs of low quality. In addition, unlike nearest neighbor correspondences
computed using DDIM inversion, UV correspondences are not guaranteed to be an
exact match in the low-dimensional feature space, which is only $64\times 64$
pixels in our model, especially around object edges. Consequently, a straight
blending of the post-attention features also creates blurry results.

Our method can handle camera rotation with reasonable quality in static
object-level scenarios, as shown in Fig. 4. In Fig. 5, we provide additional
results on the same input meshes but with different prompts. Our method shows
robustness across a diverse set of prompts and styles. Our inflated attention
modules can capture motion-specific traits, such as shadows (Row 3, Row 7),
lighting changes (Row 1), and reasonable deformations that are not modeled by
the input low-fidelity meshes (Row 1 ground regions).

#### Quantitative results

Method | Frame Consistency $\uparrow$ | Prompt Fidelity $\uparrow$
---|---|---
Per-Frame | 0.9547 | 0.3233
Pix2Video* | 0.9630 | 0.2983
TokenFlow* | 0.9822 | 0.2737
Gen1 | 0.9907 | 0.3029
Ours | 0.9845 | 0.3227
Table 1: Quantitative evaluation. Our approach offers the best frame
consistency among approaches using T2I models. As expected, only the
proprietary T2V model Gen1 observes better temporal consistency. Our method
also shows a higher prompt fidelity than relevant baselines that offer a
reasonable amount of temporal consistency. *baseline adapted to our setting.

We show quantitative evaluations for frame consistency and prompt fidelity in
Tab. 1. We observe that our frame consistency is better than all baselines
using T2I models. As expected, Pix2Video [6] suffers more from temporal
changes due to the lack of a correspondence injection module. Even though
TensorFlow’s token propagation encourages smoothness over fidelity, our method
still shows better consistency. Only the T2V model Gen1 has a better frame
consistency, which is expected because it is trained on video data. All
current T2V models, including Gen1, can only generate a few frames and are
limited in other ways. Our approach is an attempt to overcome this deficiency
using 2D T2I models exclusively. Fidelity-wise, we are slightly better than
Pix2Video [6], because it reuses the set of pre-attention features from
previous frames, while we only reuse a portion of the pre-attention features
to generate each frame. Since TokenFlow [10]’s token propagation mechanism
introduces over-smoothing effects, its results are blurry for many of the
samples, hence achieving lower fidelity scores. Our generative rendering
utilizes the benefits of pre-attention inflation to gain overall global
structural and semantic consistency, while also combining with post-attention
feature warping as local guidance. Combined with our UV noise initialization,
we outperform or are on par with all relevant baselines in the proposed
setting.

#### Ablation studies

We perform a thorough ablation study to verify our design choices, shown in
Fig. 6. We study three aspects of our pipeline: 1) comparing with pre-
attention injection only, 2) comparing with post-attention injection only, and
3) the effectiveness of UV noise initialization. We observe that directly
unifying and projecting the post-attention features without passing through
the attention module produces chaotic results (severe artifacts in “no pre-
attn”, the joker’s face and clothes regions). This observation is in line with
our hypothesis for the poor performance of the adapted TokenFlow mechanism.
When the motion is relatively large, geometrically matched features may not be
good matches in the feature space. Therefore, directly warping a feature from
one frame to another based on the geometrical correspondence has a high chance
of not producing reasonable outputs. However, when we combine post-attention
feature warping with pre-attention feature injection, the post-attention
features can act as guidance for the next network layer. Without post-
attention feature warping, purely pre-attention feature injection does not
guarantee pixel-wise consistency since the attention operation merely computes
a correspondence based on feature similarities (in “no post-attn”, the joker’s
clothes change over frames). Our UV noise initialization is well-aligned with
our setup. If we use fixed noise instead, the blending of the features may
cause unwanted artifacts (in “fixed noise”, the joker’s face has blending
artifacts and is not recognizable in the third image). Our UV noise
initialization serves generally and can work beyond our setup. Additional
details on ablations are included in the supplementary.

##  5 Discussion

In this paper, we introduce Generative Rendering, a novel zero-shot pipeline
based on 2D diffusion models for 4D-conditioned animation generation. Our
method can animate creator-defined low-fidelity meshes and motion sequences,
thereby bypassing steps requiring significant manual labor such as detailing,
texturing, physical simulation, etc. The key idea of generative rendering is
to utilize the prior within a depth-conditioned 2D diffusion model to provide
the basic structure and physical fidelity to make convincing animations. This
is accomplished by injecting the correspondences into the diffusion model
using a combination of pre- and post-attention feature injection while
unifying these features in the UV space. Our model demonstrates better frame
consistency and prompt fidelity than relevant baselines.

#### Limitation and future work

Generative rendering cannot achieve real-time animations due to the multi-step
inference of current diffusion models. However, accelerating this inference
stage has been an active research area and advances in this area, such as
consistency models [33], can be directly applied to speed up our method.
Furthermore, generative rendering is not yet able to guarantee perfect
consistency and preservation of details. This is mainly because our method
works purely in the low-dimensional latent space, which is only $64\times 64$
pixels in our model resulting in imprecise UV correspondences. For the same
reason, generative rendering may suffer from unwanted misalignment and
artifacts. We believe that applying some of findings to pre-trained video
diffusion models to augment them with 3D controllability is an exciting future
direction.

#### Conclusion

Diffusion models have emerged as state-of-the-art generative methods across
multiple domains, such as 2D, 3D, and video generation and editing. generative
rendering adds unprecedented levels of control to video generation with T2I
diffusion models by bridging the gap between the traditional rendering
pipeline and diffusion models.

#### Acknowledgement

G.W. was in part supported by Google, Samsung, and Stanford HAI.

## References

  * [1] Yuval Alaluf, Daniel Garibi, Or Patashnik, Hadar Averbuch-Elor, and Daniel Cohen-Or.  Cross-image attention for zero-shot appearance transfer.  In arXiv, 2023.
  * [2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In ECCV, 2022.
  * [3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In CVPR, 2023.
  * [4] Shengqu Cai, Eric Ryan Chan, Songyou Peng, Mohamad Shahbazi, Anton Obukhov, Luc Van Gool, and Gordon Wetzstein.  Diffdreamer: Towards consistent unsupervised single-view scene extrapolation with conditional diffusion models.  In ICCV, 2023.
  * [5] Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, and Kangxue Yin.  Texfusion: Synthesizing 3d textures with text-guided image diffusion models.  In ICCV, 2023.
  * [6] Duygu Ceylan, Chun-Hao Huang, and Niloy J. Mitra.  Pix2video: Video editing using image diffusion.  In ICCV, 2023.
  * [7] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In ICCV, 2023.
  * [8] Rafail Fridman, Amit Abecasis, Yoni Kasten, and Tali Dekel.  Scenescape: Text-driven consistent scene generation.  In arXiv, 2023.
  * [9] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji.  Preserve your own correlation: A noise prior for video diffusion models.  In ICCV, 2023.
  * [10] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  In arXiv, 2023.
  * [11] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  In arXiv, 2023.
  * [12] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans.  Imagen video: High definition video generation with diffusion models.  In arXiv, 2022.
  * [13] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In arXiv, 2022.
  * [14] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  In arXiv, 2022.
  * [15] Lukas Höllein, Ang Cao, Andrew Owens, Justin Johnson, and Matthias Nießner.  Text2room: Extracting textured 3d meshes from 2d text-to-image models.  In ICCV, 2023.
  * [16] Inbar Huberman-Spiegelglas, Vladimir Kulikov, and Tomer Michaeli.  An edit friendly ddpm noise space: Inversion and manipulations.  In arXiv, 2023.
  * [17] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  In NeurIPS, 2022.
  * [18] Michael Kass and Davide Pesare.  Coherent noise for non-photorealistic rendering.  In TOG, 2011.
  * [19] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  In arXiv, 2023.
  * [20] Xin Li, Wenqing Chu, Ye Wu, Weihang Yuan, Fanglong Liu, Qi Zhang, Fu Li, Haocheng Feng, Errui Ding, and Jingdong Wang.  Videogen: A reference-guided latent diffusion approach for high definition text-to-video generation.  In arXiv, 2023.
  * [21] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin.  Magic3d: High-resolution text-to-3d content creation.  In CVPR, 2023.
  * [22] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [23] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen.  Codef: Content deformation fields for temporally consistent video processing.  In arXiv, 2023.
  * [24] Ryan Po and Gordon Wetzstein.  Compositional 3d scene generation using locally conditioned diffusion.  In arXiv, 2023.
  * [25] Ryan Po, Wang Yifan, Vladislav Golyanik, Kfir Aberman, Jonathan T Barron, Amit H Bermano, Eric Ryan Chan, Tali Dekel, Aleksander Holynski, Angjoo Kanazawa, et al.  State of the art on diffusion models for visual computing.  In arXiv, 2023.
  * [26] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  In ICLR, 2023.
  * [27] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  In ICCV, 2023.
  * [28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In CoRR, 2021.
  * [29] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  In TPAMI, 2022.
  * [30] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or.  Texture: Text-guided texturing of 3d shapes.  In SIGGRAPH, 2023.
  * [31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [32] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman.  Make-a-video: Text-to-video generation without text-video data.  In arXiv, 2022.
  * [33] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  In arXiv, 2023.
  * [34] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In CVPR, 2023.
  * [35] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang.  Modelscope text-to-video technical report.  In arXiv, 2023.
  * [36] Tianfu Wang, Menelaos Kanakis, Konrad Schindler, Luc Van Gool, and Anton Obukhov.  Breathing new life into 3d assets with generative repainting.  In BMVC, 2023.
  * [37] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In arXiv, 2022.
  * [38] Shuai Yang, Yifan Zhou, Ziwei Liu, , and Chen Change Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.  In SIGGRAPH Asia, 2023.
  * [39] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin.  Latent video diffusion models for high-fidelity video generation with arbitrary lengths.  In arXiv, 2022.
  * [40] Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin.  Video probabilistic diffusion models in projected latent space.  In CVPR, 2023.
  * [41] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In ICCV, 2023.
  * [42] Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng.  Magicvideo: Efficient video generation with latent diffusion models.  In arXiv, 2022.

Generative Rendering: Controllable 4D-Guided Video Generation with 2D
Diffusion Models
Supplementary Material

##  Appendix A Additional qualitative results

We refer to the supplemental webpage for various qualitative results obtained
by using different 3D scenes, motions, and target prompts.

##  Appendix B Additional quantitative results

| Pairwise Frame Interval
---|---
Method | 1 | 5 | 10 | 15 | 20
Per-Frame | 0.9547 | 0.9173 | 0.9109 | 0.9145 | 0.9062
Pix2Video* | 0.9630 | 0.9503 | 0.9494 | 0.9415 | 0.9471
TokenFlow* | 0.9822 | 0.9754 | 0.9728 | 0.9706 | 0.9712
Gen1 | 0.9907 | 0.9715 | 0.9582 | 0.9624 | 0.9601
Ours | 0.9845 | 0.9815 | 0.9738 | 0.9749 | 0.9727

Table 2:  Frame consistency with different intervals. Our approach offers
better or competitive consistencies at different intervals. *: baseline
adapted to our setting.

We report CLIP [28] embedding similarities for a pair of frames separated by
different frame intervals in Tab. 2. As shown, semantic consistency achieved
by our method is stable across different intervals. This is potentially
because of our texture-based feature aggregation. This provides a canonical
representation for the objects that links frames that are temporally further
apart even in long sequences. Even though video-based methods such as Gen1 [7]
achieve temporally smoother results, we speculate that it is not easy for them
to capture such long range interactions.

##  Appendix C Robustness to video length

Our method can be scaled up to handle long input sequences. To generate a long
output video, our method requires the sampling of a sparse set of keyframes,
and the unified features in the texture space can be propagated. The only GPU
memory bottleneck in our method is the parallel processing of the keyframes
using inflated attention. Our algorithm is robust enough to generate long
videos with $>$200 frames with $<$12GB GPU memory.

##  Appendix D Additional ablations

#### Effectiveness of UV noise initialization

We find that the performance of many components in our pipeline, e.g. inflated
attention, are tightly related to the initial noise. Video editing works [6,
10, 38] typically require accurate DDIM inversion [16] as noise
initialization. Using DDIM inverted noise encourages temporal coherence and
similarities to the input video. However, since our input is textureless UV
maps and depths, DDIM inversion constantly fails and will not provide useful
information. We instead propose to utilize the canonical UV space to
initialize the noise in each frame of the sequence. In our supplemental page,
we show the effectiveness of this initialization strategy.

#### Effectiveness of latent normalization

Our latent normalization can address a large portion of the color flickering
issue, where the overall color distribution shifts randomly for different
frames. We notice that small color shifts in the latent space will be inflated
by the VAE decoder used in Stable Diffusion [31], causing the generation
process very sensitive to small differences in the latent space. The video
comparison is included in our supplemental page.

##  Appendix E Limitation and failure cases

Figure 7:  Effect of different texture resolution. If the ground’s texture
resolution is too low (2048$\times$2048), artifacts appear at ground regions
in the final rendered image compared with using a higher texture resolution
(3072$\times$3072). Prompt: a train running in a field of green grass.

Our method still suffers from several aspects. A large portion of our
inconsistency comes from the VAE decoder. As mentioned in D, small
inconsistencies in the latent space will be inflated by the VAE decoder while
transiting into RGB frames. This phenomenon scales to image details, where
small differences in the latent space will be inflated after decoding with the
VAE. Additionally, since our noise initialization and feature fusion both work
in the UV space, it could be tricky to set the texture resolutions. Setting
the texture resolution too low will create corrupted regions while too high
will cause the UV coordinates to be too far away and no blending effect will
take place. An example is shown in Fig. 7. Finally, our work does not yet
generalize to large environmental changes and dramatic perspective changes.
This is because of the highly overlapping pixel-wise matches and bad feature
projections. We believe finetuning or adding a video module to be an exciting
future direction to improve the performances on these more challenging scenes.