# VASE: Object-Centric Appearance and Shape Manipulation of Real Videos

Elia Peruzzo 1 Vidit Goel 2 Dejia Xu 3 Xingqian Xu 2 Yifan Jiang 3
Zhangyang Wang 2,3 Humphrey Shi 2,4 Nicu Sebe 1
1 University of Trento 2 Picasrt AI Research 3 UT Austin 4 Georgia Tech

###### Abstract

Recently, several works tackled the video editing task fostered by the success
of large-scale text-to-image generative models. However, most of these methods
holistically edit the frame using the text, exploiting the prior given by
foundation diffusion models and focusing on improving the temporal consistency
across frames. In this work, we introduce a framework that is object-centric
and is designed to control both the object’s appearance and, notably, to
execute precise and explicit structural modifications on the object. We build
our framework on a pre-trained image-conditioned diffusion model, integrate
layers to handle the temporal dimension, and propose training strategies and
architectural modifications to enable shape control. We evaluate our method on
the image-driven video editing task showing similar performance to the state-
of-the-art, and showcasing novel shape-editing capabilities. Further details,
code and examples are available on our project page.

Figure 1: With VASE, we can manipulate both the shape and appearance of an
object within a real video. In the top row, we present the source video,
followed by a series of edits showcasing precise adjustments in both
attributes. In the first frame (1st column), the keyframe structure is
overlayed to the object. Red areas represent regions that have been removed
from the original shape, while green regions denote newly added parts. The
driver image for appearance edits is showcased in the bottom left corner.

##  1 Introduction

In recent years, there has been a surge in both the quality and diversity of
generated content, primarily due to the deployment of Diffusion Models (DM)
[45, 46, 17] trained on vast amounts of data [43]. Remarkably, large-scale
text-to-image (T2I) models [39, 29, 42, 37] enable inexperienced users to
produce impressive results using only a textual prompt. A plethora of works
have been developed ever since, tailoring these models for more specific and
constrained use cases, enabling various editing capabilities and making them
even more user-friendly [6, 40, 41, 19]. Following a similar trend, many works
are extending these paradigms to videos. The majority of current methods rely
on a text prompt to guide the editing process [53, 5] and introduce new
techniques to improve the smoothness and temporal consistency across the
generated frames [35, 57, 12]. As highlighted in prior research [40, 10, 13],
text prompts, while intuitive, often fall short in capturing precise nuances,
leading to potential mismatches with the user’s intentions. Additionally,
existing approaches treat the video frames as a whole for editing [35, 12,
53], lacking the granularity needed to constrain the changes to a specific
region. Lastly, the majority of these approaches cannot modify the structure
of the objects in the video explicitly, as they often rely on per-frame
structural guidance from the source video [9, 6, 51], making them not suitable
for this application.

In this work, our goal is to explore other directions in the video editing
domain to empower the user with comprehensive capabilities. Specifically, we
propose VASE, a framework for video object-centric appearance and shape edits.
Our approach is _object-centric_ , designed to control one individual object
at a time while preserving the rest of the video intact. This mirrors numerous
real-world scenarios where users seek to confine edits to a particular region.
Secondly, we drive the edit using another image, as an image can convey more
details than a textual prompt. More significantly, we introduce the potential
to alter the object’s structure, providing more comprehensive editing options.
This capacity remains under-explored in previous methods, or if present, it
lacks an explicit control mechanism [23, 35]. In this work, we argue that in a
video editing setup, shape modifications should not alter the object’s shape
drastically, as this pertains more to a pure generative setup than an editing
scenario.

Our focus lies in enabling precise, user-driven shape adjustments by
manipulating a _single keyframe_ , which then extends across the entire video.
It is worth noting that our approach is not confined to particular domains
(_e.g_. humans); instead, we propose a generic framework that can modify both
rigid and articulated objects for videos in the wild. Lastly, we exclude
expensive per-video training or video-decomposition processes, which limit the
practicality of editing methods.

We base our method on a pre-trained image-conditioned diffusion model [56],
inflated with temporal layers to process videos. In order to faithfully
replicate the motion depicted in the original video, we incorporate a
ControlNet [6] that takes the optical flow derived from the source video as
input. Additionally, we unlock the explicit control over the object’s shape by
feeding the desired keyframe shape into the ControlNet.

In our initial implementation, we observed that the model could potentially
solve the task by predominantly relying on the optical flow while disregarding
the structural condition. This rendered the shape edits ineffective during
inference. To address this, we propose a novel _Joint Flow-Structure
Augmentation_ procedure to break the alignment between the optical flow and
the shape conditioning. Additionally, we introduce an auxiliary _Segmentation
Head_ at training time to force the model to rely on the structure
information. Moreover, we introduce a _Flow-Completion Network_ to generate
realistic motion in the edited regions where no displacement information is
available from the source video.

To summarize, our key contributions are as follows:

  * •

We enable object-centric video editing in the wild, controlling the shape and
the appearance of individual objects in the input video while keeping the rest
of the video unchanged. The user provides one driving image that specifies the
desired appearance and the shape of a single edited keyframe.

  * •

We design a _Joint Flow-Structure Augmentation_ pipeline, and introduce a
_Flow-Completion Network_ and a _Auxiliary Segmentation Head_. All the
components are necessary to edit the shape of the objects effectively.

  * •

We perform challenging shape edits of the target object without relying on
test-time optimization procedures. Our model is trained once and applied at
inference time, enhancing its practicality for various editing scenarios.

##  2 Related Works

Image Editing with Diffusion Models. Large-scale text-to-image (T2I) diffusion
models [39, 29, 42, 37] have gained widespread adoption as powerful generative
models. They serve as strong generative priors and can be tailored to
accommodate various forms of conditioning, along with the ability to
manipulate real images [27]. One line of works, achieves editing capabilities
in a zero-shot manner, by exploiting the internal representation of a
pretrained T2I model. Hertz et al. [15] investigate the role of cross-
attention maps in the generated images and exploit them to bind edits to a
specific region. Plug-and-Play [48] instead processes a real image and a
target text prompt, and exploits the features of self-attention blocks to
preserve the structure from the original image while applying the editing
operation. Epstein et al. [8] recently applies similar observation in
conjunction with classifier-free guidance [16] to achieve more comprehensive
editing capabilities. Differently, ControlNet [6] opts to train a hypernetwork
to control the generation of the main T2I model based on spatial cues,
enabling control with various types of signals _e.g_. depth, semantic maps, or
sketches. Concurrently, [28] explores more lightweight designs for the
hypernetwork and the combination of multiple conditioning signals. Other works
explore image-driven conditioning as an alternative to text for specifying the
desired appearance of the objects in the scene. In particular, personalization
emerged as an important research direction [40, 10, 1], where the goal is to
inject a user-provided concept in the form of images which is not present in
the training data. In a similar way, Paint-by-Example (PBE) [56] proposed the
first image-based inpainting method with coarse masks depicting the shape of
the desired object, and [7] enhances the fidelity to the driver image using
features from strong self-supervised models. [55] introduces a new encoder to
capture more nuances of the driver image and implement a “prompt-free” method,
while PAIR Diffusion [13] extracts per-object appearance representation, which
enables image-driven editing and precise control. We draw inspiration from
these methods and aim for more comprehensive editing capabilities in the video
domain.

Video Generation and Editing with Diffusion Models. Recently, there have been
several efforts to expand the achievements of text-to-image models to the
video domain [44, 11, 18, 4]. As foundational video diffusion models remain
largely unavailable to the open-source community and prohibitive to train from
scratch, many works explored available T2I models for video editing.

One line of work exploits pre-trained T2I models and adapts them to the task
in a zero-shot manner [35, 5, 21, 12, 57]. The temporal consistency of the
generated frames is generally encouraged by inflating the self-attention
blocks to process multiple frames [21, 53]. Tune-A-Video [53] involves fine-
tuning the model on the video to be edited, enabling test-time edits through
text prompts or cross-attention control [25]. In FateZero [35], the attention
maps are extracted during an initial inversion step and blended with those
generated during the editing process, confining the edit to a specific region.
FateZero allows for shape modification of the foreground object, but it lacks
a mechanism to specify the desired edit explicitly. TokenFlow [12] proposes to
propagate features of the base T2I model leveraging the optical flow extracted
from the source video. Differently, other methods opt for training on video
datasets; these models employ factorized spatio-temporal layers, integrating
pre-existing layers from a T2I model with newly initialized temporal blocks
[9, 51, 14]. While these methods demonstrate remarkable editing capabilities
with satisfactory temporal consistency, they do not explore the domain of
shape manipulation. Any potential changes in shape, if they occur, cannot be
directly controlled, which poses a limitation on their applicability.

Neural Layered Atlases (NLA). Kasten et al. [20] propose to decompose the
video in Neural Layered Atlases (NLA), exploiting the high correlation across
frames. This representation is inherently designed for editing purposes, as
the edits applied to the canonical image are propagated to the rest of the
video, with temporal consistency obtained by design. Text2Live [3] introduces
text conditioning to obtain both image and video appearance editing results
through text, while [32] improved the atlas decomposition both in terms of
reconstruction quality and computational time. Recently, Lee et al. [23]
leverages NLA representation in conjunction with an image editor and designed
a framework that propagates shape edits. A deformation field is obtained with
a semantic correspondence method between the input keyframe and the edited
keyframe. Following that, the canonical representation of the foreground
object is updated using the guidance of a T2I model through an optimization
process [34]. Despite the remarkable results, the current version lacks direct
control over the shape of the final object. Additionally, as pointed out by
[23], the pipeline is sensitive to the semantic correspondence method, which
is inherently difficult to achieve with precision. Any discrepancies or
inaccuracies in this process are reflected in the final video output.
Furthermore, NLA-based methods have a high computational cost (approximately
10 hours for a 70-frame video) and may fail in case of complex motion and
self-occlusions [20]. Drawing inspiration from [23], we aim to replicate
similar capabilities in a model that doesn’t necessitate NLA decomposition,
making it a more practical alternative.

##  3 Method

Figure 2: We enhance video editing by conditioning the synthesis of the video
on two branches, one that controls the appearance and the other responsible
for the motion and the structure of the object. To enable shape modifications,
we propose a Joint Flow-Structure Augmentation pipeline that outputs an
augmented flow which is processed by a Flow-Completion Network, before going
as input to the final ControlNet module. Furthermore, we introduce an
auxiliary loss, used to enhance the model fidelity to the input segmentation
map.

VASE is based on an image-conditioned video inpainting model. Our approach
preserves the original video’s motion by employing optical flow, and we
introduce explicit guidance to enhance structural manipulations. We recall
that the structure conditioning is provided for a single keyframe, which for
simplicity we assume to be the first frame of the sequence. An overview of our
pipeline is shown in Fig. 2.

Formally, let $\mathcal{X}\in\mathbb{R}^{T\times 3\times H\times W}$ be the
source video and $\mathcal{M}\in\\{0,1\\}^{T\times 1\times H\times W}$ the
segmentation mask of the foreground object targeted for editing. We denote
with $\mathcal{M}^{\text{bb}}\in\\{0,1\\}^{T\times 1\times H\times W}$ a
bounding box mask enclosing it. Our goal is to synthesize an edited video
$\mathcal{Y}\in\mathbb{R}^{T\times 3\times H\times W}$, where the appearance
of the foreground object is driven by $I_{\text{ref}}\in\mathbb{R}^{3\times
H\times W}$, while the structure of the object is specified by
$M_{\text{ref}}\in\\{0,1\\}^{1\times H\times W}$, which represents the shape
of the edited keyframe. We define the edited region as
$M_{\text{edit}}\coloneqq M_{\text{ref}}\neq\mathcal{M}_{0}$,
$M_{\text{edit}}\in\\{0,1\\}^{1\times H\times W}$.

###  3.1 Backbone

Inflated UNet. A significant challenge in video generation lies in achieving
temporal consistency, which involves minimizing the flickering and
inconsistencies between frames. The first step of our method is designed to
tackle this problem. We follow recent works [9, 4] and inflate a pretrained
diffusion model by introducing temporal layers; we refer to this model as 3D
Unet. Specifically, our temporal blocks are composed of a _1D-convolution_
followed by _temporal self-attention_. The temporal blocks are placed after
the original spatial blocks, and the input features are reshaped to
accommodate the temporal dimension. We base our framework on Paint-by-Example
(PBE) [56], which serves as our base image-conditioned diffusion model. We
adopt a similar self-supervised training strategy extended to videos and pre-
train the model to reconstruct the source video $\mathcal{X}$. The input to
the model consists of the masked source video
$\mathcal{X}\odot(1-\mathcal{M}^{\text{bb}})$, and the reference object
extracted from the $t$-th frame
$I_{\text{ref}}\coloneqq\mathcal{X}_{t}\odot\mathcal{M}^{\text{bb}}_{t}$. The
model is trained with the $\mathcal{L}_{\text{simple}}$ loss [39], defined as
$\mathcal{L}_{\text{simple}}=\lVert\epsilon-\epsilon_{\theta}(z_{t})\rVert_{2}^{2}$.
In this first stage, the model learns to generate a temporally consistent
sequence of frames.

Inflated ControlNet. Next, we introduce a conditioning mechanism to guide the
generation process, maintain the motion characteristics from the source video,
and simultaneously add explicit control on the shape. We extract the forward
optical flow $\mathcal{F}\in\mathbb{R}^{T-1\times 2\times H\times W}$ from the
source video using RAFT [47]. Then, we input to the network the structure of
the keyframe $M_{\text{ref}}={\mathcal{M}}_{0}$, _i.e_. the segmentation mask
of the first frame. We condition the model by incorporating a ControlNet [58],
which takes $\\{\mathcal{F},M_{\text{ref}}\\}$ as input, concatenated across
the channel dimension. We pad the optical flow in the temporal dimension and
repeat the structure conditioning $T$ times, to ensure compatibility of
dimensions. During this stage, we train the ControlNet while freezing the
Inflated UNet.

This model serves as our backbone. While it can achieve appearance editing
operations, it fails to perform shape editing control. During inference, if
$M_{\text{ref}}\neq\mathcal{M}_{0}$, the model tends to overlook it. We impute
this to the dominant signal coming from the optical flow $\mathcal{F}$, which
can be used to minimize $\mathcal{L}_{\text{simple}}$ without considering the
structure conditioning $M_{\text{ref}}$.

In the next subsections, we describe our proposed solutions to solve this
problem and enable shape control at inference time.

###  3.2 Joint Flow-Structure Augmentation (JFSA)

As a first step, we propose an augmentation procedure to jointly modify the
shape and the flow conditioning. As outlined in Sec. 1, we are focusing on
edits of the foreground object, which can be reduced to two main types: adding
a region or removing a region from the foreground. With _JFSA_ we aim to
simulate this behavior during training by modifying the optical flow
$\mathcal{F}$ and the conditioning structural mask $\mathcal{M}$ in a
systematic manner. Specifically, we approximate regions of the target object
that exhibit consistent motion across the $T$ frames by clustering the optical
flow along the temporal dimension:

$E=\mathtt{KMeans}(\mathcal{F}+\lambda\cdot X_{\text{bias}},\,N_{c})$ (1)

Let $E=\\{\mathcal{E}^{0},\ldots,\mathcal{E}^{N_{c}}\\}$ represent the set of
clustered regions, where $\mathcal{E}^{k}\in\\{0,1\\}^{T\times 1\times H\times
W}$ denotes the binary map associated with the $k$-th region. Here, $N_{c}$
stands for the number of clusters, $X_{\text{bias}}$ introduces a spatial
bias, and $\lambda$ governs the trade-off.

We consider $E$ as a collection of possible ground truth shape-edits, and
define two operations:

$\begin{split}\tilde{\mathcal{F}}&=\mathcal{F};\\\
\tilde{\mathcal{M}}&=\mathcal{M}\odot(1-\mathcal{E}^{k})\end{split}$ (2)

and,

$\begin{split}\tilde{\mathcal{F}}&=\mathcal{F}\odot(1-\mathcal{E}^{k})+\Big{(}\dfrac{1}{H\cdot
W}\sum_{i,j}\mathcal{F}(i,j)\Big{)}\cdot\mathcal{E}^{k};\\\
\tilde{\mathcal{M}}&=\mathcal{M}\end{split}$ (3)

Respectively, in Eq. 2, we address the scenario where the user intends to
remove a region from the foreground object. Meanwhile, in Eq. 3, we simulate
the scenario where the user aims to include an additional region in the
foreground object. This procedure is efficient and can be executed on the fly
during training. We randomly choose an edit region $\mathcal{E}^{k}$ and
perform one of the two operations with a specified probability
$p_{\text{augm}}$. Note that we use $\tilde{\mathcal{F}},\tilde{\mathcal{M}}$
as the updated signals, replacing the original $\mathcal{F},\mathcal{M}$, and
use them as described in Sec. 3.1.

###  3.3 Warping Flow-Completion Net (wFCN)

Inspired by the Video Inpainting literature [60], we introduce an additional
network that explicitly predicts a _complete_ optical flow before passing it
to the ControlNet. The objective of this network is to estimate the optical
flow within the editing area, thus simplifying the task for the subsequent
synthesis module. Unlike Video Inpainting methods, our approach involves using
a single mask for the initial keyframe instead of employing individual masks
for each frame in the video. To bridge this gap, we propose to employ forward
warping [30] as an initial estimation for the edited region’s location. It is
important to note that although forward warping may introduce holes or
inconsistencies, in our case, it is applied to a binary mask, mitigating these
issues. We repeatedly warp the edited region obtaining
$\tilde{\mathcal{E}}^{k}=W_{\text{sum}}(\mathcal{E}^{k}_{0},\mathcal{F})$,
$\tilde{\mathcal{E}}^{k}\in\\{0,1\\}^{T\times 1\times H\times W}$ , and
$W_{\text{sum}}$ the summation splatting operation as defined in [30]. Note
that, while at training time the ground truth temporal edit operation
$\mathcal{E}^{k}$ is available, we obtain better results by warping the mask
instead, to reduce the gap with inference. Conversely, during the inference
phase, the source optical flow $\mathcal{F}$ may be unavailable, particularly
when adding a region to the foreground object. In such cases, infilling the
missing region using the displacement of the nearest known point belonging to
the object provides a reliable approximation, effectively resolving the issue
(see _Supp. Mat._ for details). Our Warping Flow-Completion Network (wFCN)
takes as input the concatenation of the optical flow, the warped edited
region, and the structural mask, and predicts the complete optical flow
$\mathcal{F}_{\text{complete}}=\text{wFCN}(\tilde{\mathcal{E}}^{k},\mathcal{F},\tilde{\mathcal{M}})$.
We pre-train this network offline using our _JFSA_ procedure to obtain pairs
of editing regions and optical flow. During this phase, the primary driving
loss is the reconstruction loss
$\mathcal{L}_{\text{MSE}}(\mathcal{F}_{\text{complete}},\tilde{\mathcal{F}})$,
along with the other regularization losses defined in [60]. The complete
optical flow $\mathcal{F}_{\text{complete}}$ is utilized in conjunction with
$\tilde{\mathcal{M}}$ as input for the ControlNet.

###  3.4 Segmentation Head (SH)

To emphasize the importance of shape conditioning, we incorporate a
lightweight segmentation head during the training process. More specifically,
we extract the second-to-last feature of the 3D UNet and pass it through a
small convolutional decoder, which receives as additional input the embedding
of the diffusion timestep. We supervise the model with binary cross-entropy
loss
$\mathcal{L}_{\text{bce}}(\mathcal{M}_{\text{pred}},\tilde{\mathcal{M}})$,
with $\mathcal{M}_{\text{pred}}$ the predicted segmentation maps. We optimize
it together with the main loss of the diffusion model; the total loss thus
becomes
$\mathcal{L}_{\text{tot}}=\mathcal{L}_{\text{simple}}+\alpha\cdot\mathcal{L}_{\text{bce}}$.
We further exploit the segmentation predictions $\mathcal{M}_{\text{pred}}$ at
inference time, to preserve the background within the unpainted area,
performing masked DDIM sampling [39]. We refer to the _Supp. Mat._ for
additional details.

Method | Video Quality | Temporal Consistency | Image Alignment
---|---|---|---
DOVER $(\uparrow)$ | FVD$(\downarrow)$ | WE$(\downarrow)$ | CLIP-T $(\uparrow)$ | CLIP-S $(\uparrow)$ | DINO-S $(\uparrow)$ | LPIPS $(\downarrow)$
PAIR Diffusion [13] | 0.602 | 24.32 | 19.6 | 0.904 | 0.552 | 0.36 | 0.71
FILM [38] | 0.643 | 19.85 | 8.9 | 0.956 | 0.546 | 0.33 | 0.76
TokenFlow [12] | 0.675 | 18.67 | 6.1 | 0.957 | 0.501 | 0.29 | 0.81
VASE | 0.672 | 16.54 | 9.8 | 0.955 | 0.567 | 0.36 | 0.73
Table 1: Quantitative results for the task of Image Driven Appearance Editing.
FVD and WE have been rescaled by a factor of $10^{2}$, $10^{-3}$. Figure 3:
Results for the Image Driven Appearance Editing task. The driver image is
displayed in the bottom-left corner of the initial frame in the source video.
We refer the reader to the _Supp. Mat._ for video results that better showcase
the differences.

###  3.5 Implementation Details

Our foundation model is Paint-by-Example (PBE) [56], a latent diffusion model
[39] trained for the task of image-based inpainting. We inherit the appearance
conditioning mechanism from PBE, _i.e_. the driver image is embedded through
CLIP [36]. We add temporal layers to the network in the form of 1d convolution
and temporal self-attention, which are initialized following previous work
[4]. The architecture of the _Flow Completion Network_ is borrowed from
ProPainter [60], and adapted to accommodate one additional channel as input to
process the segmentation map. Lastly, the _Segmentation Head_ is implemented
as 4 convolutional layers, which upsample the second-to-last feature of the 3D
Unet decoder (_i.e_. before the final conv-out layer) by a factor of $8$. Our
training procedure can be divided into three stages: initially, we train the
3D UNet on the video dataset and keep it frozen in the next steps.
Subsequently, we introduce the ControlNet, conditioning on the original
optical flow and segmentation map from the source video. Finally, we integrate
the pretrained _wFCN_ and the _Segmentation Head_ into the model and jointly
train them along with the ControlNet using our _JFSA_ procedure.

We use YouTube-VOS [54] for training, and resize the frames at a resolution of
$448\times 256$. We train the model for a total of $400K$ iterations. During
the inference stage, we employ DDIM sampling with 50 steps to generate the
edited video. Additionally, we integrate classifier-free guidance [16] to
enhance the quality of the final output. We refer to the _Supp. Mat._ for more
comprehensive implementation details.

##  4 Experiments

We showcase the performances of VASE both qualitatively and quantitatively. We
study two scenarios: _Image-driven Appearance Editing_ , wherein the structure
of the source video remains unchanged, and the edit is driven by a reference
image, and _Joint Appearance-Shape Editing_ , where both the object’s
structure and appearance are edited simultaneously.

Method | Video Quality | Temporal Consistency | Shape Guidance | Throughput
---|---|---|---|---
DOVER $(\uparrow)$ | WE $(\downarrow)$ | CLIP-T $(\uparrow)$ | mIoU $(\uparrow)$ | Hz $(\uparrow)$
Shape-NLA [23] | 0.543 | 5.2 | 0.970 | 0.83 | 0.11
VASE | 0.565 | 10.2 | 0.962 | 0.89 | 0.35
Table 2: Quantitative comparison with Shape-NLA [23] on the Joint Appearance-
Shape Editing task. WE has been rescaled by $10^{-3}$. Figure 4: Results for
the Joint Appearance-Shape Editing task. Please, note that the shape edit is
provided only for the first keyframe, we show all of them only for
visualization purposes. We highlight the structure modifications either in red
or green.

###  4.1 Image-Driven Appearance Editing

The model is tasked with editing a target object in a source video based on
the guidance from a driver image. It is important to emphasize that both the
structure of the object and the background should remain unchanged.

Baselines. We compare our method with the following baselines. (i) _Frame
Independent Editing_ : we employ PAIR Diffusion [13] to independently edit
each frame of the source video. (ii) _Multi Frame Editing_ : we use FILM [38]
to perform temporal interpolation between frames edited using the frame
independent baseline [13]. A stride of 4 is utilized in this process,
obtaining videos with superior temporal consistency. (iii) _Text-based
Editing_ : we compare with TokenFlow [12], and use BLIP2 [24] to caption the
driver image before editing.

Metrics. We evaluate the competitors on different aspects [26]: (i) _Video
Quality_ : we rely on Fréchet Video Distance (FVD) [49, 50] to compute a score
for the overall video quality. Moreover, we follow recent work and compute a
per-video quality score using DOVER [52]. (ii) _Temporal Consistency_ : we
compute the Warping Error (WE) [22], as the pixel-wise difference between the
warped next frame and the current frame. Additionally, we measure the CLIP
similarity between adjacent frames (CLIP-T) [9, 12]. Given our focus on
editing a particular foreground object, our computations are confined to the
region that encloses it, without considering the background. (iii) _Image
Alignment_ : we evaluate the faithfulness of the edited frame to the driver
image; we use the cosine similarity based on CLIP [36] (CLIP-S) and DINO [31]
(DINO-S) features, and use the LPIPS to capture more low-level details [59].

Dataset. We create a benchmark, using 30 videos from the DAVIS dataset [33]
and selecting 3 driver images to edit each video. All the methods are
evaluated at a resolution of $448\times 256$, with $24$ fps.

Results. We report the quantitative results in Tab. 1. We can observe that
VASE archives good results in terms of overall visual quality. Furthermore, it
is generally more aligned with the driver image, missing some low-level
details compared to [13]. Lastly, while VASE shows slightly lower numbers in
terms of temporal consistency, we remark that it is not the main focus of the
paper, and better consistency is expected by training on a higher quality
training set both in terms of fps and visual quality (_e.g_. WebVid [2]).
Analyzing the qualitative results in Fig. 3, we can observe how the text-level
conditioning of TokenFlow sometimes misses important aspects of the driver
image (_e.g_. the color of the flamingo), and leaks the edit to the background
(second column). Conversely, results obtained with PAIR Diffusion exhibit
strong flickering artifacts, whereas FILM enhances the temporal consistency at
the expense of introducing distortions in the videos. For a more comprehensive
video comparison, we direct the reader to the _Supp. Mat._.

###  4.2 Joint Appearance-Shape Editing

The objective is to alter the structure of one object in a source video,
guided by the initial edited keyframe; simultaneously, a driver image is
employed to edit the appearance. Note that the edit is restricted to one
specific object, and the background should remain unaffected.

Baselines. We compare our method with [23], and denote it with Shape-NLA for
brevity. Specifically, we edit the key-frame with [13]. Next, we construct the
dense semantic mapping, by assigning each point in the edited area to either
the closest point in the background or the foreground.

Metrics. Following the previous setting, we compare the two methods on the
_Video Quality_ and _Temporal Consistency_. We use the metrics defined in Sec.
4.1, excluding FVD, as it could not be reliably computed on a limited set of
samples. Furthermore, we analyze two other aspects. (i) _Shape Control_ : to
assess if the shape edit is reflected in the final frame. Evaluating it for
the entire sequence presents challenges, primarily due to the absence of
ground truths for every frame. Consequently, our assessment focuses solely on
the first frame, calculating the mIoU. (ii) _Throughput_ : we quantify the
inference cost of the different methods, by reporting the number of frames
processed per second (Hz). It is noteworthy that, for Shape-NLA [23], the NLA
decomposition step is excluded from this computation. This step demands
roughly an additional 10 hours per video [20].

Dataset. We use a subset of our benchmark used in Sec. 4.1 composed of 10
videos. For each video, we manually craft a structural edit for the initial
frame.

Results. We report the quantitative results on Tab. 2. We can observe that
Shape-NLA [23], obtain excellent performances in term of temporal consistency
thanks to the NLA decomposition. However, the faithfulness to the edited shape
is inferior compared to our method, as shown by a lower mIoU. We can observe
it in the qualitative results displayed in Fig. 4. Shape-NLA struggles to
precisely edit the shape _e.g_. in the blackswan and bus examples. At the same
time, it produces artifacts around the edited shape, _e.g_. tiger. Lastly, the
NLA decomposition introduces errors in the foreground object (for instance the
legs in the horsejump-low case) and distortion in the background.

###  4.3 Ablation

Version | JFSA | wFCN | SH | DOVER | WE | CLIP-T | mIoU
---|---|---|---|---|---|---|---
I | ✗ | ✗ | ✗ | 0.514 | 13.2 | 0.939 | 0.79
II | ✓ | ✗ | ✗ | 0.529 | 12.2 | 0.945 | 0.84
III | ✓ | ✓ | ✗ | 0.546 | 11.7 | 0.952 | 0.87
Full | ✓ | ✓ | ✓ | 0.565 | 10.2 | 0.962 | 0.89

Table 3: Ablation study on the components of VASE. Figure 5: Qualitative
results obtained with different versions of our model.

We showcase results obtained with different variations of our model in Fig. 5,
and compare them quantitatively in Tab. 3. We employ the experimental setup
outlined in Sec. 4.2 for the comparison. Specifically, we train three versions
of our full model. In the first experiments, we keep only the backbone model
as described in Sec. 3.1 (Vers. I). We can notice that the shape modification
is ignored by the model, which focuses only on the optical flow to solve the
task. Secondly, we add the Joint Flow-Structure Augmentation procedure Sec.
3.2 (Vers. II). In this case, the model produces the desired edit in the first
frame but generates artifacts around the object and the edit vanishes in the
subsequent frames. Next, we introduce the Flow Completion Network Sec. 3.4
(Vers. III), which results in edits that are propagated to the whole sequence
but with severe artifacts around the borders. Lastly, incorporating the
Segmentation Head, we show the result of our Full model, which is the only one
achieving satisfactory results. We refer the reader to the _Supp. Mat._ for
further analysis and details.

##  5 Conclusions

In this paper, we present VASE, a framework that enables users with a high
degree of control over real video editing. We propose an object-centric
formulation, that enables appearance modifications through a driver image and
precise structural modifications with a single keyframe. While our method is
able to produce a variety of satisfactory edits, it presents limitations in
the presence of strong occlusions or significant changes in perspective for
the target object. Furthermore, achieving consistent edits in lengthy videos
remains a challenge. We identify these limitations as opportunities for future
exploration, possibly incorporating 3D information to make the framework more
robust.

## References

  * Avrahami et al. [2023] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  _arXiv preprint_ , 2023.
  * Bain et al. [2021] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In _ICCV_ , 2021.
  * Bar-Tal et al. [2022] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In _ECCV_ , 2022.
  * Blattmann et al. [2023] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In _CVPR_ , 2023.
  * Ceylan et al. [2023] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra.  Pix2video: Video editing using image diffusion.  In _ICCV_ , 2023.
  * Chen et al. [2023a] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin.  Control-a-video: Controllable text-to-video generation with diffusion models.  _arXiv preprint_ , 2023a.
  * Chen et al. [2023b] Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.  Anydoor: Zero-shot object-level image customization.  _arXiv preprint_ , 2023b.
  * Epstein et al. [2023] Dave Epstein, Allan Jabri, Ben Poole, Alexei A Efros, and Aleksander Holynski.  Diffusion self-guidance for controllable image generation.  _arXiv preprint_ , 2023.
  * Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In _ICCV_ , 2023.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion, 2022.
  * Ge et al. [2023] Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yu Liu, and Yogesh Balaji.  Preserve your own correlation: A noise prior for video diffusion models.  In _ICCV_ , 2023.
  * Geyer et al. [2023] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  _arXiv preprint_ , 2023.
  * Goel et al. [2023] Vidit Goel, Elia Peruzzo, Yifan Jiang, Dejia Xu, Nicu Sebe, Trevor Darrell, Zhangyang Wang, and Humphrey Shi.  Pair-diffusion: Object-level image editing with structure-and-appearance paired diffusion models.  _arXiv preprint_ , 2023.
  * Guo et al. [2023] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _arXiv preprint_ , 2023.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint_ , 2022.
  * Ho and Salimans [2022] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _NeurIPS_ , 2020.
  * Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint_ , 2022.
  * Huang et al. [2023] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou.  Composer: Creative and controllable image synthesis with composable conditions.  _arXiv preprint_ , 2023.
  * Kasten et al. [2021] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel.  Layered neural atlases for consistent video editing.  _ACM TOG_ , 2021.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint_ , 2023.
  * Lai et al. [2018] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsuan Yang.  Learning blind video temporal consistency.  In _ECCV_ , 2018.
  * Lee et al. [2023] Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Elizabeth Qiu, and Jia-Bin Huang.  Shape-aware text-driven layered video editing.  In _CVPR_ , 2023.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_ , 2023.
  * Liu et al. [2023a] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  _arXiv preprint_ , 2023a.
  * Liu et al. [2023b] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan.  Evalcrafter: Benchmarking and evaluating large video generation models.  _arXiv preprint_ , 2023b.
  * Meng et al. [2021] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Image synthesis and editing with stochastic differential equations.  _arXiv preprint_ , 2021.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint_ , 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint_ , 2021.
  * Niklaus and Liu [2020] Simon Niklaus and Feng Liu.  Softmax splatting for video frame interpolation.  In _CVPR_ , 2020.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint_ , 2023.
  * Ouyang et al. [2023] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen.  Codef: Content deformation fields for temporally consistent video processing.  _arXiv preprint_ , 2023.
  * Perazzi et al. [2016] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  In _CVPR_ , 2016.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint_ , 2022.
  * Qi et al. [2023] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  _arXiv preprint_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint_ , 2022.
  * Reda et al. [2022] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless.  Film: Frame interpolation for large motion.  In _ECCV_ , 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _CVPR_ , 2023.
  * Saharia et al. [2022a] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In _ACM MM_. ACM MM, 2022a.
  * Saharia et al. [2022b] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _NeurIPS_ , 2022b.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint_ , 2021.
  * Singer et al. [2022] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  _arXiv preprint_ , 2022.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , 2015.
  * Song and Ermon [2019] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  _NeurIPS_ , 2019.
  * Teed and Deng [2020] Zachary Teed and Jia Deng.  Raft: Recurrent all-pairs field transforms for optical flow.  In _ECCV_ , 2020.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _CVPR_ , 2023.
  * Unterthiner et al. [2018] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  _arXiv preprint_ , 2018.
  * Unterthiner et al. [2019] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphaël Marinier, Marcin Michalski, and Sylvain Gelly.  Fvd: A new metric for video generation.  2019\.
  * Wang et al. [2023] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  _arXiv preprint_ , 2023.
  * Wu et al. [2023a] Haoning Wu, Erli Zhang, Liang Liao, Chaofeng Chen, Jingwen Hou Hou, Annan Wang, Wenxiu Sun Sun, Qiong Yan, and Weisi Lin.  Exploring video quality assessment on user generated contents from aesthetic and technical perspectives.  In _ICCV_ , 2023a.
  * Wu et al. [2023b] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _ICCV_ , 2023b.
  * Xu et al. [2018] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang.  Youtube-vos: Sequence-to-sequence video object segmentation.  In _CVPR_ , 2018.
  * Xu et al. [2023] Xingqian Xu, Jiayi Guo, Zhangyang Wang, Gao Huang, Irfan Essa, and Humphrey Shi.  Prompt-free diffusion: Taking” text” out of text-to-image diffusion models.  _arXiv preprint_ , 2023.
  * Yang et al. [2023a] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen.  Paint by example: Exemplar-based image editing with diffusion models.  In _CVPR_ , 2023a.
  * Yang et al. [2023b] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.  _arXiv preprint_ , 2023b.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , 2023.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.
  * Zhou et al. [2023] Shangchen Zhou, Chongyi Li, Kelvin C.K Chan, and Chen Change Loy.  ProPainter: Improving propagation and transformer for video inpainting.  In _ICCV_ , 2023.

Supplementary Material

We present additional details and results for our method. Specifically, in
Appendix A, we delve into a more comprehensive discussion of the components of
VASE. We provide visualization for the _Joint Flow-Structure Augmentation_
procedure, along with the predictions of the _Segmentation Head_ and the
_Warping Flow Completion Network_. In Appendix B we collect the implementation
details of our method. Lastly, in Appendix C we provide additional qualitative
results of our model, both for the image-driven appearance editing and for the
join structure and appearance editing task. We refer the reader to the project
page for full video results.

##  Appendix A VASE components

In this section, we provide details about the main components of VASE and show
their respective effect with qualitative results. For a quantitative
comparison and an examination of the specific contribution of each component,
we refer the reader to Tab. 3 of the main paper.

Joint Flow-Structure Augmentation. This augmentation procedure simulates
structural modifications of the target object during training. To achieve
realistic edits, our approach involves clustering the optical flow along the
temporal dimension. This results in regions that exhibit consistent motion
within the $T$ frames. As outlined in Eq. 1, we introduce a regularization
term in the form of a spatial bias, to promote the assignment of spatially
neighbor pixels to the same region. The bias is computed as

$X_{\text{bias}}(i,j)=(i,j)\quad\forall i\in\\{0,\ldots,H-1\\},\,\forall
j\in\\{0,\ldots,W-1\\}$

and replicated $T-1$ times before adding it to the optical flow $\mathcal{F}$.
We employ KMeans clustering and vary the number of cluster centroids $N_{c}$
within a random range of 6 to 10. In Fig. 6 (a), we present a qualitative
example of the JFSA procedure. Here, distinct colors signify different regions
clustered by our approach.

Warping Flow Completion Network. We present details about the warping
operation conducted before the Flow Completion Network. This operation is
introduced to align with the original use case of Video Inpainting [60], where
a mask is supplied for each frame. Additionally, while not entirely precise,
this approach facilitates the task for the network. We use summation splitting
[30] prior to feeding the inputs to the flow completion network. During
inference, two cases arise. If a region is removed from the object, the
original optical flow is readily available. On the other hand, if we add a
region to the object, the flow is absent, leading to potential warping errors.
To address this issue, we employ a displacement determined by the closest
point in the object for which the flow is available. The nearest neighborhood
search is based on the Euclidean distance between the points. This provides us
with an approximation for the missing flow region, enabling us to warp the
edit region and facilitating the subsequent task for the wFCN.

We showcase an example of the complete optical flow predicted by the wFCN in
Fig. 6 (b).

Segmentation Head. We present qualitative results of the segmentation head’s
predictions during inference in Fig. 6 (c). These masks are utilized for
masked DDIM sampling, with further details provided in the subsequent
subsection.

Figure 6: Qualitative results showcasing components of VASE. (a) Illustrates
JSFA, where distinct colors signify different clustered regions. (b) Compares
the original optical flow extracted from the source video (top row), with the
complete optical flow predicted by the wFCN (bottom row). The edit region is
shown at the bottom of the original flow. (c) Shows the corresponding
predicted segmentation map.

##  Appendix B Model and Implementation Details

Architecture. Our foundation model is Paint-by-Example (PBE) [56], a latent
diffusion model [39] trained for the task of image-based inpaining. We inherit
the appearance conditioning mechanism from PBE, _i.e_. the driver image is
embedded through CLIP, and the obtained representation is further compressed
with a linear layer which reduces its dimension, to avoid copy-paste
artifacts. Next, we inflate the UNet architecture by adding temporal layers
between the original spatial layers. Specifically, the original spatial layers
of PBE process each frame independently, producing features with dimensions
$f_{\text{spatial}}\in\mathbb{R}^{(\text{bs}\cdot\text{T})\times\text{dim}\times\text{h}\times\text{w}}$.
The features are reshaped to expose the temporal dimension before being
forwarded to the temporal layers, _i.e_.
$f_{\text{temporal}}\in\mathbb{R}^{(\text{bs}\cdot\text{h}\cdot\text{w})\times\text{dim}\times\text{T}}$.
Finally, they are reshaped back to the original spatial structure. The
temporal layers consist of a 1D-Convolution with a kernel size of 3, followed
by a self-attention block, with relative positional encoding to capture the
temporal order. The ControlNet architecture mirrors the encoder branch of the
UNet, including the newly added temporal layers. We use a singular ControlNet,
which receives the optical flow and the segmentation maps concatenated across
the channel dimension.

The _Flow Completion Network_ architecture is borrowed from ProPainter [60].
We adapt the architecture and add one channel to the input layer, which
receives a concatenation of the source optical flow, the warped edited region,
and the segmentation mask.

The last component of our framework is the Segmentation Head, which is
implemented as 4 convolutional layers. It upsamples the second-to-last feature
of the 3D Unet decoder by a factor of $8\times$ and produces an output that
matches the dimensions of the input frames before the VQGAN encoding [39].
This head is trained using binary cross-entropy (bce) loss, with a weighting
factor set to $\lambda=0.05$.

Training. Our training procedure can be divided into three stages. (i) Train
the 3D Unet on a video dataset, in this stage the network learns to model the
temporal dimension and the consistency between adjacent frames. This stage is
performed for 250K iterations. (ii) Introduce the ControlNet with flow
conditioning and shape control while freezing the 3D Unet, in this way, the
network learns to preserve the motion from the optical flow. In this stage,
the model is trained for 100K iterations. (iii) Introduce _JFSA_ , _wFCN_ ,
and the _Segmentation Head_. The Flow Completion Network is pre-trained as a
stand-alone module, before plugging it into our pipeline, for which we use the
same training recipe as [60]. We perform the final joint-finetuning of the
whole pipeline for an additional 50K iterations. Additionally, during training
we randomly drop the conditioning, to enable classifier-free guidance at
inference time. Specifically, we drop the reference image $I_{\text{ref}}$
with a probability of $p_{\text{image}}=0.15$. At the same time, we drop the
flow conditioning and the mask with a probability of $p_{\text{flow}}=0.1$,
$p_{\text{mask}}=0.1$.

We train the model with a batch size of 16, across 8 NVIDIA A100 GPUs.
Training takes approximately 3 days. We keep all the other hyperparameters,
including the noise schedule as from Paint-by-Example [56].

Dataset. For training we use YouTube-VOS [54], a dataset comprising 3471
videos at 6 fps, collected from YouTube. The dataset comes with annotations of
the segmentation maps, which we use during training to build our masked source
video and the conditioning segmentation map. We resize the frames to a
resolution of $448\times 256$ and use sequences of $8$ frames to train our
model (_i.e_. $T=8$). The first frame of the sequence serves as the keyframe.
For evaluation purposes, we rely on DAVIS [33] and use it to build our
evaluation benchmark.

Inference. At inference, we apply the DDIM [17] algorithm with 50 steps for
all the visualization in this paper. Moreover, we use the masks predicted by
the Segmentation Heads to provide a coherent background in the regions that
are not edited. This is achieved with the masked DDIM sampling procedure, as
described in [39].

Finally, we enhance the visual quality of the generated results by employing
classifier-free guidance [16]. VASE is conditioned on three signals: the
driver image, the complete optical flow, and the structure conditioning. Let
$\emptyset$ denote the null conditioning signal. Our final prediction can be
expressed as:

$\begin{split}\tilde{\epsilon}_{\theta}(z_{t};I_{\text{ref}},\mathcal{M},\mathcal{F})&=\epsilon_{\theta}(z_{t};\emptyset,\emptyset,\emptyset)+\\\
+&s_{\text{image}}\cdot\bigl{(}\epsilon_{\theta}(z_{t};I_{\text{ref}},\emptyset,\emptyset)-\epsilon_{\theta}(z_{t};\emptyset,\emptyset,\emptyset)\bigr{)}\\\
+&s_{\text{mask}}\cdot\bigl{(}\epsilon_{\theta}(z_{t};I_{\text{ref}},\mathcal{M},\emptyset)-\epsilon_{\theta}(z_{t};I_{\text{ref}},\emptyset,\emptyset)\bigr{)}\\\
+&s_{\text{flow}}\cdot\bigl{(}\epsilon_{\theta}(z_{t};I_{\text{ref}},\mathcal{M},\mathcal{F})-\epsilon_{\theta}(z_{t};I_{\text{ref}},\mathcal{M},\emptyset)\bigr{)}\\\
\end{split}$ (4)

Throughout this paper, we keep these values fixed to $s_{\text{image}}=5$,
$s_{\text{mask}}=7$, $s_{\text{flow}}=6$.

Longer Video Prediction. Predicting long videos remains an open problem, with
many approaches facing challenges in achieving satisfactory results. In this
paper, our emphasis is not on improving the length of the videos or enhancing
temporal consistency but rather on introducing new tools to enhance editing
capabilities. Even though the model is trained for video sequences of length
$T=8$, we generate longer videos using the following approach. We condition
the next batch by taking the last frame of the current batch and concatenating
it in the temporal dimension to the next batch. It’s important to note that in
this case, we don’t mask the first frame but provide it clean as input to the
3D Unet. This behavior is replicated during training, where with a probability
of $p_{\text{clean}}=0.1$, we condition input to the model a clean first
frame, skipping the JFSA procedure.

##  Appendix C Qualitative Results

We report additional qualitative results of VASE and the other methods as
shown in the main paper. In Fig. 7 Fig. 8 we provide additional examples for
the joint appearance shape editing, while in Fig. 9 Fig. 10 we show results
for the task of image-driven appearance editing.

Figure 7: Results for the Joint Appearance-Shape Editing task. Please, note
that the shape edit is provided only for the first keyframe, we show all of
them only for visualization purposes. Figure 8: Results for the Joint
Appearance-Shape Editing task. Please, note that the shape edit is provided
only for the first keyframe, we show all of them only for visualization
purposes. Figure 9: Results for the Image Driven Appearance Editing task. The
driver image is displayed in the bottom-left corner of the initial frame in
the source video. Figure 10: Results for the Image Driven Appearance Editing
task. The driver image is displayed in the bottom-left corner of the initial
frame in the source video.


#1     #1#1#1     #1     #1     #1     #1#1          #1#1

[Albert, Arenas, Genaim, and PueblaAlbert
et al2011]
authorpersonElvira Albert, personPuri
Arenas, personSamir Genaim, and personGermán
Puebla. year2011.
Closed-form upper bounds in static cost
journalJournal of Automated Reasoning
volume46 (year2011), pages161–203.
Issue 2.

[Albert, Arenas, Genaim, Puebla, and
ZanardiniAlbert et al2012]
authorpersonElvira Albert, personPuri
Arenas, personSamir Genaim, personGerman Puebla,
and personDamiano Zanardini. year2012.
Cost analysis of object-oriented bytecode
journalTheoretical Computer Science
volume413, number1 (year2012),

[Albert, Genaim, and MasudAlbert
et al2013]
authorpersonElvira Albert, personSamir
Genaim, and personAbu Naser Masud.
On the Inference of Resource Usage Upper and Lower
journalACM Transactions on Computational Logic
volume14, number3 (year2013),

[Alonso-Blas and GenaimAlonso-Blas and
authorpersonDiego Esteban Alonso-Blas and
personSamir Genaim. year2012.
On the limits of the classical approach to cost
analysis. In booktitleStatic Analysis
(seriesLecture Notes in Computer Science),
editorpersonAntoine Miné and
personDavid Schmidt (Eds.), Vol. volume7460.
publisherSpringer Berlin Heidelberg, pages405–421.

authorpersonRobert Atkey.
Syntax and semantics of quantitative type theory.
In booktitleProceedings of the 33rd Annual ACM/IEEE
Symposium on Logic in Computer Science, LICS 2018 (Oxford, United Kingdom).
publisherACM Press, pages56–65.

authorpersonRalph Benzinger.
Automated higher-order complexity analysis.
journalTheoretical Computer Science
volume318, number1-2 (year2004),

authorpersonNils Anders Danielsson.
Lightweight semiformal time complexity analysis
for purely functional data structures. In
booktitleProceedings of the 35th Annual ACM SIGPLAN-SIGACT
Symposium on Principles of Programming Languages, POPL 2008,
editorpersonGeorge Necula and
personPhilip Wadler (Eds.). publisherACM Press,

[Danner and LicataDanner and Licata2020]
authorpersonNorman Danner and
personDaniel R. Licata. year2020.
titleDenotational semantics as a foundation for cost
recurrence extraction for functional languages.

[Danner, Licata, and RamyaaDanner
et al2015]
authorpersonNorman Danner, personDaniel R.
Licata, and personRamyaa Ramyaa.
Denotational cost semantics for functional
languages with inductive types. In booktitleProceedings
of the 20th ACM SIGPLAN International Conference on Functional Programming,
ICFP 2015 (Vancouver, BC, Canada),
editorpersonKathleen Fisher and
personJohn Reppy (Eds.). publisherACM Press,

[Danner, Paykin, and RoyerDanner
et al2013]
authorpersonNorman Danner, personJennifer
Paykin, and personJames S. Royer.
A static cost analysis for a higher-order
language. In booktitleProceedings of the 7th workshop on
Programming languages meets program verification, PLPV 2013,
editorpersonMatthew Might and
personDavid Van Horn (Eds.). publisherACM Press,

[Danner and RoyerDanner and Royer2007]
authorpersonNorman Danner and
personJames S. Royer. year2007.
Adventures in time and space.
journalLogical Methods in Computer Science
volume3, number9 (year2007),

authorpersonAntonio Flores-Montoya.
Upper and Lower Amortized Cost Bounds of Programs
Expressed as Cost Relations. In booktitleFM 2016: Formal
Methods (seriesLecture Notes in Computer Science),
editorpersonJohn Fitzgerald,
personConstance Heitmeyer, personStefania Gnesi,
and personAnna Philippou (Eds.), Vol. volume9995.
publisherSpringer International Publishing,

[Girard, Scedrov, and ScottGirard
et al1992]
authorpersonJean-Yves Girard, personAndre
Scedrov, and personPhilip J. Scott.
Bounded linear logic: a modular approach to
polynomial-time computability.
journalTheoretical Computer Science
volume97, number1 (year1992),

[Hoffmann, Aehlig, and HofmannHoffmann
et al2012]
authorpersonJan Hoffmann, personKlaus
Aehlig, and personMartin Hofmann.
Multivariate Amortized Resource Analysis.
journalACM Transactions on Programming Languages and
Systems volume34, number3 (year2012),

[Hoffmann, Das, and WengHoffmann
et al2017]
authorpersonJan Hoffmann, personAnkush Das,
and personShu-Chun Weng. year2017.
Towards automatic resource bound analysis for
OCaml. In booktitleProceedings of the 44th ACM
SIGPLAN Symposium on Principles of Programming Languages, POPL 2017,
editorpersonGiuseppe Castagna and
personAndrew D. Gordon (Eds.). publisherACM Press,

[Hoffmann and ShaoHoffmann and Shao2015]
authorpersonJan Hoffmann and personZhong
Shao. year2015.
Automatic static cost analysis for parallel
programs. In booktitleProgramming Languages and Systems:
24th European Symposium on Programming, ESOP 2015
(seriesLecture Notes in Computer Science),
editorpersonJan Vitek (Ed.),
Vol. volume9032. publisherSpringer-Verlag,

authorpersonMartin Hofmann.
The Strength of Non-Size Increasing Computation.
In booktitleProceedings of the 29th ACM SIGPLAN-SIGACT
Symposium on Principles of Programming Languages.

authorpersonMartin Hofmann.
Linear Types and Non-Size-Increasing Polynomial
Time Computation.
journalInformation and Computation
volume183, number1 (year2003),

[Hofmann and JostHofmann and Jost2003]
authorpersonMartin Hofmann and
personSteffen Jost. year2003.
Static prediction of heap space usage for
first-order functional programs. In booktitleProceedings
of the 30th ACM SIGPLAN-SIGACT Symposium on Principles of Programming
Languages, editorpersonAlex Aiken and
personGreg Morrisett (Eds.). publisherACM Press,

[Hudson, BowornmetHudson,
authorpersonHudson, Bowornmet.
titleComputer-Checked Recurrence Extraction for
Functional Programs.
thesistypeMaster's thesis. schoolWesleyan

[Jost, Vasconcelos, Florido, and HammondJost
et al2017]
authorpersonSteffen Jost, personPedro
Vasconcelos, personMário Florido, and
personKevin Hammond. year2017.
Type-based cost analysis for lazy functional
journalJournal of Automated Reasoning
volume59, number1 (year2017),

[Kavvos, Morehouse, Licata, and DannerKavvos
et al2019]
authorpersonG. A. Kavvos, personEdward
Morehouse, personDaniel R. Licata, and personNorman
Danner. year2019.
Recurrence Extraction for Functional Programs
through Call-by-Push-Value.
journalProceedings of the ACM on Programming
Languages volume4, numberPOPL, Article
articleno15 (year2019).

[Kincaid, Cyphert, Breck, and RepsKincaid
et al2017]
authorpersonZachary Kincaid, personJohn
Cyphert, personJason Breck, and personThomas
Reps. year2017.
Non-linear Reasoning for Invariant Synthesis.
journalProceedings of the ACM on Programming
Languages volume2, numberPOPL, Article
articleno54 (dateDec. year2017),
numpages33 pages.

[Knoth, Wang, Polikarpova, and HoffmannKnoth
et al2019]
authorpersonTristan Knoth, personDi Wang,
personNadia Polikarpova, and personJan Hoffmann.
Resource-Guided Program Synthesis. In
booktitleACM SIGPLAN Conference on Programming Language
Design and Implementation.

[Le MétayerLe Métayer1988]
authorpersonDaniel Le Métayer.
ACE: an automatic complexity evaluator.
journalACM Transactions on Programming Languages and
Systems volume10, number2 (year1988),

[Licata, Shulman, and RileyLicata
et al2017]
authorpersonDaniel R. Licata, personMichael
Shulman, and personMitchell Riley.
A fibrational framework for substructural and
modal logics. In booktitle2nd International Conference on
Formal Structures for Computation and Deduction, FSCD 2017
(seriesLeibniz International Proceedings in Informatics
(LIPIcs)), editorpersonDale Miller (Ed.),
Vol. volume84. publisherSchloss
Dagstuhl–Leibniz-Zentrum fuer Informatik, pages25:1–25:22.

authorpersonConor McBride.
I got plenty o' Nuttin'. In
booktitleA List of Successes That Can Change the World:
Essays Dedicated to Philip Wadler on the Occasion of His 60th Birthday
(seriesLecture Notes in Computer Science),
editorpersonSam Lindley, personConor
McBride, personPhil Trinder, and personDon
Sannella (Eds.), Vol. volume9600.

[Mitchell and PlotkinMitchell and
authorpersonJohn C. Mitchell and
personGordon D. Plotkin. year1985.
Abstract types have existential types. In
booktitleProceedings of the 12th ACM SIGACT-SIGPLAN
Symposium on Principles of Programming Languages, POPL 1985.

[Niu and HoffmannNiu and Hoffmann2018]
authorpersonYue Niu and personJan
Hoffmann. year2018.
Automatic Space Bound Analysis for Functional
Programs with Garbage Collection. In booktitleLPAR-22. 22nd
International Conference on Logic for Programming, Artificial Intelligence
and Reasoning (seriesEPiC Series in Computing),
editorpersonGilles Barthe, personGeoff
Sutcliffe, and personMargus Veanes (Eds.),
Vol. volume57. publisherEasyChair,

authorpersonChris Okasaki.
booktitlePurely Functional Data Structures.
publisherCambridge University Press.

[Orchard, Liepelt, and Eades IIIOrchard
et al2019]
authorpersonDominic Orchard,
personVilem-Benjamin Liepelt, and personHarley
Eades III. year2019.
Quantitative program reasoning with graded modal
journalProceedings of the ACM on Programming
Languages volume3, numberICFP
(year2019), pages110:1–110:30.

authorpersonJason Reed.
titleNames are (mostly) useless.
(year[n. d.]).

<https://www.cis.upenn.edu/ sweirich/wmm/wmm08-programme.html>

Presented at 3rd Informal ACM SIGPLAN Workshop on Mechanizing
Metatheory 2008.

authorpersonMads Rosendahl.
Automatic complexity analysis. In
booktitleProceedings of the Fourth International Conference
on Functional Programming Languages and Computer Architecture, FPCA 1989,
editorpersonJoseph E. Stoy (Ed.).
publisherACM Press, pages144–156.

authorpersonDavid Sands.
titleCalculi for Time Analysis of Functional
thesistypePh.D. Dissertation. schoolUniversity
of London.

authorpersonJon Shultis.
booktitleOn the complexity of higher-order
typeTechnical Report CU-CS-288-85.
institutionUniversity of Colorado at Boulder.

[Sleator and TarjanSleator and
authorpersonDaniel Dominic Sleator and
personRobert Endre Tarjan. year1985.
Self-adjusting binary search trees.
journalJournal of the ACM volume32,
number3 (year1985).

authorpersonKathryn Van Stone.
titleA Denotational Approach to Measuring
Complexity in Functional Programs.
thesistypePh.D. Dissertation. schoolCarnegie
Mellon University.

authorpersonRobert Endre Tarjan.
Amortized computational complexity.
journalSIAM J. Algebraic Discrete Methods
volume6, number2 (year1985),

[Wang, Wang, and ChlipalaWang
et al2017]
authorpersonPeng Wang, personDi Wang, and
personAdam Chlipala. year2017.
TiML: A functional language for practical
complexity analysis with invariants.
journalProceedings of the ACM on Programming
Languages volume1, numberOOPSLA, Article
articleno79 (year2017), numpages26 pages.

authorpersonBen Wegbreit.
Mechanical program analysis.
journalCommunications of the Association for
Computing Machinery volume18, number9
(year1975), pages528–539.