# InVi: Object Insertion In Videos Using Off-the-Shelf Diffusion Models

Nirat Saini * University of Maryland, College Park  Navaneeth Bodla  Cruise
LLC  Ashish Shrivastava  Cruise LLC  Avinash Ravichandran  Cruise LLC  Xiao
Zhang  Cruise LLC  Abhinav Shrivastava  University of Maryland, College Park
Bharat Singh  Cruise LLC

###### Abstract

We introduce InVi, an approach for inserting or replacing objects within
videos (referred to as inpainting) using off-the-shelf, text-to-image latent
diffusion models. InVi targets controlled manipulation of objects and blending
them seamlessly into a background video unlike existing video editing methods
that focus on comprehensive re-styling or entire scene alterations. To achieve
this goal, we tackle two key challenges. Firstly, for high quality control and
blending, we employ a two-step process involving inpainting and matching. This
process begins with inserting the object into a single frame using a
ControlNet-based inpainting diffusion model, and then generating subsequent
frames conditioned on features from an inpainted frame as an anchor to
minimize the domain gap between the background and the object. Secondly, to
ensure temporal coherence, we replace the diffusion model’s self-attention
layers with extended-attention layers. The anchor frame features serve as the
keys and values for these layers, enhancing consistency across frames. Our
approach removes the need for video-specific fine-tuning, presenting an
efficient and adaptable solution. Experimental results demonstrate that InVi
achieves realistic object insertion with consistent blending and coherence
across frames, outperforming existing methods.

11footnotetext: work done during internship at Cruise. Corresponding email:
<EMAIL_ADDRESS>
##  1 Introduction

The emergence of image and video generation algorithms has opened up exciting
new possibilities for utilizing generated data across various domains,
including media production, AR/VR, and synthetic data for model training
Rombach et al. (2022); Guo et al. (2023b); PNVR et al. (2023); Ramesh et al.
(2022); Esser et al. (2023); Shrivastava et al. (2017). However, unconstrained
text-to-image/video generation suffices only in a limited set of scenarios. In
practice, there is often a need for enhanced control over image/video
generation processes, encompassing aspects such as character consistency,
pose, and beyond. This need has prompted the development of numerous
algorithms in the image generation domain, including inpainting Lugmayr et al.
(2022); Rombach et al. (2022), LoRARuiz et al. (2023); Hu et al. (2022), and
ControlNet Zhang et al. (2023). These techniques ensure that the generated
images adhere to constraints such as background, style, and pose. In the realm
of video generation, algorithms such as Geyer et al. (2023); Cao et al.
(2023); Wu et al. (2023) have addressed the demand for control, but many
predominantly focus on comprehensive restyling of entire videos rather than
the nuanced task of inserting or replacing specific objects within the video –
a process commonly known as inpainting. Furthermore, while some approaches
tackle object manipulation, they often extend changes to the entire scene’s
background rather than solely concentrating on modifying the subject.

Figure 1: InVi inserts objects into a background video using a foreground
mask, a control signal (e.g., pose, canny, depth map), and a text prompt by
leveraging off-the-shelf diffusion models. It ensures that the inserted object
aligns semantically with the text, is temporally coherent in time, and also
conforms spatially to the control signal.

In this work, we focus on the tasks of adding and replacing objects in a video
(Figure 1). Unlike recent techniques such as those presented in Geyer et al.
(2023); Wu et al. (2023), we choose text-to-image diffusion models instead of
text-to-video diffusion models, as the latter necessitate significant
modifications for our specific task. Moreover, by building upon text-to-image
models, we circumvent the requirement for training on extensive video datasets
and can leverage a wide array of established text-to-image models spanning
various domains, including anime, art, photography, autonomous driving, and
more. This strategic choice enables us to take advantage of pre-trained
conditional models like inpainting Rombach et al. (2022), LoRARuiz et al.
(2023); Hu et al. (2022), ControlNet Zhang et al. (2023), and seamlessly
integrate them into our algorithm.

Existing approaches for video editing exhibit shortcomings, such as not
generating all the frames Geyer et al. (2023) or requiring expensive per-video
fine-tuning Wu et al. (2023). Methods like Tokenflow Geyer et al. (2023),
which opt for a joint synthesis approach, however, generates only a subset of
the required frames and rely on optical flow to generate the remaining ones.
This limitation arises from the challenge of synthesizing all frames jointly,
which becomes increasingly challenging due to GPU memory limitations, leading
to performance degradation as the number of frames increases. On the other
hand, methods like Tune-a-Video Wu et al. (2023) require additional temporal
layers and fine-tuning on the target video, leading to significant latency.

To tackle these challenges, we introduce InVi, a novel method for inpainting
objects in videos. Leveraging off-the-shelf text-to-image latent diffusion
models, our approach seamlessly applies to videos of any duration, eliminating
the requirement for individual fine-tuning for each video. In addressing
object inpainting in videos, our method addresses two primary challenges: (1)
Ensuring realistic blending of the inserted object in the target video,
avoiding a resemblance to its appearance in the source image. (2) Ensuring
consistency across frames during the video synthesis process.

To achieve a seamless integration of the source image into the target image,
InVi introduces a two-step inpaint and match process. Initially, the object is
inserted into a single video frame, leveraging the effectiveness of image-
based inpainting. Subsequently, the inpainted frame serves as the reference
for generating subsequent frames, ensuring that video synthesis is conditioned
on features within the domain of the target video rather than the source image
alone. To maintain coherence across frames, InVi employs an auto-regressive
architecture with extended-attention to incorporate features from the
preceding frame while generating the current frame. Through experiments
conducted on several videos from the DAVIS dataset and our own test set, which
includes novel object insertion scenarios, we observe that InVi outperforms
other methods by more than 40 points in background consistency metrics and is
the preferred choice in nearly 70% of the videos in our user study.

##  2 Related Works

Conditional video generation and editing: Based on the progress in generating
images from text with diffusion models Saharia et al. (2022); Ramesh et al.
(2022); Rombach et al. (2022), there has been an increase in works that
address video generation Guo et al. (2023a); Chen et al. (2023); Wu et al.
(2023). This has facilitated the creation of videos from textual descriptions,
which can be further refined to achieve video-to-video generation by using
attributes derived from initial video inputs. For instance, Gen-1 Esser et al.
(2023) utilizes estimated depth as a conditioning factor, while VideoComposer
Wang et al. (2023) uses a broader array of inputs such as depth, motion
vectors, and sketches. However, most of these methods need explicit training
on videos for learning motion Guo et al. (2023a); Chen et al. (2023), and
ensuring that these models generalize to arbitrary motion patterns requires
access to carefully curated large video datasets, which are relatively fewer
(or non-existent) than those available for images Schuhmann et al. (2022).
Additionally, substantial computational resources are required for the
development of these models and their derivatives for conditional generation.
To the best of our knowledge, there does not exist a text to video model which
is trained end-to-end, which supports inpainting objects in videos, while
providing support for using auxiliary conditions like pose, depth, edgemaps,
etc., as is commonly available for images. To overcome the challenges
associated with training such complex models on videos, some approaches resort
to single-image editing, subsequently extending these modifications across
video sequences by identifying and applying edits to corresponding pixels
throughout the frames and their efficacy hinges on robust tracking. Various
methods Yang et al. (2023); Gu et al. (2023) have employed techniques such as
optical flow, keypoint tracking, or other forms of motion detection to address
this challenge. However, these techniques are hard to scale to long videos,
for consistent appearance changes in objects.

Adapting Image Models for Video to Video tasks: Many methods have extended
image-to-image models for swapping objects in videos. For instance,
Khachatryan et al. (2023) modifies self-attention mechanisms in diffusion
models, while Wu et al. (2023) conducts per-video fine-tuning and employs
inversion-denoising techniques for editing purposes. MasaCtrl Cao et al.
(2023), originally developed for image editing tasks, has been extended to
video generation tasks and leverages the first frame generated as a reference
to synthesize subsequent frames in the video sequence.

Liu et al. (2023a) and Fate-Zero Qi et al. (2023) adapt image-to-image
pipelines Hertz et al. (2022); Tumanyan et al. (2023); Brooks et al. (2023)
for video editing by introducing modifications to cross-frame attention
modules, incorporating null-text inversion, and more. However, most existing
methods are limited to generating very short video clips. TokenFlow Geyer et
al. (2023) produces keyframes and employs a nearest-neighbor field on
diffusion features to extend keyframe attributes to remaining frames. However,
as the video length increases, interpolation performance may degrade due to
accumulated interpolation errors over time. In contrast, our model enhances
spatio-temporal attention Khachatryan et al. (2023); Liu et al. (2023a); Qi et
al. (2023) with anchor-based cross-frame attention, enabling the generation of
long videos with any desired number of frames. Our work also differs from
TokenFlow Geyer et al. (2023) in its support for inpainting. Geyer et al.
(2023) does not support in-painting, as it is tailored to preserve the
structure and motion of the original video and cannot handle edits like
changing the size, shape, pose or motion patterns of objects. We use similar
ideas of latent inversion of the source video, but they can be of a video from
a different domain, and we can use it’s pose or canny features to inpaint a
similar object in new videos. This ensures sharp and consistent object
insertion in new videos, while Geyer et al. (2023) fails in maintaining
sharpness of a new object, due to its optical flow propagation in the latent
space.

Figure 2:  InVi inference pipeline: (a) Given a video and object bounding
boxes, first, we crop a region around the bounding box which is inpainted. (b)
Next, we use a ControlNet-based inpainting diffusion model to inpaint the
cropped region in the first frame. (c) To ensure temporal consistency when
inpainting subsequent frames, we employ the previous frame as an anchor image.
This is achieved by adapting the self-attention block of the denoising U-Net
with extended attention. Specifically, we augment the Keys and Values of the
current frame being inpainted with those of the anchor frame, allowing for
consistent appearance. Finally, the inpainted crop is seamlessly integrated
back into the input video.

##  3 InVi

We build upon the concepts of Latent Diffusion Models Ho et al. (2020);
Rombach et al. (2022), DDIM inversion Rombach et al. (2022); Geyer et al.
(2023) and Lora Hu et al. (2022). Readers are encouraged to refer to the
methods or appendix for a more in-depth details. Given an input video
$\boldsymbol{\mathcal{I}}=[\mathbf{I}^{1},...,\mathbf{I}^{n}]$ comprising $n$
frames, a text prompt $\boldsymbol{\mathcal{P}}$ describing the desired edit
and a control sequence
$\boldsymbol{\mathcal{C}}=[\mathbf{C}^{1},...,\mathbf{C}^{n}]$, InVi generates
an edited video
$\tilde{\boldsymbol{\mathcal{I}}}=[\tilde{\mathbf{I}}^{1},...,\tilde{\mathbf{I}}^{n}]$.
As in LDM Rombach et al. (2022), the video frames are converted to latent
feature using an encoder, $E$, and the corresponding encoded features are
denoted by $[\mathbf{x}^{1},\dots,\mathbf{x}^{n}]$. Similarly, the encoded
features of the edited video are denoted by
$[{\tilde{\mathbf{x}}}^{1},\dots,{\tilde{\mathbf{x}}}^{n}]$. The edited video
aligns spatially with the control sequence $\boldsymbol{\mathcal{C}}$ and
conforms to the semantic constraints outlined in $\boldsymbol{\mathcal{P}}$.
The text prompt, $\boldsymbol{\mathcal{P}}$, offers generic semantic guidance,
influencing factors such as object appearance. Alternatively, the desired
edit’s appearance can be specified directly as an image instead of the text
prompt, for which, we leverage LoRA Hu et al. (2022). In contrast, the control
sequence $\boldsymbol{\mathcal{C}}$ provides more nuanced control, such as
pose or object shape. Various methods exist for providing spatial control,
denoted by $\boldsymbol{\mathcal{C}}$, such as depth maps, edge maps, and
normal maps for generic objects, or human poses if the object is a person
Zhang et al. (2023). Next, we will describe each of the steps in our pipeline
in more detail.

###  3.1 Generating the first-frame and pre-processing

First, given the object’s location in each frame via bounding boxes, we
extract a region of fixed resolution by expanding these bounding boxes, as
illustrated in Figure 2(a). We then insert an object into the first frame, for
which we rely on a ControlNet-based inpainting diffusion model. This can be
any off-the-shelf text to image inpainting model for prompt-based editing and
personalized model using LoRARuiz et al. (2023) for reference image based
editing. Once one frame is edited using image inpainting pipeline, we use this
generated image as an “anchor” denoted as $\mathbf{I^{\text{anc}}}$ and edit
the remaining frames.

To prepare the inputs for generating subsequent frames, we first pass the
masked image through a VAE encoder, as done in prior work Rombach et al.
(2022), compressing it into a lower-dimensional input ($64\times 64\times 4$
in our experiments). This input is then concatenated with a suitably
downsampled mask of identical dimensions ($64\times 64$), indicating the area
to be inpainted. In contrast to the inpainting pipeline in Rombach et al.
(2022), which combines these inputs with Gaussian noise (sized $64\times
64\times 4$) during inpainting, we utilize the output after DDIM inversion on
background frames as input for the inpainting model. This step is crucial for
maintaining video consistency, as DDIM inversion on the background frame
ensures a consistent noise pattern across frames. Additional conditions such
as pose or depth-map are provided to ControlNet, as outlined in Zhang et al.
(2023). A comprehensive wire diagram detailing all inputs for our pipeline is
illustrated in Figure 2(c).

###  3.2 Temporally Consistent Frame Inpainting

To propagate information from the edited anchor frame
$\mathbf{I}^{\text{anc}}$ to another video frame, we propose to use cross-
frame attention mechanisms, circumventing conventional methods such as optical
flow or explicit point tracking. Given an anchor frame,
$\mathbf{I}^{\text{anc}}$, we incorporate it as an additional input to the
diffusion model and replace the self-attention mechanism in the model with
cross-frame attention.

Specifically, we use the anchor frame features to augment keys, denoted by
$\mathbf{K}$, and values, denoted by $\mathbf{V}$, within the attention layers
of the diffusion model. We denote the key and value matrices of the $i^{th}$
frame as $\mathbf{K}_{i,l,t}$ and $\mathbf{V}_{i,l,t}$, respectively, where
$l$ is the layer index of the diffusion model and $t$ is the diffusion step.
Similarly, we denote the key and value matrices of the model obtained when the
anchor frame is passed to the model as $\mathbf{K}^{\text{anc}}_{l,t}$ and
$\mathbf{V}^{\text{anc}}_{l,t}$***For brevity, we will omit the subscripts $l$
and $t$ where context makes it clear., respectively. To edit $i$-th frame
$\mathbf{I}^{i}$, we modify the self-attention module to a cross-frame
attention using the key and value vectors of anchor frames as follows:

$l^{\text{th}}\text{layer
feature}=\texttt{Softmax}\left(\frac{\mathbf{Q}_{i,l,t}[\mathbf{K}_{i,l,t},\mathbf{K}^{\text{anc}}_{l,t}]^{T}}{\sqrt{d}}\right)[\mathbf{V}_{i,l,t},\mathbf{V}^{\text{anc}}_{l,t}],\;\forall
l,\forall t\in[1,\dots,T].$

Note that this augmentation does not change the network architecture and does
not require any learning of new parameters. Our method, as shown in Figure
2(c), utilizes softmax-generated attention scores to integrate
$\mathbf{V}^{\text{anc}}$ features from the anchor frame. This process
effectively enforces the temporal correspondence between the current frame and
the anchor frame, and facilitates the propagation of value features from the
anchor frames to the current frame through the multiplication of attention
scores with $\mathbf{V}^{\text{anc}}$. By substituting the self-attention
module with an anchor-based cross-frame attention mechanism, we achieve
temporal consistency across the edited video frames.

We could use one anchor frame for the entire video, however, this is not ideal
as the background appearance and the pose of an object gradually evolves over
time. Therefore, once we generate a frame $i$, it serves as the anchor for
generating the next frame $i+1$. This sequential process is described in
Algorithm 1.

Algorithm 1 InVi: Object Insertion in Videos

Input:

${\mathbf{X}}=[\mathbf{x}^{1}_{b},\dots,\mathbf{x}^{n}_{b}]$ $\triangleright$
Background video in latent space

$\boldsymbol{\mathcal{M}}=[\mathbf{M}^{\text{1}},\dots,\mathbf{M}^{n}]$
$\triangleright$ Downsampled input mask

$\mathbf{X}_{bm}=[\mathbf{x}^{1}_{bm},\dots,\mathbf{x}^{n}_{bm}]$
$\triangleright$ Masked background in latent space

$\boldsymbol{\mathcal{C}}=[\mathbf{C}^{1},\dots,\mathbf{C}^{n}]$
$\triangleright$ Conditional inputs

$\boldsymbol{\mathcal{P}}$, $\boldsymbol{\phi}$ $\triangleright$ Target text
prompt, ControlNet-based inpainting model

$\\{\mathbf{x}^{i}_{t}\\}_{t=1}^{T}\leftarrow\text{DDIM-
Inv}[\mathbf{x}^{i}_{b}]\quad\forall i\in[1,\dots,n],\;\forall
t\in[1,\dots,T]$
For $t=T,\dots,1$ do

$\tilde{\mathbf{x}}^{1}_{t}=\boldsymbol{\phi}(\mathbf{x}^{1}_{t},\mathbf{x}^{1}_{bm},\mathbf{M}^{1},\mathbf{C}^{1})$

$\mathbf{K}^{\text{anc}}_{l,t},\mathbf{V}^{\text{anc}}_{l,t}\leftarrow\mathbf{K}_{1,l,t},\mathbf{V}_{1,l,t}\qquad\forall
l$ $\triangleright$ save first frame features in a cache

For $i=2,\dots,n$ do

For $t=T,\dots,1$ do

load $\mathbf{K}^{\text{anc}}_{l,t},\mathbf{V}^{\text{anc}}_{l,t}$ from cache

$\tilde{\mathbf{x}}^{i}_{t}\leftarrow\boldsymbol{\phi}({\mathbf{x}^{i}_{t},\mathbf{x}^{i}_{bm_{t}},\mathbf{M}^{i}_{t},\mathbf{C}^{i}_{t},\mathbf{K}^{\text{anc}}_{t},V^{\text{anc}}_{t}})$
$\triangleright$ inpaint $i$-th frame with anchor features

save $\mathbf{K}_{i,l,t},\mathbf{V}_{i,l,t}$

$\mathbf{K}^{\text{anc}}_{l,t},\mathbf{V}^{\text{anc}}_{l,t}\leftarrow\mathbf{K}_{i,l,t},\mathbf{V}_{i,l,t}$
$\triangleright$ Update cache with $i$-th frame features

Output:
$\tilde{\mathbf{X}}=[\tilde{\mathbf{x}}_{1}^{1},\dots,\tilde{\mathbf{x}}_{1}^{n}]$
$\triangleright$ Latents for inpainted frames at $t=1$

###  3.3 Post-processing

After inpainting the object within the Region of Interest (RoI), an occasional
subtle halo effect emerges, resembling a flickering square, in the vicinity of
the inserted object. In the case of high-resolution videos, due to the limited
training of base diffusion models on such resolutions (and an order of
magnitude higher inference time), object inpainting can only be performed
within a small RoI. The subtle differences which result from VAE based
reconstruction are not very prominent (although noticeable) when the inpainted
RoI is composed with the original frame but this gets amplified in a video as
the object moves. Consequently, to achieve seamless and efficient blending for
high resolution videos, we adopt a multi-step approach. Initially, we extract
the mask of the inserted object using grounding-DINO Liu et al. (2023b) (for
detecting arbitrary classes) and SAM Kirillov et al. (2023) (getting object
masks inside bounding boxes). Once the mask is obtained, we employ dilation to
expand its boundary. Subsequently, we utilize Lama Suvorov et al. (2022) to
inpaint the pixels within this boundary, ensuring smooth blending throughout
the video sequence as shown in Figure 3. This comprehensive strategy enhances
visual coherence and minimizes any artifacts or discrepancies resulting from
the object insertion process. Note that for low resolution videos where the
entire frame can be inpainted, we do not require this step.

Figure 3: Post-processing to remove flickering square artifacts. a) Background
image. b) Initial image generated from our pipeline. c) Zoomed-in view
revealing artifacts around the inserted object. d) A trimap is generated to
facilitate seamless blending of the object into the background. e) Post-
processed frame showcasing the final result after blending the inserted object
with the background.

##  4 Experiments

Our method is evaluated across diverse datasets, including videos from the
DAVIS dataset which are used in prior work Perazzi et al. (2016), a selection
of videos from the VIRAT surveillance dataset Oh et al. (2011), as well as
human-centric videos sourced from YouTube. Additionally, we curate our own
video footage featuring cars, traffic cones, falling balls, and various moving
objects to further assess the robustness and efficacy of our method. To
replicate synthetic assets suitable for insertion into a 3D scene using
simulation engines for applications like surveillance, AR/VR, and autonomous
driving, we adopt two approaches. Firstly, we gather videos with a static
camera over extended durations, enabling the extraction of conditional inputs
from earlier time frames. Alternatively, we employ object removal software
(RunwayML) to artificially remove objects from scenes, allowing us to utilize
conditional inputs from the original video. We attach examples of these in the
supplementary material. The spatial resolution of our videos (after cropping)
is $384\\!\times\\!672$ or $512\\!\times\\!512$ pixels, and they consist of
anywhere from $24$ to $200$ frames. Our evaluation dataset comprises of $30$
text-video pairs. When training a LoRA-Dreambooth model, we train for $1200$
iterations with a rank of $96$, using a single reference image, without
setting any regularization. In our experiments, we use the inpainting version
of RealisticVision 5.0, which is based on Stable Diffusion 1.5 Rombach et al.
(2022). Our computational overhead (apart from DDIM inversion) is minimal
compared to the per-frame baseline as we only double the FLOPs, and memory in
the self-attention blocks of the transformer layers, while everything else
remains the same.

###  4.1 Baselines

We benchmark InVi against several video editing methods that swap objects
while preserving their structure. These include: (1) Fate-Zero Qi et al.
(2023), a zero-shot text-based video editing method; (2) Tune-a-Video Wu et
al. (2023), which fine-tunes the text-to-image model on the given test video;
and (3) TokenFlow Geyer et al. (2023), which edits selected anchor frames and
propagates the implicit flow from the keyframes to the rest of the video using
an off-the-shelf propagation method. We employ PnP-Diffusion Tumanyan et al.
(2022) based editing with TokenFlow. These methods alter the entire frame and
do not preserve the background. Since there are no existing video inpainting
methods that utilize off-the-shelf diffusion models, we include two additional
baselines to evaluate the inpainting performance: (1) Per-frame diffusion-
based image inpainting baseline using ControlNet; and (2) a ControlNet Zhang
et al. (2023) based inpainting pipeline for TokenFlow.

Table 1: Quantitative Results for object swapping (on the left) and object
insertion (on the right). Evaluation for background consistency, temporal
appearance consistency, and alignment with prompts.

| FateZero | Tune-a-Video | TokenFlow | InVi
---|---|---|---|---
CLIP-Text | 0.30 | 0.31 | 0.32 | 0.33
CLIP-Temp | 0.95 | 0.95 | 0.96 | 0.97
Back-L1 | 35.66 | 100.98 | 42.26 | 6.40

| Frm+Inp | TokenFlow+Inp | InVi
---|---|---|---
CLIP-Text | 0.24 | 0.26 | 0.28
CLIP-Temp | 0.96 | 0.97 | 0.98
LPIPS | 0.07 | 0.05 | 0.02

Figure 4: User Preference Study: InVi Outperforms Baseline Methods in text
alignment, background and temporal appearance consistency and overall video
quality. Figure 5: Qualitative results. The first image is a background frame
from the video undergoing inpainting. Subsequent frames depict the video with
the inserted object.

###  4.2 Quantitative Evaluation

Following previous workGeyer et al. (2023); Qi et al. (2023), we use several
metrics to evaluate various aspects of our object editing and inpainting
techniques. Firstly, we compute CLIP-Text, which represents the average CLIP
feature similarity between the generated frames and the target prompt, serving
as an indicator of video-text alignment. For assessing temporal consistency
within the video, we utilize CLIP-Temp, which measures the similarity of
consecutive frames and averages the results across the generated video. Given
the importance of maintaining background consistency while editing a specific
object in videos, we use a background mask to evaluate Back-L1 which is the
average L1 distance between each pixel across corresponding frames of original
video and edited video. Video editing is more common task, hence we compare
with existing baselines which operate off-the-shelf without any training, for
a fair comparison with our method. For inserting new objects in a video, all
baselines and our method inpaint only the object, the background remains
consistent. Hence, instead of Back-L1 we use average LPIPS Zhang et al.
(2018), which is patch based perceptual similarity score across consecutive
frames of the video. Lower LPIPS means more similarity across frames. Finally,
in addition to objective metrics, we conduct a user study to gauge the
alignment of the edited video quality with human preferences, covering aspects
such as text alignment, background changes, temporal consistency, and overall
impressions.

###  4.3 User Study

Video editing and inpainting is a subjective task, where quality of results
cannot be evaluated with quantitative metrics alone. Hence, we also conduct a
user preference study (with 15 users, 195 question responses), where users are
shown videos of baselines and our method, and are asked to pick the video with
best text alignment, background consistency (for edited video), temporal
consistency (if the inpainted object is consistent in appearance across
frames) and overall visual quality (least blurriness and extra artifacts).
Figure 4 shows that users prefer InVi across all questions $\sim 75\%$ times.
While Tokenflow Geyer et al. (2023) is preferred $\sim$15% of times across all
the qualitative categories. More details can be found in supplemental
materials.

###  4.4 Qualitative Evaluation

As depicted in Figure 7, we conduct a comparative analysis of InVi against
prominent baselines. Our approach, represented in the bottom row, demonstrates
superior performance by closely adhering to editing instructions and ensuring
temporal coherence in the edited videos. Conversely, other techniques often
struggle to achieve both objectives simultaneously. Tune-A-Video Wu et al.
(2023) expands a 2D image model into a video model and fine-tunes it to follow
the video’s movement closely. While effective for short clips, it encounters
challenges in accurately capturing movement in longer videos, resulting in
visual artifacts such as cartoonish appearances in the edited videos, as
observed in the car example. Similarly, fate-zero also exhibits artifacts and
deviates from the editing text-prompt closely. Although TokenFlow Geyer et al.
(2023) yields reasonable results overall, it fails to perform well for
inpainting. While it effectively edits rigid objects using flow, it struggles
with inserting articulated moving objects like walking people. Moreover, all
baselines exhibit inconsistencies in maintaining the background consistent
with the source video, often modifying the background along with the object to
be edited. Through a comprehensive user study and qualitative assessments, as
shown in Table 1 and Figure 5, we demonstrate that InVi excels in preserving
background consistency while inserting new objects into the scene.

Figure 6: Ablation experiments: We make simple changes to the baseline
methods. Frm+Inp conducts frame-wise inpainting using a constant seed and
prompt. Tokenflow preserves the exact structure of the original jeep (like
preserves the grills and mostly changes the color). TokenFlow+Inp combines
ControlNet along with an inpainting method, serving as a baseline for
inpainting, but leads to blurry results. TokenFlow+Inp (No Flow) removes the
nearest-neighbor field computation from Tokenflow, and keeps the sliding
window based inpainting of 2 frames at a time. Finally, InVi, which surpasses
these methods in terms of clarity, consistency, and sharpness, establishing
itself as the preferred choice for inpainting tasks. Figure 7: In our
qualitative comparison, we contrast the performance of InVi with three
baseline methods: FateZero, TokenFlow, and Tune-a-video. FateZero frequently
diverges from the editing prompt, as seen in the woman running example.
Meanwhile, both TokenFlow and Tune-a-video unintentionally modify the
background. InVi, however, consistently yields results that closely align with
the editing prompt while preserving the background.

###  4.5 Ablation

####  4.5.1 Advancing Beyond TokenFlow for Inpainting Tasks

TokenFlow Geyer et al. (2023) primarily works for video editing tasks, and
relies on optical flow in the latent space. However, as seen in Figure 6 (Row
3), it results in color leakage in edited objects and unwanted color
saturation changes in background colors. To compare with an inpainting
pipeline, we modified TokenFlow to include a 9-channel inpainting based UNet
for inference alongside ControlNet Zhang et al. (2023). This mitigates the
color leakage issues and enhances background consistency, as seen in Row 4 of
Figure 6, but leads to blurry and unrealistic video outputs. TokenFlow relies
on two main components: (i) Extended-attention, which selects and edits 5-6
frames sparsely from the video, ensuring consistent appearance across all
frames, and (ii) Flow propagation across other frames, which computes latents
for unedited frames through interpolation in the latent space based on edited
frames. Our hypothesis suggests that the blurriness observed in TokenFlow
results from flow computation. We experiment with using only extended-
attention with a sliding window, which results in sharper inpainted objects
(Row 5).

InVi edits one frame and recursively use the generated frames for editing the
remaining video, which ensures consistency in appearance throughout the video.
Because of the recursive approach, we do not need to jointly generate $K$
frames sampled across the video, but we only use the previously generated
frame while generating the next frame. Hence, our memory usage only increases
by a factor of $2$ compared to Tokenflow Geyer et al. (2023).

##  5 Conclusion and Future Work

We presented a new approach to use text-to-image models for video inpainting
tasks, using off-the-shelf models which operates without the need for video-
specific training. By harnessing DDIM inverted latents extracted from the
source video and incorporating the structural information of new objects via
conditional ControlNet inputs, InVi seamlessly inpaints new objects into
scenes. Utilizing anchor-frame based extended-attention for editing frames,
InVi ensures both consistency in appearance and structure of the inserted
object. Our method surpasses existing baselines, showcasing significant
enhancements in temporal consistency and visual fidelity. Moreover, unlike
prior methods, InVi efficiently handles longer videos with limited GPU memory
and enables the insertion of dynamic objects without requiring an explicit
motion module. One limitation of our work is that our method relies on 2D
bounding boxes in each frame, which can either be provided by the user or
estimated using the geometry of the scene. In future work, we plan to automate
the generation of these boxes using GPT based layout generation techniques, so
that it can be more broadly applicable. As our work builds upon existing image
generation methods, we inherit both the positive and negative societal impact
of such methods.

## References

  * Brooks et al. [2023] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18392–18402, 2023.
  * Cao et al. [2023] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  _arXiv preprint arXiv:2304.08465_ , 2023.
  * Chen et al. [2023] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang-Jin Lin.  Control-a-video: Controllable text-to-video generation with diffusion models.  _ArXiv_ , abs/2305.13840, 2023.  URL https://api.semanticscholar.org/CorpusID:258841645.
  * Esser et al. [2023] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7346–7356, 2023.
  * Geyer et al. [2023] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel.  Tokenflow: Consistent diffusion features for consistent video editing.  _arXiv preprint arxiv:2307.10373_ , 2023.
  * Gu et al. [2023] Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia-Wei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang.  Videoswap: Customized video subject swapping with interactive semantic point correspondence.  _ArXiv_ , abs/2312.02087, 2023.  URL https://api.semanticscholar.org/CorpusID:265609343.
  * Guo et al. [2023a] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Y. Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _ArXiv_ , abs/2307.04725, 2023a.  URL https://api.semanticscholar.org/CorpusID:259501509.
  * Guo et al. [2023b] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai.  Animatediff: Animate your personalized text-to-image diffusion models without specific tuning.  _arXiv preprint arXiv:2307.04725_ , 2023b.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_ , 2022.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint arXiv:2303.13439_ , 2023.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Liu et al. [2023a] Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia.  Video-p2p: Video editing with cross-attention control.  _arXiv preprint arXiv:2303.04761_ , 2023a.
  * Liu et al. [2023b] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023b.
  * Lugmayr et al. [2022] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11461–11471, 2022.
  * Oh et al. [2011] Sangmin Oh, Anthony J. Hoogs, A. G. Amitha Perera, Naresh P. Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, Jake K. Aggarwal, Hyungtae Lee, Larry S. Davis, Eran Swears, Xiaoyang Wang, Qiang Ji, Kishore K. Reddy, Mubarak Shah, Carl Vondrick, Hamed Pirsiavash, Deva Ramanan, Jenny Yuen, Antonio Torralba, Bi Song, Anesco Fong, Amit K. Roy-Chowdhury, and Mita Desai.  A large-scale benchmark dataset for event recognition in surveillance video.  _CVPR 2011_ , pages 3153–3160, 2011.  URL https://api.semanticscholar.org/CorpusID:263882069.
  * Perazzi et al. [2016] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus H. Gross, and Alexander Sorkine-Hornung.  A benchmark dataset and evaluation methodology for video object segmentation.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 724–732, 2016.  URL https://api.semanticscholar.org/CorpusID:1949934.
  * PNVR et al. [2023] Koutilya PNVR, Bharat Singh, Pallabi Ghosh, Behjat Siddiquie, and David Jacobs.  Ld-znet: A latent diffusion approach for text-based image segmentation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 4157–4168, October 2023.
  * Qi et al. [2023] Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen.  Fatezero: Fusing attentions for zero-shot text-based video editing.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 15886–15896, 2023.  URL https://api.semanticscholar.org/CorpusID:257557738.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _ArXiv_ , abs/2204.06125, 2022.  URL https://api.semanticscholar.org/CorpusID:248097655.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In _Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18_ , pages 234–241. Springer, 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Shrivastava et al. [2017] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb.  Learning from simulated and unsupervised images through adversarial training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2017.
  * Suvorov et al. [2022] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions.  In _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , pages 2149–2159, 2022.
  * Tumanyan et al. [2022] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1921–1930, 2022.  URL https://api.semanticscholar.org/CorpusID:253801961.
  * Tumanyan et al. [2023] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1921–1930, 2023.  URL https://api.semanticscholar.org/CorpusID:253801961.
  * Vaswani et al. [2017] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _Neural Information Processing Systems_ , 2017.  URL https://api.semanticscholar.org/CorpusID:13756489.
  * Wang et al. [2023] Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou.  Videocomposer: Compositional video synthesis with motion controllability.  _arXiv preprint arXiv:2306.02018_ , 2023.
  * Wu et al. [2023] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7623–7633, 2023.
  * Yang et al. [2023] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy.  Rerender a video: Zero-shot text-guided video-to-video translation.  _arXiv preprint arXiv:2306.07954_ , 2023.
  * Zhang et al. [2023] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  _2023 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 3813–3824, 2023.  URL https://api.semanticscholar.org/CorpusID:256827727.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2018.

##  Appendix A Preliminaries

We introduce concepts that are required to understand our methods. Readers are
encouraged to refer to the methods for a more in-depth treatment.

Diffusion models Ho et al. [2020] gradually introduce Gaussian noise to a
sample $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$ over $T$ steps, yielding noisy
samples $\mathbf{x}_{t},t=1,\dots,T$. The distribution of these noisy samples
is governed by
$q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{\alpha_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I})$,
where $\beta_{t}$ denotes the noise variance at a diffusion step $t$ and
$\alpha_{t}=1-\beta_{t}$. Eventually, this _forward process_ leads to
$\mathbf{x}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, rendering the image
$\mathbf{x}_{T}$ as white noise. Conversely, the _reverse process_ inversely
applies the aforementioned procedure through the $\theta$-parameterized
Gaussian distribution:
$p_{\theta}(\mathbf{x}_{t-1}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{t-1};\mu_{\theta}(\mathbf{x}_{t},t),\beta_{t}\mathbf{I}).$
The learning involves estimating $\mu_{\theta}$ to be able to generate a data
sample from noise in $T$ reverse process steps.

Latent Diffusion Models (LDM) Rombach et al. [2022] improved the learning and
generation process by shifting it from the image space to a latent space. The
image is encoded into the latent space using an encoder, $E$, and both the
forward and reverse diffusion processes occur in this latent space. The latent
space samples are converted back into an image samples using a decoder, $D$.
The denoising model, based on the U-Net architecture Ronneberger et al.
[2015], is composed of self-attention layers Vaswani et al. [2017] and cross-
attention layers Vaswani et al. [2017] to seamlessly integrate textual
conditions. These models are also referred to as text-to-image models as a
text prompt can be converted into tokens and used within cross attention
layers of the U-Net model.

DreamBooth-LoRA based fine-tuning Ruiz et al. [2023], Hu et al. [2022] helps
personalize the diffusion model by creating a unique prompt and “binding” it
with a specific image. To achieve this “binding”, first, we generate a prompt
with a unique identifier: “a [V] [class noun]”, where [V] denotes a unique
identifier linked to the subject and [class noun] represents a coarse class
descriptor of the subject (e.g. boy, horse, etc.). Next, we condition the
diffusion model on this prompt and fine-tune it using the LoRA Hu et al.
[2022] technique, ensuring that the prompt aligns with the provided image.
LoRA involves creating a duplicate set of the original diffusion weights,
representing them with low-rank matrices, and exclusively training these low-
rank matrices while maintaining the original network’s frozen state. The
training low-rank matrices are then merged with the original frozen weights,
preserving the architecture and keeping the inference time identical to the
original model. This approach is the same for inpainting based diffusion
models.

DDIM Inversion (DDIM-Inv) converts a clean sample $\mathbf{x}_{0}$ to its
noisy version in reverse steps:

$\small\mathbf{z}_{t+1}=\sqrt{\alpha_{t+1}}\frac{\mathbf{z}_{t}-\sqrt{1-\alpha_{t}}\epsilon_{\theta}(\mathbf{z}_{t},t,p)}{\sqrt{\alpha_{t}}}+\sqrt{1-\alpha_{t+1}}\epsilon_{\theta}(\mathbf{z}_{t},t,p),\quad
t=0,\dots,T-1.$

The difference between the forward diffusion process (FDP) and DDIM-Inv is in
the noise generation mechanism. In the FDP, noise is sampled from a Gaussian
distribution, whereas in DDIM-Inv, the noise is the output of the U-Net model.

Figure 8: Survey Preview for Object swapping videos. The users are shown 4
videos along with source video and prompt used for editing, to answer
questions about visual quality, text alignment and background consistency.

##  Appendix B User Study

We evaluate our approach with a user study, for 13 text-video pairs and 15
participants. The users were shown source video, and 3-4 methods, randomized
(with InVi included), and are expected to answer 3 questions. There are two
types of videos: (a) videos for Object swapping and (b) videos for object
insertion. In object swapping video, the source video also has the objects,
which are modified with a prompt. In Object insertion, the source video do not
have the object, and using conditioned control images, we insert a new object
in the scene. Moreover, for object swapping videos, we use existing video
editing methods are baselines: FateZero Qi et al. [2023], Tune-A-Video Wu et
al. [2023] and TokenFlow Geyer et al. [2023]. For object insertion, there are
no video inpainting pipelines using text-to-image pre-trained models. Hence we
use baselines Framewise inpainting (Frm+Inp) and TokenFlow with Controlnet and
inpainting pipeline (Tokenflow+Inp). For object swapping, we ask users the
following questions:

  * •

Which video demonstrates the highest consistency with respect to the source
video?
Choose the method which is BEST at preserving the background from the source
video.

  * •

Which video aligns most accurately with the provided text prompt?
Choose the method which BEST captures the details in the prompt (given on top
of the video).

  * •

Which video demonstrates the highest visual quality?
Choose a video which has the LEAST amount of extra artifacts (jitter, unwanted
blobs), blurriness, unrealistic lighting and flickering.

Figure 9: Survey Preview for Object Insertion videos. The users are shown 3
videos along with source video and prompt used for editing, to answer
questions about overall visual quality, text alignment and temporal
consistency.

For object insertion, we ask users the following questions:

  * •

Which video demonstrates the highest temporal consistency across new object
appearance?
Choose the video with the BEST appearance consistency overtime for the new
inserted object.

  * •

Which video aligns most accurately with the provided text prompt?
Choose the method which BEST captures the details in the prompt (given on top
of the video).

  * •

Which video demonstrates the highest visual quality?
Choose a video which has the LEAST amount of extra artifacts (jitter, unwanted
blobs), blurriness, unrealistic lighting and flickering.