# RealFill: Reference-Driven Generation for Authentic Image Completion

Luming Tang1,2 Nataniel Ruiz1 Qinghao Chu1 Yuanzhen Li1 Aleksander Hołyński1
David E. Jacobs1
Bharath Hariharan2 Yael Pritch1 Neal Wadhwa1 Kfir Aberman1 Michael Rubinstein1

1Google Research $\qquad{}^{2}$Cornell University

###### Abstract

Recent advances in generative imagery have brought forth outpainting and
inpainting models that can produce high-quality, plausible image content in
unknown regions, but the content these models hallucinate is necessarily
inauthentic, since the models lack sufficient context about the true scene. In
this work, we propose RealFill, a novel generative approach for image
completion that fills in missing regions of an image with the content that
should have been there. RealFill is a generative inpainting model that is
personalized using only a few reference images of a scene. These reference
images do not have to be aligned with the target image, and can be taken with
drastically varying viewpoints, lighting conditions, camera apertures, or
image styles. Once personalized, RealFill is able to complete a target image
with visually compelling contents that are faithful to the original scene. We
evaluate RealFill on a new image completion benchmark that covers a set of
diverse and challenging scenarios, and find that it outperforms existing
approaches by a large margin. See more results on our project page:
https://realfill.github.io.

Figure 1:  Given a few reference images that roughly capture the same scene,
and a target image with a missing region, RealFill is able to complete the
target image with image content that is faithful to the true scene. In
contrast, standard prompt-based inpainting methods hallucinate plausible but
inauthentic content due to their lack of knowledge of the original scene.

##  1 Introduction

Photographs capture frozen moments in time corresponding to ephemeral and
invaluable experiences in our lives, but can sometimes fail to do these
memories justice. In many cases, no single shot may have captured the perfect
angle, framing, timing, and composition, and unfortunately, just as the
experiences themselves cannot be revisited, these elements of the captured
images are similarly unalterable. We show one such example in Fig. 2: imagine
having taken a nearly perfect photo of your daughter dancing on stage, but her
unique and intricate crown is only barely cut out of the frame. Of course,
there are many other pictures from the performance that showcase her crown,
but they all fail to capture that precise special moment: her pose mid-dance,
her facial expression, and the perfect lighting. Given your memories of this
event and this collection of imperfect photos, you can certainly imagine the
missing parts of this perfect shot, but actually creating a complete version
of this image, e.g., to share with family and friends, is a much harder task.

In this paper, we focus on this problem, which we call Authentic Image
Completion. Given a few reference images (up to five) and one target image
that captures roughly the same scene (but in a different arrangement or
appearance), we aim to fill missing regions of the target image with high-
quality image content that is faithful to the originally captured scene. Note
that for the sake of practical benefit, we focus particularly on the more
challenging, unconstrained setting in which the target and reference images
may have very different viewpoints, environmental conditions, camera
apertures, image styles, or even moving objects.

Approaches to solve variants of this problem have been proposed using
classical geometry-based pipelines [33, 51, 49] that rely on correspondence
matching, depth estimation, and 3D transformations, followed by patch fusion
and image harmonization. These methods tend to encounter catastrophic failure
when the scene’s structure cannot be accurately estimated, e.g., when the
scene geometry is too complex or contains dynamic objects. On the other hand,
recent generative models [45, 7, 6], and in particular diffusion models [14,
35, 27], have demonstrated strong performance on the tasks of image inpainting
and outpainting [26, 1, 40]. These methods, however, struggle to recover the
genuine scene structure and fine details, since they are only guided by text
prompts, and therefore lack a mechanism for utilizing reference image content.

To this end, we present a simple yet effective reference-driven image
completion framework called RealFill. For a given scene, we first create a
personalized generative model by fine-tuning a pre-trained inpainting
diffusion model [1] on the reference and target images. This fine-tuning
process is designed such that the adapted model not only maintains a good
image prior, but also learns the contents, lighting, and style of the scene in
the input images. We then use this fine-tuned model to fill the missing
regions in the target image through a standard diffusion sampling process.
Given the stochastic nature of generative inference, we also propose
Correspondence-Based Seed Selection, to automatically select a small set of
high-quality generations by exploiting a special property of our completion
task: the fact that there should exist true correspondence between our
generated content and our reference images. Specifically, we filter out
samples that have too few keypoint correspondences with our reference images,
a filtering process that greatly limits the need for human intervention in
selecting high-quality model outputs.

As shown in Fig 1, 3, and 4, RealFill is able to very effectively inpaint or
outpaint a target image with its genuine scene content. Most importantly, our
method is able to handle large differences between reference and target
images, e.g., viewpoint, lighting, aperture, style or dynamic deformations —
differences which are very difficult for previous geometry-based approaches.
Existing benchmarks for image completion [51] mainly focus on small inpainting
tasks and minimal changes between reference and target images. In order to
quantitatively evaluate the aforementioned challenging use-case, we collect a
dataset containing 10 inpainting and 23 outpainting examples along with
corresponding ground-truth, and show that RealFill outperforms baselines by a
large margin across multiple image similarity metrics.

In summary, in our work, we propose the following contributions:

  * •

We define a new problem, named Authentic Image Completion, where, given a set
of reference images and a target image with missing regions, we seek to
complete those missing regions with content faithful to the scene observed in
the reference images. In essence, the goal is to complete the target image
with what “should have been there” rather than what “could have been there”,
as is often the case in typical generative inpainting.

  * •

We introduce RealFill, a method that aims to solve this problem by finetuning
a diffusion-based text-to-image inpainting model on reference and target
images. This model is sampled with Correspondence-Based Seed Selection to
filter ouptuts with low fidelity to the reference images. RealFill is the
first method that expands the expressive power of generative inpainting models
by conditioning the process on more than text (i.e., by adding reference
images).

  * •

We propose RealBench, a dataset for quantitative and qualitative evaluation of
authentic image completion, composed of 33 scenes spanning both inpainting and
outpainting tasks.

##  2 Related Work

Adapting Pre-trained Diffusion Models. Diffusion models [14, 35, 10] have
demonstrated superior performance in text-to-image (T2I) generation [26, 31,
27]. Recent works take advantage of this useful pre-trained image prior by
fine-tuning these models, either for added controllability, personalization,
or for specialized tasks. Personalization methods [28, 29, 8, 12] propose to
fine-tune the T2I model on a few choice images in order to achieve subject-
driven generation, allowing for arbitrary text-driven generation of a given
object or style. Other techniques instead fine-tune a T2I model to add new
conditioning signals, either for image editing [17, 4, 40] or more
controllable image generation [46, 23, 34]. This same approach has been shown
to be useful for specialized tasks [19, 25, 42, 48] such as adding camera
viewpoint conditioning, e.g., to aid in text-to-3D generation or converting a
T2I model into a generative video model. Our method shows that a pre-trained
T2I inpainting diffusion model can be adapted to perform reference-driven
image completion.

Image Completion. An enduring challenge in computer vision, image completion
aims to fill missing parts of an image with plausible content. This task is
interchangeably referred to as inpainting or outpainting depending on the
characteristics of the missing region. Traditional approaches to this problem
[3, 9, 13, 2] rely on handcrafted heuristics while more recent deep learning
based methods [16, 18, 37, 41] instead directly train end-to-end neural
networks that take original image and mask as inputs and generate the
completed image. Given the challenging nature of this problem [50], many works
[44, 7, 20, 6] propose to leverage the image prior from a pre-trained
generative model for this task. Built upon powerful T2I diffusion models,
recent diffusion-based solutions [1, 26] demonstrate strong text-driven image
completion capabilities. However, due to their sole dependence on a text
prompt (which has limited descriptive power), generated image content can
often be hard to control, resulting in tedious prompt tuning, especially when
a particular or otherwise true scene content is desired. This is one of the
main issues we aim to tackle in our work.

Reference Based Image Inpainting. Existing work for reference-based inpainting
[51, 49] or outpainting [33] usually make use of carefully tuned pipelines
containing many individual components like depth and pose estimation, image
warping, and harmonization. Each of these modules usually tackles a moderately
challenging problem itself and the resulting prediction error can, and often
does, propagate and accumulate through the pipeline. This can lead to
catastrophic failure especially in challenging cases with complex scene
geometry, changes in appearance, or scene deformation. Paint-by-Example [43]
propose to fine-tune a latent diffusion model [27] such that the generation is
conditioned on both a reference and target image. However, the image
conditioning is based on a CLIP embedding [24] of a single reference image,
and is therefore only able to capture high-level semantic information of the
reference object. In contrast, our method is the first to demonstrate multiple
reference image-driven inpainting and outpainting that is both visually
compelling and faithful to the original scene, even in cases where there are
large appearance changes between reference and target images.

##  3 Method

Figure 2:  RealFill - training and inference pipelines. The input to our
method is a target image to be filled and a few reference images of the same
scene. We first fine-tune LoRA weights of a pre-trained inpainting diffusion
model on the reference and target images (with random patches masked out).
Then, we use the adapted model to fill the desired region of the target image,
resulting in a faithful and high-quality output, e.g., the dancing girl’s
crown is recovered in the target image, despite the girl and crown exhibiting
significantly different poses and articulations when compared to any of the
reference images.

###  3.1 Reference-Driven Image Completion

Given a set of casually captured reference images (up to five), our goal is to
complete (i.e., either outpaint or inpaint) a target image of roughly the same
scene. The output image is expected to not only be plausible and
photorealistic, but to also be faithful to the reference images — recovering
content and scene detail that was present in the actual scene. In essence, we
want to achieve authentic image completion, where we generate what “should
have been there” instead of what “could have been there”. We purposefully pose
this as a broad and challenging problem with few constraints on the inputs.
For example, the images could be taken from very different viewpoints with
unknown camera poses. They could also have different lighting conditions or
styles, and the scene could potentially be non-static and have significantly
varying layout across images.

In this section, we first provide background knowledge on diffusion models and
subject-driven generation (Sec. 3.2). Then, we formally define the problem of
authentic image completion (Sec. 3.3). Finally, we present RealFill, our
method to perform reference-driven image completion with a pre-trained
diffusion image prior (Sec. 3.4).

###  3.2 Preliminaries

Diffusion models are generative models that aim to transform a Gaussian
distribution into an arbitrary target data distribution. During training,
different magnitudes of Gaussian noise are added to a clean data point $x_{0}$
to obtain noisy $x_{t}$:

$x_{t}=\sqrt{\alpha_{t}}x_{0}+(\sqrt{1-\alpha_{t}})\epsilon$ (1)

where the noise $\epsilon\sim\mathcal{N}(0,\mathbf{I})$, and
$\\{\alpha_{t}\\}_{t=1}^{T}$ define a fixed noise schedule with larger $t$
corresponding to more noise. Then, a neural network $\epsilon_{\theta}$ is
trained to predict the noise using the following loss function:

$\mathcal{L}=\mathbb{E}_{x,t,\epsilon}\left\|\mathbf{\epsilon}_{\theta}(x_{t},t,c)-\mathbf{\epsilon}\right\|^{2}_{2}$
(2)

where the generation is conditioned on some signal $c$, e.g., a language
prompt for a text-to-image model, or a masked image for an inpainting model.
During inference, starting from $x_{T}\sim\mathcal{N}(0,\mathbf{I})$,
$\epsilon_{\theta}$ is used to iteratively remove noise from $x_{t}$ to get a
less noisy $x_{t-1}$, eventually leading to a sample $x_{0}$ from the target
data distribution.

DreamBooth [28] enables T2I diffusion models to generate images of a specific
subject with semantic modifications. The core idea is to fine-tune the model
$\epsilon_{\theta}$ on a few images of the subject using the loss in Eq. 2.
Instead of fine-tuning all of the weights of the network, it is possible to
combine DreamBooth with Low Rank Adaptations (LoRA) [30, 15], for a more
memory-efficient alternative, by injecting learnable residual modules $\Delta
W$ to each network weight matrix $W$. $\Delta W$ is a composition of low-rank
matrices, i.e., $W+\Delta W=W+AB$ where $W\in\mathbb{R}^{n\times n}$,
$A\in\mathbb{R}^{n\times r}$, $B\in\mathbb{R}^{r\times n}$, $r\ll n$, and only
the added $\Delta W$ is being updated during training while model’s original
parameters $W$ stay frozen.

###  3.3 Problem Setup

Formally speaking, the model is given $n$ ($n\leq 5$) reference images
$\mathcal{X}_{ref}\coloneqq\\{I_{ref}^{k}\\}_{k=1}^{n}$, a target image
$I_{tgt}\in\mathbb{R}^{H\times W\times 3}$ and its associated binary mask
$M_{tgt}\in\\{0,1\\}^{H\times W}$, in which 1 denotes the region to fill and 0
denotes the existing area in $I_{tgt}$. The model is expected to generate a
harmonized image $I_{out}\in\mathbb{R}^{H\times W\times 3}$ whose pixels are
to remain as similar as possible to $I_{tgt}$ where the mask equals 0, while
staying faithful to the corresponding contents in $\mathcal{X}_{ref}$ where
the mask equals 1. We assume there is sufficient overlap between the contents
of $\mathcal{X}_{ref}$ and $I_{tgt}$ such that a human is able to imagine a
plausible $I_{out}$.

###  3.4 RealFill

This task is challenging for both geometry-based [51, 49] and reconstruction-
based approaches [22] because there are barely any geometric constraints
between $\mathcal{X}_{ref}$ and $I_{tgt}$, there are only a few images
available as inputs, and the reference images may have different styles,
lighting conditions, and subject poses from the target. One alternative is to
use a controllable inpainting or outpainting methods, however, these methods
are either prompt-based [27] or single-image object-driven [43], which makes
them hard to use for recovering complex scene-level structure and details.

To this end, we propose to first fine-tune a pre-trained generative model by
injecting knowledge of the scene (from a set of reference images), such that
the model is aware of the contents of the scene when generating $I_{out}$,
conditioned on $I_{tgt}$ and $M_{tgt}$.

Training. Starting from a state-of-the-art T2I diffusion inpainting model
[27], we inject LoRA weights and fine-tune it on both $\mathcal{X}_{ref}$ and
$I_{tgt}$ with randomly generated binary masks $m\in\\{0,1\\}^{H\times W}$.
The loss function is

$\mathcal{L}=\mathbb{E}_{x,t,\epsilon,m}\left\|\mathbf{\epsilon}_{\theta}(x_{t},t,p,m,(1-m)\odot
x)-\mathbf{\epsilon}\right\|^{2}_{2}$ (3)

where $x\in\mathcal{X}_{ref}\cup\\{I_{tgt}\\}$, $p$ is a fixed language
prompt, $\odot$ denotes the element-wise product and therefore $(1-m)\odot x$
is the masked clean image. For $I_{tgt}$, the loss is only calculated on the
existing region, i.e., where $M_{tgt}$’s entry equals 0. Specifically, we use
the open-sourced Stable Diffusion v2 inpainting model [1] and inject LoRA
layers into its text encoder and U-Net for fine-tuning. Following [28], we fix
$p$ to be a sentence containing a rare token, i.e., “a photo of [V]”. For each
training example, similar to [37], we generate multiple random rectangles and
take either their union or the complement of the union to get the final random
mask $m$. Our fine-tuning pipeline is illustrated in Fig. 2.

Inference. After training, we use the DDPM [14] sampler to generate an image
$I_{gen}$, conditioning the model on $p$, $I_{tgt}$ and $M_{tgt}$. However,
similar to the observation in [52], we notice that the existing region in
$I_{tgt}$ is distorted in $I_{gen}$. To resolve this, we first blur the mask
$M_{tgt}$, then use it to alpha composite $I_{gen}$ and $I_{tgt}$, leading to
the final $I_{out}$ with full recovery on the existing area and a smooth
transition at the boundary of the generated region.

Correspondence-Based Seed Selection. The diffusion inference process is
stochastic, i.e., the same input conditioning images may produce any number of
generated images depending on the input seed to the sampling process. This
stochasticity often results in variance in the quality of generated results,
often requiring human intervention to select high-quality samples. While there
exists work in identifying good samples from a collection of generated outputs
[32], this remains an open problem. Nevertheless, our proposed problem of
authentic image completion is a special case of this more general problem
statement. In particular, the reference images provide a grounding signal for
the true content of the scene, and can be used to help identify high-quality
outputs. Specifically, we find that the number of image feature
correspondences between $I_{out}$ and $\mathcal{X}_{ref}$ can be used as a
metric to roughly quantify whether the result is faithful to the reference
images. We propose Correspondence-Based Seed Selection, a process that
consists of generating a batch of outputs, i.e., $\\{I_{out}\\}$, extracting a
set of correspondences (using LoFTR [36], for example) between
$\mathcal{X}_{ref}$ and the filled region of each $I_{out}$, (i.e., where
$M_{tgt}$’s entry equals 1), and finally ranking the generated results
$\\{I_{out}\\}$ by the number of matched keypoints. This allows us to
automatically filter generations to a small set of high-quality results.
Compared to traditional seed selection approaches in other domains, our
proposed method greatly alleviates the need for human intervention in
selecting best samples.

##  4 Experiments

###  4.1 Qualitative Results

Figure 3:  Reference-based outpainting with RealFill. Given the reference
images on the left, RealFill is able to outpaint the corresponding target
images on the right. The region inside the white box is provided to the
network as known pixels, and the regions outside the white box are all
generated. Results show that RealFill produces high-quality images that are
faithful to the references, even when there are dramatic differences between
references and targets including changes in viewpoint, aperture, lighting,
image style, and object motion.

Figure 4:  Reference-based inpainting with RealFill. Given the reference
images on the left, RealFill is not only able to remove undesired objects in
the target image and reveal the occluded contents faithfully (left column),
but it is also able to insert objects into the scene despite significant
viewpoint changes between reference and target images (right column). In the
bottom left example, the aperture between reference and target images is also
different, and RealFill not only recovers the buildings behind the mug, but
also maintains the appropriate amount of blur seen in the target image.

In Fig. 3 and 4, we show that RealFill is able to convincingly outpaint and
inpaint image content that is faithful to the reference images. Notably, it is
able to handle dramatic differences in camera pose, lighting, defocus blur,
image style and even subject pose. This is because RealFill has both a good
image prior (from the pre-trained diffusion model) and knowledge of the scene
(from fine-tuning on the input images). Thus, it is able to inherit knowledge
about the contents of the scene, but generate content that fits seamlessly
into the target image.

###  4.2 Comparisons

Evaluation Dataset. Existing benchmarks for reference-driven image completion
[51] primarily focus on inpainting small regions, and assume at most very
minor changes between the reference and target images. To better evaluate our
target use-case, we create our own dataset, _RealBench_. RealBench consists of
33 scenes (23 outpainting and 10 inpainting), where each scene has a set of
reference images $\mathcal{X}_{ref}$, a target image $I_{tgt}$ to fill, a
binary mask $M_{tgt}$ indicating the missing region and the ground-truth
result $I_{gt}$. The number of reference images in each scene varies from 1 to
5. The dataset contains diverse, challenging scenarios with significant
variations between the reference and target images, such as changes in
viewpoint, defocus blur, lighting, style and subject pose.

Evaluation Metrics. We use multiple metrics to evaluate the quality and
fidelity of our model outputs. We compare the generated images with the
ground-truth target image at multiple levels of image similarity, including
PSNR, SSIM, and LPIPS [47] for low-level, DreamSim [11] for mid-level, and
DINO [5] and CLIP [24] for high-level.

For low-level metrics, we only calculate a loss on the filled-in region, i.e.,
where $M_{tgt}$ is 1. For high-level image similarity, we use the cosine
distance between the full image embeddings from CLIP and DINO. For mid-level
similarity, we use the full image embedding using DreamSim [11] which is
designed to emphasize differences in image layouts, object poses, and semantic
contents.

Baseline Approaches. We compare to two baselines: the exemplar-based image
inpainting method Paint-by-Example [43] and the popular prompt-based image
filling approach Stable Diffusion Inpainting [1]. Since Paint-by-Example only
uses one reference image during generation, we randomly pick a reference image
for each run of this baseline. Choosing an appropriate prompt for Stable
Diffusion Inpainting is a necessary component of getting a high quality
result. So, for a fair comparison, instead of using a generic prompt like “a
beautiful photo”, we manually write a long prompt that describes the scene in
detail. For example, the prompt for the first row of Fig. 5 is “two men
sitting together with a child in the middle, the man on the left is playing
guitar, the man on right is wearing a birthday hat with some stickers on it.
There is a blue decorator hanging on the wall”.

Implementation Details of RealFill. For each scene, we fine-tune the
inpainting diffusion model for 2,000 iterations with a batch size of 16 on a
single NVIDIA A100 GPU with LoRA rank 8. With a probability of 0.1, we
randomly dropout prompt $p$, mask $m$ and LoRA layers independently during
training. The learning rate is set to 2e-4 for the U-Net and 4e-5 for the text
encoder. Note that these hyper-parameters could be further tuned for each
scene to get better performance, e.g., some scenes converge more quickly may
overfit if trained for too long. However, for the sake of fair comparison, we
use a constant set of hyper-parameters for all results shown in the paper.

| Method | low-level | mid-level | high-level
---|---|---|---|---
PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | DreamSim $\downarrow$ | DINO $\uparrow$ | CLIP $\uparrow$
prompt-based | Stable Diffusion Inpaint [1] | 10.63 | 0.282 | 0.605 | 0.213 | 0.831 | 0.874
reference-based | Paint-by-Example [43] | 10.13 | 0.244 | 0.642 | 0.237 | 0.797 | 0.859
RealFill (ours) | 14.78 | 0.424 | 0.431 | 0.077 | 0.948 | 0.962

Table 1:  On RealBench, our evaluation dataset of 33 diverse challenging
scenes, including 23 outpainting and 10 inpainting tasks, RealFill outperforms
both prompt-based and reference-based baselines by a large margin for all
types of image similarity metrics.

Figure 5:  A comparison of RealFill and baseline methods. Transparent white
masks are overlayed on the unaltered known regions of the target images.
Paint-by-Example loses fidelity with the reference images because it relies on
CLIP embeddings, which only capture high-level semantic information. While
Stable Diffusion Inpainting produces plausible results, they are inconsistent
with the reference images because prompts have limited expressiveness. In
contrast, RealFill generates high-quality results that have high fidelity with
respect to the reference images.

Quantitative Comparison. We quantitatively compare our method with the
baseline methods. For each method, we report average metrics across all target
images $\\{I_{out}\\}$, where each image’s metric is itself computed from an
average of 64 stochastically generated samples.. In Tab. 1, we report these
aggregate metrics and find that RealFill outperforms all baselines by a large
margin across all levels of similarity.

Qualitative Comparison. In Fig. 5, we present a visual comparison between
RealFill and the baselines. We also show the ground-truth and input images for
each example. In order to better highlight the regions which are being
generated, we overlay a semi-transparent white mask on the ground truth and
output images, covering the known regions of the target image. RealFill not
only generates high-quality images, but also more faithfully reproduces the
scene than the baseline methods. Paint-by-Example relies on the CLIP embedding
of the reference images as the condition. This poses a challenge when dealing
with complex scenes or attempting to restore object details, since CLIP
embeddings only capture high-level semantic information. The generated results
from Stable Diffusion Inpainting are plausible on their own. However, because
natural language is limited in conveying complex visual information, they
often exhibit substantial deviations from the original scenes depicted in the
reference images.

Filtering Rate | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | DreamSim$\downarrow$ | DINO$\uparrow$ | CLIP$\uparrow$
---|---|---|---|---|---|---
0% | 14.78 | 0.424 | 0.431 | 0.077 | 0.948 | 0.962
25% | 15.01 | 0.427 | 0.421 | 0.066 | 0.955 | 0.967
50% | 15.05 | 0.427 | 0.418 | 0.063 | 0.958 | 0.969
75% | 15.10 | 0.427 | 0.417 | 0.060 | 0.961 | 0.970

Table 2:  Correspondence-Based Seed Selection results in higher quality, as
shown by the fact that filtering out samples with fewer matches results in
higher aggregate quantitative scores.

Figure 6:  Given the reference images on the left, we show multiple RealFill
outputs on the right along with the number of matched key points noted below
each image. We can see that fewer matches correlate with lower-quality outputs
that are more divergent from the ground-truth.

Correspondence-Based Seed Selection. We evaluate the effect of our proposed
correspondence-based seed selection described in Sec. 3.4. To measure the
correlation between our seed selection mechanism and high-quality results, we
rank RealFill’s outputs $\\{I_{out}\\}$ according to the number of matched
keypoints, and then filter out a certain percent of the lowest-ranked samples.
We then average the evaluation metrics only across the remaining samples. We
find that higher filtering rates like 75% greatly improve the quantitative
metrics, when compared to unfiltered results (Tab. 2). In Fig 6, we show
multiple RealFill outputs with the corresponding number of matched keypoints.
These demonstrate a clear trend, where fewer matches usually indicate lower-
quality results.

##  5 Discussion

###  5.1 Would other baselines work?

Figure 7:  Commercial image stitching software fails to produce any outputs
when there are dramatic differences between reference and target images, as in
the examples shown above with large lighting changes. In contrast, RealFill
produces faithful and high-quality results, i.e., the rooftop tank and the
balloon are both recovered even when the target images are captured at vastly
different times of day.

Image Stitching. One straight-forward approach is to utilize the
correspondence between reference and target images and stitch them together.
However, we find that this does not yield acceptable results most of the time,
even using commercial image stitching software, particularly when there are
dramatic viewpoint changes, lighting changes, or moving objects. Taking the
two scenes in Fig. 7 as example, multiple commercial software solutions
produce no output, asserting that the reference and target images do not have
sufficient correspondences. On the contrary, RealFill recovers these scenes
both faithfully and realistically.

Figure 8:  Vanilla Dreambooth, i.e., fine-tuning a standard Stable Diffusion
model on the reference images and using it to fill missing regions, leads to
drastically worse results compared to RealFill. We show different samples
using varying levels of the strength hyper-parameter.

Vanilla DreamBooth. Instead of adapting an inpainting model, another
alternative is to fine-tune a standard Stable Diffusion model on the reference
images, i.e., vanilla DreamBooth, then use the fine-tuned T2I model to inpaint
the target image [20], as implemented in the popular Diffusers library
[39]111Diffusers’ Stable Diffusion inpainting pipeline code.. However, because
this model is never trained with a masked prediction objective, it performs
much worse compared to RealFill, as shown in Fig. 8.

###  5.2 What makes RealFill work?

In order to explore why our proposed method leads to strong results,
especially on complex scenes, we make the following two hypotheses:

Figure 9:  RealFill is able to generate multiple scene variants when
conditioned on a blank image as input, e.g., people are added or removed in
the first and second rows. This suggests that the fine-tuned model can relate
the elements inside the scene in a compositional manner.

RealFill relates multiple elements in a scene. If we make the conditioning
image a blank canvas during inference, i.e., all entries of $M_{tgt}$ equal 1,
we can see in Fig. 9 that the fine-tuned model is able to generate multiple
scene variants with different structures, e.g., removing the foreground or
background object, or manipulating the object layouts. This suggest that
RealFill is able to relate the elements inside the scene in a compositional
way.

Figure 10:  When the reference and target images do not depict the same scene,
the fine-tuned model is still able to fuse the reference contents into the
target image in a semantically-reasonable way, suggesting that it captures
both real or invented correspondences between input images.

RealFill captures correspondences among input images. Even if the reference
and target images do not depict the same scene, the fine-tuned model is still
able to fuse the corresponding contents of the reference images into the
target area seamlessly, as shown in Fig. 10. This suggests that RealFill is
able to capture and utilize real or invented correspondences between reference
and target images to do generation. Previous works [38, 21] also found similar
emergent correspondence inside pre-trained Stable Diffusion models.

###  5.3 Limitations

Figure 11: (Top) RealFill fails to recover the precise 3D scene structure,
e.g., the output husky plush has different pose compared to the reference;
(Bottom) RealFill fails to handle cases that are also challenging for the base
T2I model, e.g., text.

Because RealFill needs to go through a gradient-based fine-tuning process on
input images, it is relatively slow and far from real time. Empirically, we
also find that, when the viewpoint change between reference and target images
is dramatic, RealFill fails to recover the 3D scene faithfully, especially
when there’s only a single reference image. For example, as seen in the top
row of Fig. 11, the reference image is captured from a side view while the
target is from a center view. Although the RealFill output looks plausible at
first glance, the pose of the husky is different from the reference, e.g., the
left paw should be on the gap between the cushions. Lastly, because RealFill
mainly relies on the image prior inherited from the base pre-trained model, it
also fails to handle cases where that are challenging for the base model. For
instance, Stable Diffusion is known to be less effective when it comes to
generating fine image details, such as text, human faces, or body parts. As
shown in the bottom row of Fig. 11 where the store sign is wrongly spelled,
this is also true for RealFill.

##  6 Societal Impact

This research aims to create a tool that can help users express their
creativity and improve the quality of their personal photographs through image
generation. However, advanced image generation methods can have complex
impacts on society. Our proposed method inherits some of the concerns that are
associated with this class of technology, such as the potential to alter
sensitive personal characteristics. The open source pre-trained model that we
use in our work, Stable Diffusion, exhibits some of these concerns. However,
we have not found any evidence that our method is more likely to produce
biased or harmful content than previous work. Despite these findings, it is
important to continue investigating the potential risks of image generation
technology. Future research should focus on developing methods to mitigate
bias and harmful content, and to ensure that image generation tools are used
in a responsible manner.

##  7 Conclusion

In this work, we introduce the problem of Authentic Image Completion, where
given a few reference images, we intend to complete some missing regions of a
target image with the content that “should have been there” — rather that
“what _could_ have been there”. To tackle this problem, we proposed a simple
yet effective approach called RealFill, which first fine-tunes a T2I
inpainting diffusion model on the reference and target images, and then uses
the adapted model to fill the missing regions. We show that RealFill produces
high-quality image completions that are faithful to the content in the
reference images, even when there are large differences between reference and
target images such as viewpoint, aperture, lighting, image style and object
position, pose and articulation.

Acknowledgements. We would like to thank Rundi Wu, Qianqian Wang, Viraj Shah,
Ethan Weber, Zhengqi Li, Kyle Genova, Boyang Deng, Maya Goldenberg, Noah
Snavely, Ben Poole, Ben Mildenhall, Alex Rav-Acha, Pratul Srinivasan, Dor
Verbin and Jon Barron for their valuable discussion and feedbacks, and thank
Zeya Peng, Rundi Wu, Shan Nan for their contribution to the evaluation
dataset. A special thanks to Jason Baldridge, Kihyuk Sohn, Kathy Meier-
Hellstern, and Nicole Brichtova for their feedback and support for the
project.

## References

  * [1] Stability AI.  Stable-diffusion-2-inpainting.  https://huggingface.co/stabilityai/stable-diffusion-2-inpainting.
  * [2] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman.  Patchmatch: A randomized correspondence algorithm for structural image editing.  ACM Trans. Graph., 28(3):24, 2009.
  * [3] Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester.  Image inpainting.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 417–424, 2000.
  * [4] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023.
  * [5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650–9660, 2021.
  * [6] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al.  Muse: Text-to-image generation via masked generative transformers.  arXiv preprint arXiv:2301.00704, 2023.
  * [7] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman.  Maskgit: Masked generative image transformer.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022.
  * [8] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  arXiv preprint arXiv:2304.00186, 2023.
  * [9] Antonio Criminisi, Patrick Perez, and Kentaro Toyama.  Object removal by exemplar-based inpainting.  In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., volume 2, pages II–II. IEEE, 2003.
  * [10] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in neural information processing systems, 34:8780–8794, 2021.
  * [11] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola.  Dreamsim: Learning new dimensions of human visual similarity using synthetic data.  arXiv preprint arXiv:2306.09344, 2023.
  * [12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [13] James Hays and Alexei A Efros.  Scene completion using millions of photographs.  ACM Transactions on Graphics (ToG), 26(3):4–es, 2007.
  * [14] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in neural information processing systems, 33:6840–6851, 2020.
  * [15] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * [16] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.  Globally and locally consistent image completion.  ACM Transactions on Graphics (ToG), 36(4):1–14, 2017.
  * [17] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007–6017, 2023.
  * [18] Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro.  Image inpainting for irregular holes using partial convolutions.  In Proceedings of the European conference on computer vision (ECCV), pages 85–100, 2018.
  * [19] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.  Zero-1-to-3: Zero-shot one image to 3d object, 2023.
  * [20] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  Repaint: Inpainting using denoising diffusion probabilistic models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11461–11471, 2022.
  * [21] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell.  Diffusion hyperfeatures: Searching through time and space for semantic correspondence.  arXiv preprint arXiv:2305.14334, 2023.
  * [22] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  In ECCV, 2020.
  * [23] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  arXiv preprint arXiv:2302.08453, 2023.
  * [24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [25] Amit Raj, Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani.  Dreambooth3d: Subject-driven text-to-3d generation.  arXiv preprint arXiv:2303.13508, 2023.
  * [26] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [28] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500–22510, 2023.
  * [29] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  arXiv preprint arXiv:2307.06949, 2023.
  * [30] Simo Ryu.  Low-rank adaptation for fast text-to-image diffusion fine-tuning.  https://github.com/cloneofsimo/lora.
  * [31] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [32] Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik.  It is all about where you start: Text-to-image generation with seed selection.  arXiv preprint arXiv:2304.14530, 2023.
  * [33] Qi Shan, Brian Curless, Yasutaka Furukawa, Carlos Hernandez, and Steven M Seitz.  Photo uncrop.  In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pages 16–31. Springer, 2014.
  * [34] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al.  Styledrop: Text-to-image generation in any style.  arXiv preprint arXiv:2306.00983, 2023.
  * [35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [36] Jiaming Sun, Zehong Shen, Yuang Wang, Hujun Bao, and Xiaowei Zhou.  LoFTR: Detector-free local feature matching with transformers.  CVPR, 2021.
  * [37] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2149–2159, 2022.
  * [38] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan.  Emergent correspondence from image diffusion.  arXiv preprint arXiv:2306.03881, 2023.
  * [39] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * [40] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al.  Imagen editor and editbench: Advancing and evaluating text-guided image inpainting.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18359–18369, 2023.
  * [41] Chenfei Wu, Jian Liang, Xiaowei Hu, Zhe Gan, Jianfeng Wang, Lijuan Wang, Zicheng Liu, Yuejian Fang, and Nan Duan.  Nuwa-infinity: Autoregressive over autoregressive generation for infinite visual synthesis.  arXiv preprint arXiv:2207.09814, 2022.
  * [42] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou.  Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation.  arXiv preprint arXiv:2212.11565, 2022.
  * [43] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen.  Paint by example: Exemplar-based image editing with diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18381–18391, 2023.
  * [44] Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do.  Semantic image inpainting with deep generative models.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5485–5493, 2017.
  * [45] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Generative image inpainting with contextual attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5505–5514, 2018.
  * [46] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models, 2023.
  * [47] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In CVPR, 2018.
  * [48] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu.  Unleashing text-to-image diffusion models for visual perception.  ICCV, 2023.
  * [49] Yunhan Zhao, Connelly Barnes, Yuqian Zhou, Eli Shechtman, Sohrab Amirghodsi, and Charless Fowlkes.  Geofill: Reference-based image inpainting with better geometric understanding.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1776–1786, 2023.
  * [50] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai.  Pluralistic image completion.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1438–1447, 2019.
  * [51] Yuqian Zhou, Connelly Barnes, Eli Shechtman, and Sohrab Amirghodsi.  Transfill: Reference-guided image inpainting by merging multiple color and spatial transformations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2266–2276, 2021.
  * [52] Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, and Gang Hua.  Designing a better asymmetric vqgan for stablediffusion.  arXiv preprint arXiv:2306.04632, 2023.