# Lazy Diffusion Transformer for Interactive Image Editing

Yotam Nitzan1,2 Zongze Wu1 Richard Zhang1 Eli Shechtman1 Daniel Cohen-Or2
Taesung Park1 Michaël Gharbi1
1 Adobe Research 2 Tel-Aviv University
https://lazydiffusion.github.io

###### Abstract

We introduce a novel diffusion transformer, _LazyDiffusion_ , that generates
partial image updates efficiently. Our approach targets interactive image
editing applications in which, starting from a blank canvas or an image, a
user specifies a sequence of localized image modifications using binary masks
and text prompts. Our generator operates in two phases. First, a context
encoder processes the current canvas and user mask to produce a compact global
context tailored to the region to generate. Second, conditioned on this
context, a diffusion-based transformer decoder synthesizes the masked pixels
in a “lazy” fashion, i.e., it _only_ generates the masked region. This
contrasts with previous works that either regenerate the full canvas, wasting
time and computation, or confine processing to a tight rectangular crop around
the mask, ignoring the global image context altogether. Our decoder’s runtime
scales with the mask size, which is typically small, while our encoder
introduces negligible overhead. We demonstrate that our approach is
competitive with state-of-the-art inpainting methods in terms of quality and
fidelity while providing a $10\times$ speedup for typical user interactions,
where the editing mask represents 10% of the image.

Figure 1:  Incremental image generation at $1024\times 1024$ using
_LazyDiffusion_ with 20 diffusion steps. The model generates content according
to a text prompt in an area specified by a mask. Each update generates _only_
the masked pixels, with a runtime that depends chiefly on the size of the
mask, rather than that of the image.

##  1 Introduction

Diffusion models have had remarkable successes in generating high-quality and
diverse images. They are the powerful engine behind exciting local image
editing applications based on inpainting, where a user provides a mask and a
text prompt describing a region to modify and the content to generate,
respectively [42, 55, 31]. While current approaches yield impressive results,
they are also slow and wasteful. Invisible to the end user, the inpainting
pipeline generates an entire image and then selectively utilizes only the few
pixels located within the mask, discarding all others. Although this approach
is generally common in inpainting pipelines [60, 59], its inefficiency is
particularly pronounced with diffusion models, due to their iterative sampling
procedure, precluding their usage in interactive workflows. Practitioners [50,
54] save time and computation by cropping a small rectangular region around
the mask, possibly downsampling for processing with the diffusion, then
upsampling and blending the result to fill the hole. In doing so they
compromise image quality and sacrifice the global image context, which often
leads to spatially inconsistent outputs (Compare Figs. 2(a) and 2(b)).

We propose a new generative model architecture, which we call _LazyDiffusion_.
Our approach, illustrated in Fig. 1, generates _partial_ image updates,
strictly limited to the masked region, and does so efficiently, with a cost
commensurate to the mask size. Yet, its output respects the global context
given by the observed canvas (Fig. 2(c)). To achieve this, our key idea is to
decouple the generative process into two distinct steps. First, an encoder
processes the visible canvas and mask, summarizing them into a global context
code. This encoder processes the entire canvas, but it only runs once per
mask. Second, conditioned on the global context and the user’s text prompt, a
diffusion decoder generates the next partial canvas update. This model runs
many times during the diffusion process, but unlike previous works, it only
operates on the masked region. Since, in practice, most updates cover small
areas (10–20% of the image), this yields significant computation savings, thus
making the editing experience more interactive.

Figure 2:  Comparing inpainting approaches. (a) Most works [42, 37] generate
the entire image, utilizing the full image context and fill the hole by
discarding the non-masked regions. While the outcome aligns well with the
image, the process is time-consuming. (b) generating only a lower resolution
crop around the mask is more efficient and still seamlessly blends with nearby
pixels [50, 54]. However, the inpainted content is semantically inconsistent
with the overall image context. (c) our approach ensures both global
consistency and efficient execution.

Our encoder and diffusion decoder operate in a latent space [42], for
efficiency. Both use the transformer architecture [53, 13, 35]. The
transformer architecture is particularly appealing because splitting the image
into small enough patches (tokens) enables generating arbitrarily-shaped
regions with minimal waste. The encoder processes the entire image and mask
and produces a mask-dependent context. We keep only the context tokens
corresponding to the location of the masked patches. This ensures the
downstream computation only scales with the size of the masked region, and
encourages the compressed context to represent the relationship of the masked
region to the rest of the image. At each denoising step, the decoder only
processes tokens corresponding to masked patches. While the decoder
_generates_ only the masked region, it _“sees”_ the entire image, through the
compressed context, ensuring strong coherence. The conditioning on context is
efficient and adds negligible computational overhead. In contrast, previous
methods [1, 55, 42] achieve spatial consistency by uniformly processing all
image regions, masked or not. Figure 3 illustrates the conceptual difference
between our approach and a baseline diffusion transformer.

Our approach reduces computational cost significantly for small masks, typical
in interactive editing. We achieve a speedup up to $\times 10$ over methods
processing the entire image, for mask covering 10% of the image. Additionally,
our model produces results of comparable quality, indicating that the
compressed context is rich and expressive. In an interactive image generation
context, our method amortizes the overall synthesis cost over multiple user
interactions, improving interaction latency. It also amortizes the encoder
cost when generating multiple updates for a given mask, using different input
noise or text prompt (Fig. 1, rightmost panel).

Figure 3:  Our diffusion transformer decoder (bottom) reduces synthesis
computation using two strategies. First, we compress the image context using a
separate encoder (not shown) outside the diffusion loop. Second, we only
generate tokens corresponding to the masked region to generate. In contrast,
typical diffusion transformers (top) [35, 7] maintain tokens for the entire
image throughout the diffusion process, to preserve global context. When
performing inpainting, such model generates a full-size image, most of which
is discarded in order to in-fill the hole region only. Existing convolutional
diffusion models for inpainting [42] suffer from the same drawbacks.

##  2 Related Work

Speeding up diffusion models. Diffusion models [21, 48, 46] are a significant
breakthrough in generative modeling [11, 42, 43, 40, 2] and editing [29, 1],
producing images with unparalleled quality and diversity. But they remain
costly to evaluate, due to the iterative nature of their sampling process.
Numerous methods have been developed to improve their inference time, such as
better samplers and dedicated ODE solvers [47, 22, 26, 27], distillation
techniques [24, 49, 44, 28]. The gap between recent one-step diffusion models
[58, 37, 30] and their expensive multi-step counterparts is closing. Our
approach also seeks to speed up the image synthesis process for diffusion-
based models, but our contribution is largely orthogonal and can be combined
with these optimizations: we reduce the amount of image data to process,
rather than the atomic diffusion iteration.

Transformer-based generative models. Early transformers for image generation
generate image autoregressively [41, 14, 8] in scanline order. CogView2 [12]
proposes a hierarchical transformer to improve generation speed and shows
application to text-guided image inpainting with rectangular masks. Later non-
autoregressive models like MaskGIT [6] generate images gradually, a few tokens
at a time, but they do so iteratively, generating all tokens at every
iteration and discarding the unmasked ones, which is inefficient. They focus
on sequential generation to improve image quality.

Our transformer-based model design is inspired by Masked Autoencoders (MAEs)
[17], but we reverse their asymmetric design. Our encoder processes _all_ the
tokens to produce context at the masked locations, and our decoder operates on
the masked tokens. Our decoder is a powerful diffusion transformer, recently
proposed as an alternative to the popular UNet design [52, 35]. Most relevant
to this work, DiT [35] was proposed for class-conditioned image generation and
was improved in PixArt-$\alpha$ [7] to support text-conditioning. Our
diffusion decoder is an adaptation of PixArt-$\alpha$ that additionally
conditions on the global context produced by the encoder. Masked diffusion
transformers were previously explored for representation learning [16, 56] or
for minimizing training cost [61]. Our focus is on speeding inference to
improve interactivity. Recent trends indicate that the transformer
architecture becoming central to state-of-the-start image [15] and video
generators [3], for which our method would enable faster inference and
interactive applications.

Text-guided diffusion-based image editing. Text-to-image diffusion models have
become the de-facto foundation for generative image editing methods. With user
edits typically spatially localized, significant effort has gone into
developing techniques that allow precise modifications [18, 4, 34] by
selectively manipulating internal representations, _e.g_. attention maps,
during the denoising process to affect only certain local regions without
undesirable side-effects. Another line of work adopts the formulation of
inpainting, where a mask is provided to localize the edit. Blended diffusion
[1] and DiffEdit [9] use pre-trained generation models and spatially blend
noised versions of the input into the gradual denoising process to enforce the
preservation of unmasked regions. This indirect approach often result in
artifacts, leading more recent approaches to fine-tune text-to-image models
specifically for inpainting. Starting from an image generation architecture,
GLIDE [31] and Stable Diffusion Inpaint [42] add mechanisms to additionally
condition on the mask and masked image and fine-tune the models to predict the
masked pixels. Recent advancements in this domain involve training inpainting
models with object-level masks [55] rather than random ones and possibly also
object-level text captions [57], mirroring real-world usage more closely.
These works retrofit image generation architectures for local editing, but
these models produce the full image, including regions that should not be
changed. This is inefficient in time and computing resources. Our architecture
efficiently performs local edits by generating only the masked region.

##  3 Method

Our goal is to develop an efficient diffusion generator for text-guided image
editing, whose generation cost scales with the size of the region to generate,
and which can incorporate the context of the entire image for a fixed, small
fraction of its total cost. Starting from an image $I\in\mathbb{R}^{h\times
w\times 3}$, the user specifies the region to be edited with a binary mask
$M\in\\{0,1\\}^{h\times w}$ and text prompt $\mathbf{c}$, indicating where and
what content to generate. A mask value $1$ specifies a hole to inpaint, and
$0$ for context pixels to not touch. Unless stated otherwise, we use images of
$h=w=1024$ resolution.

Following standard practice, we operate in latent space [42], a compressed
version of the RGB domain (§ 3.1). Observing that the iterative diffusion
process is the computational bottleneck in state-of-the-art generators, our
generator has a novel asymmetric encoder-decoder transformer architecture, as
illustrated in Fig. 4. The encoder (§ 3.2) compresses and summarizes the whole
image context and is only run once. The decoder (§ 3.3) is a transformer-based
diffusion denoiser that is iteratively run, but only on the masked area. As
such, computation cost and latency are proportional to the number of pixels to
synthesize, rather than the entire canvas [57, 55, 1]. This significantly
reduces computation since, for most edits, the masks are small.

Figure 4:  Overview. To generate an incremental image update, our algorithm
takes as input a user mask and a text prompt. (top) We start by transforming
the visible pixels and binary mask into patches, and pass them to a vision
transformer (ViT) encoder. We then drop all tokens, except those corresponding
to the hole region; this is our global context. (bottom) To generate the
missing pixels, we initialize a set of noise patches corresponding to the
masked region and pass them through a diffusion transformer model for several
denoising iterations, until we obtain denoised patches. Unlike previous works
[35, 7], which process the entire image, our diffusion transformer only
processes the patches required to cover the missing region. We train our
encoder and diffusion decoder jointly using a diffusion denoising objective on
the missing patches. The generated patches are then blended back into the
missing region to produce the final output. Our model operates in a pretrained
latent image space [42], but we illustrate our pipeline with RGB images for
simplicity.

###  3.1 Latent space processing

Following previous works of Latent Diffusion Models (LDM) [42], our model
operates in an intermediate latent space of $8\times$ lower resolution with
$c=4$ channels, which reduces computation without significantly impacting
visual quality. We use the pretrained latent VAE of Stable Diffusion [42],
denoting the encoder and decoder $\mathcal{E}$ and $\mathcal{D}$,
respectively. We encode the masked image as our latent input [55]:

$\begin{split}Z=\mathcal{E}\left(I\odot(1-M)\right)\hskip
5.69054pt\in\mathbb{R}^{\frac{h}{8}\times\frac{w}{8}\times c},\end{split}$ (1)

where $\odot$ represents element-wise multiplication across the spatial
dimensions.

###  3.2 Global context encoder

Encoder $E$ processes the whole image, with the goal of efficiently encoding
the information given by the visible region, so that a downstream decoder can
synthesize a visually consistent output with the context. Our encoder $E$ is a
Vision Transformers (ViT) [13]. To produce tokens, we first downsample the
mask $M$ using a learned convolution layer to match the latent spatial
dimensions, as done by Wang et al. [55]. Then, we concatenate the downsampled
mask and latent code $Z$ along the channel dimensions and and divide them into
$4\times 4$ patches, with an overlap of $1$ on each side. This yields
$N=64\times 64=4096$ patches. Then, following standard practice, we linearly
embed each patch and add positional embedding [53]. Finally, the tokens are
passed through the transformers and produce a new set of $N$ tokens. In
summary, the encoder transforms the input $Z$ and $M$ into a set of $N$ tokens
of dimension $d=1152$.

$\mathcal{T}_{\text{all}}=\\{\mathbf{\tau}_{1},\mathbf{\tau}_{2},\ldots,\mathbf{\tau}_{N}\\}=E(Z,M),~{}\mathbf{\tau}_{i}\in\mathbb{R}^{d}.$
(2)

Token dropping. The set of output tokens contain information regarding the
whole image, but using them all would cause downstream computation to scale
with respect to the input size. _Can we instead keep only a subset of tokens,
that would hold the information needed for generation?_

As the self-attention layers in the encoder transformer enable all the tokens
to interact, each individual token has the potential to encode the relevant
context of the whole image. As such, we discard the tokens corresponding to
the visible region, keeping the ones corresponding to the hole. Dropping
tokens outside the mask creates an information bottleneck that encourages $E$
to summarize the input context in a compact set of tokens and ensures the
downstream computation only scales with the size of the masked area, since the
decoder will thus only process tokens covering the hole. The tokens should
also represent the relevant information for the given location; previous works
visualizing transformers [5] suggest that this location information can be
preserved. Patches with partial holes are also included, and the visible
pixels in those patches are blended in at the output step. Formally, we
maxpool mask $M$ to a $64\times 64$ map and vectorize into a set
$\\{m_{i}\\}_{i=1}^{4096}$, where $m_{i}\in\\{0,1\\}$.

$\mathcal{T}_{\text{hole}}=\\{\mathbf{\tau}_{i}\ |\hskip
2.84526ptm_{i}=1\\}\subseteq\mathcal{T}_{\text{all}}.$ (3)

The remaining set of $N_{\text{hole}}\leq N$ tokens form our compressed global
context. This design, along other architectural choices, are evaluated in the
supplemental.

###  3.3 Incremental diffusion decoder

We synthesize the missing pixels, using a transformer-based diffusion decoder
$D$ [7, 35]. Rather than keeping a set of $N$ tokens representing the whole
image, we start with $N_{\text{hole}}$ tokens corresponding to the hole,
$\mathcal{X}_{\text{hole}}=\\{\mathbf{x}_{i}\\}$. The diffusion process
creates time-conditioned tokens
$\mathcal{X}_{\text{hole}}^{t}=\\{\mathbf{x}_{i}^{t}\\}$, where
$t\in[0,...,T]$, starting at time $T$ with features drawn from a unit
Gaussian. The decoder progressively denoises these tokens, conditioned on the
T5-encoded text prompt $\mathbf{c}$ [39] and the global context produced by
the encoder $\mathcal{T}_{\text{hole}}$:

$\mathcal{X}_{\text{hole}}^{t-1}=D\left(\mathcal{X}_{\text{hole}}^{t}\oplus\mathcal{T}_{\text{hole}};t,\mathbf{c}\right),$
(4)

where $\oplus$ denotes concatenation along the hidden dimension of
corresponding elements in each set. We find this conditioning mechanism
superior to several alternatives analyzed in Appendix B.

Blending. The final tokens $\mathcal{X}^{0}_{\text{hole}}$ are mapped back
into the latent image domain using a linear layer, and the inverse of the
patch-splitting procedure to obtain a partial latent image
$\hat{Z}_{\text{hole}}\in\mathbb{R}^{\frac{h}{8}\times\frac{w}{8}\times c}$.
The missing tokens, corresponding to visible pixels, are left uninitialized
with zeros. We combine this output with the visible latent, using pointwise
masking, to obtain the final latent composite:

$\hat{Z}=(1-M)\odot Z+M\odot\hat{Z}_{\text{hole}}.$ (5)

Finally, this is decoded by the latent decoder to produce the final RGB image
$\hat{I}=\mathcal{D}(\hat{Z})$.

These decoded results occasionally contains faintly visible seams. Previous
works performing inpainting with latent diffusion models observed this
phenomenon and addressed it with a dedicated latent decoder [62]. As their
decoder is computationally intensive, we opt to use a simple Poisson blending
postprocessing step [36] in RGB space. We discuss this challenge in greater
length in the supplemental.

Training and implementation details. For the decoder, we adopt the
PixArt-$\alpha$ [7] architecture, and add a single layer to support our
conditioning on context. We initialize all shared layers from the public
PixArt-$\alpha$ checkpoint to benefit from their pretraining. The encoder on
the other hand, is trained from scratch. The two models are trained jointly to
reconstruct masked (latent) pixels, using the Improved DDPM objective [32]. We
train our model for 100,000 iterations on 56 NVIDIA A100 GPUs, using the AdamW
optimizer [25], with a constant learning rate $2\times 10^{-5}$, weight decay
set to $3\times 10^{-2}$ and global batch size of 224. We use $T=1000$
diffusion steps during training. We generate our results using the Improved
DDPM sampler [32] with 50 steps, unless specified otherwise, and set the
classifier-free guidance scale to 4.5. All running times are measured on a
single A100 GPU. We provide further details in Appendix D.

##  4 Experiments

###  4.1 Experimental setup

The main paper primarily focuses on a text-conditioned setting, as do the
experiments that follow. However, our approach is versatile and can be applied
in other use cases as well. In the early stages of this research, we primarily
explored unconditional inpainting on the ImageNet dataset [10], which are
detailed in Appendix B.

Dataset. We train our model at $1024\times 1024$ resolution on an internal
dataset containing 220 million high-quality images, covering a wide variety of
objects and scenes. We produce masks and text prompts in a process similar to
that proposed by Xie et al. [57]. Specifically, we use an entity segmentation
model [38] to segment all objects in an image and then caption each entity
with BLIP-2 [23]. To simulate the rough and inaccurate masks created by users,
we randomly dilate the entity mask (see Appendix D for details). During
training, we randomly sample triplets of image, mask, and caption.

Baselines. We compare _LazyDiffusion_ with two inpainting baselines (already
shown in Fig. 2), which we refer to as _RegenerateImage_ and _RegenerateCrop_.
_RegenerateImage_ , is the approach found in most academic works [55, 57, 42,
37], and operates on the entire image. _RegenerateCrop_ , used in popular
software frameworks [50, 54], operates on a tight square crop around the
masked region. The crop is first resized to a fixed low-resolution before
processing and is upsampled back afterwards. Both approaches generate as many
pixels as their input contains (whether full-canvas or local crop), unlike
_LazyDiffusion_ that generates only masked patches.

To ensure a fair comparison, we utilize the PixArt-$\alpha$ architecture for
both approaches. Since there is currently no publicly available PixArt-based
inpainting models, we design and train them ourselves. We adapt PixArt for
inpainting using the same procedure employed to transform Stable Diffusion
[42] from generation to inpainting. Specifically, we incorporate the GLIDE
[31] conditioning mechanism, where the generator operates on 9 latent
channels: four channels for the latent being denoised, four channels
representing the latent of the masked input image, and the last channel
containing a downsampled version of the mask. We train two PixArt models at
$1024\times 1024$ and $512\times 512$ for _RegenerateImage_ and
_RegenerateCrop_ , respectively.

We also compare with Stable Diffusion variants of these two approaches for
reference: SDXL [37] operates on the entire $1024\times 1024$ image, while
SD2-crop [42] operates on a $512\times 512$ crop. It is important to note that
these models utilize different architectures and were trained on different
datasets, and hence are not directly comparable. We include them in this
comparison only as references for state-of-the-art quality.

Figure 5:  Comparing _LazyDiffusion_ ’s runtime to that of baselines
regenerating the entire $1024\times 1024$ image or a smaller $512\times 512$
crop around the mask. _LazyDiffusion_ is consistently faster than
_RegenerateImage_ , especially for small mask ratios typical to interactive
edits, reaching a speedup of $10\times$. Similarly, _LazyDiffusion_ is faster
than _RegenerateCrop_ for mask ratios $<25\%$. For masks greater than that
(dashed), _RegenerateCrop_ is technically faster but generates in low-
resolution and naively upsamples to match the desired resolution, harming
image quality.

###  4.2 Inference time

We illustrate the overall runtime of all methods in Fig. 5. The baselines run
is constant time, as they operate on fixed size tensors derived from the fixed
input size – full canvas for _RegenerateImage_ and a fixed-size crop for
_RegenerateCrop_. In contrast, _LazyDiffusion_ ’s runtime scales with the mask
size, because our decoder processes tensors with dimensions proportional to
the masked region. This leads to significant speedups for small masks, typical
of interactive editing applications. For example, with a mask covering 10% of
the image our model achieves a $\times 10$ speedup over _RegenerateImage_.
Similarly, _LazyDiffusion_ is also faster than _RegenerateCrop_ for masks
smaller than 25%. At mask ratio 25%, both methods generate the same number of
pixels and have comparable running times. For larger masks, _RegenerateCrop_
is faster but generates low-resolution crops and naively upsamples to native
resolution, reducing sharpness. Additionally, _RegenerateCrop_ often fails to
produce outputs that are consistent with the region outside the mask, as we
discuss below (Sec. 4.4).

While there are additional networks in the pipeline, the diffusion decoder is
the only component running multiple times, and thus dominates the runtime.
Notably, our context encoder adds a $73$ms overhead, which is dwarfed by the
cost of the diffusion loop. The latent encoder and decoder take $97$ms and
$176$ms, respectively, and the T5 text encoder $21$ms. These are shared by all
methods.

Scaling laws. Our method essentially reduces the cost of each denoising
iteration at the price of a small overhead for the context encoder, to balance
quality with context retention. As a result, our performance gains are most
striking for high diffusion step counts (typically correlated with higher
image quality), and smaller mask sizes (most frequent in interactive
applications). A single evaluation of our decoder takes $374$ms to generate
_full_ image, but only $28$ms for 10% masks — a $\times 13.4$ speedup, greater
than the encoder’s overhead. So, our method remains beneficial for few-step
[49], or even one-step models [58]. We expect the performance gains provided
by our strategy to be even more striking on costlier applications like high-
resolution image editing, or video synthesis [3].

###  4.3 Progressive generation

Diffusion models are challenging to integrate into interactive pipelines due
to their high latency.

---
Figure 6:  Progressive image editing (top) and image generation (bottom) using
_LazyDiffusion_. Each panel illustrates a generative progression compared to
the preceding state of the canvas to its left. _LazyDiffusion_ markedly
accelerates local image edits (approximately $\times 10$), rendering diffusion
models more apt for user-in-the-loop applications.

There exists an abundance of research on broadly accelerating diffusion models
[58, 49, 26], but in the context of this study, we highlight that individuals
often tackle tasks incrementally, executing operations progressively and
concentrating on local adjustments one at a time—whether it involves adding or
removing objects, refining, or retrying previous attempts. _LazyDiffusion_
significantly accelerates such local operations, making it well-suited for
interactive pipelines with a user-in-the-loop.

In Fig. 6, we showcase a couple of iterations using _LazyDiffusion_ for both
image editing and image generation, starting from a blank canvas. Furthermore,
we attach a supplemental video that showcases authentic user interactions with
both _LazyDiffusion_ and our _RegenerateImage_ baseline, highlighting the
discernible difference in running time between the two.

###  4.4 Inpainting quality

A distinctive feature of _LazyDiffusion_ is its utilization of a compressed
global context to aid inpainting. In contrast, _RegenerateImage_ utilizes the
complete global context, while _RegenerateCrop_ relies on the context provided
by pixels neighboring the mask. We now compare the results produced by these
approaches.

For quantitative evaluation, we report zero-shot FID [20] and CLIPScore [19],
which estimate similarity to real images and text-image alignment,
respectively. Additionally, we include scores for SDXL [37] and SD2-crop [42].
Despite not being directly comparable, because they use different
architectures and training data, they serve as references for state-of-the-art
quality. In Table 1, we report mean scores over a random sample of 10,000
images drawn from OpenImages [45]. Notably, text-image alignment (CLIP)
remains unaffected by the mechanism to use image context. On the FID metric,
_LazyDiffusion_ exhibits only a marginal increase compared to
_RegenerateImage_ (%4) and performs significantly better than _RegenerateCrop_
(%26).

Table 1:  Quantitative comparison of our method with the three baselines. We report zero-shot FID [20] and CLIPScore [19] on $10k$ images images from OpenImages [45]. Scores of SD2-crop [42] and SDXL [37] are not directly comparable and provided only for reference.  Method | CLIP Score $(\uparrow)$ | FID $(\downarrow)$
---|---|---
SD2-crop | 0.21 | 6.95
SDXL | 0.21 | 6.88
_RegenerateCrop_ | 0.19 | 9.35
_RegenerateImage_ | 0.19 | 7.38
_LazyDiffusion_ (Ours) | 0.19 | 7.70
---
Figure 7:  Comparing Inpainting Results: (Top) Inpainting most objects
requires relatively little semantic context. In such cases, all methods
produce reasonably good results, even those processing only a tight crop.
(Bottom) However, when inpainting an object closely related to others, such as
one bun out of many, the inpainting model requires robust semantic
understanding. Methods processing only a crop produce objects that may seem
reasonable in isolation, but do not fit well within the greater context of the
image. In contrast, _LazyDiffusion_ adeptly leverages the compressed image
context to generate high-fidelity results, comparable in quality to models
regenerating the entire image and running up to ten times slower. Additional
results are provided in Figs. 12, 11, 13 and 14.

We show qualitative comparisons in Fig. 7. Our examination reveals a
significant discrepancy in the performance of models regenerating a crop –
_RegenerateCrop_ and SD2-crop. In many instances, inpainting involves
generating an object that is visually independent of other concepts in the
image, such as adding a side of fries next to a burger. Here, models operating
on a tight crop can produce reasonable-looking objects and seamlessly blend
them with the surrounding pixels available in the crop (Fig. 7 (Top)).
However, in numerous scenarios, the goal is to add an object that is strongly
related to the existing context, such as adding another bun to a tray of buns.
Models operating solely on a crop lack knowledge of the global image and
consequently produce objects that may seem reasonable in isolation but do not
fit well within the greater image context (Fig. 7 (Bottom)). In contrast, SDXL
and _RegenerateImage_ utilize direct and full access to all image pixels to
consistently yield highly realistic results, where the generated region fits
well with the existing content. Notably, we find that _LazyDiffusion_ behaves
similarly and produces comparable results even in these challenging edge
cases. This suggests that the compressed image context is highly expressive
and encodes meaningful semantic information.

User study. We measure the models’ capability to produce highly-contextual
inpainting through a user study. For this, we curate a specialized test set
comprising scenarios that necessitate a high level of semantic image context
for effective inpainting. Specifically, we select images featuring several
closely related objects, such as a set of uniform buns on a tray.
Subsequently, we evaluate all models based on their ability to regenerate one
of these objects when masked. In this scenario, the models must rely on
visible pixels to produce a high-fidelity result. Users are presented with the
masked input image, a text prompt, and two results — ours and a baseline. They
are then asked to ”select the option in which the inpainted image, as a whole,
looks best”. We collect a total of 1778 responses from 48 unique users and
find that our method is strongly preferred over methods operating solely on a
crop and competitive with those regenerating the entire image. Specifically,
_LazyDiffusion_ is preferred over _RegenerateCrop_ in 81% of cases, over
SD2-crop in 82.5% of cases, over _RegenerateImage_ in 46.1% of cases, and over
SDXL in 48.5% of cases. These results indicate that the compressed encoder
context retains the core semantic information required even for challenging
use cases. In short, our model demonstrates competitive quality to our
conceptual upper-bound _RegenerateImage_ , but runs up to ten times faster.

Figure 8: Our model readily supports additional forms of local conditioning.
For example, similar to SDEdit [29], a user can draw a simplistic colored
sketch, providing the model shape and color information.

###  4.5 Sketch-guided inpainting

So far, our emphasis has been on generation guided solely by the mask and a
text prompt. However, in principle, our method is applicable to any localized
generation task and can accommodate other forms of conditioning, such as
sketches and edge maps. In Fig. 8, we briefly showcase this versatility by
guiding the generation with a coarse color sketch provided by the user.
Following the SDEdit [29] approach, we initiate the generation process from
the partially noised input image instead of Gaussian noise.

##  5 Conclusions, limitations and future work

We introduced a novel transformer-based encoder-decoder architecture for
interactive image generation and editing using a diffusion model. Our approach
reduces the diffusion runtime by only generating the patches corresponding to
the small region to synthesize, rather than the entire image. This is achieved
through a global context encoder that summarizes the entire image once,
outside the diffusion sampling loop, ensuring globally-consistent outputs.

Our method maintains the generation quality of state-of-the-art models, and
reduces runtime costs proportionally to the size of the region to generate.
This reduction in latency, particularly for small masks, transforms image
generation into an interactive process by spreading the generation cost across
multiple user interactions.

Our architecture does have some weaknesses. Despite operating outside the
diffusion loop, the context encoder processes the entire image, posing a
potential bottleneck for very high-resolution images due to its quadratic
scaling in input size. Addressing this limitation could enhance the
scalability and applicability of our approach to larger and more intricate
visual content. We observed that occasionally, generated results have a subtle
color shift compared to the visible image regions, leading to visible patch
boundaries. While the Poisson blending post-processing methods discussed in
Section 3.3 effectively mitigates these issues, future research is needed to
identify a more principled and systematic solution.

Acknowledgement. We are grateful to Minguk Kang, Tianwei Yin and Wei-An Lin
for technical suggestions, to Rotem Shalev-Arkushin for proofreading our draft
and offering feedback, and to Yogev Nitzan for his help running the user
study. This work was done while Yotam Nitzan was an intern at Adobe.

## References

  * Avrahami et al. [2022] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18208–18218, 2022.
  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ , 2:3, 2023.
  * Brooks et al. [2024] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.  Video generation models as world simulators.  2024\.
  * Cao et al. [2023] Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.  _arXiv preprint arXiv:2304.08465_ , 2023.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Chang et al. [2022] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman.  Maskgit: Masked generative image transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11315–11325, 2022.
  * Chen et al. [2023] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li.  Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023.
  * Chen et al. [2020] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In _International conference on machine learning_ , pages 1691–1703. PMLR, 2020.
  * Couairon et al. [2022] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  _arXiv preprint arXiv:2210.11427_ , 2022.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Ding et al. [2022] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  _Advances in Neural Information Processing Systems_ , 35:16890–16902, 2022.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Esser et al. [2021] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 12873–12883, 2021.
  * Esser et al. [2024] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach.  Scaling rectified flow transformers for high-resolution image synthesis, 2024.
  * Gao et al. [2023] Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan.  Masked diffusion transformer is a strong image synthesizer.  _arXiv preprint arXiv:2303.14389_ , 2023.
  * He et al. [2022] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 16000–16009, 2022.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Hessel et al. [2021] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  _arXiv preprint arXiv:2104.08718_ , 2021.
  * Heusel et al. [2017] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023.
  * Liu et al. [2023] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, and Qiang Liu.  Instaflow: One step is enough for high-quality diffusion-based text-to-image generation.  _arXiv preprint arXiv:2309.06380_ , 2023.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2022a] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022a.
  * Lu et al. [2022b] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv preprint arXiv:2211.01095_ , 2022b.
  * Luo et al. [2023] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  _arXiv preprint arXiv:2310.04378_ , 2023.
  * Meng et al. [2021] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  _arXiv preprint arXiv:2108.01073_ , 2021.
  * Nguyen and Tran [2023] Thuan Hoang Nguyen and Anh Tran.  Swiftbrush: One-step text-to-image diffusion model with variational score distillation.  _arXiv preprint arXiv:2312.05239_ , 2023.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Parmar et al. [2021] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu.  On buggy resizing libraries and surprising subtleties in fid calculation.  _arXiv preprint arXiv:2104.11222_ , 2021.
  * Patashnik et al. [2023] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch-Elor, and Daniel Cohen-Or.  Localizing object-level shape variations with text-to-image diffusion models.  _arXiv preprint arXiv:2303.11306_ , 2023.
  * Peebles and Xie [2023] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4195–4205, 2023.
  * Pérez et al. [2003] Patrick Pérez, Michel Gangnet, and Andrew Blake.  Poisson image editing.  In _ACM SIGGRAPH 2003 Papers_ , pages 313–318. 2003.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Qi et al. [2022] Lu Qi, Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Philip Torr, Zhe Lin, and Jiaya Jia.  Open world entity segmentation.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Razavi et al. [2019] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  _Advances in neural information processing systems_ , 32, 2019.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International conference on machine learning_ , pages 2256–2265. PMLR, 2015.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  2023\.
  * stable-diffusion webui [2024] stable-diffusion webui.  stable-diffusion-webui.  https://github.com/AUTOMATIC1111/stable-diffusion-webui, 2024.  Accessed: Jan 2024.
  * Suvorov et al. [2021] Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky.  Resolution-robust large mask inpainting with fourier convolutions.  _arXiv preprint arXiv:2109.07161_ , 2021.
  * Tevet et al. [2022] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano.  Human motion diffusion model.  _arXiv preprint arXiv:2209.14916_ , 2022.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * von Platen et al. [2022] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf.  Diffusers: State-of-the-art diffusion models.  https://github.com/huggingface/diffusers, 2022.
  * Wang et al. [2023] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al.  Imagen editor and editbench: Advancing and evaluating text-guided image inpainting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18359–18369, 2023.
  * Wei et al. [2023] Chen Wei, Karttikeya Mangalam, Po-Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer.  Diffusion models as masked autoencoders.  _arXiv preprint arXiv:2304.03283_ , 2023.
  * Xie et al. [2023] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang.  Smartbrush: Text and shape guided object inpainting with diffusion model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22428–22437, 2023.
  * Yin et al. [2023] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park.  One-step diffusion with distribution matching distillation.  _arXiv preprint arXiv:2311.18828_ , 2023.
  * Yu et al. [2019] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang.  Free-form image inpainting with gated convolution.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 4471–4480, 2019.
  * Zhao et al. [2021] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu.  Large scale image completion via co-modulated generative adversarial networks.  In _International Conference on Learning Representations (ICLR)_ , 2021.
  * Zheng et al. [2023] Hongkai Zheng, Weili Nie, Arash Vahdat, and Anima Anandkumar.  Fast training of diffusion models with masked transformers.  _arXiv preprint arXiv:2306.09305_ , 2023.
  * Zhu et al. [2023a] Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, and Gang Hua.  Designing a better asymmetric vqgan for stablediffusion.  _arXiv preprint arXiv:2306.04632_ , 2023a.
  * Zhu et al. [2023b] Zixin Zhu, Xuelu Feng, Dongdong Chen, Jianmin Bao, Le Wang, Yinpeng Chen, Lu Yuan, and Gang Hua.  Designing a better asymmetric vqgan for stablediffusion, 2023b.

##  Appendix A Supplementary Overview

In Appendix B, we conduct an ablation study, comparing our chosen architecture
with possible alternatives. Then, in Appendix C, we analyze our blending
approach at post-processing and extend the qualitative evaluation from the
main paper. Finally, in Appendix D, we offer additional implementation
details, completing the paper.

##  Appendix B Architecture Design and Ablation

Pivotal to our architectural design is compressing the visible context to
fewer tokens and utilizing it within the diffusion decoder. In the following
section, we describe the experiments leading to our eventual design.

###  B.1 Setting

While text-based inpainting serves as the primary application demonstrated in
this paper, _LazyDiffusion_ is readily applicable to a range of other local
generation applications. When designing our architecture in early stages of
this work, we applied our method to unconditional inpainting [59, 51] on
ImageNet [10] at $256\times 256$ resolution, as this setting demands
substantially less training time and resources. We adopt the masking protocol
from DeepFillV2 [59]. We use the same ViT XL/2 [13] backbone for our context
encoder and adopt DiT XL/2 [35] for the diffusion transformer. Note that the
PixArt-$\alpha$ [7] architecture, used in the main paper, is a straight-
forward adaptation of DiT to support text conditioning. Consequently, the
architectures we describe next can seamlessly use both as backbones.

###  B.2 Chosen design review

Recall that in our proposed architecture, discussed in Sec. 3, we selectively
retain only encoder output tokens corresponding to the masked region, marked
$\mathcal{T}_{\text{hole}}$. This ensures that downstream decoder computation
scales with the mask size rather than the image size. At time $t$, the decoder
denoises tokens $\mathcal{X}_{\text{hole}}^{t}$ while conditioning on the
retained context tokens. We implement the conditioning by concatenating the
context tokens to the noise tokens at the decoder’s input. Omitted from the
main paper for clarity, we prepend a linear projection layer to the diffusion
transformer backbone, projecting the concatenation of tokens to the decoder’s
hidden dimension $d$. Other than the first layer, the diffusion transformer is
then used _as-is_ to generate $k=|\mathcal{T}_{\text{hole}}|$ tokens.
Rewriting Eq. 4 from the main paper with greater detail, a single denoising
step reads as

$\mathcal{X}_{\text{hole}}^{t-1}=\text{DiT}\left(\text{linear}(\mathcal{X}_{\text{hole}}^{t}\oplus\mathcal{T}_{\text{hole}});t,\mathbf{c}\right),$
(6)

where $\oplus$ denotes concatenation along the hidden dimension. Transformers
runtime scale quadratically with the number of tokens. Thus, the runtime of
this architecture scales as $\mathcal{O}(k^{2})$. In this section, we refer to
this architecture as the “Concat Hidden” variant.

###  B.3 Alternative designs

We next describe alternative designs with the goal of ablating the two core
choices – dropping visible tokens to compress context and conditioning through
concatenation

Full context designs, utilizing the full set of $N$ encoder tokens
$\mathcal{T}_{\text{all}}$ as context:

  * •

_RegenerateImage_ – As described in the paper, we adapt DiT for inpainting
using the GLIDE [31] conditioning approach. This model represents the common
approach in local editing literature – operates on the entire canvas thus
seeing the full context but also re-generating the entire image. The runtime
complexity of this variant scales as $\mathcal{O}(N^{2})$. Note that $N>>k$.

  * •

Full-Context Cross-Attention – We add a cross-attention layer to the DiT
block, between the self-attention and MLP layers. Other than the upstream
activations, the cross-attention layer gets as input the _full_ encoder
context tokens $\mathcal{T}_{\text{all}}$. Despite “seeing” the full context,
the model generates only the $k$ masked patches. It’s runtime scales as
$\mathcal{O}(Nk)$.

Compressed context designs. Comparable to our chosen design – the following
models utilize the masked tokens $\mathcal{T}_{\text{hole}}$ as context,
generate only the masked region and have runtimes that scale with
$\mathcal{O}(k^{2})$. They differ in their mechanism to condition on the
context tokens. We experiment with simple conditioning approaches that are
applied near the input level. This prevents designs from being tightly coupled
with the specific backbone architecture, which we anticipate would facilitate
easier adaptation to future diffusion transformers.

  * •

Concat Length – The sets of tokens are concatenated over the sequence length,
rather than hidden dimension. This requires the two sets of tokens to have the
same hidden dimension. To this end, we first linearly project the context
tokens to the decoder’s hidden dimension $d$. Formally, a single denoising
step is done by

$\mathcal{X}_{\text{hole}}^{t-1}=\text{DiT}\left([\mathcal{X}_{\text{hole}}^{t},\text{linear}(\mathcal{T}_{\text{hole}})];t,\mathbf{c}\right),$
(7)

where $[\cdot,\cdot]$ represents the sequence-length concatenation.

  * •

Weighted Sum – An additional weight $w\in\mathbb{R}^{d}$ is learned, and the
input to DiT is a weighted sum of the two sets of tokens, formally

$\mathcal{X}_{\text{hole}}^{t-1}=\text{DiT}\left(\mathcal{X}_{\text{hole}}^{t}+w*\text{linear}(\mathcal{T}_{\text{hole}});t,\mathbf{c}\right).$
(8)

  * •

Compressed-Context Cross-Attention – We again add a cross-attention layer, but
here it attends only to the reduced set of tokens $\mathcal{T}_{\text{hole}}$.
To better resemble other designs in this category, incorporating the
conditioning near the input, we add the cross-attention layer only to the
first DiT block.

###  B.4 Configurations

DiT’s FLOPs are strongly negatively correlated with FID, across different
configurations [35]. To facilitate direct comparison, we slightly adjust the
XL/2 configuration for the $\mathcal{O}(k^{2})$ variants so that their FLOP
counts are similar. We provide the exact hyperparameters used with each
variant in Tab. 2 and the resulting FLOP counts as a function of mask size are
in Fig. 9(a). As can be seen, Concat Hidden, Weighted Sum and the Compressed-
Context Cross-Attention have comparable FLOPs on the entire spectrum ranging
from mask ratio of 10% to 100%. For full masks, the Concat Hidden, Weighted
Sum variants use $0.4\%$ and $0.6\%$ more FLOPs than _RegenerateImage_ ,
respectively. This implies that our conditioning introduces negligible
overhead and is well suited for using larger masks with no apparent downside.
The other three variants have strictly greater FLOP counts.

Table 2:  Hyperparameters configuration for all architecture designs. Starting from DiT’s XL/2 configuration, we slightly adapt the hyperparameters to ensure FLOP counts of $\mathcal{O}(k^{2})$ are comparable.  Runtime Complexity | Model | Layers |  Hidden Dimension
---|---|---|---
$\mathcal{O}(k^{2})$ |  |  |
Concat Hidden | 28 | 1152
Weighted Sum | 28 | 1152
Concat Length | 24 | 1024
| Cross Attention | 26 | 1152
$\mathcal{O}(Nk)$ | Cross Attention | 28 | 1152
$\mathcal{O}(N^{2})$ | _RegenerateImage_ | 28 | 1152

(a)

(b)

Figure 9:  Comparing the various architecture designs in terms of (a) FLOPs
and (b) quality, measured via FID [20]. Solid lines represent variants of our
approach – the encoder outputs a compressed context and the decoder generates
only the masked region. Dashed lines represent mechanisms in which the decoder
is conditioned on the full image context and either generates the masked
region or the entire image. The latter is the approach taken by existing
inpainting approaches [42]. The runtime complexities of different approaches
is noted in the legend. As can be seen, conditioning each generated token
directly on its corresponding compressed context token, as done for the
“Concat Hidden” and “Weighted Sum” variants, leads to superior performance,
despite using fewer FLOPs than competing approaches.

###  B.5 Results

We track the FID [20] scores across 500K training iterations for all decoder
designs and present the results in Fig. 9(b).

Initially, we observe that “Concat Hidden” and “Weighted Sum” notably
outperform all other variants. We attribute this superior performance to the
explicit one-to-one context provided by these approaches. In both cases, each
noise token is directly conditioned on the corresponding context token. In
contrast, other methods require the decoder to extract context from a set of
encoder tokens, which appears to be more challenging despite the use of
positional embedding and more expressive mechanisms such as cross-attention.

Furthermore, we note that the more computationally intensive baselines, which
leverage additional context, do not yield better results. Specifically, in the
two cross-attention variants, the one that uses compressed context is superior
to the one using full context. Our attempts to improve the performance of the
_RegenerateImage_ baseline by using a context encoder and a “Concat Hidden”
based conditioning were futile; only dropping the visible context tokens was
effective. We speculate that incorporating the full context imposes additional
complexity on the decoder’s task. In comparison, with _LazyDiffusion_ , the
information bottleneck encourages the context to be expressive but selective,
allowing the decoder to “concentrate” on synthesis only.

Interestingly, in the text-conditioned setting, _LazyDiffusion_ is not
superior in terms of quality to _RegenerateImage_. This disparity might be
explained by the lower level context required for unconditional inpainting,
which primarily involves continuing surrounding textures, compared to the
semantic context required for generating novel objects.

###  B.6 Implementation details

We train and sample all models with the EDM [22] diffusion formulation. We use
Stable Diffusion’s [42] public latent VAE. We train the encoder and decoder
jointly from scratch, on 8 NVIDIA A100 GPUs, using global batch size of 256,
using the AdamW [25] optimizer with constant learning rate of $10^{-4}$. We
sample using 40 denoising steps and classifier-free guidance scale of $4.0$.
Other details are the same as in the text-conditioned setting and are detailed
in the main paper or in Appendix D.

##  Appendix C Additional Experiments and Results

###  C.1 Blending

_LazyDiffusion_ generates only the masked regions of the latent image. To
achieve the final desired results, these regions must be composited with the
visible image regions and decoded into an image. Initially, we naively blend
the generated latent with the latent of the input image, as described in Eq. 5
in the main paper. However, we observe that passing the blended latent through
the latent decoder $\mathcal{D}$ occasionally results in poorly harmonized
images, characterized by faintly visible seams between the generated and
visible regions. This phenomenon was previously noted by Zhu et al. [63] when
performing local editing with Stable Diffusion [42]. It is conjectured that
the latent encoding loses subtle color information, hindering image
harmonization. In response, Zhu et al. proposed an alternative latent decoder
that additionally conditions on the masked input image $I\odot(1-M)$ itself
and is also significantly larger. Specifically, their decoder runs for
$800$ms, $4.5\times$ longer than the “vanilla” Stable Diffusion latent
decoder.

In our experiments, we find that simply performing Poisson blending [36] in
pixel space achieves comparable results, while running only for $35$ms on
average. Therefore, we introduce a Poisson blending post-processing step to
our pipeline. We demonstrate the harmonization issue and compare the two
approaches in Fig. 10.

Figure 10:  From partial latent generation to inpainted image. The “zero-pad
decoding” column is produced by decoding the incremental generation with zero
padding, demonstrating the object in isolation. To produce the desired
composited image, we blend the incremental generation with the latent input.
This occasionally leads to visible seams and lack of color harmonization as
seen in the “vanilla decoder” column. This issue can be solved using the
latent decoder proposed by Zhu et al. [62] or with Poisson blending [36]. We
recommend zooming in to better view the seams or lack thereof.

###  C.2 Additional Results

In Figs. 12 and 11, we extend Fig. 7 of the main paper and provide more
qualitative samples comparing _LazyDiffusion_ with the four baselines –
_RegenerateCrop_ , SD2-crop, _RegenerateImage_ and SDXL. We find that
_LazyDiffusion_ is mostly comparable to _RegenerateImage_ and SDXL even when
inpainting objects that require high semantic context, despite using a
compressed context and running up to $10\times$ faster.

Finally, in Figs. 13 and 14 we provide a non-curated set of results, with
masks and text prompts produced automatically by the segmentation and
captioning models. The main challenge we observe from these results is that
the model partially ignores the text when it conflicts with the shape of the
mask. For example, the hamburger in Fig. 13 is generated without a hat.

Figure 11:  Comparing inpainting results on objects that require modest
context, similar to Fig. 7(Top). All models usually produce reasonably good
results. Occasionally, SDXL [37] and SD2 [42] do not generate anything – a
result of their usage of random masks rather than object-level masks [57, 55].
Figure 12:  Comparing inpainting results on objects that have close semantic
relationship with the observed canvas, similar to Fig. 7(Bottom). Approaches
that only process a crop may generate objects that appear reasonable on their
own but lack coherence within the broader context of the image. In contrast,
_LazyDiffusion_ produces results comparable to those produces by methods
regenerating the entire image. Occasionally, _LazyDiffusion_ does not fully
utilize the visible context. For instance, our “sushi” result accurately
depicts the orange wrap and sesame seeds on top, consistent with other sushi
in the roll, but it features a different filling.  Figure 13:  A random set of
results produced by _LazyDiffusion_. For each input we produce three outputs
from different random seeds.  Figure 14:  A random set of results produced by
_LazyDiffusion_. For each input we produce three outputs from different random
seeds.

##  Appendix D Additional Details

Evaluation. We compute FID [20] using clean-fid [33]. For CLIPScore [19], we
report the “local” version that takes as input a crop around the generated
object and the local text, describing the object. This approach was previously
advocated by Wang et al. [55] and is more suitable for image inpainting than
using the full image and text caption for the entire image.

Architecture. As described in the main paper, we initialize our decoder with
PixArt-$\alpha$’s publicly released weights. Our decoder has an additional
linear layer, introduced in Sec. B.2, that projects the concatenation of
context and noise tokens to the decoder’s hidden dimension $d$. We initialize
this layer such that it outputs the noise tokens in its input and ignores the
context. This ensures that at initialization, if given a full mask and thus
operates on all tokens, our results are exactly equivalent to
PixArt-$\alpha$’s.

Data. As discussed in the paper, we adopt a data processing pipeline similar
to that of SmartBrush [57]. Specifically, our masks are originally produced by
an entity segmentation model [38] and are dilated to simulate the rough and
inaccurate masks created by users. First, with probability of 20% we replace
the segmentation mask with a rectangular mask corresponding to a bounding box.
Regardless, we dilate the mask by first performing Gaussian Blurring and
thresholding the output. The size of the Gaussian kernel is sampled uniformly
from $[\text{image size}/15,\text{image size}/5]$ and its standard deviation
along X and Y is sampled uniformly and independently from $[3,17]$. The
threshold is sampled uniformly from $\\{10^{-1},10^{-2},10^{-3},10^{-4}\\}$.


# Mathematical models of confirmation bias

Rose Baker
School of Business
University of Salford, UK

###### Abstract

Confirmation bias is a cognitive bias that adversely affects management
decisions, and mathematical modelling is an aid to its detailed understanding.
Bias in opinion update about the value of a parameter is modelled here
assuming that observations are discounted depending on their distance from
prior opinion. The models allow belief persistence, attitude polarization, and
the irrational primacy effect to be explored. A general framework for
exploring large-sample properties of these models is given, and an attempt
made to classify the models. An interesting result is that in some models the
influence of an observation always increases with distance from the prior
opinion, whereas in others observations greatly at odds with prior opinion are
given very little weight. The models could be useful to those exploring these
phenomena in detail.

## Keywords

Decision theory; cognitive bias; confirmation bias; log-gamma distribution;
directional discounting.

##  1 Introduction

###  1.1 Managerial relevance of confirmation bias and its modelling

Decision-making is essential to management: French (1991, 1995) gives a
description. However, management decisions, like all others, are plagued by a
vast host of cognitive biases. Sutherland (2013) gives a comprehensive account
(the first edition of his work dates from 1992). Among cognitive biases,
confirmation bias, the tendency to interpret data so as to confirm one’s prior
beliefs, is arguably the chief. It has been quoted as causing belief
perseverance, attitude polarization, and the irrational primacy effect
(earlier observations have more impact).

An example is the launch of a new product line. The CEO initially ‘knows’ it
is a good idea, and the subsequent feasibility research is interpreted so as
to support this belief, rendering the research valueless. Another is the
recruitment of new staff, where a battery of aptitude tests cannot dislodge
the hirers’ original impression of the potential new recruit, impressions
perhaps based on unconscious bias.

The relevance of management studying and becoming knowledgeable about
confirmation bias is evident. In general, mathematical modelling can be the
path to more detailed understanding of phenomena, such as has been the case
with infectious diseases. Models can offer detailed predictions, but more
importantly they can give insight and help to focus further study. Here,
however, little mathematical modelling has been done.

###  1.2 Modelling confirmation bias

Previous work includes Rabin and Schrag (1999), who considered a series of
Bernoulli trials, where signals that conflict with current belief are
sometimes misinterpreted. Gerber and Green (1999) considered biased learning,
e.g. normally-distributed observations, where the random variable $X$ is taken
as $\alpha X$ if above a neutral position at zero. This comes close to the
ideas presented here. Zimper and Ludwig (2009) develop models of Bayesian
learning using Choquet expected utility theory, and Jern et al (2014)
considered belief polarization arising from Bayesian networks. Allahverdyan
and Galstyan (2014) considered a non-Bayesian model. This work extends Baker
(2021).

Here, we discuss some simple Bayesian models of the evolution of belief about
observable quantities, such as an individual’s blood pressure, or an
organization’s sales figures. In these models bias is caused by undue
discounting of observations that contradict prior belief.

We first consider rational updating of opinion in the light of observation,
where Bayes’ theorem tells us how we ought to update our prior belief.

Let the probability density function (pdf) of prior belief about a parameter
$\theta$ be $f(\theta)$, and the likelihood (probability or pdf) of an
observation $X$ be ${\cal L}(X|\theta)$. The the posterior pdf of the
distribution of $\theta$ is

$f(\theta|X)=\frac{{\cal L}(X|\theta)f(\theta)}{\int{\cal
L}(X|\phi)f(\phi)\,\mbox{d}\phi}.$ (1)

In this work, the likelihood is that for a normal distribution with mean $\mu$
and variance $\sigma^{2}$. Unless the prior belief $f$ is certainty, after $n$
observations the posterior distribution will tend to a narrow Gaussian centred
on $\mu$ with variance approximately $\sigma^{2}/n$.

Bayesian meta-analysis provides an example of this kind in which opinion is
updated in the light of a series of observations (studies). In general, the
problem with using (1) is that we do not really know $\sigma^{2}$. The problem
is thus with the likelihood function rather than the prior belief. In general,
we must interpret the world in the light of our current (prior) knowledge, and
in a world of fake news and hype, claimed results must often be discounted to
some extent. Even in meta-analysis, where a series of results with well-
estimated variances is available, in over half of analyses the true error is
found to exceed the quoted one, as judged by the scatter of the results. In
medicine this may be caused by different patient mixes or different operating
procedures, but the same problem arises in other contexts, such as physics.
Thus discounting data is not irrational, although it can lead to the
persistence of prior belief even given a lot of contrary evidence.

There is a great deal of irrational and bizarre belief in the world, but this
work is concerned with individuals and organizations that are at least trying
to draw correct conclusions from data, but may still fall victim to
confirmation bias.

The next section introduces a simple model of bias, then a general framework
for analysing models is given. Finally there is an attempt to classify models
and a discussion of primacy, followed by some conclusions.

##  2 Models of bias

Imagine a series of $n$ independent observations $X_{i}$, where $X_{i}\sim
N[\mu,\sigma^{2}]$. For example, the random variable could be blood pressure
or some other medical measurement for an individual, or a quantity of
corporate interest such as a sales figure. There are more complex cases where
variances $\sigma_{i}^{2}$ differ, or where $\mu$ or $\sigma^{2}$ changes with
time, but for simplicity these are not considered.

We wish to find a posterior distribution for the centre of location. Using (1)
starting from a normally-distributed prior with mean $\mu_{0}$, variance
$\sigma_{0}^{2}$, after $n$ observations the posterior distribution would be
normal, with variance $v$ given by $1/v=n/\sigma^{2}+1/\sigma_{0}^{2}$, mean
$(n\bar{X}/\sigma^{2}+\mu_{0}/\sigma_{0}^{2})/v$. Asymptotically as
$n\rightarrow\infty$ we can ignore prior belief. However, with biased
observation the posterior distribution will be different.

Denote the centre of location by $\theta$ and the mean of the posterior
distribution of $\theta$ by $\lambda$

In these models, observations are discounted preferentially in one direction,
which to coin a phrase we could call ‘directional discounting’. Intuitively,
we can see that results in the ‘wrong’ direction might be less-liked. Utility
theory and prospect theory (e.g. Kahneman, 2012) teach that the pain of losing
is greater than the satisfaction of an equivalent gain.

The discounting has here been taken as ‘low is good’, but the models can
easily be changed to the opposite. The true distribution of observations is
normal, with the mean taken as zero without loss of generality. Hence here the
mean of the posterior distribution is also the asymptotic bias $\lambda$.

Several models are explored, chosen for realism and relative ease of
computation of the bias $\lambda$. The asymptotic properties are explored as
$n\rightarrow\infty$. In real life there will often be only a few
observations, but the asymptotic properties will still shed light on how
biased observation behaves.

Given a formula for $\hat{\lambda}$ after $n$ observations, a quantity of
interest is the influence $\Delta\hat{\lambda}$ of an observation $X$. This is
here defined as the change in $\hat{\lambda}$ following from adding a new
$n+1$th observation $X$, when $n$ is very large. In the unbiased case, this
increases linearly with $X-\lambda$. It is convenient to use
$n\Delta\hat{\lambda}$, which tends to a limit.

Some simple models are presented, followed by a ‘taxonomy’ of models.
Computations were carried out via purpose-written Fortran95 programs, using
the Numerical Algorithms Group (NAG) library for random-number generation and
special function evaluation.

###  2.1 Exponential Model

The simplest model is that an individual discounts observations $X$
exponentially, so that the variance $\sigma^{2}$ is taken as
$\sigma^{2}\exp(\beta X/\sigma)$. These discounted observations update the
prior opinion. If $\beta>0$, high observations are regarded as less
trustworthy. This simple model is quite tractable but it ignores the fact that
bias should be relative to current belief, i.e. the variance should really be
$\sigma^{2}\exp\\{\beta(X-\theta)/\sigma\\}$, of which more later. Figure 1
shows the evolution of the mean and standard deviation of the posterior
distribution under this model. The sudden downward jumps are caused by low
observations, which have a corresponding low perceived variance and large
influence. The approach to the asymptotic limit can be seen.

Figure 1: Posterior mean and standard deviation of blood pressure, starting at
a prior mean of 120 mm Hg, sd. 5, with random observations of mean 140, sd.
10, and with exponential discounting, $\beta=0.2$.

The asymptotic distribution of belief will be normal, and the posterior
distribution mean $\lambda$ and variance $v$ can be found after some large
number $n$ of observations using the expressions for a weighted mean

$\hat{\mu}=\frac{\sum_{i=1}^{n}X_{i}/\sigma_{i}^{2}}{\sum_{i=1}^{n}1/\sigma_{i}^{2}}$
(2)

and its variance

$v=\frac{1}{\sum_{i=1}^{n}1/\sigma_{i}^{2}}.$ (3)

This gives here

$\hat{\lambda}=\frac{\sum_{i=1}^{n}X_{i}\exp(-\beta
X_{i}/\sigma)}{\sum_{i=1}^{n}\exp(-\beta X_{i}/\sigma)}.$ (4)

As $n\rightarrow\infty$ the numerator and denominator tend to
$n\text{E}\\{X\exp(-\beta X/\sigma)\\},\text{E}\\{\exp(-\beta X/\sigma)\\}$
respectively. Using the results

$\text{E}\\{\exp(-\beta
X/\sigma)\\}=\frac{1}{\sqrt{2\pi\sigma^{2}}}\int_{-\infty}^{\infty}\exp(-x^{2}/2\sigma^{2}-\beta
x/\sigma)\,\mbox{d}x=\exp(\beta^{2}/2),$ (5) $\text{E}\\{X\exp(-\beta
X/\sigma)\\}=\frac{1}{\sqrt{2\pi\sigma^{2}}}\int_{-\infty}^{\infty}x\exp(-x^{2}/2\sigma^{2}-\beta
x/\sigma)\,\mbox{d}x=-\beta\sigma\exp(\beta^{2}/2),$

we have that

$\hat{\lambda}\rightarrow\lambda=-\beta\sigma.$ (6)

This shows that the mean of the posterior distribution is biased downwards if
$\beta>0$ and vice versa, by an amount that increases with the standard
deviation $\sigma$ of the observations.

The perceived (subjective) estimate of the variance of $\hat{\lambda}$ from
(3) is

$v=n\sigma^{2}/\text{E}\\{\exp(-\beta
X/\sigma)\\}=(\sigma^{2}/n)\exp(-\beta^{2}/2),$

which is smaller than the unbiased estimate of $\sigma^{2}/n$. Since

$\hat{\lambda}-\lambda=\frac{\sum_{i=1}^{n}(X_{i}-\lambda)\exp(-\beta
X_{i}/\sigma)}{\sum_{i=1}^{n}\exp(-\beta X_{i}/\sigma)}.$ (7)

the true variance of $\hat{\lambda}$ is given by

$\text{var}(\hat{\lambda})=\frac{\text{E}\\{\exp(-2\beta
X/\sigma)(X-\lambda)^{2}\\}}{n\text{E}\\{\exp(-\beta X/\sigma)\\}^{2}},$

which evaluates to

$\text{var}(\hat{\lambda})=(1+\beta^{2})\exp(\beta^{2})(\sigma^{2}/n).$ (8)

Whereas the subjective (perceived) variance was smaller than the unbiased
variance, the true variance is larger, by a factor that increases rapidly with
the bias $\beta$.

Finally, the distribution of $\hat{\lambda}$ is asymptotically normal, from
(7). The numerator is a sum of random variates which is normally distributed
by the Central Limit Theorem (CLT). The denominator does not alter this, as it
tends to a constant.

Hence under this simple model, discounting high or low observations biases the
mean of the posterior distribution by an amount that increases with the
standard deviation. In this case discounting reduces the subjective variance
of the posterior distribution, although that does not always happen. It also
increases the actual variance, which does always happen.

The influence function is readily computed from (4) as

$n\Delta\hat{\lambda}=\exp\\{-\beta(X/\sigma+\beta/2)\\}(X-\lambda).$ (9)

This is plotted in figure 2 for $\beta=\sigma=1$. For $X>\lambda$ it shows
surprising behaviour. As $X$ increases, influence first increases as one might
expect, but it then decreases after $X-\lambda=\sigma/\beta$, i.e. after
$X>0$. The discounting is so heavy that very high observations are effectively
ignored. Paradoxically, stronger evidence is largely ignored as being too far
from prior belief.

Figure 2: The influence function $n\Delta\hat{\lambda}$ for the exponential
model with $\beta=1$ and for the log-gamma distribution model, ditto.

####  2.1.1 Beta distribution model

We can apply this type of exponential model when the observations are not
normally distributed random variables such as sales figures, but are
probabilities or percentages, for example the efficacy of a vaccine expressed
as a proportion rendered immune. Then the discounting factor can be the odds
ratio $\\{(1-p)/p\\}^{\gamma}=\exp\\{\ln(\frac{1-p}{p})\gamma\\}$, where
higher probabilities are discounted. The posterior distribution mean can be
estimated as

$\hat{\lambda}=\frac{\sum_{i=1}^{n}p_{i}\\{(1-p_{i})/p_{i}\\}^{\gamma}}{\sum_{i=1}^{n}\\{(1-p_{i})/p_{i}\\}^{\gamma}},$

which on assuming the $p_{i}$ distributed as a beta distribution with
parameters $\alpha,\beta$ and taking expectations as $n\rightarrow\infty$
becomes

$B(\alpha+1-\gamma,\beta+\gamma)/B(\alpha-\gamma,\beta+\gamma)=\frac{\alpha-\gamma}{\alpha+\beta}=\mu-\frac{\gamma}{\alpha+\beta},$

where $B$ denotes the beta function. The variance of the posterior
distribution is

$v=\frac{\alpha\beta}{(\alpha+\beta)^{2}(\alpha+\beta+1)}\frac{B(\alpha-\gamma,\beta+\gamma)}{B(\alpha,\beta)}\frac{1}{n}.$

Note that we must have $\gamma<\alpha,\gamma>-\beta$.

##  3 A likelihood-based approach

The approach used for the exponential model and beta-distribution models
cannot be used in general, where the bias is relative to the prior belief
$\theta$. The simplest general way to derive the asymptotic behaviour of
models is as follows. For a specific prior belief $\theta$ we find the log-
likelihood $\ell(X_{1}\cdots X_{n}|\theta)$. Asymptotically, after a large
number $n$ of observations, this tends to its expected value
$\ell/n\rightarrow\text{E}\\{\ell(X|\lambda)\\}$ under the true distribution
for $X\sim N[0,\sigma^{2}]$. Because the likelihood becomes strongly peaked,
the (subjective) variance of $\hat{\lambda}$ is given by

$1/v=-\partial^{2}\text{E}(\ell)/\partial\lambda^{2}.$ (10)

This tends to zero as $n\rightarrow\infty$, with the asymptotic value of
$\lambda$ being that which maximises the likelihood, i.e. we require

$\partial\text{E}(\ell)/\partial\lambda=0.$ (11)

This approach yields the asymptotic bias $\lambda$ and the subjective
(perceived) error. It also yields the the true variance
$\text{var}(\hat{\lambda})$ of $\hat{\lambda}$. This can be found by expanding
the score function about $\lambda$:

$\partial\ell/\partial\lambda|_{\hat{\lambda}}=0\simeq\partial\ell/\partial\lambda|_{\lambda}+\partial^{2}\ell/\partial\lambda^{2}|_{\lambda}(\hat{\lambda}-\lambda).$

Hence

$\hat{\lambda}-\lambda=-\frac{\partial\ell/\partial\lambda|_{\lambda}}{\partial^{2}\ell/\partial\lambda^{2}|_{\lambda}}$
(12)

The influence function can be found using this large-sample approximation for
$\hat{\lambda}$.

From (12)

$\text{var}\hat{\lambda}=\frac{\text{E}\\{(\partial\ell/\partial\lambda)^{2}\\}}{\text{E}(\partial^{2}\ell/\partial\lambda^{2})^{2}}.$
(13)

Normally the Bartlett identity
$\text{E}(\partial^{2}\ell/\partial\theta^{2})=-\text{E}\\{(\partial\ell/\partial\theta)^{2}\\}$
would mean that these two forms for $v$ and $\text{var}(\hat{\lambda})$ are
the same, but here this is not so because the likelihood function is wrongly
specified, i.e. the expectation is not taken over the subjective distribution.
The maximum-likelihood estimator tends to the posterior mean as
$n\rightarrow\infty$, so the formulae above are the required variances.

With this framework, we can examine the ‘relative exponential’ model, which
puts right the unrealistic feature of the exponential bias model, that the
weight given to an observation did not depend on the opinion. Correcting this,
with an opinion $\theta$ we obtain the log-likelihood

$\ell=-\frac{\sum_{i=1}^{n}(X_{i}-\lambda)^{2}\exp(-\beta(X_{i}-\lambda)/\sigma)}{2\sigma^{2}}.$

As $\lambda\rightarrow-\infty$, $\ell\rightarrow 0$, so the bias would be
infinite and negative if $\beta>0$. In reality, there is also the prior
opinion which makes the posterior belief behave properly, but whose effect
diminishes as $n\rightarrow\infty$, so the bias becomes increasingly negative
as the number of observations increases. Thus such extreme discounting does
not lead to a stable bias $\lambda$ as sample size increases.

###  3.1 The ‘sweet spot’ model

When discounting is not directional, there is no bias, but the subjective and
true variance of the estimator $\hat{\lambda}$ change.

Consider a subjective distribution of $X$ that is a t-distribution with
$\nu=1/\beta$ degrees of freedom. Then the log-likelihood is

$\ell(\lambda)=-(1+\beta)\sum_{i=1}^{n}\frac{\ln(1+\beta(X_{i}-\lambda)^{2}/\sigma^{2})}{2\beta}.$

As $\beta\rightarrow 0$ we regain the normal log-likelihood function.

Asymptotically

$\ell(\lambda)/n\rightarrow(1+\beta)\text{E}\\{\frac{-\ln(1+\beta(X-\lambda)^{2}/\sigma^{2}}{2\beta}\\},$

and the peak occurs at

$\text{E}\\{\frac{X-\lambda}{1+\beta(X-\lambda)^{2}/\sigma^{2}}\\}=0.$

The only solution is $\lambda=0$, as expected.

The computations of subjective and true variances require integrals
$I_{1}(\beta),I_{2}(\beta)$, where

$I_{n}(\beta)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\frac{\exp(-x^{2}/2)\,\mbox{d}x}{(1+\beta
x^{2})^{n}}.$

These integrals can be expressed analytically and are derived in the appendix.
Given these functions, the variance of the posterior distribution $v$ is given
by

$v=-(1/n)\\{\partial^{2}\text{E}(\ell)/\partial\lambda^{2}|_{\lambda=0}\\}^{-1}=\frac{\sigma^{2}/n}{(1+\beta)(2I_{2}-I_{1})}.$

From (13), we have that

$\text{var}(\hat{\lambda})=(\sigma^{2}/n)\frac{I_{1}-I_{2}}{\beta(2I_{2}-I_{1})^{2}}.$

Figure 3 shows the subjective and objective variances as a function of the
biasing coefficient $\beta$. It can be seen that for large $\beta$, the
subjective variance declines whereas the objective variance exceeds it and
increases with $\beta$.

Figure 3: Subjective and actual asymptotic variance for the posterior
distribution mean for the sweet-spot model, as a function of the amount of
bias $\beta$.

###  3.2 Constant variance model

Hence the variance is scaled by $\gamma>1$ if $X>\lambda$, else scaled by
$\beta<1$. We can take $\beta=1$ so that the variance is not altered for
$X<\lambda$, or $\beta=1/\gamma$. Hence

$\text{E}(\ell)/n\rightarrow\left\\{\begin{array}[]{ll}\text{E}\\{-(X-\lambda)^{2}/2\sigma^{2}\beta\\}&\mbox{if
$X\leq\lambda$}\\\ \text{E}\\{-(X-\lambda)^{2}/2\sigma^{2}\gamma\\}&\mbox{if
$X>\lambda,$ }\end{array}\right.$

where constants have been discarded. Differentiating, we have that

$n^{-1}\text{E}(\partial\ell/\partial\lambda)=(1/\gamma-1/\beta)\frac{\exp(-\lambda^{2}/2\sigma^{2})}{\sqrt{2\pi\sigma^{2}}}-(\lambda/\sigma^{2})\\{\frac{\Phi(-\lambda/\sigma)}{\gamma}+\frac{\Phi(\lambda/\sigma)}{\beta}\\},$

whee $\Phi$ is the normal distribution function, and setting the expectation
to zero gives an equation for the asymptotic value of $\lambda$. Also,

$n^{-1}\text{E}(\partial^{2}\ell/\partial\lambda^{2})=-\sigma^{-2}\\{\frac{\Phi(\lambda/\sigma)}{\beta}+\frac{\Phi(-\lambda/\sigma)}{\gamma}\\}.$

There is a solution with $\lambda<0$ for $\gamma>\beta$.

The most efficient method to compute $\lambda$ is to use Newton-Raphson
iteration

$\lambda_{n+1}=\lambda_{n}-\frac{\text{E}(\partial\ell/\partial\lambda)}{\text{E}(\partial^{2}\ell/\partial\lambda^{2})|_{\lambda=\lambda_{n}}}.$

This typically converges in 5 or 6 iterations, starting from $\lambda=0$.
Since the equation $\text{E}(\partial\ell/\partial\lambda)=0$ contains only
the ratio $\lambda/\sigma$, it follows that $\lambda\propto\sigma$.

The variance of the posterior distribution is

$v=\frac{\sigma^{2}/n}{\frac{\Phi(\lambda/\sigma)}{\beta}+\frac{\Phi(-\lambda/\sigma)}{\gamma}}.$

This can be greater or smaller than $\sigma^{2}/n$. For $\gamma>\beta$ it is
straightforward to derive lower and upper bounds $v>(\sigma^{2}/n)/2\gamma$,
$v<2\gamma(\sigma^{2}/n)$.

The true variance of $\hat{\lambda}$ can be found from
$\partial\ell/\partial\lambda=0$ by applying (13). This yields

$\hat{\lambda}-\lambda=n^{-1}\frac{\sum_{i=0}^{n_{1}}(X_{i}-\lambda)/\beta+\sum_{j=1}^{n_{2}}(Y_{j}-\lambda)/\gamma}{n_{1}/\beta+n_{2}/\gamma},$

where the $X_{i}<\lambda$ and $Y_{j}>\lambda$. After some manipulation and
simplification, this yields

$\text{var}(\hat{\lambda})=n^{-1}\frac{\\{\sigma^{2}(\Phi(\lambda/\sigma)/\beta^{2}+\Phi(-\lambda/\sigma)/\gamma^{2}\\}-\lambda^{2}/\beta\gamma}{\\{\Phi(\lambda/\sigma)/\beta+\Phi(-\lambda/\sigma)/\gamma\\}^{2}}.$

This reduces to $\sigma^{2}/n$ when $\beta=\gamma$.

The influence function is derivable as

$n\Delta\hat{\lambda}=\frac{Y-\lambda}{(\gamma/\beta)\Phi(\lambda/\sigma)+\Phi(-\lambda/\sigma)}.$

This is simply a straight line.

###  3.3 Log-gamma distribution model

Here the subjective distribution for $X$ is log-gamma (see e.g. Johnson et al
1995), so that the log-likelihood is

$\ell=\sum_{i=1}^{n}\\{1-\exp(-\beta(X_{i}-\lambda)/\sigma)-(\beta/\sigma)(X_{i}-\lambda),$
(14)

and tends to

$\ell/n\rightarrow\text{E}\\{\frac{1-\exp(-\beta(X-\lambda)/\sigma)-\beta(X-\lambda)/\sigma}{\beta^{2}}\\},$

ignoring constants. As $\beta\rightarrow 0$ this tends to the correct normal
distribution. Setting $\partial\ell/\partial\lambda=0$ gives
$\lambda=-\beta\sigma/2$. From (10) the variance $v=\sigma^{2}/n$, and is
unchanged from the variance that results from using the correct likelihood
with $\beta=0$.

From (13) and using (5), we have that the true variance is

$\text{var}(\hat{\lambda})=\frac{\sigma^{2}}{n}\frac{\exp(\beta^{2})-1}{\beta^{2}},$

From (12), the influence can be derived as

$n\Delta\hat{\lambda}=(\sigma/\beta)\\{1-\exp(-\beta(X-\lambda)/\sigma)\\}.$

Unlike the exponential model case, the influence here always increases with
$X$, although sub-linearly. Figure 2 shows the function.

##  4 Attitude polarization

Consider the probability that the centre of location is less than a threshold
$L$, i.e. $\lambda<L$. Writing $\hat{\lambda}=\lambda+Z\sigma_{\lambda}$,
where $\sigma_{\lambda}$ is the standard deviation of $\hat{\lambda}$ and $Z$
a standard normal random variable, this probability is

$p=\Phi\\{(\hat{\lambda}-L)/\sigma_{\lambda}\\}.$

As $\sigma_{\lambda}\propto n^{-1/2}$, $p$ will tend to 0 or 1 as
$n\rightarrow\infty$, depending on the sign of $\lambda-L$. Hence opinion will
become increasingly polarized after more and more evidence is observed.

##  5 Classification of models

Several models were explored. In general, with directional discounting
whatever the model the bias of the mean of the posterior distribution tends to
a small multiple of the standard deviation $\sigma$ of the observations, and
the variance of the posterior distribution decreases proportionally as
$\sigma^{2}/n$. It may be larger or smaller than $\sigma^{2}/n$, but the
decrease means that attitude polarization can occur. An optimist who tends to
believe the high sales figures and discount the low ones as aberrations will
become increasingly certain that the figures are high, whereas a pessimist
will become increasingly certain of the opposite conclusion.

The following properties can be distinguished:

  1. 1.

The bias tends to an asymptotic limit in the models given here, but if
discounting is strong enough, as in the relative exponential model, it does
not. Hence models can be distinguished by whether or not there is an
asymptotic limit to bias.

  2. 2.

The perceived variance is larger in one direction, but may or may not be
smaller than the true variance of the observations in the other direction. For
the constant variance model with $\beta=1$ low observations are not unduly
prized. However, for the exponential and log-gamma models they are. This also
distinguishes models.

  3. 3.

For some models such as the constant variance model and the log-gamma model,
there is a subjective distribution of observations, whereas for others such as
the exponential model, there is not. Models without a subjective distribution
such as the exponential bias model can give an asymptotic fixed bias however,
as long as the expectation of the log-likelihood function is properly behaved.

  4. 4.

The discounting may be such that higher observations above the mean always
have more influence than lower ones, so that
$\partial\Delta\hat{\lambda}/\partial X>0$, or so extreme that eventually
higher observations have less influence than lower ones. Figure 2 illustrates
the two types of behaviour. This property seems the best way to order models,
based on how much the influence function departs from a straight line. Thus in
increasing order of severity of bias, we would have: the constant variance
model, the log-gamma model, the exponential model, followed by models without
an asymptotic limit to bias.

##  6 Primacy

Earlier observations have more weight. This could be modelled by taking the
perceived variance as $\sigma^{2}(\sigma^{2}/v)^{\xi}$, where if $\xi>0$, the
variance is smaller when the prior distribution has high variance. As
observations accumulate, $v$ decreases and the perceived variance increases.

Given a normally-distributed random variable, the variance of the posterior
distribution $v^{\prime}$ is given by

$\frac{1}{v^{\prime}}=\frac{1}{v}+\frac{1}{\sigma^{2}(\sigma^{2}/v)^{\xi}}.$

We can see how this updating scheme behaves asymptotically when $v$ is small
Then if $u=1/v$, we have approximately that

$\,\mbox{d}u/dt=u^{-\xi}/\sigma^{2+2\xi},$

where $t$, time, is the number of observations. Hence after $n$ observations,

$v=(\sigma^{2\xi+2}/(\xi+1)n)^{1/(\xi+1)},$

showing that the posterior variance decreases as $n^{-1/(\xi+1)}$. As $\xi$
increases from zero, the decrease is slower. Hence because initial
observations have more weight and later ones have less weight, posterior
variance is slow to reduce.

##  7 Conclusions

Some statistical models of confirmation or myside bias have been presented.
Most of the models cover the case where observations are normally distributed,
and observations to one side of the believed value are discounted to some
extent, while those on the other side may be prized (given unduly high
weight). Such models embody the persistence of prior belief, leading
eventually to a posterior mean biased by the order of the standard deviation
of the distribution of observations. The variance of the posterior
distribution decreases as the reciprocal of sample size, so those believing in
high or low results will polarize opinions, each becoming more certain of
their opposite beliefs as evidence accumulates. The primacy effect can also be
accommodated in this framework, where earlier evidence carries more weight
than does later evidence. This could occur if the weight given to an
observation is taken to depend on the extent to which it decreases one’s
ignorance.

This type of model can be applied where rather than an opinion about a
continuous quantity such as blood pressure, we have belief in a proposition,
such as that anthropogenic climate change is occurring. Here we imagine an
underlying observation, and the probability that the proposition is true is
the posterior probability to one side of a fixed point $L$. Alternatively, one
could use the beta distribution model presented earlier.

This type of modelling would be useful for psychological research into
confirmation bias and for assessing its prevalence in an organization. It
would be necessary to elicit belief, e.g. in a proposition as evidence
accumulates. It would then be possible to fit the models given here, and to
estimate the parameters. However, we may still be far from such research, and
the value of this work may be chiefly to stimulate fruitful debate about the
detailed way in which confirmation bias operates.

## References

  * [1] Allahverdyan A. E. and Galstyan A. (2014). Opinion dynamics with confirmation bias, Plos One 9 (7)
  * [2] aker, R. (2021). The irrational persistence of prior beliefs, Mathematics Today, 57 (4), 132-134.
  * [3] French, S. (1991). Recent mathematical developments in decision analysis, IMA journal of Management Mathematics, 3 (1),1-12.
  * [4] French, S. (1995). An introduction to decision theory and prescriptive decision analysis, IMA journal of Management Mathematics, 6 (2),239-247.
  * [5] Gerber, A. and Green, D. (1999). Misperceptions about perceptual bias, Annual Review of Political Science 2,189-210.
  * [6] Gradshteyn I. S. and Ryzhik I. M. (2015). Table of integrals, series, and products, 8th ed., Academic Press, Waltham.
  * [7] Jern, A., Chang, K, K. and Kemp C. (2014). Belief polarization is not always irrational, Psychological Review 121 (2), 206-224.
  * [8] Johnson, N. L., Kotz, S. and Balakrishnan, N. (1995), Continuous univariate distributions vol. 2, Wiley, New York.
  * [9] Kahneman, D. (2012). Thinking, Fast and Slow, Penguin, New York.
  * [10] Rabin, M. and Schrag, J. L. (1999), First impressions matter: a model of confirmatory bias, The Quarterly Journal of Economics, 114 (1), 37-82.
  * [11] Sutherland, S. (2013). Irrationality: the Enemy Within, Pinter & Martin Ltd,Berwick-upon-Tweed.
  * [12] Zimper, A. and Ludwig, A. (2009). On attitude polarization under Bayesian learning with non-additive beliefs, Journal of risk and uncertainty 39, 181-212.

## Appendix: derivation of integrals for the ‘sweet-spot’ model

The integrals $I_{1},I_{2}$ are derived, where

$I_{n}(\beta)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\frac{\exp(-x^{2}/2)\,\mbox{d}x}{(1+\beta
x^{2})^{n}}.$

These are needed for computing subjective and true variances in the ‘sweet-
spot’ model.

The integral $I_{1}$ is given in Gradshteyn and Ryzhik (2015) as result 3.466
(1), albeit in different notation. On changing variable, we have

$I_{n}(\alpha)=\frac{\sqrt{\alpha}}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\frac{\exp(-\alpha
y^{2}/2)\,\mbox{d}y}{(1+y^{2})^{n}},$

where $\alpha=1/\beta$. Write

$J_{n}(\alpha)=\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}\frac{\exp(-\alpha(1+y^{2}/2))\,\mbox{d}y}{(1+y^{2})^{n}},$

and differentiate with respect to $\alpha$ to obtain
$\,\mbox{d}J_{1}/\,\mbox{d}\alpha=-\exp(-\alpha/2)/(2\sqrt{\alpha})$.
Integrating,

$J_{1}(\alpha)=-\int_{0}^{\alpha}\frac{\exp(-z/2)\,\mbox{d}z}{2\sqrt{z}}+J_{1}(0).$

Using $J_{1}(0)=\sqrt{\pi/2}$ and evaluating the integral, we obtain $J_{1}$
and finally $I_{1}$ as

$I_{1}=\frac{\sqrt{2\pi}}{\sqrt{\beta}}\Phi(-\sqrt{1/\beta})\exp(1/2\beta).$

The asymptotic expansion of $\Phi$ shows that $I_{1}\rightarrow 1$ as
$\beta\rightarrow 0$ as it must.

The integral $I_{2}$ is not given in Gradshteyn and Ryzhik. We have
$\,\mbox{d}J_{2}/\,\mbox{d}\alpha=-\frac{1}{2}J_{1}=\sqrt{2\pi}\Phi(-\sqrt{\alpha})$.
From this,

$J_{2}(\alpha)=\frac{\sqrt{\pi}}{\sqrt{2}}\int_{0}^{\alpha}\Phi(-\sqrt{x})\,\mbox{d}x+J_{2}(0).$

Since $J_{2}(0)=\frac{\sqrt{\pi}}{2\sqrt{2}}$, $J_{2}(\alpha)$ can be
evaluated by twice integrating by parts. Finally,

$I_{2}=\frac{1}{2\beta}-\frac{\sqrt{\pi}}{\sqrt{2}}\frac{(1-\beta)}{\beta^{3/2}}\exp(1/2\beta)\Phi(-1/\sqrt{\beta}).$

The asymptotic expansion of $\Phi$ to second order shows that $I_{2}(0)=1$ as
it must.