# Analogist: Out-of-the-box Visual In-Context Learning with Image Diffusion
Model

Zheng Gu<EMAIL_ADDRESS>0000-0001-9914-3922 City University of Hong
Kong and State Key Lab for Novel Software Technology, Nanjing UniversityChina
,  Shiyuan Yang<EMAIL_ADDRESS>0000-0001-8213-5803 City University
of Hong Kong and Tianjin UniversityChina ,  Jing Liao<EMAIL_ADDRESS>0000-0001-7014-5377 City University of Hong KongChina ,  Jing Huo
<EMAIL_ADDRESS>0000-0002-8504-455X State Key Lab for Novel Software
Technology, Nanjing UniversityChina and  Yang Gao<EMAIL_ADDRESS>0000-0002-2488-1813 State Key Lab for Novel Software Technology, Nanjing
UniversityChina

###### Abstract.

Visual In-Context Learning (ICL) has emerged as a promising research area due
to its capability to accomplish various tasks with limited example pairs
through analogical reasoning. However, training-based visual ICL has
limitations in its ability to generalize to unseen tasks and requires the
collection of a diverse task dataset. On the other hand, existing methods in
the inference-based visual ICL category solely rely on textual prompts, which
fail to capture fine-grained contextual information from given examples and
can be time-consuming when converting from images to text prompts. To address
these challenges, we propose Analogist, a novel inference-based visual ICL
approach that exploits both visual and textual prompting techniques using a
text-to-image diffusion model pretrained for image inpainting. For visual
prompting, we propose a self-attention cloning (SAC) method to guide the fine-
grained structural-level analogy between image examples. For textual
prompting, we leverage GPT-4V’s visual reasoning capability to efficiently
generate text prompts and introduce a cross-attention masking (CAM) operation
to enhance the accuracy of semantic-level analogy guided by text prompts. Our
method is out-of-the-box and does not require fine-tuning or optimization. It
is also generic and flexible, enabling a wide range of visual tasks to be
performed in an in-context manner. Extensive experiments demonstrate the
superiority of our method over existing approaches, both qualitatively and
quantitatively. Our project webpage is available at
https://analogist2d.github.io.

Visual In-Context Learning, Diffusion Models, Image Transformation

††ccs: Computing methodologies Image processing

Figure 1. Examples of in-context visual generation by our method using a
pretrained Stable Diffusion Inpainting model are demonstrated. With an example
image pair $A$ and $A^{\prime}$, illustrating a visual transformation, and a
query image $B$, our method enhances the model’s capacity for visual in-
context comprehension, producing a reasonable output $B^{\prime}$ that follows
the same visual pattern. Source images: ImageNet (Deng et al., 2009), LOL
(Chen et al., 2018), InstructPix2Pix (Brooks et al., 2023), TongYi QianWen
APP, UBC-Fashion (Zablotskaia et al., 2019), ScanNet (Dai et al., 2017), DAVIS
(Perazzi et al., 2016), DALLE-3 (Betker et al., 2023).

##  1\. Introduction

As one of the most popular research topics in the recent field of natural
language processing (NLP), in-context learning (ICL) represents a paradigm
wherein large language models (LLMs) acquire the ability to learn tasks based
on a limited set of demonstrative examples (Dong et al., 2022). Unlike
supervised learning, ICL directly generates predictions using pretrained LLMs
(Brown et al., 2020). This paradigm offers an interpretable interface for
interacting with LLMs through language demonstrations, mirroring human
decision-making by learning through analogies and similar experiences. ICL
significantly lowers computational costs for adapting models to new tasks,
making language-model-as-a-service feasible and enabling practical
applications in large-scale, real-world tasks such as machine translation (Xu
et al., 2023), information extraction (He et al., 2023), and complexity
reasoning (Wei et al., 2022).

Following the success of NLP, research in visual In-Context Learning has
entered its embryonic stage of exploration (Yang et al., 2023a; Bai et al.,
2023). Specifically, when the demonstration is a pair of images $A$ and
$A^{\prime}$, visual in-context learning can be considered as an image analogy
problem (Hertzmann et al., 2001). This involves analogizing the observed
transformation from $A$ to $A^{\prime}$ and applying it onto a query image
$B$, resulting in $B^{\prime}$. This analogy capability holds significant
potential in computer graphics and vision tasks (Šubrtová et al., 2023; Parmar
et al., 2023; Cao et al., 2023). For example, as shown in Figure 1, with just
a single pair of examples without training on a large dataset, the pretrained
model can perform tasks ranging from low-level tasks such as colorization,
deblurring, denoising, etc., to high-level tasks such as image editing, image
translation, motion transfer, etc. Visual ICL also offers significant
potential in enhancing creative workflows. Designers can leverage a model to
learn design ideas such as color themes, typography, and visual motifs from an
example pair and adapt them analogously to different contents.

Existing visual ICL works fall into two categories: training-based and
inference-based. Training-based methods train the generative model on diverse
in-context tasks (Wang et al., 2023a; Najdenkoska et al., 2023). The ICL
capabilities primarily exhibit tasks similar to their training tasks and have
limitations when applied to unseen tasks. Moreover, collecting and organizing
the data into in-context task format is laborious. Inference-based methods
conduct ICL via appropriate prompting the model during inference, possessing
better generalizability. However, existing methods (Šubrtová et al., 2023;
Nguyen et al., 2023) convert the given images into textual prompts, falling
short in two aspects. First, the textual prompting is coarse-grained and
cannot cover the detailed information presented in the image examples. Second,
textual inversion from images requires iterative optimization, which is still
time-consuming.

In this work, we propose Analogist, a novel inference-based visual ICL
approach, to address the aforementioned challenges. We introduce both visual
and textual prompting techniques on a pretrained text-to-image diffusion
model.

Firstly, we introduce a novel visual prompting technique to overcome the
coarse-granularity issue in textual prompting. Inspired by MAEVQGAN (Bar et
al., 2022), we formulate the ICL task as an image inpainting task by arranging
the exemplary image pair $A$ and $A^{\prime}$, the query image $B$, and the
unknown image $B^{\prime}$ in a $2\times 2$ grid. Then, we utilize a
pretrained diffusion inpainting model to fill in the region of $B^{\prime}$.
To guide the inpainting process with fine-grained visual contextual
information, we propose a self-attention cloning (SAC) method. This method
clones the self-attention maps between $A$ and $B$ to the self-attention maps
between $A^{\prime}$ and $B^{\prime}$ during the forward propagation of the
diffusion inpainting model. Since the self-attention maps represent similarity
between pixels, the SAC method effectively helps learn structural-level
relationships between $A$ and $B$, which are then applied to $A^{\prime}$ to
generate $B^{\prime}$ analogically.

In addition to visual prompting offering structural-level guidance, we
incorporate textual prompting to offer semantic-level guidance by providing
appropriate text prompts to the inpainting model. However, unlike previous
methods (Šubrtová et al., 2023; Nguyen et al., 2023) that rely on time-
consuming textual inversion optimization, we propose utilizing GPT-4V’s visual
reasoning capability to analyze the semantic transformation between $A$ and
$A^{\prime}$ and apply it analogically to $B$ to generate a textual
description of $B^{\prime}$. This is facilitated by our well-designed
graphical and textual instructions fed into GPT-4V. Furthermore, we introduce
a cross-attention masking (CAM) operation to restrict the interaction between
text and image to the $B^{\prime}$ region only, which ensures that the textual
prompt more accurately guides the generation of $B^{\prime}$.

With both semantic-level (coarse-grained) and structural-level (fine-grained)
contextual information respectively provided by textual and visual prompting
techniques, our approach is capable of performing a wide range of visual tasks
in an in-context manner, as illustrated in Figure 1. Our approach is an out-
of-the-box solution that only requires one forward step of a pretrained
diffusion model, without the need for fine-tuning or optimization. Extensive
experiments and comparisons across different tasks have confirmed that our
method outperforms existing training-based and inference-based visual ICL
methods, both qualitatively and quantitatively. Our method is primarily
designed for applications where the input $A$ and $A^{\prime}$ are spatially
aligned. Nonetheless, we show that it holds promise for applications in
misaligned scenarios as well. In summary, our contributions can be summarized
as follows:

  * •

We introduce Analogist, an out-of-the-box approach for visual in-context
learning that utilizes a pretrained diffusion inpainting model along with
effective visual and textual prompting techniques.

  * •

In visual prompting, we propose a Self-Attention Cloning (SAC) method that
effectively guides the image inpainting model to exploit fine-grained
contextual information in the $2\times 2$ grid visual prompt.

  * •

In textual prompting, we propose to efficiently generate textual prompts using
GPT-4V and enhance the accuracy of textual guidance by introducing a Cross-
Attention Masking (CAM) operation.

##  2\. Related Work

###  2.1. Visual In-context Learning

Inspired by the taxonomy in Dong et al. (2022), we categorize current visual
in-context learning into two groups, training-based and inference-based, based
on the criterion of whether the model is trained on in-context tasks.

##### Training-based Methods

Training-based methods train (or finetune) the model on diverse in-context
tasks. Painter (Wang et al., 2023b) uses paired input and output images as
visual prompts to train a Vision Transformer (Dosovitskiy et al., 2020), which
enables the model to learn and perform a wide range of vision tasks. The
follow-up work SegGPT (Wang et al., 2023c) extends the in-context learning
capabilities of Painter specifically for precise and adaptable segmentation
across various domains. More recently, several work progressively exhibits the
ICL ability of state-of-the-art diffusion models (Rombach et al., 2022).
PromptDiffusion (Wang et al., 2023a) introduces ControlNet (Zhang et al.,
2023) to tune a pretrained Stable Diffusion on six manually designed vision-
language tasks. The proposed method is able to generalize to similar,
contextually related unseen tasks. However, it poses challenge for users to
offer detailed and precise text descriptions. ImageBrush (SUN et al., 2023)
introduces a novel framework for image manipulation using in-context visual
instructions, rather than natural language. An additional prompt encoder is
introduced to translate the visual changes depicted in the example images into
text features to guide the inpainting model. ImageBrush is built on a
diffusion-based inpainting model and trained on several vision datasets. The
above training-based methods necessitate the construction of high-quality and
diverse tasks, making the pipeline laborious and inflexible. Meanwhile, the
test tasks should ideally bear some similarity to the training tasks,
suggesting opportunities for improving generalizability.

Figure 2. Overview of the proposed Analogist. A visual demonstration is
defined by an example pair $A$ (woman holding a cat) and $A^{\prime}$ (the
same woman holding a tiger). Given a new image $B$ (another cat), we format
these three images into a $2\times 2$ grid and tackle this problem by fill the
missing image via a pretrained Stable Diffusion inpainting model. We employ
GPT-4V to provide a proper text description (i.e., “close-up of a tiger’s
face”) to further guide the inpainting process. During the process of model
inference, Self-Attention Cloning (SAC) and Cross-Attention Masking (CAM) are
introduced to encourage the model concentrate on the visual and textual
prompts, thus enhance its in-context learning capacities. Source image:
InstructPix2Pix (Brooks et al., 2023).

##### Inference-based Methods

Instead of tuning the model parameters, inference-based methods inspire the
model’s understanding on the given demonstrations during inference time. Among
them, MAEVQGAN (Bar et al., 2022) innovatively proposes a visual prompting
format of inpainting the missing patch in a $2\times 2$ grid-like image. The
model is pre-trained on figures from computer vision papers which are
typically in a regular grid pattern and emerges with ICL capability. However,
the generation effects are not entirely satisfactory due to limitations in
dataset size and model capacity in comparison with the latest diffusion
models. VISII (Nguyen et al., 2023) considers the demonstration as images
before and after image editing. This approach estimates the editing
instruction based on a pretrained text-based image editing model (Brooks et
al., 2023), producing results with higher quality. However, reverse-
engineering the textual description of the differences between two images
through optimization remains time-consuming. What’s more, by transferring
visual information to coarse-grained text, the generation process is merely
driven by textual descriptions. The role of visual prompting is not fully
leveraged, leading to inaccurate contextual understanding.

Our work falls into the category of inference-based methods and, notably,
eliminates the need for additional optimization steps. Instead of solely
relying on textual prompts, our approach leverages both textual and visual
prompting. This allows us to respectively understand semantic-level and
structural-level contextual information for visual ICL. Besides, our method
utilizes GPT-4V to get textual prompts instead of textual inversion.

###  2.2. Image Analogies

Defined by $A:A^{\prime}::B:B^{\prime}$, the goal of image analogies
(Hertzmann et al., 2001) is to find an “analogous” image $B^{\prime}$ that
relates to $B$ in the same way as $A^{\prime}$ relates to $A$. Such idea can
be extended in various ways of image synthesis (Diamanti et al., 2015;
Jamriška et al., 2019; Liao et al., 2017; Yuan et al., 2024). Recently, DIA
(Šubrtová et al., 2023) investigates the image analogies task with Diffusion
model. This method estimates the CLIP features of the given images. The CLIP
features are injected into a pretrained text-to-image diffusion model to
provide in-context guidance. DIA is capable of executing example-based image
editing that encompasses complex, higher-level contextual or structural
relationships. However, since the goal of CLIP is to align image and text
spaces, the estimated features are high level and struggle to capture detailed
image information.

Our work aims to tackle the problem of image analogies in the paradigm of
visual in-context learning. Different from traditional texture synthesis
approaches (Hertzmann et al., 2001; Liao et al., 2017), the analogy is
achieved by prompting a pre-trained text-to-image diffusion model and can be
applied to more applications such as low-level tasks, manipulation tasks, and
vision tasks.

###  2.3. Prompt-based Image Editing

Recent multimodal approaches have demonstrated superior text-image feature
alignment capabilities (Radford et al., 2021; Li et al., 2022), leading to a
series of works on prompt-based image editing. Previous GAN-based methods
perform manipulation in the latent space via GAN inversion (Xia et al., 2022;
Patashnik et al., 2021; Baykal et al., 2023). More recent methods utilize
text-to-image diffusion models to attain leading outcomes (Cao et al., 2023;
Brooks et al., 2023; Parmar et al., 2023). However, these methods struggle to
do image analogy task since they take textual descriptions as input, which is
not sufficiently intuitive and accurate to depict details related to the image
structure. In contrast, our work takes a pair of images as demonstration
input, utilizes self-attention to provide structure-related information, and
automatically acquires the corresponding textual description through GPT-4V.

##  3\. Preliminary

Since our approach utilizes a pretrained Stable Diffusion inpainting model, we
briefly review latent Stable Diffusion in Section 3.1 as well as the Stable
Diffusion inpainting model in Section 3.2.

###  3.1. Latent Diffusion Models.

Denoising Diffusion Probabilistic Models (DDPM) (Ho et al., 2020) are a class
of generative models that gradually convert random noise into structured data
through a series of reverse diffusion steps based on a Markov chain. Latent
Diffusion Models (LDM) like Stable Diffusion (SD) (Rombach et al., 2022)
enhances DDPM by employing an encoder $E$ to map high-dimensional data $x$
into lower-dimensional latent space $z=E(x)$. The generation of Stable
Diffusion can be guided by an additional text embedding $c(y)$ encoded by CLIP
(Radford et al., 2021) and a text prompt $y$. During training, an UNet model,
parameterized by $\theta$, is optimized to eliminate the noise $\epsilon$
introduced into $z_{t}$:

(1) $\mathcal{L}=\mathbb{E}_{z\sim
E(x),y,\epsilon\sim\mathcal{N}(0,1),t}\left[{\left\|{\epsilon-\epsilon_{\theta}(z_{t},t,c(y))}\right\|}^{2}_{2}\right].$

During inference, a randomly sampled latent $z_{T}\sim\mathcal{N}(0,1)$ is
progressively denoised through the model to produce a clean latent
representation $z_{0}$ by

(2)
$z_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left[z_{t}-\frac{1-\alpha_{t}}{1-\sqrt{\bar{\alpha}_{t}}}\epsilon_{\theta}\left(z_{t},t,c(y)\right)\right],$

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{t}$. Subsequently, the clean
latent is fed into the decoder to obtain the generated image $D(z_{0})$.

###  3.2. Stable Diffusion Inpainting Model

We apply our method over the pretrained Stable Diffusion inpainting model,
which is fine-tuned to boasts an additional feature of image inpainting. The
forward process of the inpainting pipeline is as follows:

(3)
$z_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left[z_{t}-\frac{1-\alpha_{t}}{1-\sqrt{\bar{\alpha}_{t}}}\epsilon_{\theta}\left(z_{t},t,c(y),E(I_{m}),M\right)\right],$

The UNet is updated to include five extra input channels – four dedicated to
the encoded masked-image $E(I_{m})$ and one for the mask $M$ itself. These two
extra inputs are concated with $z_{t}$ to fed into the UNet to predict the
noise at each time step.

##  4\. Method

The goal of ICL is to encourage pretrained model to learn tasks given only a
few examples in the form of demonstration (Dong et al., 2022). Specific to the
image domain, the demonstration is defined as an example image pair $A$ and
$A^{\prime}$, where $A^{\prime}$ is the result obtained by applying a certain
visual effect or transformation to $A$. Given a new query image $B$, the model
is expected to apply the same effect to $B$, thus creating a new image
$B^{\prime}$, so that $A:A^{\prime}::B:B^{\prime}$ (Hertzmann et al., 2001).
This process demonstrates the model’s understanding and replication of visual
transformations from a given demonstration to a new context, exhibiting the
ICL ability.

As illustrated in Figure 2, to address this issue, we approach it from both
visual structural-level (Section 4.1) and textual semantic-level (Section 4.2)
perspectives. For visual prompting (red region in Figure 2), we formulate the
input images into a 2x2 grid image, utilizing a pretrained diffusion
inpainting model to fill in the missing region in Section 4.1.1. To introduce
more fine-grained visual information, we propose Self-Attention Cloning (SAC)
in Section 4.1.2. For textual prompting (blue region in Figure 2), GPT-4V is
elaborated to provide semantic-level guidance to the generation process in
Section 4.2.1. To foster semantic correspondence between the inpainted image
and the text prompt, we propose Cross-Attention Masking (CAM) in Section
4.2.2.

Figure 3. Visualization of the attention relationships. Given an anchor point
on image $A$ (shown in red, green, and blue colors), we calculate the
attention values between this point and all regions of image $B$. Soucre
image: InstructPix2Pix (Brooks et al., 2023).

###  4.1. Visual Prompting

To introduce fine-grained structural-level visual guidance in the in-context
inference process, we construct a visual prompt in the form of a $2\times 2$
grid-like image for the pretrained inpainting model, and provide visual
contextual information by cloning the self-attention associations between the
given images.

####  4.1.1. $2\times 2$-grid Prompting

Image inpainting models fill in unknown areas of an image based on its known
regions, which naturally aligns with the concept of ICL. As shown in Figure 2,
to take advantage of this property, we first rearrange the input images $A$,
$A^{\prime}$, and $B$ into a single $2\times 2$ grid-like image, denoted as
$I$. Image $B$ is pasted to the bottom right corner of the grid image, getting
image $I^{\prime}$. We extract the features of the pasted image,
$E(I^{\prime})$, and add noise to it via diffusion forward process, getting
the initial $x_{T}$. To align with the interface of the pretrained model, a
mask image $M$ is simultaneously generated. In this mask, the bottom right
region is entirely ones, while the remaining regions are zeros. At each
timestep $t$, the latent $x_{t}\in\mathbb{R}^{b\times 4\times h\times w}$ is
concatenated with the feature $E(I)\in\mathbb{R}^{b\times 4\times h\times w}$
and mask $M\in\mathbb{R}^{b\times 1\times h\times w}$, constructing the input
of the UNet. By establishing such a $2\times 2$-grid prompt, we encourage the
model to fill in the content of unknown area ($B^{\prime}$) based on the
contextual regions ($A$, $A^{\prime}$, and $B$) in the image.

Figure 4. Detailed illustration of self-attention cloning (SAC). The sub self-
attention map $\mathcal{M}_{s}(A^{\prime},B^{\prime})$ is set as the value of
$\mathcal{M}_{s}(A,B)$, denoting cloning the relation between $A$ and $B$ to
that of $A^{\prime}$ and $B^{\prime}$.

####  4.1.2. Self-Attention Cloning

The key of in-context learning is to recognize task instruction from the given
demonstration. Previous inference-based work extract the visual instructions
through cross-attention injection, which could only provides coarse and
imprecise guidance. Differently, we introduce fine-grained structural-aware
contextual information via self-attention.

Our motivation comes from an observation that the Diffusion model accurately
constructs associations between different positions in the known areas through
self-attention. We show the visualization of self-attention relations in
Figure 3. We calculate the attention values between key semantic positions
(e.g., the eyes, mouth, and flower in the first row and the spire, building,
and the background grassland in the second row) in $A$ and all regions in $B$.
The results demonstrate that the visual associations between images can be
accurately identified through self-attention, which could be more accurate
than abstract semantic text prompts as guidance. Based on this observation, we
propose to use self-attention as a structural-level prior to guide the in-
context generation procedure by modulating self-attention in UNet. We show an
example in Figure 2 of translating a cat into a tiger. The relative positional
relationship of the tiger in $B^{\prime}$ and the tiger in $A^{\prime}$ should
be consistent with the relative positional relationship of the two cats in $B$
and $A$.

We present detailed illustration of the proposed self-attention cloning (SAC)
in Figure 4. Denote the image feature before self-attention as
$F_{i}\in\mathbb{R}^{h\times w\times c}$. The self-attention map
$\mathcal{M}_{s}\in\mathbb{R}^{hw\times hw}$ records the similarity of each
position on the entire image with other positions, which also includes the
similarities between $A$ and $B$, as well as between $A^{\prime}$ and
$B^{\prime}$. We extract the sub self-attention map
$\mathcal{M}_{s}(A,B)\in\mathbb{R}^{\frac{hw}{4}\times\frac{hw}{4}}$ and
assign its value to
$\mathcal{M}_{s}(A^{\prime},B^{\prime})\in\mathbb{R}^{\frac{hw}{4}\times\frac{hw}{4}}$:

(4) $\mathcal{M}_{s}(A^{\prime},B^{\prime}):=\mathcal{M}_{s}(A,B)\cdot s,$

where $s$ is a coefficient used to balance the degree of preserving the
structure of image $B$ and the degree of applying transformations. We perform
the self-attention cloning operation before softmax to prevent the original
self-attention results being excessively affected.

Figure 5. Detailed illustration of cross-attention masking (CAM). The sub
cross-attention map between text embedding and regions $A$, $A^{\prime}$, and
$B$ are set to zero, making the semantic guidance more focused on region
$B^{\prime}$.

###  4.2. Textual Prompting

Cloning self-attention effectively manages basic in-context visual guidance,
yet the diffusion model’s celebrated text-to-image feature remains
underutilized to provide semantic-level guidance. To address this, we utilize
GPT-4V’s visual reasoning abilities (Yang et al., 2023a) to provide semantic
guidance to the inpainting model.

####  4.2.1. GPT-4V Prompting

We prompt GPT-4V to generate a coherent text description to aid the inpainting
process. Considering the consistency of the entire pipeline, we feed the whole
$2x2$ grid-like image directly into GPT-4V with a pre-designed problem
Description, as depicted in Figure 2. We employ two carefully-designed
graphical instructions to make it easier for GPT-4V to understand the task.
Firstly, inspired by (Yang et al., 2023b), we place a letter mark ($A$,
$A^{\prime}$, $B$, $B^{\prime}$) in the top-left corner of each grid cell.
Secondly, we add prominent arrow markers ($\rightarrow$) between $A$ and
$A^{\prime}$, as well as between $B$ and $B^{\prime}$, to indicate the
relationship between the two images. These approaches introduce structured,
easily identifiable reference points, facilitating more effective and accurate
responses to queries involving visual content. Then, GPT-4V is asked to
perform an analogy and output the text description for $B^{\prime}$. Finally,
we use GPT-4V’s answer as the semantic-level positive text prompt to reinforce
the model’s ICL capabilities. We also employ negative text prompts (i.e.,
“Messy, Disordered, Chaotic, Cluttered, Haphazard, Unkempt, Scattered,
Disheveled, Tangled, Random”) to prevent the diffusion model from generating
irregular and illogical results. These two prompts work cooperatively to
inject semantic-level guidance into the model.

Figure 6. Comparison with other baseline methods, each row indicates one task,
given the input image pair $A$, $A^{\prime}$ and query image $B$. Since
MAEVQGAN (Bar et al., 2022) does not take text as input and DIA (Šubrtová et
al., 2023) and VISII (Nguyen et al., 2023) estimate the text prompts by extra
optimization, the text prompts generated by GPT-4V prompting are only used by
PromptDiffusion (Wang et al., 2023a) and Analogist. Source images: ImageNet
(Deng et al., 2009), LOL (Chen et al., 2018), InstructPix2Pix (Brooks et al.,
2023), UBC-Fashion (Zablotskaia et al., 2019), ScanNet (Dai et al., 2017),
DAVIS (Perazzi et al., 2016).

Figure 7. Examples of results generated by the proposed Analogist on different
tasks. In each example, the image $A$ and $A^{\prime}$ are shown in the first
column, the image $B$ and generated image $B^{\prime}$ is shown in the second
and third column. The text prompt generated via GPT-4V is shown below each
example. Source ImageNet (Deng et al., 2009), InstructPix2Pix (Brooks et al.,
2023), ScanNet (Dai et al., 2017), DAVIS (Perazzi et al., 2016).

Figure 8. Comparison with ImageBrush (SUN et al., 2023). The result of
ImageBrush in the first three tasks are from the original paper and the result
of the last three tasks are provided by the authors of ImageBrush. Source
images: InstructPix2Pix (Brooks et al., 2023), UBC-Fashion (Zablotskaia et
al., 2019), ScanNet (Dai et al., 2017), DAVIS (Perazzi et al., 2016).

####  4.2.2. Cross-Attention Masking

Note that the prompt obtained from GPT-4V is specifically tailored for
$B^{\prime}$, yet the textual guidance impacts the entire image through cross-
attention in the UNet. To address this issue, we propose cross-attention
masking (CAM): in cross-attention layers, we restrict the text interacts only
with the region corresponding to $B^{\prime}$. Specifically, suppose the
cross-attention map as $\mathcal{M}_{c}\in\mathbb{R}^{hw\times L}$, where $L$
denotes the length of text embedding. We repurpose the indices of different
regions identified in the previous SAC process and set the attention values
between the text and regions other than $B^{\prime}$ (i.e., $A$, $A^{\prime}$,
and $B$) to zero:

(5)
$\mathcal{M}_{c}(A):=0;\mathcal{M}_{c}(A^{\prime}):=0;\mathcal{M}_{c}(B):=0.$

As illustrated in Figure 5, we utilize the attention map post-softmax, as we
are completely obstructing the relationship between the text and regions
outside of $B^{\prime}$.

As for the attention map indexing in SAC and CAM, due to the fixed positions
of each image, we are able to pre-calculate the indices required for
extracting the necessary sub-attention maps (e.g., $\mathcal{M}_{s}(A,B)$ and
$\mathcal{M}_{c}(A)$) from the entire attention map. This pre-determination
streamlines the entire pipeline, enhancing its simplicity and efficiency.

##  5\. Experiments

###  5.1. Implementation Details

We implement our work in PyTorch (Paszke et al., 2019). The input images $A$,
$A^{\prime}$, $B$ are resized to $256\times 256$ and spatially combined to
form a $512\times 512$ grid-like image. We used a publicly available Stable
Diffusion inpainting model111https://huggingface.co/runwayml/stable-diffusion-
inpainting. The model is initialized with SD1.2 and trained on inpainting
task, therefore capable of inpainting the missing areas specified by a mask.
The UNet architecture contains 16 blocks, each consists of one cross-attention
and one self-attention. We perform SAC and CAM from layer 3 to 10 at all
timesteps in the diffusion process. The scale for classifier-free guidance is
set at $15$. The coefficient for self-attention cloning $s=1.3$ in all
experiments except for skeleton-to-image where $s=1.4$. All experiments are
conducted on an RTX 3090 GPU.

###  5.2. Evaluation Setup

##### Dataset

We employ the following three major categories, totaling ten tasks to evaluate
the effectiveness of the proposed method quantitatively: low-level tasks,
manipulation tasks, and more challenging vision tasks.

  * •

Low-level tasks. We test out method on four low-level tasks, i.e., image
colorization, image deblurring, image denoising, and image enhancement. For
the first three tasks, we sample in-the-wild images from ImageNet (Deng et
al., 2009) and apply corresponding transformations (i.e., grayscale, gaussian
blurry, adding noise). For image enhancement, we use the LOL dataset (Chen et
al., 2018), which consists of low/normal-light image pairs. We collect 100
samples for each low-level task.

  * •

Manipulation tasks. We select three kind of image manipulation tasks (i.e.,
image editing, image translation, and style transfer) from the CLIP-filtered
subset processed by InstructPix2Pix (Brooks et al., 2023). Since the dataset
is constructed for general image editing, we split the samples into three
tasks based on the keywords. Instructions containing “add”, “remove” are
considered as image editing tasks, those with “make, turn, change” are image
translation tasks. Each manipulation task contains 200 samples.

  * •

Vision tasks. We select three more challenging vision tasks for evaluation:
skeleton-to-image generation from UBC-Fas-hion (Zablotskaia et al., 2019),
mask-to-image generation from ScanNet (Dai et al., 2017), and image inpainting
from DAVIS dataset (Perazzi et al., 2016). Each task contains 200 samples.

By developing these three major categories, we can evaluate if the pretrained
model is capable of understanding, processing, and utilizing visual
information across various levels, while also evaluating its ability to
generalize effectively across these tasks.

##### Baseline methods

We take four methods, MAEVQGAN (Bar et al., 2022), PromptDiffusion (Wang et
al., 2023a), DIA (Šubrtová et al., 2023) and VISII (Nguyen et al., 2023) as
our baseline. All baseline methods utilize the official implementations and
checkpoints provided. Since PromptDiffusion (Wang et al., 2023a) requires text
as part of its input, but most of the test datasets (such as low-level) do not
have paired text descriptions, we input the same text prompts as ours that
obtained from GPT-4V into PromptDiffusion (Wang et al., 2023a) to ensure a
fair comparison.

##### Evaluation Metrics

We evaluate the model’s ICL capacity via the CLIP direction similarity between
the demonstration and the produced results. We utilize the Image Encoder from
CLIP to extract the image features of $A$, $A^{\prime}$, $B$, and the
generated $B^{\prime}$. Then, we calculate the cosine similarity between the
directional changes from $A$ to $A^{\prime}$ and from $B$ to $B^{\prime}$. The
higher the similarity, the more consistent the inferred $B^{\prime}$ is with
the transformation effects applied to $A$. Due to the generation diversity of
diffusion models, we do not compare pixel-level metrics like SSIM and PSNR.
Instead, we calculate FID between the generated $B^{\prime}$ images and the
ground truth images. In order to obtain more accurate result, we merge all the
data in each major category to calculate the FID values for comparison.

###  5.3. Qualitative Results

Figure 6 presents comparison of our method with the baselines on all of the
ten tasks. For MAEVQGAN (Bar et al., 2022), due to the lack of specific
structuring of training data into the form of tasks and the absence of textual
guidance, the quality of the generated output is relatively poor, especially
for high-level tasks like manipulation. For PromptDiffusion (Wang et al.,
2023a), the bias in training task (i.e., image-to-HED, HED-to-image)
significantly impacts the ICL generalizability of the model. As shown in the
example of deblur and translation, the results tend to produce line drawings
similar with edge detection results. For the other two inference-based methods
DIA (Šubrtová et al., 2023) and VISII (Nguyen et al., 2023), they conduct in-
context learning through the estimated text solely, making it difficult
provide sufficiently accurate prompt information to generate the correct
results. Our method takes into account guidance at both the visual and
semantic levels, which can produce accurate and reasonable in-context outputs.
Notice that GPT-4V prompting may struggle with vision tasks, giving coarse
descriptions. For example, “person in dress standing” in the skeleton-to-image
example does not give the detailed description that what pose the woman should
be standing in. However, thanks to the proposed SAC operation, these
structure-aware in-context information can be still captured and utilized to
produce the correct results. Figure 7 shows further results of Analogist on
these tasks, demonstrating the ICL capabilities of our proposed method. More
randomly selected results are shown in supplementary materials.

Additionally, we conducted a comparison with ImageBrush (SUN et al., 2023).
Since ImageBrush has not released the code, the comparison is made in the
range of training tasks of ImageBrush. As shown in Figure 8, it is worth
noting that our method is more effective at preserving the details in Image
$B$. Especially in manipulation tasks, the color of the aurora, the contour
structure of the animals, and the texture on the clothing are better
preserved. This is because our proposed visual and textual prompting contain
more detailed in-context information. On the three vision tasks, we achieve
competitive results with ImageBrush. Note that our model is not fine-tuned
specifically for these tasks, which demonstrate our superiority of in-context
generalizability as an inference-based method.

Table 1. Quantitative comparison on different category of tasks with previous ICL approaches. We report the cosine similarity between CLIP direction from $A$ to $A^{\prime}$ and from $B$ to $B^{\prime}$. Higher similarity represents more contextually appropriate generated results. The best results are highlighted. Category | Task |  MAEVQGAN |  PromptDiffusion |  DIA |  VISII |  Analogist
---|---|---|---|---|---|---
Low level tasks | Colorization |  0.0558 |  0.1283 |  0.0066 |  0.1061 |  0.1797
Deblur |  -0.0961 |  0.0251 |  -0.1337 |  0.0081 |  0.0608
Denoise |  -0.0389 |  0.1612 |  0.1212 |  0.1098 |  0.2391
Enhancement |  0.1120 |  0.1551 |  -0.1443 |  0.2181 |  0.2251
Manipulation tasks | Image Editing |  0.1600 |  0.1768 |  0.0922 |  0.2181 |  0.1800
Image Translation |  0.2526 |  0.2426 |  0.1617 |  0.2965 |  0.3136
Style Transfer |  0.2274 |  0.2336 |  0.1515 |  0.2687 |  0.2455
Vision tasks | Skeleton-to-image |  0.4452 |  0.6150 |  0.2874 |  0.5201 |  0.7334
Mask-to-image |  0.4467 |  0.3984 |  0.1590 |  0.3071 |  0.5531
Inpainting |  -0.0357 |  0.0014 |  -0.0511 |  0.0619 |  0.1013
Average |  |  0.1529 |  0.2137 |  0.0650 |  0.2104 |  0.2832
Table 2. Comparison of FID between the generated $B^{\prime}$s and the ground-truth images. The best results are highlighted. Our method outperforms previous methods in terms of all the three task categories. Method | Low-level | Manipulation | Vision
---|---|---|---
MAEVQGAN | 181.48 | 143.19 | 169.74
PromptDiffusion | 180.39 | 111.79 | 159.02
DIA | 173.10 | 103.39 | 191.51
VISII | 140.39 | 88.36 | 138.44
Analogist | 114.15 | 85.67 | 96.67
Table 3. User study results. In each task, we report the average percentage of selected result by the users. The best results are highlighted. Our approach garnered the highest number of selections. Method | Low-level | Manipulation | Vision
---|---|---|---
MAEVQGAN | 3.51% | 3.45% | 0.87%
PromptDiffusion | 5.33% | 14.99% | 9.09%
DIA | 4.88% | 3.32% | 0.43%
VISII | 20.18% | 18.30% | 15.58%
Analogist | 66.10% | 59.95% | 74.03%

###  5.4. Quantitative Comparisons

##### CLIP Direction

We compute the following CLIP direction similarity,
$cos[(\mathcal{E}(B^{\prime})-\mathcal{E}(B)),(\mathcal{E}(A^{\prime})-\mathcal{E}(A))]$,
to evaluate how faithfully the transformations provided by the model adhere to
the transformations contained in the given examples. The results are shown in
in Table 1. Note that VISII (Nguyen et al., 2023) achieves acceptable results
in manipulation tasks since the model it utilizes is pretrained on this ip2p
dataset (Brooks et al., 2023). Overall, our method demonstrates superior ICL
capabilities across all these tasks.

##### Fréchet inception distance (FID)

We calculate FID between generated images and ground truth on the entire major
category. The results are shown in Table 2. The proposed Analogist outperforms
all baselines across the three major tasks. Notice that VISII (Nguyen et al.,
2023) outperforms other baselines on manipulation tasks. This is because VISII
leverages an InstructPix2Pix (Brooks et al., 2023) model which is pretrained
on the same dataset, making it more familiar with generating data of similar
quality.

##### User Study

We conduct a user study to evaluate the perceptual performance of our method.
The user study consisted of 50 questions, with 42 participants involved,
containing all of the 10 kind of tasks. In each question, first, we presented
the participants with images $A$ and $A^{\prime}$, asking them to analyze the
changes between them. Then, we provided image $B$ and tasked them with
predicting the expected transformation of $B$ following the same pattern.
Subsequently, we displayed the outputs generated by different methods for this
task, and the participants were required to select the one they deemed most
consistent with the identified pattern and of the highest generative quality.
We report the average selection result for the three major tasks: low-level
tasks, manipulation tasks, and vision tasks in Table 3. Our proposed method
exhibited the highest rate of being chosen among all of the three tasks.

Figure 9. Ablation on the proposed components. An input $2\times 2$ image grid
is inpainted by: (a) pretrained SD Inpainting model with random noise as
input, (b) initializing $B^{\prime}$ as noised $B$, (c) adding negative
prompt, (d-1) adding self-attention cloning (SAC) by
$\mathcal{M}_{s}(B,B^{\prime}):=\mathcal{M}_{s}(A,A^{\prime})$, (d-2) adding
SAC by $\mathcal{M}_{s}(A^{\prime},B^{\prime}):=\mathcal{M}_{s}(A,B)$, (e)
adding GPT-4V prompting without cross-attention masking (CAM), and (f) adding
CAM (the full approach). Source images: The $1^{st}$ row are generated by
DALLE-3 (Betker et al., 2023) and all others are from InstructPix2Pix (Brooks
et al., 2023).

Figure 10. Ablation on the graphical instructions in GPT-4V prompting. By
adding marks and arrows, the identity and relation of the task becomes more
obvious, making it easier for GPT-4V to produce proper text prompt. Source
images: InstructPix2Pix (Brooks et al., 2023).

Figure 11. Ablation on hyper-parameters. In the first row, lower coefficient
$s$ produces results more like $B$, while higher $s$ transfers more feature of
$A^{\prime}$. In the second row, performing SAC and CAM at middle layers
($16\times 16$) of the UNet achieves balance between structure preserving and
transformation applying. Source images: InstructPix2Pix (Brooks et al., 2023).

###  5.5. Ablation Study

##### Effectiveness of proposed components

To evaluate the effectiveness of the proposed components, we conduct a series
of ablation studies. The ablation results are presented in Figure 9. (a) The
baseline model of pretrained inpainting model generates rough and low-quality
results. (b) By pasting $B$ to the bottom right corner of the grid image, the
outputs are more structurally consistent with $B$. (c) Adding negative prompts
helps to stabilize the generation process and avoid messy results. (d-1)
Crucially, when operating self-attention cloning by
$\mathcal{M}_{s}(B,B^{\prime}):=\mathcal{M}_{s}(A,A^{\prime})$, the model
retains the information from $B$, but is unable to extract accurate context
from $A^{\prime}$ to infer the same transformation result. (d-2) When
executing SAC by
$\mathcal{M}_{s}(A^{\prime},B^{\prime}):=\mathcal{M}_{s}(A,B)$, the model is
required to keep the structural relation between $A$ and $B$ consistent, after
they have been transformed into $A^{\prime}$ and $B^{\prime}$. Thus, we use
(d-2) instead of (d-1). (e) When adding textual prompts from GPT-4V in the
whole grid image, the model rarely focuses the text guidance on the target
inpainting area $B^{\prime}$. (f) Finally, with the proposed CAM, our full
approach not only maintained respectable generation quality but also
successfully identified the necessary visual editing (adding sunglasses),
effects (applying a cubist style), and transformations (changing church into
mosque) for the ICL task.

##### GPT-4V Prompting

We ablate on the designed graphical instructions that used to hint GPT-4V in
Figure 10. Without adding the visual marks on the grid image, GPT-4V may not
know the corresponding relationship of the given images, therefore is unable
to correctly analyze the content according to the instructions. By explicitly
marking the positions of images ($A$, $A^{\prime}$, $B$, and $B^{\prime}$) on
the constructed grid image, GPT-4V conveniently understands the information
contained in the pictures. Meanwhile, the introduced arrows from $A$ to
$A^{\prime}$ and $B$ to $B^{\prime}$ successfully demonstrate the
transformation relations, making it more acceptable for GPT-4V to produce the
ideal response of adding a “pagoda in the snowy forest”. This text prompt will
introduce semantic contextual information for the pretrained model to
understand the task. Note that our method is generic and supports other
vision-language models (Zhu et al., 2023) as well.

Figure 12. Given the same image $A$ and $B$ in the first column, and different
$A^{\prime}$s, our method is able to recognize the contextual relation between
$A$ and $A^{\prime}$ and produce the output $B^{\prime}$ images accordingly.
Source image: $A$ and $B$ are from ImageBrush (SUN et al., 2023).
$\\{A_{1}^{\prime},A_{2}^{\prime},A_{3}^{\prime},A_{4}^{\prime}\\}$ are
generated using MasaCtrl (Cao et al., 2023).

Table 4. Comparison of inference time taken to perform one ICL task for different methods. Compared to existing methods, our method does not require training on a specific task and additional optimization. Method | Inference time
---|---
MAEVQGAN (Bar et al., 2022) | 0.4s
PromptDiffusion (Wang et al., 2023a) | 4s
DIA (Šubrtová et al., 2023) | 258s
VISII (Nguyen et al., 2023) | 685s
Analogist (ours) | 4s

##### Hyper-parameters

We present ablation on the parameter sensitivity of our proposed method in
Figure 11. As for the SAC coefficient $s$, utilizing a smaller $s$ value
($s=0.5$) results in an output more closely resembling the original Image $B$,
whereas a larger value ($s=1.3$) tends to imbue the result with
characteristics of $A^{\prime}$. However, excessively large coefficients
($s=1.8$) leads to an overly unbalanced attention map, which in turn reduces
the quality of generation. We also ablate the selection of UNet layers in
which we perform SAC and CAM. The results indicate that it is necessary to
perform operations simultaneously in both the encoder and the decoder.
Furthermore, if the operations are performed at a shallow level (high
resolution), the outcome is merely a simple replication of some colors and
coarse textures, leading to poor quality. If the operations are performed at a
deeper level (low resolution), the excessive compression of information leads
to the generated result being similar to the original image $B$. In our
experiments, we perform SAC and CAM at a middle level of the UNet layers.

###  5.6. Analysis

##### Different In-context examples

A model with contextual reasoning abilities should be able to produce
different results based on different in-context examples, when given the same
input. To verify that our approach has such capabilities, we conducted the
following experiment as shown in Figure 12. Given the same image $A$ as an
image of wolves, we first translate $A$ into different example outputs
$\left\\{A^{\prime}_{1},A^{\prime}_{2},A^{\prime}_{3},A^{\prime}_{4}\right\\}$
using MasaCtrl (Cao et al., 2023), obtaining different animals like lion,
tiger, dog, and panda. We construct different ICL tasks, keeping the image $A$
and $B$ being the same, while varying the image $A^{\prime}$s. Our method is
able to recognize the translation from $A$ to $A^{\prime}$ accordingly and
generate the corresponding animals in $B^{\prime}$, demonstrating the ICL
capacity of our Analogist.

##### Inference Runtime

In this section, we compare the execution time for different ICL methods
performed once. Our experiment is conducted on an RTX 3090 GPU, and we
calculated the time taken to generate one image. The result is shown in Tab 4.
MAEVQGAN (Bar et al., 2022) is the least time-consuming, taking 0.4 seconds,
since it is generating very few tokens without the need of iteratively
denoising. Our method Analogist takes about 4 second, the same as
PromptDiffusion (Wang et al., 2023a), which is typically the standard sampling
time for Diffusion models, but does not require specific fine-tuning. As for
the previous inference-baesd methods DIA (Šubrtová et al., 2023) and VISII
(Nguyen et al., 2023), it takes rather long time (i.e., 258 seconds and 685
seconds) for these two methods to estimate the CLIP feature and editing
instruction respectively.

Figure 13. Examples of application for tasks where $A$ and $A^{\prime}$ are
aligned. The text prompts generated by GPT-4V is shown below each example.
utput images are highlighted. Source image: Photo-to-caricature images are
from CariMe (Gu et al., 2021). Sketch-to-portrait images are from
DeepFaceDrawing (Chen et al., 2020). Normal-to-RGB images are from Trevithick
et al. (2024). Icon images are from IconShop (Wu et al., 2023).

Figure 14. Illustration of the pipeline for tasks in which $A$ is aligned with
$B$ instead of $A^{\prime}$. We swap the positions of $A^{\prime}$ and $B$ in
the grid image. Through this way, we simplify the problem into aligned tasks.
Source images: generated by DALLE-3 (Betker et al., 2023).

Figure 15. Examples of application for tasks where $A$ and $B$ are aligned.
The text prompts of GPT-4V are shown below each example. Output images are
highlighted. Source images: The example images of the first motion transfer
case are from Chang et al. (2023). The other three example images are
generated by DALLE-3 (Betker et al., 2023).

Figure 16. Examples of application for tasks where $A$, $A^{\prime}$ and $B$
are all misaligned. We test our method without SAC, only CAM is applied.
Output images are highlighted. Source images: MAEVQGAN (Bar et al., 2022).

##  6\. Application

In this section, we extend Analogist to three categories of applications: (a)
$A$ and $A^{\prime}$ are aligned, (b) $A$ and $B$ are aligned, and (c) $A$,
$A^{\prime}$, and $B$ are all misaligned. For (b) and (c), we make adjustments
to our method accordingly.

###  6.1. $A$ and $A^{\prime}$ are aligned

Under the condition that $A$ and $A^{\prime}$ are aligned, we show example of
applications in Figure 13, e.g., photo-to-caricature, sketch-to-portrait,
normal-to-RGB, and icon-to-image tasks. The results show that our method is
able to generate reasonable results on these tasks. Notice that there are
slight structural changes between $A$ and $A^{\prime}$ for photo-to-caricature
and icon-to-image. However, our method is still robust to these minor issues
since we are providing in-context information from both structural and
semantic levels.

###  6.2. $A$ and $B$ are aligned

We make it possible to address tasks where $A$ is aligned with $B$ instead of
$A^{\prime}$. We give an example of object multiplication in Figure 14, where
$A$ contains one brick and $A^{\prime}$ contains a brick stack. This problem
can not be done through our original pipeline. To tackle this problem, we swap
the positions of $A^{\prime}$ and $B$ in the grid image, constructing a new
grid image where $A^{\prime}$ contains one brick and $B$ contains a stack of
bricks. In this way, we simplify the task into one where $A$ and $A^{\prime}$
are aligned again, i.e., changing the task of turning one brick into brick
stack into the task of changing bricks into golden bricks. This strategy can
be applied to tasks like motion transfer and image analogy where $A$ and
$A^{\prime}$ are misaligned in figure 15. We also demonstrate our method’s
ability of addressing tasks with multiple translations like both motion
editing and style transfer, and object multiplication with editing.

###  6.3. $A$, $A^{\prime}$, and $B$ are all misaligned

We extend our method on tasks where $A$, $A^{\prime}$, and $B$ are all
misaligned in Figure 16, such as changing a circle to a square, resizing a big
circle to a smaller one, extrapolating new content of numbers and letters. We
test our method without SAC to prevent incorrect structure guidance. Analogist
produces reasonable results and outperforms MAEVQGAN. It should be pointed out
that the quality of long sequence letter generation still have room to improve
due to notorious tendency of diffusion models to struggle with generating
high-quality text. Nevertheless, we believe these results demonstrate the pre-
trained generative models have ample potential of in-context ability to be
further tapped.

(a) Example of inaccurate prompt by GPT-4V. The expected right prompt is shown
above the image with the critical words marked green. The prompt given by
GPT-4V is shown below with the wrong words in red.

(b) Failure examples of generating unnatural images on which the model is
rarely seen during the pretraining stage, for example, normal maps and
abstract icons.

(c) Example of $A$, $A^{\prime}$, and $B$ are all misaligned, where SAC is not
applicable.

Figure 17. Example of failure cases. (a) GPT-4V fails to accurately deduce the
correct textual prompt from the given grid images when the transformation
(adding a polar bear) or category (elephant, instead of lion) is ambiguous.
(b) The model fails to generate unnatural images like normal maps or icons
even though given the right text prompt. (c) The proposed SAC struggles with
tasks where $A$, $A^{\prime}$, and $B$ are all misaligned. Source image:
Trevithick et al. (2024), IconShop (Wu et al., 2023), and DALLE-3 (Betker et
al., 2023).

##  7\. Limitation

Although our approach enhances in-context learning abilities, it’s important
to consider two possible limitations. Firstly, the inpainting model might be
misled by incorrect text descriptions. In Figure 17(a), when the
transformation from $A$ to $A^{\prime}$ is minor (i.e., the added object in
the first case is small and easily overlooked), GPT-4V fails to recognize it.
The second case shows an style transfer task of drawing “a sketch of
elephant”. However, GPT-4V recognizes the object as a lion instead of an
elephant, leading to inaccurate guidance. The potential solution could be
leaving an interface for users to monitor and customize the text prompts in
real time.

Secondly, the model struggles with producing data that it seldom sees during
the training stage. As shown in Figure 17(b), when asked to produce unnatural
images like normal map and line-drawing icons, the model fails to generate
accurate results since most of its training data are natural RGB images. On
the other hand, it explains our method’s mediocre performance on vision tasks
compared to ImageBrush (SUN et al., 2023). We believe this could potentially
be achieved by demanding a more powerful pretrained base model.

Finally, the proposed self-attention cloning may struggle with scenario in
which $A$, $A^{\prime}$, and $B$ are all misaligned as shown in Figure 17(c).
The structural-level information is not applicable in this case. One possible
solution is to rely on semantic-level information to produce the
transformation as discussed in Section 6.3.

##  8\. Conclusion

Addressing the limitations of inaccurate instruction and tedious optimization
of existing inference-based methods, we introduced Analogist, a novel approach
for visual In-Context Learning (ICL) combining visual and textual prompting.
The proposed method utilizes a text-to-image diffusion model pretrained for
image inpainting, making it an out-of-the-box solution for a wide range of
visual tasks. We innovate with Self-Attention Cloning (SAC) for visual
prompting, enabling fine-grained structural-level analogy, and leverage
GPT-4V’s visual reasoning for efficient textual prompting, supplemented by
Cross-Attention Masking (CAM) for enhanced semantic-level analogy accuracy.
Our approach, without the need for extra training or optimization,
demonstrates superior performance in both qualitative and quantitative
measures, showcasing robust ICL capabilities.

###### Acknowledgements.

This work was supported in part by the National Natural Science Foundation of
China under Grant 62276128, Grant 62192783 in part by the Collaborative
Innovation Center of Novel Software Technology and Industrialization, and a
GRF grant from the Research Grants Council (RGC) of the Hong Kong Special
Administrative Region, China [Project No. CityU 11216122].

## References

  * (1)
  * Bai et al. (2023) Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. 2023.  Sequential Modeling Enables Scalable Learning for Large Vision Models.  _arXiv preprint arXiv:2312.00785_ (2023).
  * Bar et al. (2022) Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. 2022.  Visual prompting via image inpainting.  _Advances in Neural Information Processing Systems_ 35 (2022), 25005–25017.
  * Baykal et al. (2023) Ahmet Canberk Baykal, Abdul Basit Anees, Duygu Ceylan, Erkut Erdem, Aykut Erdem, and Deniz Yuret. 2023.  CLIP-guided StyleGAN Inversion for Text-driven Real Image Editing.  _ACM Transactions on Graphics_ 42, 5 (2023), 1–18.
  * Betker et al. (2023) James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. 2023\.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf_ 2 (2023), 3.
  * Brooks et al. (2023) Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023.  Instructpix2pix: Learning to follow image editing instructions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18392–18402.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in Neural Information Processing Systems_ 33 (2020), 1877–1901.
  * Cao et al. (2023) Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. 2023.  Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 22560–22570.
  * Chang et al. (2023) Di Chang, Yichun Shi, Quankai Gao, Jessica Fu, Hongyi Xu, Guoxian Song, Qing Yan, Xiao Yang, and Mohammad Soleymani. 2023.  MagicDance: Realistic Human Dance Video Generation with Motions & Facial Expressions Transfer.  _arXiv preprint arXiv:2311.12052_ (2023).
  * Chen et al. (2020) Shu-Yu Chen, Wanchao Su, Lin Gao, Shihong Xia, and Hongbo Fu. 2020.  DeepFaceDrawing: Deep generation of face images from sketches.  _ACM Transactions on Graphics_ 39, 4 (2020), 72–1.
  * Chen et al. (2018) Wei Chen, Wang Wenjing, Yang Wenhan, and Liu Jiaying. 2018.  Deep Retinex Decomposition for Low-Light Enhancement. In _British Machine Vision Conference_. British Machine Vision Association.
  * Dai et al. (2017) Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 2017.  Scannet: Richly-annotated 3d reconstructions of indoor scenes. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 5828–5839.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009.  Imagenet: A large-scale hierarchical image database. In _2009 IEEE Conference on Computer Vision and Pattern Recognition_. Ieee, 248–255.
  * Diamanti et al. (2015) Olga Diamanti, Connelly Barnes, Sylvain Paris, Eli Shechtman, and Olga Sorkine-Hornung. 2015.  Synthesis of Complex Image Appearance from Limited Exemplars.  _ACM Transactions on Graphics_ (Mar 2015), 1–14.  https://doi.org/10.1145/2699641
  * Dong et al. (2022) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ (2022).
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Gu et al. (2021) Zheng Gu, Chuanqi Dong, Jing Huo, Wenbin Li, and Yang Gao. 2021.  CariMe: Unpaired caricature generation with multiple exaggerations.  _IEEE Transactions on Multimedia_ 24 (2021), 2673–2686.
  * He et al. (2023) Jiabang He, Lei Wang, Yi Hu, Ning Liu, Hui Liu, Xing Xu, and Heng Tao Shen. 2023.  ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 19485–19494.
  * Hertzmann et al. (2001) Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. 2001.  Image analogies. In _Proceedings of the 28th annual conference on Computer graphics and interactive techniques_.  https://doi.org/10.1145/383259.383295
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Jamriška et al. (2019) Ondřej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. 2019.  Stylizing video by example.  _ACM Transactions on Graphics_ (Aug 2019), 1–11.  https://doi.org/10.1145/3306346.3323006
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _International Conference on Machine Learning_. PMLR, 12888–12900.
  * Liao et al. (2017) Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017.  Visual atribute transfer through deep image analogy.  _ACM Transactions on Graphics_ 36, 4 (2017), 120.
  * Najdenkoska et al. (2023) Ivona Najdenkoska, Animesh Sinha, Abhimanyu Dubey, Dhruv Mahajan, Vignesh Ramanathan, and Filip Radenovic. 2023.  Context Diffusion: In-Context Aware Image Generation.  _arXiv preprint arXiv:2312.03584_ (2023).
  * Nguyen et al. (2023) Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee. 2023.  Visual Instruction Inversion: Image Editing via Image Prompting. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=l9BsCh8ikK
  * Parmar et al. (2023) Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. 2023.  Zero-shot image-to-image translation. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–11.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in Neural Information Processing Systems_ 32 (2019).
  * Patashnik et al. (2021) Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021.  Styleclip: Text-driven manipulation of stylegan imagery. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2085–2094.
  * Perazzi et al. (2016) Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung. 2016.  A benchmark dataset and evaluation methodology for video object segmentation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 724–732.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10684–10695.
  * Šubrtová et al. (2023) Adéla Šubrtová, Michal Lukáč, Jan Čech, David Futschik, Eli Shechtman, and Daniel Sỳkora. 2023.  Diffusion Image Analogies. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–10.
  * SUN et al. (2023) Yasheng SUN, Yifan Yang, Houwen Peng, Yifei Shen, Yuqing Yang, Han Hu, Lili Qiu, and Hideki Koike. 2023.  ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=EmOIP3t9nk
  * Trevithick et al. (2024) Alex Trevithick, Matthew Chan, Towaki Takikawa, Umar Iqbal, Shalini De Mello, Manmohan Chandraker, Ravi Ramamoorthi, and Koki Nagano. 2024.  What You See is What You GAN: Rendering Every Pixel for High-Fidelity Geometry in 3D GANs.  _arXiv preprint arXiv:2401.02411_ (2024).
  * Wang et al. (2023b) Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. 2023b.  Images speak in images: A generalist painter for in-context visual learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6830–6839.
  * Wang et al. (2023c) Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. 2023c.  SegGPT: Towards Segmenting Everything in Context. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 1130–1140.
  * Wang et al. (2023a) Zhendong Wang, Yifan Jiang, Yadong Lu, yelong shen, Pengcheng He, Weizhu Chen, Zhangyang Wang, and Mingyuan Zhou. 2023a.  In-Context Learning Unlocked for Diffusion Models. In _Thirty-seventh Conference on Neural Information Processing Systems_.  https://openreview.net/forum?id=6BZS2EAkns
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022\.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ 35 (2022), 24824–24837.
  * Wu et al. (2023) Ronghuan Wu, Wanchao Su, Kede Ma, and Jing Liao. 2023.  IconShop: Text-Guided Vector Icon Synthesis with Autoregressive Transformers.  _ACM Transactions on Graphics_ 42, 6 (2023), 1–14.
  * Xia et al. (2022) Weihao Xia, Yulun Zhang, Yujiu Yang, Jing-Hao Xue, Bolei Zhou, and Ming-Hsuan Yang. 2022.  Gan inversion: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 3 (2022), 3121–3138.
  * Xu et al. (2023) Canwen Xu, Yichong Xu, Shuohang Wang, Yang Liu, Chenguang Zhu, and Julian McAuley. 2023.  Small models are valuable plug-ins for large language models.  _arXiv preprint arXiv:2305.08848_ (2023).
  * Yang et al. (2023b) Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023b.  Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v.  _arXiv preprint arXiv:2310.11441_ (2023).
  * Yang et al. (2023a) Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. 2023a.  The dawn of lmms: Preliminary explorations with gpt-4v (ision).  _arXiv preprint arXiv:2309.17421_ 9, 1 (2023).
  * Yuan et al. (2024) Liang Yuan, Dingkun Yan, Suguru Saito, and Issei Fujishiro. 2024.  DiffMat: Latent diffusion models for image-guided material generation.  _Visual Informatics_ (2024).
  * Zablotskaia et al. (2019) Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. 2019.  Dwnet: Dense warp-based network for pose-guided human video generation.  _arXiv preprint arXiv:1910.09139_ (2019).
  * Zhang et al. (2023) Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. 2023.  Adding conditional control to text-to-image diffusion models. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 3836–3847.
  * Zhu et al. (2023) Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023.  MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. In _The Twelfth International Conference on Learning Representations_.