Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing

Ekaterina Iakovleva1Equal contribution. Order decided randomly.
Fabio Pizzati2^*
Philip Torr2   Stéphane Lathuilière1

1 LTCI, Télécom-Paris, Institut Polytechnique de Paris

2 University of Oxford

Text-based editing diffusion models exhibit limited performance when the user's input instruction is ambiguous. To solve this problem, we propose Specify ANd Edit (), a zero-shot inference pipeline for diffusion-based editing systems. We use a large language model (LLM) to decompose the input instruction into specific instructions, well-defined interventions to apply to the input image to satisfy the user's request. We benefit from the LLM-derived instructions along the original one, thanks to a novel denoising guidance strategy specifically designed for the task. Our experiments with three baselines and on two datasets demonstrate the benefits of in all setups. Moreover, our pipeline improves the interpretability of editing models, and boosts the output diversity. We also demonstrate that our approach can be applied to any edit, whether ambiguous or not. Our code is public at <https://github.com/fabvio/SANE>.=-1

§ INTRODUCTION

Recent advances in text-to-image generation have attracted a lot of attention in the research community and beyond it. This success is primarily due to development of text-conditioned diffusion models [46, 48, 45, 40, 10] which allow to translate textual concepts, described in natural language in the form of text prompts, into semantically coherent visualizations. Besides image synthesis, text-conditioned diffusion models have demonstrated strong performance on the image editing task [31, 21, 6, 49, 27, 56], where users describe in plain text the editing instructions to apply to input images.

Problem definition. Abstract user instructions may lead existing editing diffusion models to failure (top). solves this problem by decomposing input instructions into specific ones, satisfying the user's request by integrating detailed edits in the editing process (bottom). is completely zero-shot, with no training required.=-1

Despite the remarkable success of text-conditioned editing methods, in this work, we start from the observation that these approaches usually fail to succesfully edit images when the edit prompt provided by the user is ambiguous.
To illustrate this, consider the example in Figure <ref>, which presents an editing task for a picture of a dog with the user instruction “Make the dog cool." What does it mean for a dog to look cool? Is there only one way for a dog to appear cool?
To answer these questions, the editing model requires a nuanced contextual understanding. The same cool adjective would suggest entirely different modifications if the subject were an inanimate object, like furniture, or a landscape. Moreover, there are multiple ways to make the dog look cool (e.g., adding glasses, making it squint, changing the surroundings), all of which are equally valid.
To address the multimodal nature of this task, editing models need reasoning and abstraction capabilities that current editing diffusion models lack [16].

To address ambiguous input instructions, we propose our method Specify ANd Edit (),  which leverages the reasoning capacities and the general knowledge of Large Language Models (LLMs)  [7, 36, 37, 5, 3]. More precisely, breaks down the input edit into a series of specific instructions that, when applied together, transform the input into well-defined editing tasks. Through this process, known as specification, we reverse the abstractions associated with input instructions. By incorporating various specific details, we effectively reduce the overall ambiguity of the input instructions.
After that, we condition a pre-trained editing diffusion model using both the specific and original ambiguous instructions. More precisely, we start with inferring individual noise estimations for each specific instruction using the chosen diffusion model. These noise estimations are then combined using a novel strategy described in method-instruction-combination. Finally, the combined noise, along with the noise predicted from conditioning on the initial ambiguous instruction, is used in a modified classifier-free guidance [23]. This allows to preserve the fidelity to the original user indication, while guiding the process with the specific interventions. Among benefits on performance, this allows us to provide the specific instructions to the user at inference time, potentially raising the interpretability of the editing instruction. Notably, can be applied to an arbitrary pre-trained instruction-based diffusion model in a zero-shot manner.

In short, our contributions are: (i) We propose the first editing method designed specifically to address  ambiguous instructions. (ii) We introduce an LLM-based instruction decomposition pipeline, and a conditioning mechanism for editing diffusion models combining ambiguous and specific instructions, specifically designed for the task. Our approach requires no training. (iii) We perform extensive experiments on two datasets, with three state-of-the-art methods, outperforming all. We provide additional results on the properties of , and ablations.

§ RELATED WORK

Diffusion-based Image Editing
Diffusion models [50, 22, 46, 51] have achieved remarkable results in generative image modeling by representing the generative process as a series of denoising steps. Conditioning these models on text has enabled controllable text-to-image synthesis [40, 46, 45, 48],
as well as development of different diffusion-based editing systems [11, 31, 40, 35, 42, 8]. Some systems rely on image inversion technique [51, 12] to provide edited versions of an input scene [52, 54, 39, 38, 8, 29]. Although effective, these techniques are normally compute-intensive due to the inversion process. The seminal InstructPix2Pix [6] solved this probelm by finetuning a diffusion model on instruction prompts, benefiting from synthetic pairs of images for training. Although many built on this result [49, 18, 56, 27, 55], the impact of ambiguous instructions on instruction-based editing models is still not explored, motivating the proposed .=-1

Large Language Models.
LLMs [7, 36, 37, 5, 3] are not only capable of human-like text completion, but are also successfully solving various reasoning tasks [4, 7, 9, 3, 1, 47]. This is achieved by applying various reasoning and prompting techniques, e.g.
In-Context Learning (ICL) [7], where the model is given a few task examples in the form of demonstration [13].
Another important direction of research is Visual-Language Models (VLMs) [44, 2, 32, 25, 43, 33] which aim to connect visual and language spaces.
In this work, we use multimodal GPT-4o [41] to caption original images, and to decompose ambiguous instructions into sets of specific instructions using ICL and captions.
There are several editing systems relevant to , which jointly finetune diffusion models and VLMs to enhance input instructions [16, 26].
While addressing a similar problem of commonsense reasoning for image editing, these models rely on implicit concept interpretations learned by VLMs. In contrast, is zero-shot and relies on explicit concept decomposition performed by GPT-4o.

Multi-instruction Editing
There are several works that investigate multi-instruction scenarios for text-to-image generation and editing [19, 34, 14, 28, 16, 26, 55]. In image editing, the diffusion model is given a pre-defined set of instructions to follow. FoI [19] extracts a region of interest for each of them, and restricts InstructPix2Pix [6] to remain within the union of these regions. Instead of editing an image in one step, EMILIE [28] applies InstructPix2Pix iteratively, one instruction at a time. To avoid artifacts, the authors propose to remain in the latent space of the diffusion model, and to decode only the last edited latent variable.
CCA [20] builds a multi-agent system that takes a composite instruction as input, then splits it into elementary instructions and iteratively applies them using an LLM, a VLM, and several editing diffusion models.
In this work, we also consider sets of instructions, however, is the first to address instruction decomposition for ambiguous instructions.

inference pipeline. We prompt an LLM to map an ambiguous input instruction $c$ to a set of specific instructions $\mathcal{S}$ (left). We provide a description of $x$ as context, in addition to $c$. Once $\mathcal{S}$ instructions are extracted, we use them in addition to $c$ in the denoising loop of an editing diffusion model (right). For each iteration, we estimate the noise in $z_{t}$ by conditioning the diffusion U-Net $f_\theta$ on all instructions. We them combine all specific instructions in a single noise estimation, that we later use in classifier-free guidance (CFG). We update the noise $z_t$ to $z_{t-1}$ following standard approaches. After $T$ iterations, we obtain the output image $\tilde{x}=\mathcal{D}(z_0)$.=-1

§ METHOD

This section introduces the pipeline. First, we present notations and preliminaries for diffusion-based image editing in Section <ref>. After that, we introduce our LLM-based instruction decomposition in Section <ref>, and our
novel instruction combination strategy in Section <ref>.

§.§ Background on Diffusion-based Image Editing

We consider an instruction-based diffusion model such as InstructPix2Pix [6]. The purpose of such models is to edit an input image $x$ while following a user-defined editing instruction $c$, in order to generate an edited image $\tilde{x}$. The edited image respects the input instruction while preserving the appearance of the original image. Following existing work [6, 56, 27] we use a latent diffusion model [46], where the denoising operation is performed in the latent space of an autoencoder with encoder $\mathcal{E}$ and decoder $\mathcal{D}$. These models typically include a denoising U-Net $f_\theta$.
At inference time, $\tilde{x}$ is produced by iterativly denoising a sampled Gaussian noise $z_T$ with $f_\theta$ over $T$ iterations. In other words, at step $t$, $f_\theta$ is used to estimate the noise $\epsilon_t$ in a corresponding noisy latent $z_t$. The latent $z_t$ is then updated to $z_{t-1}$, by removing the estimated noise $\epsilon_t$ and reintroducing Gaussian noise with lower intensity, following strategies from literature [22, 51, 30].
This is repeated for each $t \in [1, T]$, resulting in the final image $\tilde{x}=\mathcal{D}(z_0)$.

In instruction-based models, noise estimation is conditioned on the instruction $c$, which describes the desired changes, and on the input $x$, to enforce consistency with the original image. This noise is denoted as $\epsilon_t = f_\theta(z_t, \mathcal{E}(x), c)$. In practice, instruction-based diffusion models benefit from classifier-free guidance [23] to boost consistency towards the instruction and the input image [6].
This means that for each $t$, $z_t$ is denoised using $\tilde{\epsilon}_t$, rather than $\epsilon_t$, with $\tilde{\epsilon}_t$ being a combination of three terms: unconditioned, conditioned on the image, and conditioned on the instruction $c$:

\begin{equation}\label{eq:cfgdef}
\tilde{\epsilon}_t = \epsilon^\text{U}_t + \epsilon^\text{I}_t + \epsilon^\text{C}_t,
\end{equation}

\begin{equation}\label{eq:cfg}
\begin{aligned}
&\epsilon^\text{U}_t = f_{\theta}(z_t, \varnothing, \varnothing),\\
&\epsilon^\text{I}_t = w^\text{I}\cdot(f_\theta(z_t, \mathcal{E}(x), \varnothing) - f_\theta(z_t, \varnothing, \varnothing)),\\
&\epsilon^\text{C}_t = w^\text{C}\cdot(\epsilon_t - f_\theta(z_t, \mathcal{E}(x), \varnothing)).
\end{aligned}
\end{equation}

In Equation (<ref>), $\varnothing$ indicates that the conditioning element is replaced with zeros, and $w^\text{I}$ and $ w^\text{C}$ are guidance strength parameters, controlling the conditioning strength on $x$ and $c$, respectively.

§.§ Instruction Specification

We noticed that directly applying ambiguous input instructions as $c$ may lead to limited editing performance. This is due to the ambiguity of $c$ which can be represented by multiple editing interventions. Let us assume that $x$ represents a cat, while $c=\text{``\texttt{make the cat look funny}''}$ is a user instruction, as in Figure <ref> (left).
As mentioned in Section <ref>, we aim to map $c$ with an LLM to a set of specific and interpretable instructions that would satisfy the user's request, e.g. map $c$ to  “”.

Formally, we want to extract from $c$ a set of $N$ editing instructions $\mathcal{S} = \{s_i\}_{i=1}^{N}$, where each $s_i$ is a specific instruction describing one modification consistent with $c$, to apply to the input scene.
As shown in Figure <ref> (left), we prompt an LLM to map $c$ to $N$ specific instructions, providing a rich caption of $x$ as context.
Our prompt contains general descriptions of ambiguous and specific modifications, and two in-context learning examples [7] of ambiguous input instructions associated with desired specific instructions.
Furthermore, we set additional restrictions on the content and formatting style of the output instructions, to preserve image consistency and to simplify parsing. Due to limited space, we report the full prompt in the appendix.
Notably, obtained specific instructions $\mathcal{S}$ are available to the user during model inference, providing insights on how the input instruction $c$ is respected. This also allows to explicitly show how $c$ impacts the input image $x$, which increases interpretability of the image editing process.

§.§ Instructions Combination

After extracting decomposition $\mathcal{S} = \{s_i\}_{i=1}^{N}$, we can use it
to guide the image editing process. Our idea is to include $\mathcal{S}$ in the denoising procedure of $f_\theta$ along the original ambiguous instruction $c$. Intuitively, this
the state of solutions satisfying the required editing $c$, allowing the model to focus on the selected specific interventions $\mathcal{S}$. On the other hand, including $c$ in the denoising process allows the diffusion model to synthesize complementary elements necessary for satisfying the user request, but not included in $\mathcal{S}$. Hence, we propose a combination strategy that aggregate $c$ and each $s_i \in \mathcal{S}$, balancing the influence of the specific instructions without losing consistency with $c$.

We start by conditioning the denoising process on each specific instruction, to isolate their effects. Hence, for each $s_i \in \mathcal{S}$ we extract the estimated noise $\epsilon_{i,t}^s$ at timestep $t$:

\begin{equation}
\epsilon_{i, t}^s = f_\theta(z_t, \mathcal{E}(x), s_i), \quad \forall i \in [1, N].
\end{equation}

This results in the set of estimated noises $\{\epsilon_{i, t}^{s}\}_{i=1}^N$, one for each $s_i \in \mathcal{S}$.
Next, we aim to combine this set of noise estimations into a single noise estimate that aggregates all specific instructions. Simple averaging would diminish the impact of specific instructions that affect particular regions by blending them with other noise estimates. Therefore, we propose an alternative aggregation scheme, assuming that each image region is predominantly affected by a single specific instruction. To identify the spatial locations where the instruction $s_i$ most significantly impacts the diffusion process, we calculate the absolute difference between the estimated noises in the set and the noise obtained with conditioning only on the input image. Formally, it can be written as:

\begin{equation}
\Delta \epsilon_{i,t}^s = |\epsilon_{i,t}^s - f_\theta(z_t, \mathcal{E}(x), \varnothing)|.
\end{equation}

We then aggregate these in a mask $M_t$, capturing the index of the most significant element across $i$:

\begin{equation}
M_t = \arg\max_{i} \Delta \epsilon_{i,t}^s
\end{equation}

Finally, to aggregate the noises $\epsilon_{i,t}^s$ based on the mask $M_t$, we use $M_t$ to select the most significant estimated noise for each spatial location. The aggregated noise $\bar{\epsilon}_t^s$ can be computed as follows:

\begin{equation}
\Bar{\epsilon}_t^s = \sum_{i} \mathbb{I}(M_t = i) \cdot \epsilon_{i,t}^s,\label{eq:masking}
\end{equation}

where $\mathbb{I}(M_t = i)$ is an indicator function that is 1 if $M_t$ equals $i$, and 0 otherwise. Similarly to existing literature [34], we use classifier-free guidance [23] for instruction combination, redefining Equation (<ref>) as:

\begin{equation}
\tilde{\epsilon}_t = \epsilon_t^\text{U} + \epsilon^\text{I}_t + \epsilon^\text{C}_t + \epsilon^\text{S}_t,
\end{equation}

where the classifier-free guidance term for the specific instructions is given by:

\begin{equation}
\epsilon^\text{S}_t = w^\text{s}(\bar{\epsilon}_t^s - f_\theta(z_t, \mathcal{E}(x), \varnothing)).
\end{equation}

The process is shown in Figure <ref>, center. We apply this for every iteration $t \in [1, T]$.

1c    1c    3cEMU-Edit     3cIP2P data

Method     $N$     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     $\text{\textbf{CLIP}}_\Delta\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     $\text{\textbf{CLIP}}_\Delta\uparrow$
4*90IP2P     Baseline     -     0.2923     0.8810     0.1203     0.2903     0.9027     0.1724

3*     1     0.2961     0.7779     0.1705     0.2886     0.8196     0.1999

2     0.2962     0.7654     0.1785     0.2910     0.8107     0.2063

3     0.2968     0.7531     0.1858     0.2935     0.8101     0.2057
4*90MB    Baseline     -     0.2888     0.7858     0.1618     0.2855     0.7934     0.2028
3*      1     0.2948     0.7994     0.1661     0.2848     0.7970     0.1992

2     0.2952     0.8134     0.1669     0.2870     0.8068     0.1996

3     0.3006     0.8209     0.1655     0.2878     0.8056     0.1998
4*90HQEdit    Baseline     -     0.2675     0.6501     0.1417     0.2785     0.7035     0.1848
3*      1     0.2725     0.6546     0.1458     0.2734     0.6997     0.1759

2     0.2789     0.6802     0.1433     0.2782     0.7057     0.1842

3     0.2823     0.6870     0.1474     0.2818     0.7136     0.1855

Editing quality results

GPT evaluation

Quantitative comparison. In tab:editing-quality, we compare against baselines by selecting different editing models and applying on top of them. We show results for $N=\{1, 2, 3\}$, evaluating $\text{CLIP}_\text{d}$, $\text{CLIP}_\text{i}$ and $\text{CLIP}_\Delta$ on all. consistently improve image quality performance across datasets and models. In tab:gpt-eval, we use GPT-4o to evaluate the quality of edited images, always outperforming baselines.

Baseline     $N=1$     $N=2$     $N=3$

Original image

Instructions $\rightarrow$     Change the scene into the alps in winter time     Cover the ground with snow     Add snowy mountain peaks in the background     Add snow-covered pine trees along the track

Instructions $\rightarrow$     Make the photo seem like it was taken in a fancy restaurant     Replace the plate with a gourmet presentation plate     Add a garnish of fresh herbs to the bacon and eggs     Add elegant silverware next to the plate

Instructions $\rightarrow$     Change this to a New York city street     Replace the road with a busy New York city street     Add tall buildings in the background     Add yellow taxis surrounding the green car

Qualitative results. Using on top of IP2P helps to respect the ambiguous instruction (underlined in grey) by adding specific elements into the input scene. We show how increasing the number of specific instructions (orange, cyan, purple) adds important details to the scene, ignored by the baseline. Examples of such details are the snow on the ground (first row), the herbs garnish (second row), and the taxis (third row). Colored boxed in the header indicate the instruction used for each column.=-1

§ EXPERIMENTS

§.§ Experimental Setup

Baselines   We test by adapting the editing diffusion models trained in InstructPix2Pix [6] (IP2P), MB [56] (MB) and HQEdit [27]. We use the default inference hyperparameters for all methods: $w^\text{C}=7, w^\text{I}=1.5$. We select with a grid search $w^\text{S}=7$ for IP2P, $w^\text{S}=5$ for MB, and $w^\text{S}=9$ for HQEdit. We test with $N=\{1, 2, 3\}$, limiting the number of instructions due to increased computational time associated with higher $N$ values. We incrementally build $\mathcal{S}$ for increasing $N$ values, in such a way that $\mathcal{S}$ with lower $N$ are subsets of those with higher $N$. We use 30 diffusion steps for image generation. Images are $512\times512$.

We test on two datasets. We first consider the global split of the EMU-Edit dataset, including 220 real images with ambiguous instructions satisfying our definition [49]. We also introduce a subset of 370 images and editing instructions from the IP2P dataset [6], following related works [19]. We call this set IP2P data. To focus our evaluation on ambiguous instructions, we manually classify the 370 samples. We process instructions in both datasets with GPT-4o to extract $\mathcal{S}$.=-1

Metrics   We evaluate the quality of edited images with three CLIP-based [44] metrics: input image preservation, editing strength and adherence to the edit. First, we use $\text{CLIP}_\text{i}$ to measure input image preservation
as in [19], which is the CLIP space cosine similarity between $x$ and $\tilde{x}$. This captures how similar is $\tilde{x}$ to the original $x$. Then, we measure the editing strehgth $\text{CLIP}_d$ as the cosine distance between the CLIP image embedding of $\tilde{x}$ and the text embedding of the final caption, following [44]. This assesses the fidelity of the edited image $\tilde{x}$ to the final caption. Finally, we use the directional similarity of [17], referred to as $\text{CLIP}_\Delta$, to measure adherence to the edit. For that, we first process each pair $(x, c)$ with GPT-4o to obtain a short initial caption, describing the input scene, and a final caption, describing the desired scene after the editing. For instance, if the initial caption is “” and $c$ is “”, a final caption may be “”. $\text{CLIP}_\Delta$ is evaluated as the cosine similarity between the difference of the CLIP image embeddings of $x$ and $\tilde{x}$, and the difference of the text embeddings of the initial and final captions.
This compares the change in the image to the change in the caption.
In addition to these three metrics, we also use TIFA [24] to evaluate the effectiveness of $s_i \in \mathcal{S}$, as well as LPIPS [57] and DreamSim [15] to evaluate image diversity. Finally, we evaluate pairwise image preferences with GPT-4o.

§.§ Editing Performance

Here, we compare against the state of the art. For each of the three editing diffusion models, we evaluate their performance with and without applied on top of them. We evaluate models on EMU-Edit and IP2P data both quantitatively (Table <ref>) and qualitatively (Figure <ref>).=-1

CLIP Metrics We present results of CLIP-based metrics in Table <ref>. Overall, we observe performance improvement across all metrics, models and datasets, advocating the advantages of . Notably, performance increases with $N$, with models using $N=3$ specific instructions performing best. In particular, we report the biggest improvements in IP2P, where we report for /Baseline 0.1858/0.1203 on EMU-Edit for $N=3$. This suggests that our instruction decomposition helps to follow the ambiguous instruction $c$. Moreover, we observe an increase in image consistency in MB and HQEdit, where we report improvements in $\text{CLIP}_i$ metric. We attribute this behavior to our decomposition strategy. Indeed, while general instructions $c$ alone may lead to ambiguous edits impacting the entire scene, injecting $s_i \in \mathcal{S}$ for inference guides the editing process on spatially-constrained edits. These still convey the desired modifications, as proved by the improvements in $\text{CLIP}_\Delta$ and $\text{CLIP}_\text{d}$. In IP2P, the Baseline reports the highest $\text{CLIP}_\text{i}$.
While it might seem as if Baseline outperforms all other methods, in reality we observe that IP2P may fail to perform any change to $x$ when the input instruction is too ambiguous, thus artificially inflating the $\text{CLIP}_\text{i}$ metric. This is also confirmed by the lower performance in $\text{CLIP}_d$ and $\text{CLIP}_\Delta$.

GPT Evaluation
We use GPT-4o to measure pairwise preference for our against the chosen baselines.
We choose the configuration with $N=3$, since it yields the best performance in Table <ref>. For each image-instruction pair $(x, c)$ on each dataset, we prompt GPT to choose the best edited image $\tilde{x}$ between one of the baselines and . We ask to take into account the fidelity to the editing instruction, the quality and realism of the generated image, and the content preservation from the original picture while making the decision. The original image $x$ is also provided for reference. The prompt is shown in the appendix. We report the average of GPT choices for all model pairs in Table <ref>. As before, we significantly outperform all baselines. In particular, we beat the MB baseline on IP2P data (/Baseline is 66.0%/34.0%). This is especially interesting, considering that it was the only setup where SANE was not outperforming the baseline $\text{CLIP}_\Delta$ in Table <ref>, reporting 0.1998/0.2028.
This result proves that we can benefit from the improved quality of the generated images, even when the fidelity to $c$ is slightly penalized. We attribute this observation to the improved precision of our editing process.=-1

Qualitative Results
We show qualitative results in Figure <ref>. We use IP2P as a baseline model, and we add on top of it with $N=\{1, 2, 3\}$ instructions. We use the same hyperparameters and random seed for the baseline and . gradually adds details with the specific instructions to respect the ambiguous instruction. Interestingly, adding more specific instructions (higher $N$) brings advantages of removing editing artifacts.
This is especially evident in the third row, where the last two specific instructions help to remove the background artifacts generated by IP2P. We believe such artifacts are generated by the ambiguity of the input instruction. Specific instructions help to constrain the solution space for the editing task, improving the robustness to such undesired visual effects.=-1

1c    1c    2cEMU-Edit

Method     $N$     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{$\Delta$}}\uparrow$
4*90IP2P     Baseline     -     0.2860     0.0861
3*     1     0.2854     0.1304

2     0.2851     0.1436

3     0.2878     0.1518
4*90MB    Baseline     -     0.2725     0.1108
3*      1     0.2831     0.1240

2     0.2876     0.1282

3     0.2926     0.1284
4*90HQEdit    Baseline     -     0.2522     0.1090
3*      1     0.2854     0.1238

2     0.2889     0.1306

3     0.2907     0.1326

CLIP metrics

TIFA score

Specific instructions effect. We evaluate how much and baselines respect the instructions in $\mathcal{S}$. With CLIP metrics tab:interp-clip and TIFA tab:interp-tifa, we verify that we improve fidelity to $\mathcal{S}$.=-1

§.§ Properties

In this section, we discuss additional properties of , namely
the impact of instructions in $\mathcal{S}$ (Table <ref>) and its effects on the variability of outputs (Figure <ref>). We use real images from EMU-Edit for all evaluations.

Effects of Specific Instructions
To prove that is working correctly, we need to evaluate whether specific instructions are applied. As mentioned in Section <ref>,
this would also enable interpretability of the editing process:
at inference time, we can provide specific instructions $\mathcal{S} = \{s_i\}_{i=1}^{N}$ to the user, to explain how the editing instruction $c$ has been applied.
For this reason we quantify how much the edited image $\tilde{x}$ respects each instruction in a reference set. We average, for each $\tilde{x}$ and for $N=\{1, 2, 3\}$, $\text{CLIP}_\text{d}$ and $\text{CLIP}_\Delta$ for each $s_i$ in the reference set. We take as reference set the $\mathcal{S}$ obtained with $N=3$, to fairly compare all method and baselines.
Results in Table <ref> show that we consistently and considerably outperform all baselines, demonstrating ability of to effectively apply specific instructions.
As expected, performance increases for higher $N$, with $N=3$ setups consistently outperforming in all scenarios. This means that is exploiting each instruction in $\mathcal{S}$ for editing. Additionally, we use TIFA [24] to generate questions related to each $s_i$ with an LLM. We then evaluate if $s_i$ are applied answering the set of question with visual-question answering. For more details, we refer to [24]. Due to high evaluation costs, we report results only for the setup with $N=3$. Our evaluations in <ref> confirm that correctly exploits the $\mathcal{S}$ instructions.

Quantitative evaluation

Image     Baseline

Pixel difference visualization

Variability. We quantify variability of generated samples with LPIPS and DreamSim, evaluating the average distance to the input image fig:variability-plots. For both, higher values imply higher variability. Above Ours bars, we report $N$. In fig:variability-viz, we show the average pixel difference of original and synthesized images for the reported instruction. Using allows to modify more pixels.

Image Variability
To evaluate variability of edited images, we randomly select 20 input images from EMU-Edit, and produce 10 edited samples for each of them using and baselines. We evaluate the mean LPIPS [57] and DreamSim [15] between input images $x$ and edited images $\tilde{x}$. Higher metrics imply higher variability of the output. LPIPS is particularly sensitive to low-level differences [57], while DreamSim captures semantic variability [15]. Results in Figure <ref> show that we outperform the baseline (Base) for IP2P and HQEdit.
The best performing setups are $N=1$ with HQedit and $N=2$ with IP2P. We speculate that higher $N$ increases the probability of two samples having at least one overlap in generated specific instructions.
We observe that MB [56] baseline outputs highly variable outputs,
which we attribute to the quality of its training set [56]. We display in Figure <ref> the average pixelwise difference between $x$ and $\tilde{x}$ for IP2P and IP2P+.
produces edits that are better distributed spatially, including regions ignored by other models (, the dog fur).=-1

§.§ Ablation Studies

We focus our ablations on instruction combination strategies (Table <ref>), also providing insights on our design choices (Table <ref>)
and on the effectiveness of on different types of input instructions (Table <ref>).=-1

Instruction Combination
We combine specific instructions as presented in Section <ref>. Here, we study the effectiveness of alternative solutions for instruction combination. We first propose a naive Prompt Concatenation baseline, where we concatenate the text of the instruction $c$ with all $s_i \in \mathcal{S}$, using commas as separator. We then perform the editing using the obtained concatenated instruction. We also combine the effects of $c$ and all $s_i \in \mathcal{S}$ with Composable Diffusion [34]. We refer to the original paper [34] for details. For a fair comparison, we set $w^\text{C}$ and $w^\text{S}$ as weights for $c$ and each $s_i$, respectively. We test with $N\!=\!3$. From results in Table <ref>, we infer that our strategy for instruction combination outperforms other strategies. We explain this with the hierarchical nature of instructions: we preserve the fidelity to the original $c$ by design, aggregating the effects of different specific instructions in a single noise estimation.

1c    3cEMU-Edit

Method     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     $\text{\textbf{CLIP}}_\Delta\uparrow$
3*90IP2P     Prompt concat     0.2933     0.8188     0.1439

Comp. Diffusion     0.2944     0.7299     0.1847

Ours     0.2968     0.7531     0.1858
3*90MB     Prompt concat     0.2890     0.7802     0.1639

Comp. Diffusion     0.2990     0.8210     0.1649

Ours     0.3006     0.8209     0.1655
3*90HQEdit     Prompt concat     0.2720     0.6086     0.1512

Comp. Diffusion     0.2753     0.6547     0.1398

Ours     0.2823     0.6870     0.1474

Instruction combination baselines. We outperform two baselines for instruction combination, as shown by CLIP metrics. Only our approach allows to benefit from the distinctions between ambiguous and specific instructions.

Impact of Design Choices
Here, we analyze the effect of several design choices. First, we study the impact of the initial instruction $c$ on the performance. One may argue that, since each $s_i$ is related to $c$, it may be sufficient to apply $\mathcal{S} = \{s_i\}_{i=1}^{N}$ without $c$ to achieve good editing. We test on EMU-Edit with $N=3$ and setting $w^T=0$, removing $c$ guidance. Results in Table <ref> (“SANE w/o $c$”) prove that although w/o $c$ can outperform certain baselines ($\text{CLIP}_\Delta$ = 0.1552 on IP2P), using $c$ for denoising is important to achieve the best performance. Then, we replace the masking operation described in Equation (<ref>) with a naive averaging of the estimated noises
In Table <ref>, we show that this alternative strategy, reported as “w/ avg”, yields lower performance on CLIP metrics.=-1

with Non-ambiguous Instructions
is based on the assumption that the input prompt is ambiguous. We now propose a preliminary evaluation on non-ambiguous instructions to show that: 1) can also perform well on non-ambiguous prompts; 2) The gain brought by is higher on ambiguous instructions, thus validating the motivation behind the design of .

To this aim, we propose the following experiment. We consider the first set of 1199 images from the IP2P dataset which can be either ambiguous or not. We then use GPT-4o to identify ambiguous instructions leading to 696 ambiguous and 503 specific $(x, c)$ pairs. The prompt is reported in the appendix. For both types of pairs (subsets Amb. and Spec.), we process the samples using IP2P with and without on top of it, leading to the results in Table <ref>. We also report the relative change with respect to the baseline.

Experiment on non-ambiguous instructions show that achieves  higher adherence to the input edit (higher $\text{CLIP}_\Delta$) while achieving similar preservation of the input image (similar $\text{CLIP}_d$). It demonstrates that can be effective as a general-purpose editing method, applied to an arbitrary input instruction, and that using LLM can boost the performance.

In addition, the improvement for ambiguous instructions is higher (+17.5% in $\text{CLIP}_\Delta$) compared to improvement for specific ones (+10.8%), supporting the motivation for our work. Notably, the absolute performance on specific instructions is higher, confirming our initial observation that editing models have difficulties with ambiguous inputs. =-1

3c    3cEMU-Edit

Method     $N$     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     $\text{\textbf{CLIP}}_\Delta\uparrow$
0px4*90IP2P    Baseline     -     0.2923     0.8810     0.1203

1*     1*w/o $c$    3    0.2883     0.7796     0.1552

1*     1*w/ $\text{avg}$    3    0.2974     0.8190     0.1587

0.2968    0.7531     0.1858

0px4*90MB    Baseline     -     0.2888     0.7858     0.1618

1*     1*w/o $c$    3    0.2954     0.8795     0.1375

1*     1*w/ $\text{avg}$    3    0.2978     0.8087     0.1681

0.3006    0.8209    0.1655

0px4*90HQEdit    Baseline     -     0.2675     0.6501     0.1417

1*     1*w/o $c$    3   0.2795     0.6983     0.1321

1*     1*w/ $\text{avg}$    3   0.2807     0.6852     0.1442

0.2823    0.6870     0.1474

Design choices. Removing the ambiguous instruction $c$  (SAGE w/o $c$) results in degraded performance, proving that editing models benefit from both specific and ambiguous instructions. Also, replacing the masking operation with averaging (SAGE w/ avg) results in worse editing. We highlight first and second best.=-1

Prompts     Method     1c$\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     1c$\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     1c$\text{\textbf{CLIP}}_\Delta\uparrow$
-3px4*90IP2P   2*Amb.     Baseline     0.2946     0.8992     0.1504

0.2998 +1.76%     0.8225 -8.53%     0.1767 +17.5%
2*Spec.     Baseline     0.3044     0.9000     0.2012

0.3044 (+0.00%)     0.8302 -7.75%     0.2228 +10.8%

Evaluation on non-ambiguous prompts. We report beneficial effects of on both ambiguous (Amb.) and specific instructions (Spec.)=-1

§ CONCLUSIONS

We have introduced Specify ANd Edit (), a zero-shot inference pipeline that improves the performance of diffusion-based text-to-image editing methods with ambiguous instructions. By using an LLM to decompose instructions into specific interventions, enhances both interpretability and editing quality. Our experiments show consistent performance improvements and increased output diversity. Moreover, is versatile, and it can benefit both ambiguous and clear editing tasks. In the future, we plan to address the limitations of , such as the difficulty in handling a high number of specific instructions and the lack of guarantee that each specific instruction is actually applied (see appendix).=-1

§ ACKNOWLEDGEMENTS

EI and SL are supported by the French National Research Agency (ANR) with the ANR-20-CE23-0027 project. FP is funded by KAUST (Grant DFR07910). PT is supported by UKRI grant: Turing AI Fellowship EP/W002981/1, and by the Royal Academy of Engineering under the Research Chair and Senior Research Fellowships scheme.

[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
arXiv, 2023.

[2]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.
Flamingo: a visual language model for few-shot learning.
NeurIPS, 2022.

[3]
Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al.
Gemini: A family of highly capable multimodal models.
arXiv, 2023.

[4]
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.
Palm 2 technical report.
arXiv, 2023.

[5]
Introducing the next generation of claude, 2024.
Accessed: 2024-07-13.

[6]
Tim Brooks, Aleksander Holynski, and Alexei A Efros.
Instructpix2pix: Learning to follow image editing instructions.
In CVPR, 2023.

[7]
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners.
NeurIPS, 2020.

[8]
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng.
Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.
In ICCV, 2023.

[9]
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.
Palm: Scaling language modeling with pathways.
JMLR, 2023.

[10]
Guillaume Couairon, Marlene Careil, Matthieu Cord, Stéphane Lathuiliere, and Jakob Verbeek.
Zero-shot spatial layout conditioning for text-to-image diffusion models.
In ICCV, 2023.

[11]
Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.
Diffedit: Diffusion-based semantic image editing with mask guidance.
arXiv, 2022.

[12]
Prafulla Dhariwal and Alexander Nichol.
Diffusion models beat gans on image synthesis.
In NeurIPS, 2021.

[13]
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.
A survey on in-context learning.
arXiv, 2022.

[14]
Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou.
Ranni: Taming text-to-image diffusion for accurate instruction following.
In CVPR, 2024.

[15]
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola.
Dreamsim: Learning new dimensions of human visual similarity using synthetic data.
In NeurIPS, 2023.

[16]
Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan.
Guiding instruction-based image editing via multimodal large language models.
In ICLR, 2024.

[17]
Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.
Stylegan-nada: Clip-guided domain adaptation of image generators.
ACM TOG, 2022.

[18]
Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li, Han Hu, et al.
Instructdiffusion: A generalist modeling interface for vision tasks.
In CVPR, 2024.

[19]
Qin Guo and Tianwei Lin.
Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation.
In CVPR, 2024.

[20]
Tiankai Hang, Shuyang Gu, Dong Chen, Xin Geng, and Baining Guo.
Cca: Collaborative competitive agents for image editing.
arXiv, 2024.

[21]
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.
Prompt-to-prompt image editing with cross attention control.
arXiv, 2022.

[22]
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
In NeurIPS, 2020.

[23]
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
In NeurIPS Workshops, 2021.

[24]
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith.
Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.
In ICCV, 2023.

[25]
Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang.
Tag2text: Guiding vision-language model via image tagging.
ICLR, 2024.

[26]
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al.
Smartedit: Exploring complex instruction-based image editing with multimodal large language models.
In CVPR, 2024.

[27]
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie.
Hq-edit: A high-quality dataset for instruction-based image editing.
arXiv, 2024.

[28]
KJ Joseph, Prateksha Udhayanan, Tripti Shukla, Aishwarya Agarwal, Srikrishna Karanam, Koustava Goswami, and Balaji Vasan Srinivasan.
Iterative multi-granular image editing using diffusion models.
In WACV, 2024.

[29]
Xuan Ju, Ailing Zeng, Yuxuan Bian, Shaoteng Liu, and Qiang Xu.
Direct inversion: Boosting diffusion-based editing with 3 lines of code.
arXiv, 2023.

[30]
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.
Elucidating the design space of diffusion-based generative models.
NeurIPS, 2022.

[31]
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.
Imagic: Text-based real image editing with diffusion models.
In CVPR, 2023.

[32]
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In ICML, 2023.

[33]
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
NeurIPS, 2024.

[34]
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.
Compositional visual generation with composable diffusion models.
In ECCV, 2022.

[35]
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
Sdedit: Guided image synthesis and editing with stochastic differential equations.
arXiv, 2021.

[36]
Meta AI.
Introducing meta llama 3: The most capable openly available llm to date, 2024.
Accessed: 2024-07-06.

[37]
Mistral AI Team.
Mistral-7b-instruct-v0.3, 2024.
Accessed: 2024-07-06.

[38]
Daiki Miyake, Akihiro Iohara, Yu Saito, and Toshiyuki Tanaka.
Negative-prompt inversion: Fast image inversion for editing with text-guided diffusion models.
arXiv, 2023.

[39]
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.
Null-text inversion for editing real images using guided diffusion models.
In CVPR, 2023.

[40]
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.
Glide: Towards photorealistic image generation and editing with text-guided diffusion models.
arXiv, 2021.

[41]
Open AI.
Hello gpt-4o, 2024.

[42]
Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu.
Zero-shot image-to-image translation.
In SIGGRAPH, 2023.

[43]
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei.
Kosmos-2: Grounding multimodal large language models to the world.
arXiv, 2023.

[44]
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In ICML, 2021.

[45]
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
arXiv, 2022.

[46]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In CVPR, 2022.

[47]
Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M Pawan Kumar, Emilien Dupont, Francisco JR Ruiz, Jordan S Ellenberg, Pengming Wang, Omar Fawzi, et al.
Mathematical discoveries from program search with large language models.
Nature, 2024.

[48]
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.
Photorealistic text-to-image diffusion models with deep language understanding.
NeurIPS, 2022.

[49]
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman.
Emu edit: Precise image editing via recognition and generation tasks.
In CVPR, 2024.

[50]
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.
Deep unsupervised learning using nonequilibrium thermodynamics.
In ICML, 2015.

[51]
Jiaming Song, Chenlin Meng, and Stefano Ermon.
Denoising diffusion implicit models.
In ICLR, 2021.

[52]
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.
Plug-and-play diffusion features for text-driven image-to-image translation.
In CVPR, 2023.

[53]
Laurens Van der Maaten and Geoffrey Hinton.
Visualizing data using t-sne.
JMLR, 2008.

[54]
Bram Wallace, Akash Gokul, and Nikhil Naik.
Edict: Exact diffusion inversion via coupled transformations.
In CVPR, 2023.

[55]
Qian Wang, Biao Zhang, Michael Birsak, and Peter Wonka.
Instructedit: Improving automatic masks for diffusion-based image editing with user instructions.
arXiv, 2023.

[56]
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su.
Magicbrush: A manually annotated dataset for instruction-guided image editing.
NeurIPS, 2024.

[57]
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.
The unreasonable effectiveness of deep features as a perceptual metric.
In CVPR, 2018.

+m +m +m

+m +m +m

Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing

In the main paper, we present , a novel zero-shot pipeline for instruction-based diffusion models, used to boost performance on ambiguous instructions. In this appendix, we propose additional information to complement our paper. We discuss limitations in Section <ref>. Then, in Section <ref> we report the LLM prompts used for each task. Finally, in Section <ref> we
provide additional results, including new experiments and qualitative evaluations.

§ LIMITATIONS

In this section, we discuss the limitations of . First, despite the observation that specific instructions $\mathcal{S}=\{s_{i}\}_{i=1}^{N}$ improve model performance on the initial instruction $c$, there is no guarantee that all of these specific instructions will always be respected. In our experience, the hire is $N$, the more the editing process tends to avoid following one or more specific instructions $s_{i}$. This is expected, since increasing the number of instructions makes the editing task more challenging for the diffusion model. To provide a transparent evaluation, we include an additional experiment quantifying how a single editing instruction is respected in Section <ref>. Moreover, brings additional computational costs due to multiple denoising operations. We quantify these effects as well in Section <ref>.

§ PROMPTS

Here, we report all the prompts used in the paper. To make it easier to understand our prompt engineering choices, we color differently the parts of the prompts designed for encouraging specific behaviour in the LLM. In particular, we use blue for indicating the task-related instruction to the model, orange for parsing-related instructions, green for in-context learning, and red for additional task-specific instructions to mitigate the presence of errors, established via trial-and-error.

Instruction Decomposition
Our instruction decomposition prompt used in Section <ref> is reported below. The  is obtained by using the captioning prompt, shown in the next paragraph.

[colback=gray!5!white, colframe=gray!75!black, title=Decomposition prompt, breakable, boxrule=0.5mm, colbacktitle=gray!75!black, coltitle=white]

You are a helpful assistant for image editing.
I will provide you with a caption that describes an image,
and an editing instruction that represents an ambiguous modification
of the scene. Your task is to propose specific modifications.
You can ask to add or replace elements in the scene,
proposing consistent modification that agree with the
ambiguous instruction.

Be concise and output your instructions without
further considerations or reasoning, one local modification per line.
Do not output any other text than the suggested outputs, do not write
“suggested output:”.=-1

I am going to provide some examples now.

Caption: a photo of a urban scenario, with cars.

ambiguous instruction: make the scene vintage.

Suggested output: replace the cars with old cars

Caption: a photo of a dog running on the grass.

ambiguous instruction: make it look funny.

Suggested output: add a hat to the dog.

The main subject of the scene must stay the same. For instance, if the photo is describing a cat
as the main subject, you cannot replace the cat with another animal. You should NEVER remove elements.
Only propose instructions targeting elements that appear in the caption, without imagining anything else.=-1

Now, provide  outputs for the following caption and subjective instruction.
Caption:  ambiguous instruction

Captioning prompt
To evaluate $\text{CLIP}_\Delta$ (Section <ref>), we generate simultaneously the captions of the initial and the final scenes with GPT-4o. Please note that we use only the caption of the original image in our inference pipeline (Section <ref>).
However, we observe that generating both captions simultaneously improves their consistency, allowing us to better evaluate $\text{CLIP}_\Delta$ in Section <ref>.
Here, we report the prompt used for the captioning task. We input the ambiguous instruction $c$, along with the input image $x$. The prompt is:

[colback=gray!5!white, colframe=gray!75!black, title=Captioning prompt, breakable, boxrule=0.5mm, colbacktitle=gray!75!black, coltitle=white]

I am going to provide an input image and an editing instruction. You should propose 1) a caption that describes accurately the input image, max 10 words, focusing only on visual content 2) a caption that encompasses how the image should look like after applying the instruction. The instruction is: . The image is

Try to keep these captions as compact as possible. The captions should be as similar as possible to each other.

You should reply following the format:
1. “caption 1”
2. “caption 2”

Just reply with the captions without reasoning or considerations.

GPT evaluation
Here, we report the prompt used for evaluating image quality with GPT-4o (Section <ref>).
We indicate with $\tilde{x}_\text{baseline}$ and $\tilde{x}_\text{ours}$ the editing results of baselines and , respectively.

[colback=gray!5!white, colframe=gray!75!black, title=GPT evaluation prompt, breakable, boxrule=0.5mm, colbacktitle=gray!75!black, coltitle=white]

I'm going to provide three pictures and one editing textual instruction. The first is an original picture.
The second and the third pictures are edited pictures, where image editing methods are applying transformations to the original picture by following the instruction.
The image editing methods identifiers are A and B. You should tell me what is the editing method that produces the best edited image.
For your evaluation, you should balance how much the edited image respects the instruction, the quality and realism of the generated image, and the content preservation from the original picture.

Reply with A or B only without further text.
The images are ordered in this way: original image, the image of method A, the image of method B.

Now, provide your answer for the input images and the instruction:  images: , .

Ambiguous instruction selection
Here, we present the prompt used to automatically select samples with abstract instructions for Table <ref>.

[colback=gray!5!white, colframe=gray!75!black, title=Instruction selection prompt, breakable, boxrule=0.5mm, colbacktitle=gray!75!black, coltitle=white]
You are a helpful assistant for image editing.
I will provide you with an editing instruction that requires certain modification of the scene in the image.
Your task is to decide whether this instruction represents an abstract instruction or a specific instruction.

Here are some examples:

ambiguous instruction: ‘Change the image so it apears old and musty.’

ambiguous instruction: ‘Make it a snowy day.’

ambiguous instruction: ‘Change the image so the players look like zombies.’

Specific instruction: ’Add the word ‘tray’ in white to the bottom of the image.'

Specific instruction: ‘Change the sheep into a calf.’

Specific instruction: ‘Draw this in an oil painting style.’

Now, tell me if the following instruction is ambiguous or specific. The instruction is: .

Before answering, motivate your decision by reasoning about the properties of this instruction.

Your response should start with either ‘Response: ambiguous.’ or ‘Response: specific.

§ ADDITIONAL RESULTS

In this section we provide complementary results and analysis for various aspects of the proposed pipeline.
First, we provide additional evaluation of the model fidelity to specific instructions $\mathcal{S}$. Then, we share insights on the embedding distribution for ambiguous and specific instructions. Finally, we provide additional qualitative results for all methods discussed in Section <ref>.=-1

3c    3cEMU-Edit

Method     LLM     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     $\text{\textbf{CLIP}}_\Delta\uparrow$
-3px4*90IP2P    Baseline     -     0.2923     0.8810     0.1203
3*     3*    GPT-4o    0.2968     0.7531     0.1858

LLaMA3-instruct     0.2903     0.7597    0.1657

Mistral v0.3     0.2955     0.7520     0.1771

-0px4*90MB    Baseline     -     0.2888     0.7858     0.1618
3*     3*    GPT-4o    0.3006     0.8209     0.1655

LLaMA3-instruct     0.3001     0.8299     0.1605

Mistral v0.3     0.2943     0.7977     0.1688

-3px4*90HQEdit    Baseline     -     0.2675     0.6501     0.1417
3*     3*    GPT-4o    0.2823     0.6870     0.1474

LLaMA3-instruct    0.2808     0.6793     0.1336

Mistral v0.3     0.2772     0.6521     0.1394

Performance with other LLMs. Open source alternatives to GPT-4o such as LLaMA3-instruct and Mistral v0.3 perform competitively on the instruction decomposition task.

Impact of the Language Model
We replace GPT-4o with LLaMA3-instruct [36] and Mistral v0.3 [37], and evaluate with $N=3$.
Results in Table <ref> report only slight decrease in performance.
Interestingly, decomposing instructions wtih LLaMA3-instruct seems to promote image consistency, outperforming GPT-4o on $\text{CLIP}_{i}$ using IP2P (0.7597) and MB (0.8299). More important, this proves that can be used in conjunction with open source models,
promoting accessibility and reproducibility of our results.

How much each specific instruction is respected?
In Section <ref> of the main paper, we highlight that the fidelity to specific instructions is an important property of . Indeed, if we respect specific instructions during editing, we can provide them to the user, improving interpretability of the editing process. In the main paper (Table <ref>), we consider the reference sets $\mathcal{S}$  of specific instructions with $N=3$, and compare the average fidelity of the baselines and the proposed with $N=\{1, 2, 3\}$ to each of those pre-defined $3$ instructions.
This allows us to conclude that each specific instruction has an impact on the final image transformation represented by that reference set $\mathcal{S}$ with $N=3$.
However, we also need to evaluate for each $N$ the fidelity of to the set of specific instructions actually being used in the denoising process.
In a similar vein to Table <ref>, in Table <ref> we report the average over $N$ CLIP-based metrics with respect to each specific instruction $s_{i}$, but considering only the corresponding set of $N$ instructions.
This means that we evaluate fidelity with respect to a single specific instruction if $N=1$, to two if $N=2$, and to three if $N=3$. Considering that the baselines do not use specific instructions for inference, it is impossible to compare with them in this setup, thus motivating the setup chosen for Table <ref>. As visible from the results, the fidelity to each specific instruction decreases with higher $N$. This is expected, since with more instructions to follow, the editing task is more challenging, and the editing effects might overlap.
However, we highlight that the $N=3$ setup still performs best in Table <ref>, resulting in a trade-off for the choice of $N$. We empirically find $N=3$ to be a good value.

1c    1c    2cEMU-Edit

Method     $N$     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{$\Delta$}}\uparrow$
3*90IP2P     3*     1     0.2990     0.1679

2     0.2907     0.156

3     0.2878     0.1518
3*90MB     3*      1     0.2930     0.1551

2     0.2924     0.1408

3      0.2926     0.1284
3*90HQEdit    3*      1     0.2970     0.1593

2     0.2944     0.1455

3     0.2907     0.1326

Impact of all the applied specific instructions We evaluate how much the applied specific instructions influence the performance of . As expected, with an increasing number of instructions, the fidelity to all the instruction set $\mathcal{S}$ decreases.

1c    3cIP2P data$\rightarrow$EMU-Edit     3cEMU-Edit$\rightarrow$IP2P data

Method     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     $\text{\textbf{CLIP}}_\Delta\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{d}}\uparrow$     $\text{\textbf{CLIP}}_\text{\textbf{i}}\uparrow$     $\text{\textbf{CLIP}}_\Delta\uparrow$
2*90IP2P     Baseline     0.2906     0.8789     0.1076     0.2937     0.9084     0.1745

0.2939     0.7828     0.1468     0.2985     0.837     0.2005
2*90MB    Baseline     0.292     0.8163     0.142     0.2872     0.8067     0.186

0.2998     0.8301     0.1457     0.296     0.8297     0.1816
2*90HQEdit     Baseline     0.2662     0.6765     0.1184     0.2747     0.7226     0.1754

0.2853     0.6904     0.1288     0.2895     0.7435     0.1735

Evaluation on cross-datasets. For each dataset, we select a new cross-dataset of ambiguous instructions using the nearest neighbor classifier fitted on the other dataset. The results demonstrate the robustness of across different datasets

t-SNE visualisation of instruction embeddings. We compute embeddings for all instructions from the EMU-Edit dataset and for ambiguous instructions from the IP2P dataset, and we visualise them using t-SNE. We show that embeddings form clusters corresponding to the task types (splits) in EMU-Edit.

Analysis of instruction embeddings
In Section <ref> we extensively evaluate our on two datasets: EMU-Edit [49] and IP2P data [6]. To analyze the instruction space of these datasets, we embed all instructions into the vectors of length $3072$ using  by OpenAI, and visualise these embeddings using t-SNE [53]. In Figure <ref>, we show all splits from EMU-Edit against ambiguous instructions from IP2P data. Interestingly, instruction embeddings from EMU-Edit form distinct clusters corresponding to the types of the task (splits). This shows that the instruction space has a complex structure which depends on the semantics encoded by instructions. Moreover, the set of ambiguous instructions from EMU-Edit (global_EMU-Edit) is completely included in the set of ambiguous instructions from IP2P (ambiguous_IP2P), showing the affinity of these sets. To evaluate cross-dataset robustness of our , we take advantage of this affinity, and perform cross-dataset ambiguous instruction classification by using ambiguous instructions from one dataset to fit a nearest neighbor classifier, and by selecting with this classifier the ambiguous instructions from the other dataset. We denote the selected instructions as “cross-datasets” IP2P data$\rightarrow$EMU-Edit (for EMU-Edit classified with IP2P data) and EMU-Edit$\rightarrow$IP2P data (for IP2P classified with EMU-Edit). We evaluate our on these cross-datasets for $N=3$ and compare it against the same baselines as in Section <ref>. The results in Table <ref> not only demonstrate the robustness of our model across different datasets, but also show that such cross-dataset classification can be used as an alternative ambiguous instruction selection strategy that does not require any prompting.

Computational costs for different $N$. We measure the average time for editing one image with multiple $N$ configurations. $N$ and processing times are directly proportional.

Computational times
implies an increased computational load due to multiple denoising operations required for processing specific instructions. We measure the time required to produce an edited image $\tilde{x}$ depending on the number of specific instructions $N$. For each point in Figure <ref>, we average the processing time over 10 images, using InstructPix2Pix as baseline on NVIDIA 4090 GPU. As seen, increasing $N$ brings a considerable computation overload. This, in addition to results in Table <ref>,
justify the schoice to limit the number of specifc instructions to $N=\{1, 2, 3\}$ in our work.

Additional qualitative results
We provide additional qualitative results for applied on top of all three baselines. In particular, in Figure <ref> we show more editing examples using InstructPix2Pix as a baseline, complementing qualitative results in the main paper (Figure <ref>), while in Figure <ref> and Figure <ref> we report results on MagicBrush and HQEdit baselines, respectively, which are not included in the main manuscript due to limited space.

Baseline     $N=1$     $N=2$     $N=3$

Original image

Instructions $\rightarrow$     Change the image so it apears old and musty.     Apply a sepia filter to the entire image     Add visible rust to the refrigerator     Add cracks and stains to the stove

Instructions $\rightarrow$     Change the image so it appears to be set in the distant future     Add futuristic buildings in the background     Replace the road with a modern hover road     Add holographic signs around the police box

Instructions $\rightarrow$     Have the image look like it was taken during a power outage     Dim the lighting in the scene     Replace the background with darkness     Add a candle on the table

Instructions $\rightarrow$     Change the time of the day to night     Replace the bright sky with a starry night sky     Add a moon in the background     Add streetlights along the dirt road

Additional qualitative results for InstructPix2Pix.

Baseline     $N=1$     $N=2$     $N=3$

Original image

Instructions $\rightarrow$     Change the image to appear like a fire in a forest     Replace the grassy field with a burning forest background     Add flames around the Ducati motorcycle     Add smoke in the sky above the forest

Instructions $\rightarrow$     Make it a snowy day      Add snow on the ground around the motorcycles     Add snow to the roofs and ledges of the museum building     Add falling snowflakes throughout the scene

Instructions $\rightarrow$     Put this inside of a cave      Replace the background with a cave interior     Add rocky textures to the monitor, keyboard, mouse, and speakers     Adjust the lighting to darker, more cave-like conditions

Instructions $\rightarrow$     Change this to a restaurant kitchen      Replace the kitchen table with an industrial stainless-steel worktable     Add commercial-grade ovens and stoves     Add multiple professional kitchen knives and utensils on the countertop

Qualitative results for MagicBrush.

Baseline     $N=1$     $N=2$     $N=3$

Original image

Instructions $\rightarrow$     Make the scene seem like it is an outdoor picnic     Add a picnic blanket under the sandwich     Add trees in the background     Add a basket with fruits beside the sandwich

Instructions $\rightarrow$     Change this to a snowstorm     Add falling snow to the scene     Add snow accumulation to the cart and cow     Replace the clear sky with a cloudy, snow-filled sky

Instructions $\rightarrow$     Change this to night time with lots of stars     Change the sky to a dark, star-filled sky     Adjust the lighting to create moonlit shadows     Add a glowing moon in the background

Instructions $\rightarrow$     Make the image appear to be in the Sahara desert     Replace the snowy hill with sand dunes     Change the man's red jacket to a lighter, desert-appropriate color     Add a blazing sun in the sky

Qualitative results for HQEdit.