# InstructPix2Pix: Learning to Follow Image Editing Instructions

Tim Brooks* Aleksander Holynski* Alexei A. Efros
University of California, Berkeley

###### Abstract

We propose a method for editing images from human instructions: given an input
image and a written instruction that tells the model what to do, our model
follows these instructions to edit the image. To obtain training data for this
problem, we combine the knowledge of two large pretrained models—a language
model (GPT-3) and a text-to-image model (Stable Diffusion)—to generate a large
dataset of image editing examples. Our conditional diffusion model,
InstructPix2Pix, is trained on our generated data, and generalizes to real
images and user-written instructions at inference time. Since it performs
edits in the forward pass and does not require per-example fine-tuning or
inversion, our model edits images quickly, in a matter of seconds. We show
compelling editing results for a diverse collection of input images and
written instructions.

00footnotetext: *Denotes equal contribution
More results on our project page: timothybrooks.com/instruct-pix2pix

Figure 1:  Given an image and an instruction for how to edit that image, our
model performs the appropriate edit. Our model does not require full
descriptions for the input or output image, and edits images in the forward
pass without per-example inversion or fine-tuning.

##  1 Introduction

We present a method for teaching a generative model to follow human-written
instructions for image editing. Since training data for this task is difficult
to acquire at scale, we propose an approach for generating a paired dataset
that combines multiple large models pretrained on different modalities: a
large language model (GPT-3 [7]) and a text-to-image model (Stable Diffusion
[52]). These two models capture complementary knowledge about language and
images that can be combined to create paired training data for a task spanning
both modalities.

Using our generated paired data, we train a conditional diffusion model that,
given an input image and a text instruction for how to edit it, generates the
edited image. Our model directly performs the image edit in the forward pass,
and does not require any additional example images, full descriptions of the
input/output images, or per-example finetuning. Despite being trained entirely
on synthetic examples (i.e., both generated written instructions and generated
imagery), our model achieves zero-shot generalization to both arbitrary _real_
images and natural human-written instructions. Our model enables intuitive
image editing that can follow human instructions to perform a diverse
collection of edits: replacing objects, changing the style of an image,
changing the setting, the artistic medium, among others. Selected examples can
be found in Figure 1.

Figure 2: Our method consists of two parts: generating an image editing
dataset, and training a diffusion model on that dataset. (a) We first use a
finetuned GPT-3 to generate instructions and edited captions. (b) We then use
StableDiffusion [52] in combination with Prompt-to-Prompt [17] to generate
pairs of images from pairs of captions. We use this procedure to create a
dataset (c) of over 450,000 training examples. (d) Finally, our
InstructPix2Pix diffusion model is trained on our generated data to edit
images from instructions. At inference time, our model generalizes to edit
real images from human-written instructions.

##  2 Prior work

##### Composing large pretrained models

Recent work has shown that large pretrained models can be combined to solve
multimodal tasks that no one model can perform alone, such as image captioning
and visual question answering (tasks that require the knowledge of both a
large language model and a text-image model). Techniques for combining
pretrained models include joint finetuning on a new task [4, 41, 68, 34],
communication through prompting [70, 63], composing probability distributions
of energy-based models [11, 38], guiding one model with feedback from another
[62], and iterative optimization [35]. Our method is similar to prior work in
that it leverages the complementary abilities of two pretrained models—GPT-3
[7]) and Stable Diffusion [52]—but differs in that we use these models to
generate paired multi-modal training data.

##### Diffusion-based generative models

Recent advances in diffusion models [60] have enabled state-of-the-art image
synthesis [61, 18, 10, 19, 56, 54] as well as generative models of other
modalities such as video [21, 59], audio [31], text [36] and network
parameters [46]. Recent text-to-image diffusion models [42, 52, 49, 55] have
shown to generate realistic images from arbitrary text captions.

##### Generative models for image editing

Image editing models traditionally targeted a single editing task such as
style transfer [15, 16] or translation between image domains [24, 72, 22, 37,
43]. Numerous editing approaches invert [1, 2, 3, 12] or encode [64, 51, 8]
images into a latent space (e.g., StyleGAN [26, 27]) where they can be edited
by manipulating latent vectors. Recent models have leveraged CLIP [48]
embeddings to guide image editing using text [32, 45, 71, 14, 42, 29, 5, 9].
We compare with one of these methods, Text2Live [6], an editing method that
optimizes for an additive image layer that maximizes a CLIP similarity
objective.

Recent works have used pretrained text-to-image diffusion models for image
editing [39, 28, 5, 49, 17]. While some text-to-image models natively have the
ability to edit images (e.g., DALLE-2 can create variations of images, inpaint
regions, and manipulate the CLIP embedding [49]), using these models for
_targeted_ editing is non-trivial, because in most cases they offer no
guarantees that similar text prompts will yield similar images. Recent work by
Hertz et al. [17] tackles this issue with Prompt-to-Prompt, a method for
assimilating the generated images for similar text prompts, such that isolated
edits can be made to a generated image. We use this method in generating
training data. To edit non-generated (i.e., real) imagery, SDEdit [39] uses a
pretrained model to noise and denoise an input image with a new target prompt.
We compare with SDEdit as a baseline. Other recent works perform local
inpainting given a caption and user-drawn mask [49, 5], generate new images of
a specific object or concept learned from a small collection of images [13,
53], or perform editing by inverting (and fine-tuning) a single image, and
subsequently regenerating with a new text description [28]. In contrast to
these approaches, our model takes only a single image and an instruction for
how to edit that image (i.e., not a full description of any image), and
performs the edit directly in the forward pass without need for a user-drawn
mask, additional images, or per-example inversion or finetuning.

##### Learning to follow instructions

Our method differs from existing text-based image editing works [39, 53, 13,
17, 28, 6] in that it enables editing from instructions that tell the model
what action to perform, as opposed to text labels, captions or descriptions of
input/output images. A key benefit of following editing instructions is that
the user can just tell the model exactly what to do in natural written text.
There is no need for the user to provide extra information, such as example
images or descriptions of visual content that remains constant between the
input and output images. Instructions are expressive, precise, and intuitive
to write, allowing the user to easily isolate specific objects or visual
attributes to change. Our goal to follow written image editing instructions is
inspired by recent work teaching large language models to better follow human
instructions for language tasks [44, 40, 69].

##### Training data generation with generative models

Deep models typically require large amounts of training data. Internet data
collections are often suitable, but may not exist in the form necessary for
supervision, e.g., paired data of particular modalities. As generative models
continue to improve, there is growing interest in their use as a source of
cheap and plentiful training data for downstream tasks [58, 50, 66, 65, 33,
47]. In this paper, we use two different off-the-shelf generative models
(language, text-to-image) to produce training data for our editing model.

##  3 Method

We treat instruction-based image editing as a supervised learning problem: (1)
first, we generate a paired training dataset of text editing instructions and
images before/after the edit (Sec. 3.1, Fig. 2a-c), then (2) we train an image
editing diffusion model on this generated dataset (Sec. 3.2, Fig 2d). Despite
being trained with generated images and editing instructions, our model is
able to generalize to editing _real_ images using arbitrary human-written
instructions. See Fig. 2 for an overview of our method.

###  3.1 Generating a Multi-modal Training Dataset

We combine the abilities of two large-scale pretrained models that operate on
different modalities—a large language model [7] and a text-to-image model
[52]—to generate a multi-modal training dataset containing text editing
instructions and the corresponding images before and after the edit. In the
following two sections, we describe in detail the two steps of this process.
In Section 3.1.1, we describe the process of fine-tuning GPT-3 [7] to generate
a collection of text edits: given a prompt describing an image, produce a text
instruction describing a change to be made and a prompt describing the image
after that change (Figure 2a). Then, in Section 3.1.2, we describe the process
of converting the two text prompts (i.e., before and after the edit) into a
pair of corresponding images using a text-to-image model [52] (Figure 2b).

####  3.1.1 Generating Instructions and Paired Captions

|  Input LAION caption | Edit instruction |  Edited caption
---|---|---|---
Human-written (700 edits) |  Yefim Volkov, Misty Morning | make it afternoon |  Yefim Volkov, Misty Afternoon
girl with horse at sunset | change the background to a city |  girl with horse at sunset in front of city
painting-of-forest-and-pond | Without the water. |  painting-of-forest
… | … |  …
GPT-3 generated ($>$450,000 edits) |  Alex Hill, Original oil painting on canvas, Moonlight Bay | in the style of a coloring book |  Alex Hill, Original coloring book illustration, Moonlight Bay
The great elf city of Rivendell, sitting atop a waterfall as cascades of water spill around it | Add a giant red dragon |  The great elf city of Rivendell, sitting atop a waterfall as cascades of water spill around it with a giant red dragon flying overhead
Kate Hudson arriving at the Golden Globes 2015 | make her look like a zombie |  Zombie Kate Hudson arriving at the Golden Globes 2015
… | … |  …

Table 1: We label a small text dataset, finetune GPT-3, and use that finetuned
model to generate a large dataset of text triplets. As the input caption for
both the labeled and generated examples, we use real image captions from
LAION. Highlighted text is generated by GPT-3.

We first operate entirely in the text domain, where we leverage a large
language model to take in image captions and produce editing instructions and
the resulting text captions after the edit. For example, as shown in Figure
2a, provided the input caption _“photograph of a girl riding a horse”_ , our
language model can generate both a plausible edit instruction _“have her ride
a dragon”_ and an appropriately modified output caption _“photograph of a girl
riding a dragon”_. Operating in the text domain enables us to generate a large
and diverse collection of edits, while maintaining correspondence between the
image changes and text instructions.

Our model is trained by finetuning GPT-3 on a relatively small human-written
dataset of editing triplets: (1) input captions, (2) edit instructions, (3)
output captions. To produce the fine-tuning dataset, we sampled 700 input
captions from the LAION-Aesthetics V2 6.5+ [57] dataset and manually wrote
instructions and output captions. See Table 3.1.1a for examples of our written
instructions and output captions. Using this data, we fine-tuned the GPT-3
Davinci model for a single epoch using the default training parameters.

Benefiting from GPT-3’s immense knowledge and ability to generalize, our
finetuned model is able to generate creative yet sensible instructions and
captions. See Table 3.1.1b for example GPT-3 generated data. Our dataset is
created by generating a large number of edits and output captions using this
trained model, where the input captions are real image captions from LAION-
Aesthetics (excluding samples with duplicate captions or duplicate image
URLs). We chose the LAION dataset due to its large size, diversity of content
(including references to proper nouns and popular culture), and variety of
mediums (photographs, paintings, digital artwork). A potential drawback of
LAION is that it is quite noisy and contains a number of nonsensical or
undescriptive captions—however, we found that dataset noise is mitigated
through a combination of dataset filtering (Section 3.1.2) and classifier-free
guidance (Section 3.2.1). Our final corpus of generated instructions and
captions consists of $454,445$ examples.

####  3.1.2 Generating Paired Images from Paired Captions

(a) Without Prompt-to-Prompt.

(b) With Prompt-to-Prompt.

Figure 3: Pair of images generated using StableDiffusion [52] with and without
Prompt-to-Prompt [17]. For both, the corresponding captions are “photograph of
a girl riding a horse” and “photograph of a girl riding a dragon”.

Next, we use a pretrained text-to-image model to transform a pair of captions
(referring to the image before and after the edit) into a pair of images. One
challenge in turning a pair of captions into a pair of corresponding images is
that text-to-image models provide no guarantees about image consistency, even
under very minor changes of the conditioning prompt. For example, two very
similar prompts: “a picture of a cat” and “a picture of a black cat” may
produce wildly different images of cats. This is unsuitable for our purposes,
where we intend to use this paired data as supervision for training a model to
edit images (and not produce a different random image). We therefore use
Prompt-to-Prompt [17], a recent method aimed at encouraging multiple
generations from a text-to-image diffusion model to be similar. This is done
through borrowed cross attention weights in some number of denoising steps.
Figure 3 shows a comparison of sampled images with and without Prompt-to-
Prompt.

While this greatly helps assimilate generated images, different edits may
require different amounts of change in image-space. For instance, changes of
larger magnitude, such as those which change large-scale image structure
(e.g., moving objects around, replacing with objects of different shapes), may
require less similarity in the generated image pair. Fortunately, Prompt-to-
Prompt has as a parameter that can control the similarity between the two
images: the fraction of denoising steps $p$ with shared attention weights.
Unfortunately, identifying an optimal value of $p$ from only the captions and
edit text is difficult. We therefore generate $100$ sample pairs of images per
caption-pair, each with a random $p\sim\mathcal{U}(0.1,0.9)$, and filter these
samples by using a CLIP-based metric: the directional similarity in CLIP space
as introduced by Gal _et al._ [14]. This metric measures the consistency of
the change between the two images (in CLIP space) with the change between the
two image captions. Performing this filtering not only helps maximize the
diversity and quality of our image pairs, but also makes our data generation
more robust to failures of Prompt-to-Prompt and Stable Diffusion.

###  3.2 InstructPix2Pix

We use our generated training data to train a conditional diffusion model that
edits images from written instructions. We base our model on Stable Diffusion,
a large-scale text-to-image latent diffusion model.

Diffusion models [60] learn to generate data samples through a sequence of
denoising autoencoders that estimate the score [23] of a data distribution (a
direction pointing toward higher density data). Latent diffusion [52] improves
the efficiency and quality of diffusion models by operating in the latent
space of a pretrained variational autoencoder [30] with encoder $\mathcal{E}$
and decoder $\mathcal{D}$. For an image $x$, the diffusion process adds noise
to the encoded latent $z=\mathcal{E}(x)$ producing a noisy latent $z_{t}$
where the noise level increases over timesteps $t\in T$. We learn a network
$\epsilon_{\theta}$ that predicts the noise added to the noisy latent $z_{t}$
given image conditioning $c_{I}$ and text instruction conditioning $c_{T}$. We
minimize the following latent diffusion objective:

$L=\mathbb{E}_{\mathcal{E}(x),\mathcal{E}(c_{I}),c_{T},\epsilon\sim\mathcal{N}(0,1),t}\Big{[}\|\epsilon-\epsilon_{\theta}(z_{t},t,\mathcal{E}(c_{I}),c_{T}))\|_{2}^{2}\Big{]}$
(1)

Wang _et al._ [67] show that fine-tuning a large image diffusion models
outperforms training a model from scratch for image translation tasks,
especially when paired training data is limited. We therefore initialize the
weights of our model with a pretrained Stable Diffusion checkpoint, leveraging
its vast text-to-image generation capabilities. To support image conditioning,
we add additional input channels to the first convolutional layer,
concatenating $z_{t}$ and $\mathcal{E}(c_{I})$. All available weights of the
diffusion model are initialized from the pretrained checkpoints, and weights
that operate on the newly added input channels are initialized to zero. We
reuse the same text conditioning mechanism that was originally intended for
captions to instead take as input the text edit instruction $c_{T}$.
Additional training details are provided in the supplemental material.

| $s_{T}=3$ | $s_{T}=7.5$ | $s_{T}=15$
---|---|---|---
$s_{I}=1.0$ |  |  |
$s_{I}=1.2$ |  |  |
$s_{I}=1.6$ |  |  |

Edit instruction: _“Turn him into a cyborg!”_

Figure 4: Classifier-free guidance weights over two conditional inputs.
$s_{I}$ controls similarity with the input image, while $s_{T}$ controls
consistency with the edit instruction.

####  3.2.1 Classifier-free Guidance for Two Conditionings

Classifier-free diffusion guidance [20] is a method for trading off the
quality and diversity of samples generated by a diffusion model. It is
commonly used in class-conditional and text-conditional image generation to
improve the visual quality of generated images and to make sampled images
better correspond with their conditioning. Classifier-free guidance
effectively shifts probability mass toward data where an implicit classifier
$p_{\theta}(c|z_{t})$ assigns high likelihood to the conditioning $c$. The
implementation of classifier-free guidance involves jointly training the
diffusion model for conditional and unconditional denoising, and combining the
two score estimates at inference time. Training for unconditional denoising is
done by simply setting the conditioning to a fixed null value $c=\varnothing$
at some frequency during training. At inference time, with a guidance scale
$s\geq 1$, the modified score estimate $\tilde{e_{\theta}}(z_{t},c)$ is
extrapolated in the direction toward the conditional $e_{\theta}(z_{t},c)$ and
away from the unconditional $e_{\theta}(z_{t},\varnothing)$.

$\tilde{e_{\theta}}(z_{t},c)=e_{\theta}(z_{t},\varnothing)+s\cdot(e_{\theta}(z_{t},c)-e_{\theta}(z_{t},\varnothing))$
(2)

For our task, the score network $e_{\theta}(z_{t},c_{I},c_{T})$ has two
conditionings: the input image $c_{I}$ and text instruction $c_{T}$. We find
if beneficial to leverage classifier-free guidance with respect to both
conditionings. Liu _et al._ [38] demonstrate that a conditional diffusion
model can compose score estimates from multiple different conditioning values.
We apply the same concept to our model with two separate conditioning inputs.
During training, we randomly set only $c_{I}=\varnothing_{I}$ for 5% of
examples, only $c_{T}=\varnothing_{T}$ for 5% of examples, and both
$c_{I}=\varnothing_{I}$ and $c_{T}=\varnothing_{T}$ for 5% of examples. Our
model is therefore capable of conditional or unconditional denoising with
respect to both or either conditional inputs. We introduce two guidance
scales, $s_{I}$ and $s_{T}$, which can be adjusted to trade off how strongly
the generated samples correspond with the input image and how strongly they
correspond with the edit instruction. Our modified score estimate is as
follows:

$\begin{split}\tilde{e_{\theta}}(z_{t},c_{I},c_{T})=&\>e_{\theta}(z_{t},\varnothing,\varnothing)\\\
&+s_{I}\cdot(e_{\theta}(z_{t},c_{I},\varnothing)-e_{\theta}(z_{t},\varnothing,\varnothing))\\\
&+s_{T}\cdot(e_{\theta}(z_{t},c_{I},c_{T})-e_{\theta}(z_{t},c_{I},\varnothing))\end{split}$
(3)

In Figure 4, we show the effects of these two parameters on generated samples.
See Appendix B for details of our classifier-free guidance formulation.

##  4 Results

|  |  |  |
---|---|---|---|---
Input | _“Make it a Modigliani painting”_ | _“Make it a Miro painting”_ | _“Make it an Egyptian sculpture”_ | _“Make it a marble roman sculpture”_

Figure 5: _Mona Lisa_ transformed into various artistic mediums.

|  |
---|---|---
Input | _“Put them in outer space”_ | _“Turn the humans into robots”_

Figure 6: _The Creation of Adam_ with new context and subjects (generated at
768 resolution).

|  |  |
---|---|---|---
_“Make it Paris”_ | _“Make it Hong Kong”_ | _“Make it Manhattan”_ | _“Make it Prague”_
|  |  |
_“Make it evening”_ | _“Put them on roller skates”_ | _“Turn this into 1900s”_ | _“Make it underwater”_
|  |  |
_“Make it Minecraft”_ | _“Turn this into the space age”_ | _“Make them into Alexander Calder sculptures”_ | _“Make it a Claymation”_

Figure 7: The iconic Beatles _Abbey Road_ album cover transformed in a variety
of ways.

We show instruction-based image editing results on a diverse set of real
photographs and artwork, for a variety of types of edits and instruction
wordings. See Figures 1, 5, 6, 7, 11, 12, 15, 16, 17, 18, and 3.1.1 for
selected results. Our model successfully performs many challenging edits,
including replacing objects, changing seasons and weather, replacing
backgrounds, modifying material attributes, converting artistic medium, and a
variety of others.

We compare our method qualitatively with a couple recent works, SDEdit [39]
and Text2Live [6]. Our model follows instructions for how to edit the image,
but prior works (including these baseline methods) expect descriptions of the
image (or edit layer). Therefore, we provide them with the “after-edit” text
caption instead of the edit instruction. We also compare our method
quantitatively with SDEdit, using two metrics measuring image consistency and
edit quality, further described in Section 4.1. Finally, we show ablations on
how the size and quality of generated training data affect our model’s
performance in Section 4.2.

Figure 8: We plot the trade-off between consistency with the input image (Y-axis) and consistency with the edit (X-axis). For both metrics, higher is better. For both methods, we fix text guidance to 7.5, and vary our $s_{I}\in[1.0,2.2]$ and SDEdit’s strength (the amount of denoising) between $[0.3,0.9]$. Input |  |  | SDEdit-OC [39] | T2L [6] |  |  | SDEdit-E [39] | Ours
---|---|---|---|---|---|---|---|---
|  |  |  |  |  |
|  |  | “Dali Painting of Nimbus Cloud…” |  |  | “make it look like a Dali Painting”
|  |  |  |  |  |
|  |  | “Crowned alias Grace. (Photo by […]/Netflix)” |  |  | “add a crown”
|  |  |  |  |  |
|  |  | “The Road Leads to the Ocean by Ben Heine” |  |  | “have the road lead to the ocean”
|  |  |  |  |  |
|  |  | “Industrial design bedroom furniture…” |  |  | “add a bedroom”
Figure 9: Comparison with other editing methods. The input is transformed
either by edit string (last two columns) or the ground-truth output image
caption (middle two columns). We compare our method against two recent works,
SDEdit [39] and Text2Live [6]. We show SDEdit in two configurations:
conditioned on the output caption (OP) and conditioned on the edit string (E).

###  4.1 Baseline comparisons

We provide qualitative comparisons with SDEdit [39] and Text2Live [6], as well
as quantitative comparisons with SDEdit. SDEdit [39] is a technique for
editing images with a pretrained diffusion model, where a partially noised
image is passed as input and denoised to produce a new edited image. We
compare with the public Stable Diffusion implementation of SDEdit. Text2Live
[6] is a technique for editing images by generating a color+opacity
augmentation layer, conditioned on a text prompt. We compare with the public
implementation released by the authors.

We compare with both methods qualitatively in Figure 9. We notice that while
SDEdit works reasonably well for cases where content remains approximately
constant and style is changed, it struggles to preserve identity and isolate
individual objects, especially when larger changes are desired. Additionally,
it requires a full output description of the desired image, rather than an
editing instruction. On the other hand, while Text2Live is able to produce
convincing results for edits involving additive layers, its formulation limits
the categories of edits that it can handle.

Quantitative comparisons with SDEdit are shown in Figure 8. We plot the
tradeoff between two metrics, cosine similarity of CLIP image embeddings (how
much the edited image agrees with the input image) and the directional CLIP
similarity introduced by [14] (how much the change in text captions agrees
with the change in the images). These are competing metrics—increasing the
degree to which the output images correspond to a desired edit will reduce
their similarity (consistency) with the input image. Still, we find that when
comparing our method with SDEdit, our results have notably higher image
consistency (CLIP image similarity) for the same directional similarity
values.

Figure 10: We compare ablated variants of our model (smaller training dataset,
no CLIP filtering) by fixing $s_{T}$ and sweeping values of
$s_{I}\in[1.0,2.2]$. Our proposed configuration performs best.  Figure 11:
Applying our model recurrently with different instructions results in
compounded edits.

Input

Edit instruction: _“in a race car video game”_

Figure 12: By varying the latent noise, our model can produce many possible
image edits for the same input image and instruction.

_“Zoom into the image”_

_“Move it to Mars”_

_“Color the tie blue”_

_“Have the people swap places”_

Figure 13: Failure cases. Left to right: our model is not capable of
performing viewpoint changes, can make undesired excessive changes to the
image, can sometimes fail to isolate the specified object, and has difficulty
reorganizing or swapping objects with each other.

###  4.2 Ablations

In Figure 10, we provide quantitative ablations for both our choice of dataset
size and our dataset filtering approach described in Section 3.1. We find that
decreasing the size of the dataset typically results in decreased ability to
perform larger (i.e., more significant) image edits, instead only performing
subtle or stylistic image adjustments (and thus, maintaining a high image
similarity score, but a low directional score). Removing the CLIP filtering
from our dataset generation has a different effect: the overall image
consistency with the input image is reduced.

We also provide an analysis of the effect of our two classifier-free guidance
scales in Figure 4. Increasing $s_{T}$ results in a stronger edit applied to
the image (i.e., the output agrees more with the instruction), and increasing
$s_{I}$ can help preserve the spatial structure of the input image (i.e., the
output agrees more with the input image). We find that values of $s_{T}$ in
the range $5-10$ and values of $s_{I}$ in the range $1-1.5$ typically produce
the best results. In practice, and for the results shown in the paper, we find
it beneficial to adjust guidance weights for each example to get the best
balance between consistency and edit strength.

##  5 Discussion

We demonstrate an approach that combines two large pretrained models, a large
language model and a text-to-image model, to generate a dataset for training a
diffusion model to follow written image editing instructions. While our method
is able to produce a wide variety of compelling edits to images, including
style, medium, and other contextual changes, there still remain a number of
limitations.

Our model is limited by the visual quality of the generated dataset, and
therefore by the diffusion model used to generate the imagery (in this case,
Stable Diffusion [52]). Furthermore, our method’s ability to generalize to new
edits and make correct associations between visual changes and text
instructions is limited by the human-written instructions used to fine-tune
GPT-3 [7], by the ability of GPT-3 to create instructions and modify captions,
and by the ability of Prompt-to-Prompt [17] to modify generated images. In
particular, our model struggles with counting numbers of objects and with
spatial reasoning (e.g., _“move it to the left of the image”_ , _“swap their
positions”_ , or _“put two cups on the table and one on the chair”_), just as
in Stable Diffusion and Prompt-to-Prompt. Examples of failures can be found in
Figure 13. Furthermore, there are well-documented biases in the data and the
pretrained models that our method is based upon, and therefore the edited
images from our method may inherit these biases or introduce other biases
(Figure 14).

Aside from mitigating the above limitations, our work also opens up questions,
such as: how to follow instructions for spatial reasoning, how to combine
instructions with other conditioning modalities like user interaction, and how
to evaluate instruction-based image editing. Incorporating human feedback to
improve the model is another important area of future work, and strategies
like human-in-the-loop reinforcement learning could be applied to improve
alignment between our model and human intentions.

##### Acknowledgments

We thank Ilija Radosavovic, William Peebles, Allan Jabri, Dave Epstein, Kfir
Aberman, Amanda Buster, and David Salesin. Tim Brooks is funded by an NSF
Graduate Research Fellowship. Additional funding provided by a research grant
from SAP and a gift from Google.

Input

_“Make them look like flight attendants”_

_“Make them look like doctors”_

Figure 14: Our method reflects biases from the data and models it is based
upon, such as correlations between profession and gender.

Input

_“Make it a grocery store”_

Figure 15: Leighton’s _Lady in a Garden_ moved to a new setting.

Input

_“Convert to a realistic photo”_

_“Turn into a 3D model”_

Figure 16: Van Gogh’s _Self-Portrait with a Straw Hat_ in different mediums.

Input

_“Add boats on the water”_

_“Replace the mountains with a city skyline”_

Figure 17: A landscape photograph shown with different contextual edits. Note
that isolated changes also bring along accompanying contextual effects: the
addition of boats also adds wind ripples in the water, and the added city
skyline is reflected on the lake.

Input

_“It is now midnight”_

_“Add a beautiful sunset”_

Figure 18: A photograph of a cityscape edited to show different times of day.

Input

_“Apply face paint”_

_“What would she look like as a bearded man?”_

_“Put on a pair of sunglasses”_

_“She should look 100 years old”_

_“What if she were in an anime?”_

_“Make her terrifying”_

_“Make her more sad”_

_“Make her James Bond”_

_“Turn her into Dwayne The Rock Johnson”_

Figure 19: Vermeer’s _Girl with a Pearl Earring_ with a variety of edits.

## References

  * [1] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432–4441, 2019.
  * [2] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan++: How to edit the embedded images?  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8296–8305, 2020.
  * [3] Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano.  Hyperstyle: Stylegan inversion with hypernetworks for real image editing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18511–18521, 2022.
  * [4] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  arXiv preprint arXiv:2204.14198, 2022.
  * [5] Omri Avrahami, Dani Lischinski, and Ohad Fried.  Blended diffusion for text-driven editing of natural images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18208–18218, 2022.
  * [6] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In European Conference on Computer Vision, pages 707–723. Springer, 2022.
  * [7] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [8] Lucy Chai, Jonas Wulff, and Phillip Isola.  Using latent space regression to analyze and leverage compositionality in gans.  In International Conference on Learning Representations, 2021.
  * [9] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff.  Vqgan-clip: Open domain image generation and editing with natural language guidance.  In European Conference on Computer Vision, pages 88–105. Springer, 2022.
  * [10] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [11] Yilun Du, Shuang Li, and Igor Mordatch.  Compositional visual generation with energy based models.  Advances in Neural Information Processing Systems, 33:6637–6647, 2020.
  * [12] Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros.  Blobgan: Spatially disentangled scene representations.  In European Conference on Computer Vision, pages 616–635. Springer, 2022.
  * [13] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [14] Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  ACM Transactions on Graphics (TOG), 41(4):1–13, 2022.
  * [15] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  A neural algorithm of artistic style.  arXiv preprint arXiv:1508.06576, 2015.
  * [16] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  * [17] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [18] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [19] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  J. Mach. Learn. Res., 23:47–1, 2022.
  * [20] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv preprint arXiv:2207.12598, 2022.
  * [21] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  arXiv preprint arXiv:2204.03458, 2022.
  * [22] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz.  Multimodal unsupervised image-to-image translation.  In ECCV, 2018.
  * [23] Aapo Hyvärinen and Peter Dayan.  Estimation of non-normalized statistical models by score matching.  Journal of Machine Learning Research, 6(4), 2005.
  * [24] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  CVPR, 2017.
  * [25] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  arXiv preprint arXiv:2206.00364, 2022.
  * [26] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  * [27] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8110–8119, 2020.
  * [28] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  arXiv preprint arXiv:2210.09276, 2022.
  * [29] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2426–2435, 2022.
  * [30] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [31] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  arXiv preprint arXiv:2009.09761, 2020.
  * [32] Gihyun Kwon and Jong Chul Ye.  Clipstyler: Image style transfer with a single text condition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18062–18071, 2022.
  * [33] Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Sanja Fidler, and Antonio Torralba.  Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21330–21340, 2022.
  * [34] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [35] Shuang Li, Yilun Du, Joshua B Tenenbaum, Antonio Torralba, and Igor Mordatch.  Composing ensembles of pre-trained models via iterative consensus.  arXiv preprint arXiv:2210.11522, 2022.
  * [36] Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori B Hashimoto.  Diffusion-lm improves controllable text generation.  arXiv preprint arXiv:2205.14217, 2022.
  * [37] Ming-Yu Liu, Xun Huang, Arun Mallya, Tero Karras, Timo Aila, Jaakko Lehtinen, and Jan Kautz.  Few-shot unsupervised image-to-image translation.  In IEEE International Conference on Computer Vision (ICCV), 2019\.
  * [38] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  arXiv preprint arXiv:2206.01714, 2022.
  * [39] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [40] Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi.  Cross-task generalization via natural language crowdsourcing instructions.  arXiv preprint arXiv:2104.08773, 2021.
  * [41] Ron Mokady, Amir Hertz, and Amit H Bermano.  Clipcap: Clip prefix for image captioning.  arXiv preprint arXiv:2111.09734, 2021.
  * [42] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [43] Utkarsh Ojha, Yijun Li, Cynthia Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, and Richard Zhang.  Few-shot image generation via cross-domain correspondence.  In CVPR, 2021.
  * [44] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  arXiv preprint arXiv:2203.02155, 2022.
  * [45] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2085–2094, October 2021.
  * [46] William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik.  Learning to learn with generative models of neural network checkpoints.  arXiv preprint arXiv:2209.12892, 2022.
  * [47] William Peebles, Jun-Yan Zhu, Richard Zhang, Antonio Torralba, Alexei A Efros, and Eli Shechtman.  Gan-supervised dense visual alignment.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13470–13481, 2022.
  * [48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [49] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [50] Suman Ravuri and Oriol Vinyals.  Classification accuracy score for conditional generative models.  Advances in neural information processing systems, 32, 2019.
  * [51] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287–2296, 2021.
  * [52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [53] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [54] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  In ACM SIGGRAPH 2022 Conference Proceedings, pages 1–10, 2022.
  * [55] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [56] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [57] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  arXiv preprint arXiv:2210.08402, 2022.
  * [58] Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb.  Learning from simulated and unsupervised images through adversarial training.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2107–2116, 2017.
  * [59] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al.  Make-a-video: Text-to-video generation without text-video data.  arXiv preprint arXiv:2209.14792, 2022.
  * [60] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265, Lille, France, 07–09 Jul 2015\. PMLR.
  * [61] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  Advances in Neural Information Processing Systems, 32, 2019.
  * [62] Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf.  Zero-shot image-to-text generation for visual-semantic arithmetic.  arXiv preprint arXiv:2111.14447, 2021.
  * [63] Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi.  Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training.  arXiv preprint arXiv:2210.08773, 2022.
  * [64] Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or.  Designing an encoder for stylegan image manipulation.  ACM Transactions on Graphics (TOG), 40(4):1–14, 2021.
  * [65] Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn.  Repurposing gans for one-shot semantic part segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4475–4485, 2021.
  * [66] Yuri Viazovetskyi, Vladimir Ivashkin, and Evgeny Kashin.  Stylegan2 distillation for feed-forward image manipulation.  In European conference on computer vision, pages 170–186. Springer, 2020.
  * [67] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen.  Pretraining is all you need for image-to-image translation.  arXiv preprint arXiv:2205.12952, 2022.
  * [68] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao.  Simvlm: Simple visual language model pretraining with weak supervision.  arXiv preprint arXiv:2108.10904, 2021.
  * [69] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  arXiv preprint arXiv:2109.01652, 2021.
  * [70] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  arXiv preprint arXiv:2204.00598, 2022.
  * [71] Wanfeng Zheng, Qiang Li, Xiaoyan Guo, Pengfei Wan, and Zhongyuan Wang.  Bridging clip and stylegan through latent alignment for image editing.  arXiv preprint arXiv:2210.04506, 2022.
  * [72] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.

##  Appendix A Implementation Details

###  A.1 Instruction and Caption Generation

We finetune GPT3 to generate edit instructions and edited captions. The text
prompt used during fine-tuning is the input caption concatenated with "\n##\n"
as a separator token. The text completion is a concatenation of the
instruction and edited caption with "\n%%\n" as a separator token in between
the two and "\nEND" appended to the end as the stop token. During inference,
we sample text completions given new input captions using temperature=0.7 and
frequency_penalty=0.1. We exclude generations where the input and output
captions are the same.

###  A.2 Paired Image Generation

We generate paired before/after training images from paired before/after
captions using Stable Diffusion [52] in combination with Prompt-to-Prompt
[17]. We use exponential moving average (EMA) weights of the Stable Diffusion
v1.5 checkpoint and the improved ft-MSE autoencoder weights. We generate
images with 100 denoising steps using an Euler ancestral sampler with
denoising variance schedule proposed by Kerras _et al._ [25]. We ensure the
same latent noise is used for both images in each generated pair (for initial
noise as well as noise introduced during stochastic sampling).

Prompt-to-Prompt replaces cross-attention weights in the second generated
image differently based on the specific edit type: word swap, adding a phrase,
increasing or decreasing weight of a word. We instead replaced _self_
-attention weights of the second image for the first $p$ fraction of steps,
and use the same attention weight replacement strategy for all edits.

We generation $100$ pairs of images for each pair of captions. We filter
training data for an image-image CLIP threshold of 0.75 to ensure images are
not too different, an image-caption CLIP threshold of 0.2 to ensure images
correspond with their captions, and a directional CLIP similarity of 0.2 to
ensure the change in before/after captions correspond with the change in
before/after images. For each each pair of captions, we sort any image pairs
that pass all filters by the directional CLIP similarity and keep up to 4
examples.

###  A.3 Training InstructPix2Pix

We train our image editing model for 10,000 steps on $8\times$ 40GB NVIDIA
A100 GPUs over $25.5$ hours. We train at $256\times 256$ resolution with a
total batch size of 1024. We apply random horizontal flip augmentation and
crop augmentation where images are first resized randomly between 256 and 288
pixels and then cropped to 256. We use a learning rate of $10^{-4}$ (without
any learning rate warm up). We initialize our model from EMA weights of the
Stable Diffusion v1.5 checkpoint, and adopt other training settings from the
public Stable Diffusion code base.

While our model is trained at $256\times 256$ resolution, we find it
generalized well to $512\times 512$ resolution at inference time, and generate
results in this paper at $512$ resolution with 100 denoising steps using an
Euler ancestral sampler with denoising variance schedule proposed by Kerras
_et al._ [25]. Editing an image with our model takes roughly $9$ seconds on an
A100 GPU.

##  Appendix B Classifier-free Guidance Details

As discussed in Section 3.2.1, we apply classifier-free guidance with respect
to two conditionings: the input image $c_{I}$ and the text instruction
$c_{T}$. We introduce separate guidance scales $s_{I}$ and $s_{T}$ that enable
separately trading off the strength of each conditioning. Below is the
modified score estimate for our model with classifier-free guidance (copied
from Equation 3):

$\begin{split}\tilde{e_{\theta}}(z_{t},c_{I},c_{T})=&\>e_{\theta}(z_{t},\varnothing,\varnothing)\\\
&+s_{I}\cdot(e_{\theta}(z_{t},c_{I},\varnothing)-e_{\theta}(z_{t},\varnothing,\varnothing))\\\
&+s_{T}\cdot(e_{\theta}(z_{t},c_{I},c_{T})-e_{\theta}(z_{t},c_{I},\varnothing))\end{split}$

Our generative model learns $P(z|c_{I},c_{T})$, the probability distribution
of image latents $z=\mathcal{E}(x)$ conditioned on an input image $c_{I}$ and
a text instruction $c_{T}$. We arrive at our particular classifier-free
guidance formulation by expressing the conditional probability as follows:

$P(z|c_{T},c_{I})=\frac{P(z,c_{T},c_{I})}{P(c_{T},c_{I})}=\frac{P(c_{T}|c_{I},z)P(c_{I}|z)P(z)}{P(c_{T},c_{I})}$

Diffusion models estimate the score [23] of the data distribution, i.e., the
derivative of the log probability. Taking the logarithm gives us the following
expression:

$\begin{split}\log(P(z|c_{T},c_{I}))=&\>\log(P(c_{T}|c_{I},z))+\log(P(c_{I}|z))\\\
&+\log(P(z))-\log(P(c_{T},c_{I}))\end{split}$

Taking the derivative and rearranging we attain:

$\begin{split}\nabla_{z}\log(P(z|c_{T},c_{I}))=&\>\nabla_{z}\log(P(z))\\\
&+\nabla_{z}\log(P(c_{I}|z))\\\ &+\nabla_{z}\log(P(c_{T}|c_{I},z))\end{split}$

This corresponds with the terms in our classifier-free guidance formulation in
Equation 3. Our guidance scale $s_{I}$ effectively shifts probability mass
toward data where an implicit classifier $p_{\theta}(c_{I}|z_{t})$ assigns
high likelihood to the image conditioning $c_{I}$, and our guidance scale
$s_{T}$ effectively shifts probability mass toward data where an implicit
classifier $p_{\theta}(c_{T}|c_{I},z_{t})$ assigns high likelihood to the text
instruction conditioning $c_{T}$. Our model is capable of learning these
implicit classifiers by taking the differences between estimates with and
without the respective conditional input. Note there are multiple possible
formulations such as switching the positions of $c_{T}$ and $c_{I}$ variables.
We found that our particular decomposition works better for our use case in
practice.


# Improving Segmentation for Technical Support Problems

Kushal Chauhan
ABV-IIITM, Gwalior
<EMAIL_ADDRESS>
&Abhirut Gupta
IBM Research AI
<EMAIL_ADDRESS>
Work done at IBM Research during a summer internship Now at Google

###### Abstract

Technical support problems are often long and complex. They typically contain
user descriptions of the problem, the setup, and steps for attempted
resolution. Often they also contain various non-natural language text elements
like outputs of commands, snippets of code, error messages or stack traces.
These elements contain potentially crucial information for problem resolution.
However, they cannot be correctly parsed by tools designed for natural
language. In this paper, we address the problem of segmentation for technical
support questions. We formulate the problem as a sequence labelling task, and
study the performance of state of the art approaches. We compare this against
an intuitive contextual sentence-level classification baseline, and a state of
the art supervised text-segmentation approach. We also introduce a novel
component of combining contextual embeddings from multiple language models
pre-trained on different data sources, which achieves a marked improvement
over using embeddings from a single pre-trained language model. Finally, we
also demonstrate the usefulness of such segmentation with improvements on the
downstream task of answer retrieval.

##  1 Introduction

Problems, reported by users of software or hardware products - called tickets
or cases, are often long and complex. Along with a description of the problem,
users often report the setup, steps they have tried at mitigating the problem,
and explicit requests. These problems also contain various non-natural
language elements like snippets of code or commands tried, outputs of commands
or software tools, error messages or stack traces, contents of log files or
configuration files, and lists of key-value pairs. Figure 1 shows a sample
support problem from AskUbuntu 111https://askubuntu.com/, where all such
segments are labeled.

Figure 1: Various non-natural language segments labelled from a problem on
AskUbuntu

While these segments are important sources of information for the human
reader, they are difficult to handle for systems built to automatically answer
support problems. As noted in Gupta et al. (2018), the non-natural language
segments lead to parsing mistakes, and errors in the understanding of support
problems. Correctly identifying these segments can also augment problem
understanding. For instance, a retrieval engine with error messages and their
solutions indexed in distinct fields would return better results with a
fielded query containing just the error message from the ticket. Specialized
tools for log analysis He et al. (2016) could also be run specifically on the
identified log segment of problems.

(a)

(b)

Figure 2: Sample problems from Ask Ubuntu with $\langle code\rangle$ tag used
to present (a) an error message, and (b) contents of a configuration file

In this paper, we aim to address the problem of identifying and extracting
these non-natural language segments from support tickets. In particular, we
choose to focus on the following six segment labels which appear often in
support tickets (also shown in Figure 1):

  * •

Command / Code: Includes terminal commands and programming code snippets

  * •

Command Output / Info Message: Includes outputs of successful command/code
executions

  * •

Error Message / Stack Trace: Includes error traces resulting from unsuccessful
command/code executions

  * •

File Content (Not Code): Includes contents of log files, configuration files,
etc. which do not contain programming source code

  * •

Path / URL: Includes file paths or webpage URLs

  * •

Semi-Structured Information: Includes text which is structured in the form of
key-value pairs, lists, etc., often used to convey system configurations or
lists of components

We formulate the problem as a sequence labelling task, with word-level tags
used to encode segments. To leverage the rich literature of supervised
approaches in this framework, we also create a dataset with segments tagged
for questions from AskUbuntu 222Data available at
https://github.com/kushalchauhan98/ticket-segmentation.

Our contributions are as follows -

  1. 1.

We introduce a novel task towards understanding technical support problems,
which has implications on a variety of downstream applications. We also
release a tagged dataset of problems for the task.

  2. 2.

We benchmark the performance of state of the art sequence labelling models on
the task, studying their performance and limitations. This hopefully provides
direction for future research.

  3. 3.

Given the relatively small size of tagged data, we also explore pre-training
based approaches. Our model leverages activations from multiple language
models pre-trained on different data sources, and we show how they can be used
to improve performance on the task.

##  2 Related Work

Understanding technical support problems is a particularly difficult task,
owing to the long text of problems. In Gupta et al. (2018), the authors
propose that understanding can be approached by extracting attributes of the
ticket that correspond to the description of the problem (symptom), steps
taken for mitigation (attempt), and explicit requests (intent). They also
propose a dependency parser-based approach for extracting these attributes.
However, while this approach pays attention to the semantics of the problem,
the syntactical idiosyncrasies are ignored.

The idea of segmenting of questions for improvements on downstream tasks is
not new. In Wang et al. (2010), the authors propose an unsupervised graph-
based approach for segmenting questions from Community Question Answering
(cQA) websites into sub-questions and their related context sentences. The
authors demonstrate improvements in question retrieval by using these segments
for more granular similarity matching. Chrupała (2013) uses representations
from a character-level language model for segmenting code spans in Stack
Overflow posts. The author uses $\langle code\rangle$ tags in HTML sources of
posts for supervised training of a character level sequence labelling model.
However, the $\langle code\rangle$ tags in the posts usually include all forms
of non-natural language text like code snippets, command outputs, error
messages or stack traces, and file paths (See Fig 2). The resulting level of
granularity is thus insufficient for effective application in downstream tasks
such as automated problem resolution. The task of text-segmentation in itself
has been well studied in the literature, with popular unsupervised approaches
like TextTiling Hearst (1997) and C99 Choi (2000). While, the problem of
ticket segmentation, as defined by us, involves both segmenting and
identifying segment types, we compare the performance of a more recent
supervised segmentation approach Koshorek et al. (2018) against our proposed
model.

Significant amount of work has been done on using sequence labelling
approaches for text segmentation tasks Huang et al. (2015); Chiu and Nichols
(2016); Lample et al. (2016); Ma and Hovy (2016); Rei et al. (2016); Peters et
al. (2017). In Wang et al. (2018) the authors use ELMo embeddings and a
biLSTM-CRF based architecture with self-attention for the task of neural
discourse segmentation. We adopt a similar architecture, and explore the
effect of using pre-trained contextual embeddings on our task. Given the fact
that different segments in technical support problems have very different
vocabularies, we also explore leveraging pre-trained Language Models on a
variety of different datasets.

##  3 Data

| #Questions | Avg. #Words | Avg. #Spans
---|---|---|---
| Total | CC | CO | ES | FC | SS | PU
Dataset | 1317 | 897.37 | 4.86 | 2.13 | 1.20 | 0.62 | 0.30 | 0.14 | 0.46
Train | 1053 | 899.33 | 4.91 | 2.14 | 1.20 | 0.63 | 0.30 | 0.14 | 0.49
Val | 131 | 783.43 | 4.67 | 2.17 | 1.04 | 0.66 | 0.26 | 0.19 | 0.36
Test | 133 | 994.10 | 4.64 | 2.08 | 1.36 | 0.47 | 0.35 | 0.09 | 0.28
Table 1: Statistics of the tagged dataset for segmentation with average number
of words and spans per question. The last 6 columns contain average number of
spans for each tag type - CC: Command/Code, CO: Command Output, ES: Error
Message/Stack Trace, FC: File Content, SS: Semi-structured Information, PU:
Path/URL Figure 3: Relative frequencies of each tag in the dataset. Figure 4:
Confusion Matrix to show the word-level agreement between annotations of 2
annotators on 50 questions. The relatively large off-diagonal values represent
the inherent difficulty in the task. Abbreviations for tags - CC:
Command/Code, CO: Command Output, ES: Error Message/Stack Trace, FC: File
Content, SS: Semi-structured Information, PU: Path/URL

Our dataset is derived from English questions on Ask Ubuntu. Questions posted
on the website are similar to proprietary tech support tickets (in terms of
question length, number of keywords/noun phrases, etc). We would like to point
out that while posts on the website support the $\langle code\rangle$ HTML
tag, it is not granular enough for our downstream tasks. These tags are also
often abused to present snippets of command outputs/error messages/file paths
etc. Figure 2 shows examples of such questions. We also do not use other
metadata available (like turn-based information) with the data dump because
these are not available with proprietary tickets.

Tagging is performed at the word level, and we use the BIO tagging scheme. We
have a pair of Begin and Inside tags for each of the 6 non-natural language
segments, and natural language segments are labelled O, totalling to 13 tags.
We use the Doccano tool 333https://github.com/chakki-works/doccano for
labelling, which provides better support for labelling long chunks in big
documents compared to other popular sequence labelling annotation tools.

We obtain labelling for 1,317 questions, totalling to 11,580 spans (including
spans labelled as O) and over 1.18 million words. We divide the data into
80:10:10 train, val, and test splits, at random. High-level statistics for the
dataset are presented in Table 1. Figure 3 shows the average number of words
per tag in the dataset. The tags Command Output and Error Message are
relatively infrequent (1.2 and 0.6 per question) compared to the tag Command
Code (2.1 per question), however, they cover a much larger fraction of words
because they tend to be quite verbose.

In Figure 4 we show the inter-annotator agreement between two annotators on 50
questions. Few of the label pairs with large off-diagonal values include -

  * •

Command Output \- Error Message, which is understandable, as error messages
are often interspersed in successful program runs. Conversely, unsuccessful
program runs often contain a long train of success messages, only ending in
one or few error logs.

  * •

Command Output \- Semi-Structured Information and File Content \- Semi-
Structured Information. This kind of confusion is due to the presence of
network configurations, commands to view these, and files that contain these.
They’re often stored in configuration files as “key-value” pairs

  * •

Command Output \- File Content. This particular confusion stems from the “cat”
command, and its use to view the contents of files.

The low inter-annotator agreement ($\kappa=0.7637$) illustrates the inherent
difficulty of the task. At this point, it’s important to note that while
there’s some confusion in identifying labels for these segments, the need for
these separate labels stems from downstream tasks.

##  4 Model

Figure 5: Model architecture for segmenting technical support problems.

Given a technical support question, we formulate the segmentation problem as a
sequence labelling task. It is an intuitive choice, given its efficacy for
similar text segmentation problems like discourse segmentation Wang et al.
(2018) and chunking Peters et al. (2017). Figure 5 presents an overview of our
model. We explore different embeddings for each word (character-based
embeddings, pre-trained embeddings, and pre-trained contextual embeddings).
These word embeddings are then fed to a bi-directional GRU for encoding
context. On the output of the GRU layer, we explore the effect of attention.
Finally, the representations are passed to a CRF to decode the segment labels.
We also study the impact of combining pre-trained contextual embeddings from
multiple language models, trained on different data sources. In the rest of
this section we detail individual components of the model.

###  4.1 Word Embeddings

For distributed word representations, we use skip-gram based word2vec
embeddings (Mikolov et al., 2013) trained on all the questions from Ask
Ubuntu. We also look at fastText word embeddings (Bojanowski et al., 2017),
which enrich word vectors by using subword information, emitting plausible
word representations for unseen or rare words, giving us a significant gain.
We use a 300-dimensional embedding from both word2vec and fastText.

###  4.2 Character Embeddings

In addition to the word-level features we also use bi-directional LSTM based
character-level features similar to Chiu and Nichols (2016), Lample et al.
(2016), and Ma and Hovy (2016). These features encode rich character level
information which can improve performance, especially in syntactic tasks. We
obtain an 80-dimensional representation for each word through the character
bi-LSTM, which is the concatenation of the last hidden state of the forward
and backward LSTMs.

###  4.3 Contextual Embeddings from Language Models

Pre-trained contextual embeddings have been shown to work well on a wide
variety of NLP tasks. In domains with relatively small task-specific training
data, the gains have been substantial (McCann et al., 2017; Akbik et al.,
2018; Peters et al., 2017). We also include contextual embeddings from the
pre-trained bi-directional language model in ELMo (Peters et al., 2018).

We observe that the non-natural language segments exhibit wide differences in
syntactic and semantic structure, as is evident from Fig 1. We propose
contextual embeddings from multiple language models; each trained on a
different data source - English text, code snippets, config/log file contents.
We hypothesize that combined embeddings from language models trained on
separate data sources can capture word relationships better and can give
richer word representations, as opposed to a single model trained on a large
English corpora.

For combining multiple contextual embeddings, we explore two techniques - (1)
a naive concatenation, and (2) a weighted sum, with weights learned from
context-independent DME (Dynamic Meta-Embeddings) and context-dependent CDME
(Contextualised Dynamic Meta-Embeddings) self-attention mechanisms as proposed
by Kiela et al. (2018).

####  4.3.1 DME and CDME

When using embeddings from $n$ different LMs for a training instance with $s$
tokens $\left\\{\mathbf{t}_{j}\right\\}_{j=1}^{s}$, we get contextual
embeddings
$\left\\{\mathbf{w}_{i,j}\right\\}_{j=1}^{s}\in\mathbb{R}^{d_{i}}(i=1,2,\ldots,n)$.

For computing the weighted sum, the embeddings from multiple LMs are first
projected to a common $d^{\prime}$-dimensional space by learned linear
functions:

$\mathbf{w}_{i,j}^{\prime}=\mathbf{P}_{i}\mathbf{w}_{i,j}+\mathbf{b}_{i}(i=1,2,\ldots,n)$
(1)

where $\mathbf{P}_{i}\in\mathbb{R}^{d^{\prime}\times d_{i}}$ and
$\mathbf{b}_{i}\in\mathbb{R}^{d^{\prime}}$. The projected embeddings are then
combined with a weighted sum

$\mathbf{w}_{j}^{DME}=\sum_{i=1}^{n}\alpha_{i,j}\mathbf{w}_{i,j}^{\prime}$ (2)

where $\alpha_{i,j}=g(\\{\mathbf{w}_{i,j}^{\prime}\\}_{j=1}^{s})$ are scalar
weights. In DME, they are learned with the self-attention mechanism:

$\alpha_{i,j}=g\left(\mathbf{w}_{i,j}^{\prime}\right)=\phi\left(\mathbf{a}\cdot\mathbf{w}_{i,j}^{\prime}+b\right)$
(3)

where $\mathbf{a}\in\mathbb{R}^{d^{\prime}}$ and $b\in\mathbb{R}$ are learned
parameters and $\phi$ is the softmax function.

For CDME, the self-attention mechanism is made context-dependent:

$\left.\alpha_{i,j}=g\left(\\{\mathbf{w}_{i,j}^{\prime}\right\\}_{j=1}^{s}\right)=\phi\left(\mathbf{a}\cdot\mathbf{h}_{j}+b\right)$
(4)

where $\mathbf{h}_{j}\in\mathbb{R}^{2m}$ is the $j^{th}$ hidden state of a bi-
directional LSTM which takes $\\{\mathbf{w}_{i,j}^{\prime}\\}_{j=1}^{s}$ as
input, $\mathbf{a}\in\mathbb{R}^{2m}$ and $b\in\mathbb{R}$. $m$ is the number
of hidden units in this LSTM, and it is set to 2 as in the original paper.

####  4.3.2 Data Sources for pre-trained LMs

In addition to the pre-trained ELMo model, we train three additional language
models on different data sources. Each of these are also trained with the ELMo
architecture. The pre-trained model emits word embeddings of size 1024, while
each of our domain-specific models emit embeddings of size 256.

  * •

Code LM: This LM was trained on a concatenation of all text inside the
$\langle code\rangle$ tags of Ask Ubuntu, Super User, and Unix Stack Exchange
posts. The total size of this corpus was approximately 400 MB.

  * •

Prog LM: This LM was trained on a corpus containing programming source code
that was compiled from various code repositories on GitHub. Approximately 275
MB in size, it includes sources in most popular languages such as C, C++,
Python, Java, Go, JavaScript, and Bash.

  * •

Config LM: This LM was trained on a corpus of configuration and log files
present in the system folders of Mac OS and Ubuntu installations. The total
size of the corpus was about 60 MB.

###  4.4 Attention

In Wang et al. (2018), the authors experiment with a restricted attention
mechanism on top of the LSTM hidden representations. This is not appropriate
for our task since the questions are fairly long (averaging around 900 words)
and signals indicating the start or end of a segment might appear far away.
Since RNNs are known to be poor at modelling very long-distance dependencies,
we also experiment with the inclusion of the Scaled Dot-Product Attention
layer (Vaswani et al., 2017) on top of the bi-directional GRU. This attention
layer requires the computation of 3 matrices (Key, Query, Value) from the RNN
hidden states, which entails a large number of extra parameters to be learned.
Therefore, we also try a version of attention where all the three matrices are
set equal to the hidden states of the GRU. We call these two approaches
“weighted” and “un-weighted” attention, in our experiments.

##  5 Experimental Setup

With the setup above, we study the performance of various model components on
the task of segmenting support problems. To put the performance in
perspective, we also compare against three baselines detailed in Section 5.1.
The evaluation metrics are carefully selected, avoiding an exact evaluation of
such long and noisy segments, and rewarding partial retrieval of segments. The
chosen evaluation metric is discussed in Section 5.2. Finally, to demonstrate
the usefulness of the task, we evaluate the performance of answer retrieval
with segmentation (Section 5.3).

All baselines and sequence labelling models are trained on the train split,
and fine-tuned on the validation split. For the baselines, we only tune the
regularization strength parameter. For the sequence labelling model, we tune
the dropout and recurrent dropout parameters, as well as the learning rate.
Our best performing models have a dropout of 0.3, recurrent dropout of 0, and
learning rate of 1e-3. All results are then reported on the test split.

###  5.1 Baseline

The task of segmenting technical support problems can be thought to be
comprised of two distinct sub-tasks - (1) segmentation of text, (2)
identification of the segment label. With these in mind, we propose 3 baseline
methods -

  1. 1.

Sentence Only Baseline - Segmentation is done trivially with newlines and
sentence boundaries serving as segment boundaries. The label for a segment is
determined using just the current sentence as input.

  2. 2.

Sentence Context Baseline - Segmentation is done identically to the Sentence
Only baseline. The label for a segment is determined using the immediate
neighbouring sentences along with the current sentence as input.

  3. 3.

Supervised Text Segmentation Baseline - Segments are identified with the
supervised algorithm for segmenting text as described in Koshorek et al.
(2018). The label for each segment is identified with all the text contained
in it as input.

For training the supervised text segmentation model from Koshorek et al.
(2018) we use the whole data dump from AskUbuntu, with the $\langle
code\rangle$ and $\langle/code\rangle$ html tags serving as segment
boundaries.

For identifying segments (in all three baselines) we use a Logistic Regression
classifier with representation from ELMo as input features. Segment
representations are created by mean pooling the contextual representation of
the comprising words from ELMo.

###  5.2 Evaluation Metrics

Segments in our dataset are typically quite long, therefore evaluation based
on an exact match is quite harsh. Keeping this in mind, we resort to soft
precision and recall metrics. We adopt proportional overlap based metrics,
used for the task of opinion expression detection, as proposed by Johansson
and Moschitti (2010).

Towards the calculation of soft precision and recall, consider two spans $s$
and $s^{\prime}$ with labels $l$ and $l^{\prime}$ respectively. The span
coverage, $c$, is defined as how well $s^{\prime}$ is covered by $s$:

$c\left(s,s^{\prime}\right)=\frac{\left|s\cap
s^{\prime}\right|}{\left|s^{\prime}\right|}\;\text{if $l=l^{\prime}$, 0
otherwise}$ (5)

Using span coverage, the span set coverage of a set of spans $\boldsymbol{S}$
with respect to another set of spans $\boldsymbol{S^{\prime}}$ is computed as
follows:

$C\left(\boldsymbol{S},\boldsymbol{S^{\prime}}\right)=\sum_{s_{j}\in\boldsymbol{S}}\sum_{s_{k}^{\prime}\in\boldsymbol{S^{\prime}}}c\left(s_{j},s_{k}^{\prime}\right)$
(6)

Using the span set coverage, we can now define the soft precision $P$ and
recall $R$ of a predicted set of spans $\hat{\boldsymbol{S}}$ with respect to
the gold standard set of spans $\boldsymbol{S}$:

$P(\boldsymbol{S},\hat{\boldsymbol{S}})=\frac{C(\boldsymbol{S},\hat{\boldsymbol{S}})}{|\hat{\boldsymbol{S}}|}\quad
R(\boldsymbol{S},\hat{\boldsymbol{S}})=\frac{C(\hat{\boldsymbol{S}},\boldsymbol{S})}{|\boldsymbol{S}|}$
(7)

In this equation, the operator $|\cdot|$ counts the no. of spans in the span
set.

###  5.3 Retrieval

An important task in the automation of technical support is the retrieval of
the most relevant answer document for a given ticket (from a corpus of product
documentation, FAQ docs, frequent procedures). In this experiment we
demonstrate the usefulness of segmenting support tickets towards this goal. We
index the text of about 250,000 answers from AskUbuntu with ElasticSearch
444https://www.elastic.co/products/elasticsearch. Answers with a large number
of downvotes, and very short answers are ignored. We use questions from our
annotated dataset as search queries. We then compare the retrieval performance
of querying with the whole question against a query with separate fields
corresponding to each segment. In the fielded query, we set different boost
values for the identified segments. Boosting a specific segment of the
question with a higher value causes it to have more significance in the
relevance score calculation in ElasticSearch. To decide the boost values, we
calculate the average percentage word overlap between a segment in the
question and its correct answer from AskUbuntu on the train and val sets. To
compare retrieval performance, we evaluate the Mean Reciprocal Rank (MRR) of
the correct answer for questions in the test set.

##  6 Results

Model | P | R | F1
---|---|---|---
Sent. Only Baseline | 47.77 | 31.75 | 38.15
Sent. Context Baseline | 52.52 | 34.03 | 41.3
Supervised Text Segmentation Baseline | 44.13 | 40.43 | 42.20
SL w/o LM embeddings | 74.57 | 75.51 | 75.04
SL + pre-trained ELMo | 76.88 | 74.49 | 75.67
SL + CDME combined pre-trained Embeddings | 78.30 | 79.29 | 78.80
Table 2: Results comparing the three baselines against variants of our
sequence labelling model. The best performing variant uses CDME to combine
pre-trained embeddings from multiple language models trained on different
datasources.

Table 2 presents evaluation metrics for the three baselines against three
variants of our sequence labelling model. The first variant does not use pre-
trained embeddings from language models, the second uses just pre-trained
ELMo, while the third combines pre-trained embeddings from multiple language
models using CDME. All three variants use fastText for word embeddings (refer
Section 6.1), character-based embeddings, and do not have attention mechanism
before the final CRF layer (refer Section 6.2).

As one would expect, the Context Baseline performs much better than the
Sentence Only Baseline. The sequence labelling models, however, outperform
both the baselines by a huge margin, demonstrating the effectiveness of the
model on the task. Specifically, the best performance is achieved by combining
pre-trained embeddings from multiple language models trained on different data
sources. It significantly outperforms the model using embeddings from a single
pre-trained model on English (explored in Section 6.3).

In the following section we present results from the various model components
we explored.

###  6.1 Effect of fastText

Row 1 and 4 in Table 3 presents the comparison between models using word
embeddings from word2vec and fastText. Both word2vec and fastText embeddings
are trained on all posts in the Ask Ubuntu dataset. As we can see, fastText
gives a marked improvement over using embeddings from word2vec. This is
probably due to the nature of the vocabulary in our task. Since large portions
of questions are spans of command output or error messages a lot of tokens
appear very rarely. In fact, out of the 62,501 unique tokens in the dataset,
57% appear just once, and 78% appear 3 or fewer times. However, the characters
in these tokens are probably very informative (for example “http” in a token
would signal that the token is a URL). Therefore, fastText, which uses n-grams
from a token to compute embeddings, would emit more meaningful
representations.

As a simple experiment, we check the similarity of two URLs from the dataset
that appear just once - http://paste.ubuntu.com/1403448/ and
http://paste.ubuntu.com/14545476/. While the cosine similarity of Word2Vec
vectors for the two is $-0.07$, the similarity between the fastText vectors is
$0.99$.

Model | P | R | F1
---|---|---|---
Word2Vec (w/o Attn) | 65.20 | 58.59 | 61.72
\+ weighted Attn. | 62.34 | 57.0 | 59.55
\+ un-weighted Attn. | 69.21 | 56.15 | 62.0
fastText | 74.57 | 75.51 | 75.04
Table 3: Results for experiments between using Word2Vec and fastText
embeddings. Also includes results of using attention on top of the model with
Word2Vec. Since attention results were not promising, we did not repeat them
with fastText.

###  6.2 Effect of Attention

Given the long tickets in our dataset, and un-reasonably long lengths of spans
for labels like command output or error messages, we explored the usefulness
of attention in our model. We used the Scaled Dot-Product Attention as in
Vaswani et al. (2017). Rows 2 and 3 in Table 3 present the results of using
attention. We find that weighted attention actually hurts performance. This
could be because of the large number of extra parameters introduced in the
calculation of Key, Value, and Query matrices. While the un-weighted version
gets around this by using the bi-directional GRU hidden states as all 3
matrices, it doesn’t improve results significantly either.

###  6.3 Effect of Contextual Pre-Trained Embeddings

As detailed in Section 4.3, we explore the impact of pre-trained contextual
embeddings. We also test our hypothesis, that combining pre-trained embeddings
from different data sources would perform better on our task than using
embeddings from a language model trained on a single data source. The
combination is also performed in two ways - naive concatenation of embeddings
from all language models, and weighted combination using DME and CDME as in
Kiela et al. (2018).

Model | P | R | F1
---|---|---|---
No Pretraining | 74.57 | 75.51 | 75.04
Simple Concat - 1 (en) | 76.88 | 74.49 | 75.67
Simple Concat - 2 (en + config) | 77.67 | 76.12 | 76.89
Simple Concat - 3 (en + code + config) | 79.64 | 77.72 | 78.67
Simple Concat - 4 (ALL) | 76.05 | 76.65 | 76.35
DME | 77.42 | 75.82 | 76.61
CDME | 78.30 | 79.29 | 78.80
Table 4: Results comparing the models using various pre-trained embeddings.
The en data source is the downloaded pre-trained ELMo model. For simple
concatenation, we present the results for the best model at each n
combinations of data sources. For example, when concatenating any 2
datasources, the en + config combination gives the best performance.

Table 4 summarizes these results. For the simple concatenation method, we
present results for the best $n$-way combination of embeddings from different
data sources, for each $n$ (1, 2, 3, and 4). We find that combining embeddings
from multiple language models trained on different data sources considerably
outperforms using embeddings from a single pre-trained model (using both the
naive concatenation and CDME). This is an artifact of the support problems
containing large sections of non-natural language text. We also find that
contextual weighting does better than a simple concatenation.

###  6.4 Retrieval of the Correct Answer

Method | MRR
---|---
Full Question | 0.292
Segmented Question - Gold | 0.300
Segmented Question - Predicted | 0.298
Table 5: Retrieval results, comparing the performance of querying with the
full question against segmented question (gold segments and predicted
segments)

Table 5 presents results for the retrieval experiment. We show that weighing
identified segments of the question with separate weights improves retrieval
of the correct answer over a query with all tokens from the question. We also
present results from the gold annotations of segments for these questions, as
an upper-bound of the performance improvement we can hope to achieve.

##  7 Conclusion

In this paper, we introduce and address an important problem towards a better
understanding of support tickets - segmentation of various non-natural
language segments. We create an annotated dataset for the task, on questions
from the publicly available website, Ask Ubuntu. We also study the performance
of the most recent Recurrent Neural Network-based approaches to sequence
labelling, on this task. In the end, we propose the novel idea of combining
pre-trained embeddings from language models trained on different data sources,
which substantially improves performance. We also demonstrate the usefulness
of the task with improvements in retrieval of the correct answer. Our future
research direction includes a thorough study of differences in this dataset
with actual tickets, and potential for transfer. It is still valuable to study
models on open datasets, however, as these are readily available to the
community.

## References

  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 1638–1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ , 5:135–146.
  * Chiu and Nichols (2016) Jason P.C. Chiu and Eric Nichols. 2016.  Named entity recognition with bidirectional LSTM-CNNs.  _Transactions of the Association for Computational Linguistics_ , 4:357–370.
  * Choi (2000) Freddy Y. Y. Choi. 2000.  Advances in domain independent linear text segmentation.  In _Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference_ , NAACL 2000, page 26–33, USA. Association for Computational Linguistics.
  * Chrupała (2013) Grzegorz Chrupała. 2013.  Text segmentation with character-level text embeddings.  Workshop on Deep Learning for Audio, Speech and Language Processing, ICML 2013, Atlanta, United States.
  * Gupta et al. (2018) Abhirut Gupta, Anupama Ray, Gargi Dasgupta, Gautam Singh, Pooja Aggarwal, and Prateeti Mohapatra. 2018.  Semantic parsing for technical support questions.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3251–3259, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * He et al. (2016) Shilin He, Jieming Zhu, Pinjia He, and Michael R Lyu. 2016.  Experience report: system log analysis for anomaly detection.  In _2016 IEEE 27th International Symposium on Software Reliability Engineering (ISSRE)_ , pages 207–218. IEEE.
  * Hearst (1997) Marti A. Hearst. 1997.  TextTiling: Segmenting text into multi-paragraph subtopic passages.  _Computational Linguistics_ , 23(1):33–64.
  * Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015.  Bidirectional LSTM-CRF Models for Sequence Tagging.  _arXiv e-prints_ , page arXiv:1508.01991.
  * Johansson and Moschitti (2010) Richard Johansson and Alessandro Moschitti. 2010.  Syntactic and semantic structure for opinion expression detection.  In _Proceedings of the Fourteenth Conference on Computational Natural Language Learning_ , CoNLL ’10, page 67–76, USA. Association for Computational Linguistics.
  * Kiela et al. (2018) Douwe Kiela, Changhan Wang, and Kyunghyun Cho. 2018.  Dynamic meta-embeddings for improved sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1466–1477, Brussels, Belgium. Association for Computational Linguistics.
  * Koshorek et al. (2018) Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018.  Text segmentation as a supervised learning task.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 469–473, New Orleans, Louisiana. Association for Computational Linguistics.
  * Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016.  Neural architectures for named entity recognition.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 260–270, San Diego, California. Association for Computational Linguistics.
  * Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016.  End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1064–1074, Berlin, Germany. Association for Computational Linguistics.
  * McCann et al. (2017) Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017.  Learned in translation: Contextualized word vectors.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6297–6308, Red Hook, NY, USA. Curran Associates Inc.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  In _Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’13, pages 3111–3119, USA. Curran Associates Inc.
  * Peters et al. (2017) Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017.  Semi-supervised sequence tagging with bidirectional language models.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1756–1765, Vancouver, Canada. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Rei et al. (2016) Marek Rei, Gamal Crichton, and Sampo Pyysalo. 2016.  Attending to characters in neural sequence labeling models.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers_ , pages 309–318, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_ , pages 5998–6008. Curran Associates, Inc.
  * Wang et al. (2010) Kai Wang, Zhao-Yan Ming, Xia Hu, and Tat-Seng Chua. 2010.  Segmentation of multi-sentence questions: Towards effective question retrieval in CQA services.  In _Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’10, page 387–394, New York, NY, USA. Association for Computing Machinery.
  * Wang et al. (2018) Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018.  Toward fast and accurate neural discourse segmentation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 962–967, Brussels, Belgium. Association for Computational Linguistics.