#  Instruct-Imagen: Image Generation with Multi-modal Instruction

Hexiang Hu♠⋆ Kelvin C.K. Chan♢⋆ Yu-Chuan Su♢⋆ Wenhu Chen♠⋆
Yandong Li♢ Kihyuk Sohn♢ Yang Zhao♢ Xue Ben♢ Boqing Gong♢
William Cohen♠ Ming-Wei Chang♠ Xuhui Jia♢
♠Google DeepMind ♢Google Research
<EMAIL_ADDRESS>
###### Abstract

This paper presents Instruct-Imagen, a model that tackles heterogeneous image
generation tasks and generalizes across unseen tasks. We introduce multi-modal
instruction for image generation, a task representation articulating a range
of generation intents with precision. It uses natural language to amalgamate
disparate modalities (_e.g_., text, edge, style, subject, _etc_.), such that
abundant generation intents can be standardized in a uniform format.

We then build Instruct-Imagen by fine-tuning a pre-trained text-to-image
diffusion model with two stages. First, we adapt the model using the
retrieval-augmented training, to enhance model’s capabilities to ground its
generation on external multi-modal context. Subsequently, we fine-tune the
adapted model on diverse image generation tasks that requires vision-language
understanding (_e.g_., subject-driven generation, _etc_.), each paired with a
multi-modal instruction encapsulating the task’s essence. Human evaluation on
various image generation datasets reveals that Instruct-Imagen matches or
surpasses prior task-specific models in-domain and demonstrates promising
generalization to unseen and more complex tasks. Our evaluation suite will be
made publicly available.

Figure 1:  Zero-shot generalization of Instruct-Imagen. Our model understands
the multi-modal instruction (left) to generate image (right) that reflects the
complex and unseen image transformation.

$\star$$\star$footnotetext:  These authors contributed equally to this work.

##  1 Introduction

The advent of generative artificial intelligence (GenAI) has ushered in an era
of significant advancements in image generation, primarily through text-to-
image models. Existing models such as Stable Diffusion [35], DreamBooth [37],
StyleDrop [42], ControlNet [50] mainly focus on accepting specific instruction
modality like text prompt, subject, style, edge, _etc_. Their ability to
comprehend more complex instructions involving multiple modalities (_e.g_.,
subject + mask + style) is yet to show, not to mention its ability to
generalize to unseen instructions [20].

Unlike the language generation [45, 27, 11, 2, 27], image generation
inherently involves multimodality. In the realm of human artistry, the
painting process often integrates various modalities to achieve the desired
outcome. A painter might start with a rough sketch to outline the composition,
then apply a specific style, like impressionism, for details on texture and
color. They may also use photographs or live models as subject references,
blending these elements to create an expressive piece of art. Communicating
the multi-modal complexities behind such an “image generation” procedure is
challenging, even among humans.

Can we effectively communicate the multi-modal complexities to models? To
address this challenge, we introduce multi-modal instruction in image
generation. This approach interleaves and adheres information from different
modalities, expressing the conditions for image generation (refer to Figure 1
left for examples). Specifically, multi-modal instruction enhances language
instructions, _i.e_., “render an instance of subject images adopting the style
of style image, such that…”, by integrating information from other modalities
(_e.g_., subject and style) to describe the objective of generating a
customized image of the given subject in the provided visual style. As such,
prior image generation tasks with multi-modal conditions can be efficiently
communicated in a human intuitive interface (see § 2).

We then build our model, _i.e_., Instruct-Imagen, employing a two-stage
training approach, to first enhance model’s ability to process multi-modal
instructions, and then faithfully follow the multi-modal user intents. This
involved initially adapting a pre-trained text-to-image model to handle
additional multi-modal inputs, followed by fine-tuning it to accurately
respond to multi-modal instructions. Particularly, we begin by continuing the
text-to-image generation training of a pre-trained diffusion model,
supplemented by similar (image, text) contexts retrieved from a web-scale
(image, text) corpus [6]. In the second stage, we fine-tune the model on
diverse image generation tasks, each paired with multi-modal instructions that
encapsulate the essence of the task. Consequently, Instruct-Imagen excels in
merging diverse modal inputs like sketches and visual styles with textual
directives, producing contextually accurate and visually compelling images.

As illustrated in Figure 1, Instruct-Imagen demonstrates strong capability of
understanding the sophisticated multi-modal instruction to generate the images
faithful to the human intention, even when the instruction combination has
never been observed before. Human studies establishes that Instruct-Imagen not
only matches but, in several instances, surpasses prior task-specific models
within their domains. More significantly, it exhibits a promising
generalization capability when applied to unseen and more complex image
generation tasks.

We summarize our contributions as follows:

  * •

We introduce multi-modal instruction, a task representation universally
represents instruction from multiple modalities, _e.g_., text, edge, mask,
style, subject, _etc_.

  * •

We propose to perform retrieval-augmented training and multi-modal
instruction-tuning to adapt the pre-trained text-to-image models to follow
multi-modal instructions.

  * •

We build Instruct-Imagen, a unified model that tackles heterogeneous image
generation tasks, surpassing the several state-of-the-arts in their domains.

  * •

More substantially, Instruct-Imagen generalizes to unseen and complex tasks,
without any ad hoc design.

##  2 Multi-modal Instructions for Generation

In this section, we start with discussing the preliminary on diffusion models
with input conditions. Then we introduce the format of multi-modal
instruction, and discuss how prior image generation tasks can be unified in
this framework.

Diffusion Models with Input Conditions. Diffusion models [41, 35, 38] are
latent variable models, parameterized by $\Theta$, in the form of
$p_{\Theta}(\bm{x}_{0}):=\int p_{\Theta}(\bm{x}_{0:T})d\bm{x}_{1:T}$, where
$\bm{x}_{1},\cdots,\bm{x}_{T}$ are “noised” latent versions of the input image
$\bm{x}_{0}\sim q(\bm{x}_{0})$. Note that the dimension of both latent and the
image are the same throughout the entire process, with
$\bm{x}_{0:T}\in\mathbb{R}^{d}$ and $d$ indicating the data dimension. The
process that computes the posterior distribution $q(\bm{x}_{1:T}|\bm{x}_{0})$
is called the diffusion process, and is implemented as a predefined Markov
chain that gradually adds Gaussian noise to the data according to a schedule
$\beta_{t}$:

$\displaystyle
q(\bm{x}_{1:T}|\bm{x}_{0})=\prod_{t=1}^{T}q(\bm{x}_{t}|\bm{x}_{t-1});$ (1)
$\displaystyle
q(\bm{x}_{t}|\bm{x}_{t-1}):=\mathcal{N}(\bm{x}_{t};\sqrt{1-\beta_{t}}\bm{x}_{t-1},\beta_{t}\bm{I})$
(2)

Diffusion models are trained to learn the image distribution by reversing the
diffusion Markov chain. Theoretically, this reduces to learning to denoise
$\bm{x}_{t}\sim q(\bm{x}_{t}|\bm{x}_{0})$ into $\bm{x}_{0}$, with a time re-
weighted square error loss [15]:

$\mathbb{E}_{(\bm{x}_{0},\bm{c})\sim
D}\\{\mathbb{E}_{\bm{\epsilon},t}[w_{t}\cdot||\hat{\bm{x}}_{\theta}(\bm{x}_{t},\bm{c})-\bm{x}_{0}||_{2}^{2}]\\}$
(3)

where $D$ is the training dataset containing (image, condition) =
$(\bm{x}_{0},\bm{c})$ pairs. In the text-to-image models, the condition
$\bm{c}$ are often the embeddings of input text prompt, from pre-trained text
embedding models (_e.g_., T5 [32]).

Figure 2: Illustration on how multi-modal intruction uniformly express
existing image generation tasks and extends to new tasks. Examples in this
figure are retrieved from [50, 7, 42]

Unified Multi-modal Instruction. While multi-modality information is necessary
for extended image generation applications, and had been explored in prior
works [50, 42, 7, 22, 37], _etc_., there was not such a format in the
literature that allows generalization. Instead, models often make ad-hoc
design to integrate information from other modalities. For example, ControlNet
[50] combines the input $\bm{x}_{t}$ with a transformed spatial control map
feature to form the new input for reverse diffusion. Such modality and task
specific design, while effective in-domain, is challenging to generalize to
other tasks (_e.g_., stylization). Therefore, we propose the multi-modal
instruction, a new format where language are used to explicitly state the
objective behind tasks, with references to multi-modal conditions.

There are two key components in the proposed instruction format: (1) the
payload text instruction that provides detailed description of the task
objective, with reference marker (_e.g_., [ref#?]). (2) a multi-modal context
with (marker + text, image) pairs. The model then employ a shared instruction
understanding model to consume both the text instruction and the multi-modal
context, regardless of the specific modality in the context. Figure 2
showcased three examples of how this format represents various prior
generation tasks, showing its compatibility to prior image generation tasks.
More importantly, the flexibility of language allows multi-modal instructions
to extend to new tasks, without any modality & task specific design.

##  3 Instruct-Imagen

Figure 3: Overview of the two-staged training pipeline for the proposed
Instruct-Imagen model.

In this section, we first discuss how Instruct-Imagen encodes the input multi-
modal instruction, and how the encoding is leveraged for generation (see §
3.1). Then we introduce the two staged training framework for Instruct-Imagen
in § 3.2. In Figure 3, we present the high-level design of the Instruct-
Imagen, alongside with an overview of its training procedure.

###  3.1 Imagen with Multi-modal Instruction

The foundation of Instruct-Imagen is the multi-modal instruction, which
uniformly represents prior image generation tasks, while remains its
capability to extend to novel and complex tasks. Based on it, we designed the
model architecture that extends a pre-trained text-to-image diffusion models,
_i.e_., a cascaded diffusion model [16], to allow it fully conditioned on the
input multi-modal instruction.

Cascaded Backbone Text-to-Image Model. We used a version of Imagen [38] pre-
trained on internal data sources, which inherents the cascaded text-to-image
diffusion model (see Figure 3 left), as the founding for adaptation to
Instruct-Imagen. The full model has two sub-components: (1) a text-to-image
that generates $128\times$ resolution images from text prompt only, and (2) a
text-conditioned super-resolution model that scales the $128$ resolution up to
high fidelity $1024\times$ images. In the scope of this work, we only consider
training and adapting the 128 resolution text-to-image network, for the sake
of efficiency and clarity. Particularly, the backbone model is a convolutional
UNet [36] with bottleneck, with a paired down-sampling encoder and up-sampling
decoder. The text are then embedded with a pre-trained T5-XXL model [32]. The
embeddings are then input to the down-sampling encoder as condition, and to
the cross-attention on bottleneck representation as enhanced reference.

Encoding Multi-modal Instruction. We adapt the above mentioned cascaded text-
to-image model via maximally reusing the pre-trained text-to-image model for
encoding the multi-modal instruction, and only introduce one cross-attention
layer that conditions the bottleneck representation of UNet with the embedded
multi-modal context the (key, value) pairs. This grows the number of
parameters of our model from 2.51$B$ to 2.76$B$ ($\sim$10%). This design is in
principle similar to the nearest neighbor UNet presented in [6] (but with the
nested encoding on the multi-modal context). Figure 3 (right) illustrates the
dataflow of how a multi-modal instruction is encoded by the Instruct-Imagen.
Here, the payload text instruction is encoded the same way as normal text
input in backbone model. The multi-modal context, _i.e_., both (marker + text,
image) pairs, are first encoded using the down-sampling encoder, same as how
backbone text-to-image model encodes the bottleneck representation, and then
provided as (key, value) pairs for the new cross-attention layer to condition
on. The up-sampling decoder then takes the outcome feature representation to
perform the reverse diffusion.

###  3.2 Training Instruct-Imagen in Two Stages

Our training pipeline is two staged, with the first stage to continue the
text-to-image generation, with augmentation of retrieved neighbor (image,
text) pairs. Then in the second stage, we fine-tune the output model from
first stage on a mixture of diverse image generation tasks, each paired with
corresponding multi-modal instructions. In both training stages, the model are
optimized end-to-end.

Retrieval-augmented Text-to-image Training. The most important research
question for Instruct-Imagen is how to train the model to condition on multi-
modal inputs for its generation, since these tasks deviate from the standard
text-to-image pre-training. A straight-forward thinking is to mine naturally
distributed multi-modal Internet data [1, 52] (such as Wikipedia articles with
images) and train models to use the interleaved (image, text) data to generate
the desired output image. However, this is inadequate to train models with
superior alignment, because the input multi-modal content are often not
relevant to the production of the output image. For example, in the Wikipedia
article, _i.e_., the US president, the headline text, summary text and info-
box images (_i.e_., Biden’s picture) are not informative to generate the image
of Franklin D. Roosevelt. Thus, training model using such data often leads to
ignorance of the multi-modal context.

To alleviate this issue, we employ the training data similar to re-imagen [6],
such that the model can learn to look at the relevant but not duplicated
neighboring multi-modal context when generating image according to the current
text prompt. Particularly, the model would be presented with portraits of
Franklin D. Roosevelt at other occurrences, when asked to generate his
presence delivering the radio address in 1933. A model capable of processing
multi-modal inputs can leverage other Roosevelt images to generate the scene,
instead of memorizing his appearance.

To achieve this, we construct the retrieval-augmented training dataset via
domain-specific clustering of Web (image, text) pairs. First, we processed the
web scale image-text corpus (_i.e_., WebLI [8, 9]) to remove low quality
images (in image quality scores [43]), classified images from specific
clusters (_e.g_., art, products, animals, scenery, _etc_.) via image-text
matching, and performed image clustering within each classified sub-cluster,
using the embeddings from CLIP [31] model. For each mined image cluster, we
took the top 10 nearest neighbor candidates, and performed near-duplication
removal via removing images with high similarity and images with the same
metadata (_e.g_., URL). We then truncate the image cluster to have the size of
5 images (discarded clusters with less than 5 images). As an outcome, this
process produced 8.08 M (image, text) clusters, with 5 pairs per cluster.
During the training, one (image, text) pair is sampled as the input and target
for the Instruct-Imagen, and three other (image, text) pairs are sampled as
the multi-modal context. Additionally, we performed the condition dropout as
[38, 35] but with two independent drop situations: (1) dropping both the input
text and multi-modal context; and (2) dropping only the multi-modal context,
each dropout situation occurs at 10% chance.

Task | Input | Dataset | #Examples | Ratio
---|---|---|---|---
Txt2Img | txt | Internal Data | $5\mathrm{M}$ | $0.15$
WikiArt | $0.1\mathrm{M}$ | $0.05$
Control2Img | depth_img+txt |  Depth WebLI [8] | $5.7\mathrm{M}$ | $0.06$
mask_img+txt |  Mask WebLI [8] | $5.7\mathrm{M}$ | $0.06$
edge_img+txt |  Edge WebLI [8] | $5.7\mathrm{M}$ | $0.06$
Sketch2Image [23] | $15\mathrm{K}$ | $0.02$
Subject Txt2img | sub_imgs+txt |  SuTI dataset [7] | $0.75\mathrm{M}$ | $0.30$
Celeb-A [25] | $0.1\mathrm{M}$ | $0.05$
Celeb-HQ [19] | $0.1\mathrm{M}$ | $0.05$
Style Txt2img | sty_img+txt | Derived from WikiArt | $0.1\mathrm{M}$ | $0.10$
Style Transfer | sty_img+ctn_img | WikiArt + Internal Data | $1\mathrm{M}$ | $0.10$
Table 1: Details of the instruction-tuning datasets and mixing ratio.

Multi-modal instruction-tuning for Image Generation. We prepared 11 image
generation datasets via either re-using existing dataset or synthesizing the
input or target image, which formed 5 task categories, for multi-modal
instruction-tuning. For each dataset, we prompted the GPT-4 [27] to generate
100 rephrased instruction templates with high variation, and validated the
semantic correctness of them manually. We defer the qualitative examples of
each dataset and its associated instruction to the appendix. The Table 1
presents the detailed information about task group, model input conditions,
and data statistics for each prepared dataset, with details below:

  * •

Text-to-image Generation. We processes two datasets for instructed text-to-
image generation: an internal high-quality natural image dataset with manual
caption; and an art specific dataset crawled from WikiArt (using the pipeline
in [44]), with the caption generated by PaLI [8]. Both datasets are augmented
with sampled instruction.

  * •

Control2Image Generation. We followed [50] to prepare the control signals
(_e.g_., depth map, mask, and edge), based on a subset of the WebLI [8].
Specifically, we use MiDas [34] for depth estimation, HED [46] for edge
extraction, and salient object [30] for mask. To improve robustness with
different edge styles, we also employed edge-to-image data from a sketch
dataset [23].

  * •

Subject-driven Generation. We consider two data sources for subjects: general
objects and human instances, for subject-driven generation. Particularly, we
use the subject-driven dataset introduced in SuTI [7] for general object
learning, and the celebrity face datasets [25, 19] to learn face rendering.
For face rendering, we group the faces of the same person and caption them
with PaLI [8], then we use one sampled example as the input/target, and the
rest as multi-modal context. All datasets then join the instruction templates,
with reference markers inserted to refer the multi-modal context.

  * •

Styled Generation. Styled generation is a task that generalizes over the
StyleDrop [42], with a style image and text as input, styled image following
the text as output. To collect such data, we used images from WikiArt as the
collection of style images to train StyleDrop models, and then use the manual
captions from the internal text-to-image dataset to sample images as the
target styled image. We employ a CLIP model to filter out examples that fails
the alignment with either style image or the caption. Then multi-modal
instructions are created via combining the instruction template with style
image and the caption, such that the style image is correctly referred.

  * •

Style Transfer. Similarly, we construct the style transfer dataset via
combining style images from our WikiArt crawl and content images from the
internal dataset (with the captions discarded). Particularly, we employ a
simple style transfer model [13], which allows fast and large-scale
generation, to blend the style image with the content image. These data are
then augmented with instructions.

During the instruction-tuning stage, we fine-tune the output model of the
retrieval-augmented training on the multi-task mixed dataset, with the mixture
ratio specified in Table 1.

##  4 Related Work

Instruction-Tuning. Instruction tuning was first introduced in FLAN [45],
which finetunes a large language model (LLM) on instructions to significantly
improve its zero-shot learning performance on unseen tasks. Chung et al.
extended the work at scale [11], showing extraordinary generalization to
numerous NLP tasks. In general, the instruction data plays a pivotal role in
the finetuned LLM [51]. This success experience in text instruction tuning was
then introduced to the vision-language models [24, 4, 9], which enables
generalization across tasks such as recognition and Visual QAs [14, 26, 17,
10]. While a concurrent work has explored image generation with multi-modal
inputs [28], this paper presents an new initiative to investigate multi-modal
instruction tuning for image generation models.

Controlled Image Synthesis. Recent advancements in text-to-image generative
models [33, 3, 38, 5, 6, 35, 48, 47] have showcased impressive capabilities in
various domains, including creativity, photorealism, diversity, and coherence.
A critical aspect of these advancements is controllability, which has been
enhanced by adapting these models to specific subjects [37, 7], styles [42],
masks [50], _etc_. For example, DreamBooth [37] fine-tunes a text-to-image
model on a limited set of images to better capture the nuances of a specific
subject. Additionally, ControlNet [50] introduces the ability to condition on
a variety of control signals, including depth maps and doodles, by fine-tuning
an auxiliary encoder with the appropriate data pairs. Despite these
advancements, a common limitation persists: these models often specialize in
specific modalities, leaving the generalization to novel modalities and their
combinations unexplored. To address this gap, we introduce Instruct-Imagen, a
novel model designed to understand complex relationships and generalize
effectively to unencountered tasks.

Depth2ImgMask2ImgEdge2ImgSty Gen.Sub Gen.Txt2ImgFace Gen.Sty
Trans.$0$$20$$40$$60$$80$$100$N/A$21$$64$$54$$45$$67$$67$$46$$50$$54$$68$$37$$65$$54$$54$$37$$0$$65$$52$$67$$71$$67$$76$$60$$90$$79$$82$$88$$81$$76$$76$$56$(a)
In-domain EvaluationHuman Score $O$ ($\times 100$)Single-TaskMulti-TaskPrior
MtdInstruct-ImagenSty+SubMulti
SubCtrl+SubCtrl+Sty$48$$54$$36$$36$$33$$43$$32$$11$$58$$53$$60$$63$(b) Zero-
shot Evaluation Figure 4: Human Study on prior methods, baselines, and
Instruct-Imagen. Instruct-Imagen can perform on par or better comparing to the
baselines and prior methods, with best generalization capability to novel
tasks. Instruct-Imagen does not require any fine-tuning for all tasks
(particularly style/subject-related), and inferences at an average speed of
18.2 seconds per example (on TPUv4). Figure 5:  Comparison on a subset of in-
domain tasks. Examples generated from prior methods, baselines, and Instruct-
Imagen. We visualize the multi-modal instruction for human intuitive
understanding (models are evaluated with in-distribution inputs).

##  5 Experiments

In this section, we first introduce the experimental setup, the human
evaluation protocol, and comparative baseline systems in § 5.1. We then
present the main results in § 5.2, highlighting advantages of Instruct-Imagen
in tackling multiple in-domain tasks and challenging unseen tasks. In § 5.3,
we perform an in-depth analysis to study the design of Instruct-Imagen, and
the model’s failure mode.

###  5.1 Experimental Setup

We evaluate our models with two setups, _i.e_., in-domain task evaluation and
zero-shot task evaluation, where the later setup is strictly more challenging
than the former. Particularly, we re-use the recently introduced conditional
image generation benchmark, _i.e_., ImagenHub [20], for evaluating text-to-
image generation. We also employ other datasets to cover in-domain evaluation:
We adopt the DreamBench [37, 7] v1 & v2 as our subject-driven evaluation data;
We use the style images from StyleDrop [42] for style evaluation; We use hold-
out style images from WikiArt [44] and content images from CustomConcept101
[21] for style transfer. We use the evaluation data of WebLI [8] for
control2image (_i.e_., mask, edge, depth) evaluation. For face evaluation, we
evaluate on the validation set of hold-out human in CelebA [25] and CelebA-HQ
[19].

For zero-shot tasks, we either adopt the existing evaluation (_i.e_.,
CustomConcept101 [21] for multi-subject, on the [20]’s split) or construct the
evaluation ourself (_e.g_., subject + control, style + control, style +
subject) by adopting examples from corresponding in-domain task datasets. We
refer the readers to the appendix for complete information about evaluation
datasets. The complete evaluation suite would be made publicly available for
future study and comparison.

Figure 6:  Comparison on a subset of zero-shot tasks. Examples generated from
prior methods, the baseline, and instruct-imagen. We visualize the multi-modal
instruction for human intuitive understanding (models are evaluated with in-
distribution inputs).

Baseline Models. We compare Instruct-Imagen with three category of baseline
models: (1) Prior State-of-the-art method (2) Single-task model (3) Multi-task
model. Since no single prior model can handle all image generation tasks, we
make comparison to different prior method on each task. Particularly, we
compare to: SDXL [29] for text-to-image generation; ControlNet [50] for
edge/depth-to-image generation; Ghiasi  _et al_. [13] for style transfer;
StyleDrop [42] for styled generation; SuTI [7] for subject-driven generation;
and TamingEncoder [18] for face generation. Note that we marked prior method
on Mask2Img task with N/A due to lack of public model. For zero-shot tasks, we
compare to: KOSMOS-G [28] for styled subject generation and multi-subject
generation; and BLIPDiffusion [22] for the other two tasks, given its
capability on accepting multi-modal inputs.

The single-task and multi-task models share the same model architecture as
Instruct-Imagen, but do not have access to the multi-modal instruction during
fine-tuning and inference. Instead, they accept the raw multi-modal inputs
from each task. Additionally, the single-task model requires an independent
model for each task, thereby inducing $7\times$ more parameters than Instruct-
Imagen.

Human Evaluation. We follow the same evaluation protocol as [20] to conduct
systematic human study. Each sample is rated by at least three raters for
their semantic consistency score (SC) and perceptual quality score (PQ). The
score in each category are $\\{0,0.5,1\\}$, where $0$ means inconsistent /
extremely poor quality and $1$ means totally consistent / high quality
respectively. Note that semantic consistency is defined as the score of the
least consistent condition when there are multiple conditions. The final human
score is defined as $O{=}\sqrt{SC{\times}PQ}$. We recruit eight huamn raters
and train them following the
guidelines111https://imagenhub.readthedocs.io/en/latest/Guidelines/humaneval.html
in ImagenHub [20]. Each method is evaluated independently, but we assign the
same rater for samples generated by different methods given the same input to
ensure evaluation calibrated per example.

###  5.2 Main Results

Figure 4 compares Instruct-Imagen with our baselines and prior methods,
showing it achieves similar or superior results in terms of in-domain
evaluation and zero-shot evaluation (the breakdown of $SC$ and $PQ$ is
detailed in the appendix). It suggests that multi-modal instruction training
enhances performance in tasks with limited training data, such as stylized
generation, while maintaining effectiveness in data-rich tasks, such as
photorealistic imaging. Without multi-modal instruction training, our multi-
task baseline tends to yield inferior image quality and text alignment. For
instance, in the in-context stylization example of the Figure 5, the multi-
task baseline struggles to differentiate style from subject, and replicate the
subject in its generation. For similar reason, it generates 0 performance in
the task of style transfer. This observation underscores the value of
instruction tuning.

Distinct from many current approaches that rely on task-specific methods
(_e.g_., StyleDrop [42] \+ DreamBooth [37]) or training [21], Instruct-Imagen
efficiently manages compositional tasks by leveraging instructions that
combines the objectives from individual tasks, and inference in-context (no
fine-tuning required, which takes $18.2$ seconds per example). As shown in
Figure 6, Instruct-Imagen consistently outperforms others in instruction
following and output quality. Furthermore, in the presence of multiple
references in the multi-modal context, the multi-task baseline model fails to
correspond the text instructions to the references, resulting in the ignorance
of some multi-modal conditions. These results further demonstrate the efficacy
of the proposed model. More qualitative visualization in the appendix.

###  5.3 Model Analysis & Ablation Study

Besides the main results, we also perform studies to explore the limit of
Instruct-Imagen, ablate important design of its training, and analyze its
failure mode.

Figure 7:  Instruct-Imagen for masked image editing When fine-tuned on MagicBrush [49], although Instruct-Imagen can edit the image as instructed (_i.e_., see the $128\times 128$ output), the super-resolution model fails to capture details from the input image, and causes the inconsistency.  Method | Setup | Human Score | Accuracy
---|---|---|---
SDXL-inpainting | - |  0.43000000 |  0.25000000
Imagen | Fine-tuned |  0.37000000 |  0.10000000
Instruct-Imagen | Fine-tuned |  0.72 (+0.35) |  0.57 (+0.47)
Table 2: Masked Image Editing Evaluation on ImagenHub [20].

Fine-tuned Instruct-Imagen can edit image. Aside from zero-shot compositional
tasks, another advantage of Instruct-Imagen lies in its adaptability to new
tasks. Particularly, we fine-tuned Instruct-Imagen on the MagicBrush dataset
[49] ($\sim 9K$ examples) for $10K$ steps, and evaluated on the masked image
editing data by ImagenHub [20]. We report the results using the overall score
[20] ($O$), and the accuracy (_i.e_., % of examples where $SC{=}1$). As a
result, Table 2 presents a comparison between prior methods (SDXL-inpainting
[29]), fine-tuned Imagen model (has been retrieval-augmented trained but
without instruction tuning), and fine-tuned Instruct-Imagen. It shows that
once fine-tuned, Instruct-Imagen can perform significantly better than the
baseline method, and also method specifically designed for mask-based image
editing. However, the fine-tuned Instruct-Imagen introduces artifacts into
edited images, particularly in high-resolution outputs after super-resolution,
as depicted in Figure 7. This occurs due to the model’s lack of prior learning
in pixel-accurate copying from context to output, a task significantly
distinct from other Instruct-Imagen tasks.

Retrieval-augmented training helps generalization. We compare variants of
Instruct-Imagen in terms of whether performing retrieval augmented training
and report results in Table 3. It shows the retrieval augmented training is an
important step to obtain superior empirical results, in terms of both in-
domain and zero-shot evaluation. This validates our hypothesis that retrieval
augmented training benefits representing and handling multi-modal context.

Figure 8:  Failure mode of Instruct-Imagen. The most common failure of Instruct-Imagen is its incapability to follow each control condition in the instruction faithfully.  Method | In-domain Eval | Zero-shot Eval
---|---|---
w/o Retrieval-augmented |  0.55000000 |  0.53000000
w/ Retrieval-augmented |  0.79 (+0.25) |  0.59 (+0.06)
Table 3: Ablation study on retrieval-augmented training. We report the average
in-domain and zero-shot eval scores $O$.

Failure mode of Instruct-Imagen. One common pattern we found in Instruct-
Imagen (when attempting more complex multi-modal instructions, with at least 3
multi-modal conditions) is its failure to follow instruction in the
generation. Particularly, the model can accomplish the generation to satisfy
only a subset of conditions specified in the multi-modal instruction. The
first example in Figure 8 shows the model is capable of handling the style and
subject to some extent, but not generate the output in the shape that the mask
specified. In the second example, the model can generate the “plushie in front
of barn” in the given style, but fails to reserve the barn’s appearance.

Figure 9: Additional Qualitative Evaluation of Instruct-Imagen on Control +
Style Generation. Figure 10: Additional Qualitative Evaluation of Instruct-
Imagen on Multi-Subject Generation. Figure 11: Additional Qualitative
Evaluation of Instruct-Imagen on Control + Subject Generation. Figure 12:
Additional Qualitative Evaluation of Instruct-Imagen on Styled Subject
Generation.

Additional qualitative results of Instruct-Imagen. Here, we provide additional
qualitative visualization on more diverse and sophisticated multi-modal
instructions (images are unseen in the model training), to explore the limit
of Instruct-Imagen. Particularly, Figure 9, Figure 10, Figure 11, and Figure
12 jointly presents complex tasks that is unseen during the training. We defer
more comprehensive view of in-domain image generation results to the appendix,
in the Figure 13. Note that we do not provide qualitative results on face
generation due to lack of consent from the original dataset owner.

##  6 Discussion

We introduce Instruct-Imagen, an image generation model that comprehends
multi-modal instruction to accomplish a variety of visual generative tasks. It
marks an initial but significant leap forward general-purpose visual
generative model, via allowing not only in-domain image generation, but also
zero-shot image generation on unseen and complex instructions. While opening
up a new research direction, Instruct-Imagen can not handle image editing
tasks in zero-shot. A key limitation is its lack of pixel consistency with
input images, hindering the inclusion of additional tasks like in-painting and
image editing in the instruction-tuning. This issue stems from the use of a
cascaded diffusion model, which depends on a low-resolution model for crucial
decisions like layout and object semantics. Such a low-resolution model
struggles with both accessing high-resolution input details and reproducing
them in the output, leading to artifacts in the generated image — because the
super resolution model has to hallucinate the details. Based on this
observation, we believe that one promising future direction is developing
diffusion models that operate at the raw image resolution.

## Acknowledgement

We thank Zhiwei Deng, Jason Baldridge, Nando de Freitas for reviewing an early
version of this paper in depth, with valuable comments and suggestions.
Special thanks to Han Zhang for project idea discussion in the early stage of
this project. We also thank Irina Blok for providing a style image used in our
evaluation.

## Broader Impact

Text-to-image generation models like Imagen [38] and Stable Diffusion [29]
present ethical concerns, including social bias. Instruct-Imagen, using
similar Web-scale datasets, faces these same issues. Instruct-Imagen’s
retrieval-augmented training and multi-modal instruction-tuning have notably
enhanced image controllability and attribution. This control can be beneficial
or harmful. A risk is using Instruct-Imagen for malicious activities, such as
creating misleading images of people. Conversely, it offers advantages, like
reducing image hallucination and improving relevance to user intent. It also
benefits minority communities by effectively generating images of less-known
landmarks, foods, and cultural artifacts, addressing the bias in AI systems.
To mitigate public risks, we’ll be cautious with code and API releases. Future
work will focus on a responsible use framework, weighing the benefits of
research transparency against the dangers of open access, ensuring safe and
beneficial usage.

## References

  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In _NeurIPS_ , 2022.
  * Anil et al. [2023] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  PaLm 2 technical report.  _arXiv preprint arXiv:2305.10403_ , 2023.
  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Brooks Tim, Jianfeng Wan, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu, and Yunxin Jiao.  Improving image generation with better captions.  _Technical Report_ , 2023.
  * Bitton et al. [2023] Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt.  VisIT-Bench: A benchmark for vision-language instruction following inspired by real-world use.  _arXiv preprint arXiv:2308.06595_ , 2023.
  * Chang et al. [2023] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al.  Muse: Text-to-image generation via masked generative transformers.  _arXiv preprint arXiv:2301.00704_ , 2023.
  * Chen et al. [2022a] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen.  Re-Imagen: Retrieval-augmented text-to-image generator.  _arXiv preprint arXiv:2209.14491_ , 2022a.
  * Chen et al. [2023a] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W Cohen.  Subject-driven text-to-image generation via apprenticeship learning.  In _NeurIPS_ , 2023a.
  * Chen et al. [2022b] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  PaLI: A jointly-scaled multilingual language-image model.  In _ICLR_ , 2022b.
  * Chen et al. [2023b] Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Carlos Riquelme Ruiz, Sebastian Goodman, Xiao Wang, Yi Tay, et al.  Pali-x: On scaling up a multilingual vision and language model.  _arXiv preprint arXiv:2305.18565_ , 2023b.
  * Chen et al. [2023c] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming-Wei Chang.  Can pre-trained vision and language models answer visual information-seeking questions?  _arXiv preprint arXiv:2302.11713_ , 2023c.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_ , 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  In _NeurIPS_ , 2021.
  * Ghiasi et al. [2017] Golnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens.  Exploring the structure of a real-time, arbitrary neural artistic stylization network.  In _BMVC_ , 2017.
  * Goyal et al. [2017] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In _CVPR_ , 2017.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _NeruIPS_ , 2020.
  * Ho et al. [2022] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  _JMLR_ , 2022.
  * Hu et al. [2023] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang.  Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities.  _arXiv preprint arXiv:2302.11154_ , 2023.
  * Jia et al. [2023] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  _arXiv preprint arXiv:2304.02642_ , 2023.
  * Karras et al. [2017] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  _arXiv preprint arXiv:1710.10196_ , 2017.
  * Ku et al. [2023] Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen.  ImagenHub: Standardizing the evaluation of conditional image generation models.  _arXiv preprint arXiv:2310.01596_ , 2023.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _CVPR_ , 2023.
  * Li et al. [2023] Dongxu Li, Junnan Li, and Steven CH Hoi.  BLIP-Diffusion: Pre-trained subject representation for controllable text-to-image generation and editing.  _arXiv preprint arXiv:2305.14720_ , 2023.
  * Li et al. [2019] Mengtian Li, Zhe Lin, Radomir Mech, Ersin Yumer, and Deva Ramanan.  Photo-Sketching: Inferring contour drawings from images.  In _WACV_ , 2019.
  * Liu et al. [2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _arXiv preprint arXiv:2304.08485_ , 2023.
  * Liu et al. [2018] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Large-scale celebfaces attributes (CelebA) dataset.  _Retrieved August_ , 2018.
  * Marino et al. [2019] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  OK-VQA: A visual question answering benchmark requiring external knowledge.  In _CVPR_ , 2019.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Pan et al. [2023] Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, and Furu Wei.  Kosmos-G: Generating images in context with multimodal large language models.  _arXiv preprint arXiv:2310.02992_ , 2023.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  SDXL: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Qin et al. [2019] Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand.  Basnet: Boundary-aware salient object detection.  In _CVPR_ , 2019.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_ , 2020.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Ranftl et al. [2020] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun.  Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer.  _TPAMI_ , 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In _MICCAI_ , 2015.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  DreamBooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _CVPR_ , 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  In _NeurIPS_ , 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Shazeer and Stern [2018] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _ICML_ , 2018.
  * Sohl-Dickstein et al. [2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _ICML_ , 2015.
  * Sohn et al. [2023] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al.  StyleDrop: Text-to-image generation in any style.  _arXiv preprint arXiv:2306.00983_ , 2023.
  * Talebi and Milanfar [2018] Hossein Talebi and Peyman Milanfar.  Nima: Neural image assessment.  _TIP_ , 2018.
  * Tan et al. [2019] Wei Ren Tan, Chee Seng Chan, Hernan Aguirre, and Kiyoshi Tanaka.  Improved artgan for conditional synthesis of natural image and artwork.  _TIP_ , 2019.
  * Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_ , 2021.
  * ”Xie and Tu [2015] Saining ”Xie and Zhuowen” Tu.  Holistically-nested edge detection.  In _ICCV_ , 2015.
  * Yu et al. [2021] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu.  Vector-quantized image modeling with improved vqgan.  _arXiv preprint arXiv:2110.04627_ , 2021.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2022.
  * Zhang et al. [2023a] Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su.  MagicBrush: A manually annotated dataset for instruction-guided image editing.  _arXiv preprint arXiv:2306.10012_ , 2023a.
  * Zhang et al. [2023b] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , 2023b.
  * Zhang et al. [2023c] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al.  Instruction tuning for large language models: A survey.  _arXiv preprint arXiv:2308.10792_ , 2023c.
  * Zhu et al. [2023] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal c4: An open, billion-scale corpus of images interleaved with text.  _arXiv preprint arXiv:2304.06939_ , 2023.

##  Appendix A Appendix

| Single-Task | Multi-Task | Prior Mtd. | Instruct-Imagen
---|---|---|---|---
| $SC_{avg}$ | $PQ_{avg}$ | Overall | $SC_{avg}$ | $PQ_{avg}$ | Overall | $SC_{avg}$ | $PQ_{avg}$ | Overall | $SC_{avg}$ | $PQ_{avg}$ | Overall
In-domain Evaluation
Depth2Img | 0.09 | 0.65 | 0.24 | 0.51 | 0.37 | 0.44 | 0.64 | 0.55 | 0.59 | 0.86 | 0.66 | 0.75
Mask2Img | 0.79 | 0.60 | 0.68 | 0.67 | 0.53 | 0.60 | 0.50 | 0.41 | 0.45 | 0.87 | 0.70 | 0.78
Edge2Img | 0.73 | 0.51 | 0.61 | 0.46 | 0.33 | 0.39 | 0.48 | 0.58 | 0.53 | 0.84 | 0.71 | 0.77
Sty Gen. | 0.44 | 0.46 | 0.45 | 0.60 | 0.70 | 0.65 | 0.64 | 0.71 | 0.67 | 0.85 | 0.92 | 0.88
Sub Gen. | 0.69 | 0.66 | 0.67 | 0.53 | 0.59 | 0.56 | 0.69 | 0.70 | 0.70 | 0.81 | 0.82 | 0.81
Txt2Img | 0.68 | 0.68 | 0.68 | 0.58 | 0.51 | 0.55 | 0.64 | 0.71 | 0.67 | 0.77 | 0.76 | 0.76
Face Gen. | 0.18 | 0.77 | 0.37 | 0.45 | 0.34 | 0.39 | 0.66 | 0.80 | 0.72 | 0.69 | 0.86 | 0.77
Sty Trans. | 0.43 | 0.43 | 0.43 | 0.00 | 0.49 | 0.00 | 0.58 | 0.56 | 0.57 | 0.55 | 0.50 | 0.53
Average | 0.50 | 0.59 | 0.52 | 0.48 | 0.48 | 0.45 | 0.60 | 0.63 | 0.61 | 0.78 | 0.74 | 0.76
Zero-shot Evaluation
Sty+Sub | - | - | - | 0.72 | 0.32 | 0.48 | 0.61 | 0.18 | 0.33 | 0.79 | 0.43 | 0.58
Multi Sub | - | - | - | 0.73 | 0.40 | 0.54 | 0.65 | 0.29 | 0.43 | 0.77 | 0.36 | 0.53
Ctrl+Sub | - | - | - | 0.54 | 0.24 | 0.36 | 0.46 | 0.23 | 0.32 | 0.61 | 0.59 | 0.60
Ctrl+Sty | - | - | - | 0.59 | 0.22 | 0.36 | 0.18 | 0.06 | 0.11 | 0.74 | 0.54 | 0.63
Average | - | - | - | 0.64 | 0.30 | 0.44 | 0.48 | 0.19 | 0.30 | 0.73 | 0.48 | 0.59
Table 4: Full evaluation results. Figure 13: Additional Qualitative Evaluation
on Instruct-Imagen for In-domain Tasks. We do not visualize the outputs of the
face generation task due to lack of consent from the original persons.

In the appendix, we present details omitted from the main paper due to the
page limit. In § A.1, we first present additional image generation results of
Instruct-Imagen, both quantitatively and qualitatively. In § A.2, we then
discuss the details of model architecture, the training related details of
Instruct-Imagen and the inference specification. In § A.3, we provide
additional details about the retrieval-augmented pre-training and multimodal
instruction tuning dataset.

###  A.1 Additional Experimental Results

####  A.1.1 Complete Quantitative Evaluation

Table 4 shows the complete evaluation results, including the breakdown of
semantic consistency and perceptual quality. In addition to the numbers shown
in the main paper, we also report the additional average performance over all
methods on in-domain tasks and zero-shot tasks. We observe that Instruct-
Imagen is better than both the prior methods and our proposed baseline methods
in most tasks.

####  A.1.2 More Qualitative Evaluation

As aforementioned, Figure 13 presents more in-domain image generation output
from Instruct-Imagen.

###  A.2 Implementation Details

####  A.2.1 Model Architecture

Our base model design is similar to the Imagen [38], with a few key
modifications. First, we’ve shifted from a three-stage cascaded diffusion
model to a two-stage cascaded diffusion model. Concretely, in this setup, the
text-to-image generation model first produces $128{\times}128$ images (instead
of the $64{\times}64$ in [38]), and then subsequently upsampled to
$1024{\times}1024$ by only one super-resolution model. This adjustment allows
more detailed and information-rich outputs from the image generation model. As
aforementioned, the focus of this work is to adapt and fine-tune the text-to-
image generation model to comprehend multi-modal instruction. Secondly, rather
than employing one DBlock / UBlock per-resolution with multiple ResNetBlocks
in each DBlock / UBlock, we’ve opted for multiple DBlock / UBlock for each
resolution, consistently using $\text{numResNetBlocksPerBlock}{=}1$. This
design choice enables us to incorporate more attention layers, a critical
aspect for our model. Finally, we’ve increased the model size, as elaborated
below.

To process the multi-modal instruction, we repurpose the downsample network
within the text-to-image model as an encoder to extract latent features from
the multi-modal instruction. These features, derived from the final layer, are
integrated into the text-to-image generation model by introducing a cross-
attention layer into each DBlock / UBlock, similar to the text cross-attention
in Imagen [38]. Comprehensive architectural details for both the text-to-image
and super-resolution models can be found in Table 5.

|  | Text-to-Image | Super-Resolution
---|---|---|---
Model size |  2.76$B$ |  581$M$
DBlock-1 | Resolution | $128\rightarrow 64$ | $1024\rightarrow 512$
#Blocks | 8 | 2
OutChannels | 512 | 128
Attention | - | -
DBlock-2 | Resolution | $64\rightarrow 32$ | $512\rightarrow 256$
#Blocks | 8 | 4
OutChannels | 1024 | 256
Attention | - | -
DBlock-3 | Resolution | $32\rightarrow 16$ | $256\rightarrow 128$
#Blocks | 8 | 8
OutChannels | 2048 | 512
Attention | Text Instr + | -
Multi-modal Ctx
DBlock-4 | Resolution | $|$ | $128\rightarrow 64$
#Blocks | $|$ | 8
OutChannels | $|$ | 1024
Attention | $\downarrow$ | Text Instr.
UBlock-4 | Resolution | $|$ | $64\rightarrow 128$
#Blocks | $|$ | 8
OutChannels | $|$ | 512
Attention | $\downarrow$ | Text Instr.
UBlock-3 | Resolution | $16\rightarrow 32$ | $128\rightarrow 256$
#Blocks | 8 | 8
OutChannels | 1024 | 256
Attention | Text Instr + | -
Multi-modal Ctx
UBlock-2 | Resolution | $32\rightarrow 64$ | $256\rightarrow 512$
#Blocks | 8 | 4
OutChannels | 512 | 128
Attention | - | -
UBlock-1 | Resolution | $64\rightarrow 128$ | $512\rightarrow 1024$
#Blocks | 8 | 2
OutChannels | 3 | 3
Attention | - | -
Table 5: Model architecture of the Backbone U-Network. Note that the Text-to-
Image network do not have DBlock-4 and UBlock-4.

####  A.2.2 Optimization & Inference

The model is trained to predict v utilizing the standard L2 loss in accordance
with [39]. For all experiments, the Adafactor optimizer [40] with
$\beta_{1}{=}0.9$ and $\beta_{2}{=}0.999$ is employed, maintaining a
consistent learning rate of $10^{-4}$, along with a warm-up phase comprising
$10,000$ steps. The model undergoes training for 500k steps in retrieval-
augmented training and 400k steps in multi-modal instruction tuning, utilizing
a batch size of 512. Following [12], we utilize the moving average (with
weight decay rate $0.9999$) for the model weights used in inference. We use
the PaxML222https://github.com/google/paxml framework and train the models on
64 TPUv4. During inference, the sampling schedule requires 256 timesteps,
employing DDPM and cosine noise schedule. We employ an oscillating classifier-
free guidance schedule, alternating between a guidance at the scale of $25.0$
and no guidance every consecutive step.

###  A.3 Details on the Training Dataset

####  A.3.1 Retrieval-augmented Training Dataset

| Prompt | Context | Target
---|---|---|---
Multi-modal Context |  | Vibrant and colorful sunset
---
captured at Marshalls Beach,
San Francisco California
|  |
---|---
Fernando de Noronha | dusk of storm on
Brazil, $\ldots$ | the beach of Laga$\ldots$
|
---
Context Dropout |  | Seedless watermelon slices
---
on a table selective focus.
|
---
|
---
Figure 14:  Data for Retrieval-Augmented Training. We present two training
situations: (1) the case where multi-modal context are presented to the model
when generating the image; and (2) the case where multi-modal context are
dropped during the training.

In the retrieval-augmented training, there are two data situations being
presented to the model: (1) the model receives an input of text and a multi-
modal context consists of several relevant (image, text) pairs, and outputs
the target image. (2) the model receives an input text and outputs the target
image (with multi-modal context dropped at 10% of chances). The former data
situation represents the task of synthesising a given visual concept, using
the text and context, whereas the later situation presents the conventional
text-to-image synthesis. As an outcome, the trained Instruct-Imagen can
preserve the capability of text-to-image generation, while learning the new
context-dependent image generation skill. Please refer to Figure 14 for
concrete examples from these two learning situations.

####  A.3.2 Multi-modal Instruction-tuning Datasets

| Instruction | Target
---|---|---
(a) Natural Images |  | Bring forth an image based on the caption,
---
British short hair cat and golden retriever.
|
---
(b) Art Images |  | Generate this artwork: The Triumph of Hope,
---
an allegorical painting by Erasmus Quellinus
The Younger in the Baroque style.
|
---
Figure 15:  Text-to-Image Data for Instruction-Tuning..  | Instruction | Context | Target
---|---|---|---
(a) Depth |  | Create an image using
---
[ref#1] depth map as a reference
and following the below description:
A black and white puppy in a sunflower field.
| [ref#1] depth map
---
|
---
(b) Mask |  | Generate an image by taking cues from
---
[ref#1] object mask as a reference
and following this caption:
A pizza on top of a mountain peak.
| [ref#1] object mask
---
|
---
(b) Edge |  | Let [ref#1] edge image guide
---
you in crafting an image
that fulfills this description -
A stuffed animal on a beach blanket.
| [ref#1] edge image
---
|
---
Figure 16:  Control-Related Data for Instruction-Tuning. | Instruction | Context | Target
---|---|---|---
(a) General Subjects |  | Synthesize an image that
---
integrates the caption’s meaning,
featuring [ref#1] A stack of towels:
A stack of towels on a sandy beach.
| [ref#1] A stack of towels
---
|
---
(b) Faces |  | Produce a facial image with [ref#1] reference
---
image and reflects the caption:
A female with long black hair in a tight
braid is smiling and looking interested.
| reference image [ref#1]
---
|
---
Figure 17:  Subject-Related Data for Instruction-Tuning. The face image is anonymized to protect the privacy.  | Instruction | Context 1 | Context 2 | Target
---|---|---|---|---
(b) Style-to-Image |  | Create an image using
---
[ref#1] Realism style
in tune with the caption
Beautiful pink Lily
flower in the pond
in the national Park.
| [ref#1] Realism
---
|
---
|
---
| Generate an image in
---
[ref#1] Tonalism style
following the caption:
Many people walking around
at a fruit market.
| [ref#1] Tonalism
---
|
---
|
---
(c) Style Transfer |  | Recreate the content of
---
[ref#2] content image
using the style of
[ref#1] Symbolism.
| [ref#1] Symbolism
---
| [ref#2] content image
---
|
---
Figure 18:  Style-Related Data for Instruction-Tuning.

Subsequent to the retrieval-augmented training, we perform instruction-tuning
using multi-modal instructions. In this work, we adopt 9 different tasks,
which divides into five general categories.

Text-to-image Generation. We require the model to generate both natural and
art images to balance its learning of the two domains. To achieve this, we use
two datasets for instructed text-to-image generation: an internal high-quality
natural image dataset with manual caption; and an art specific dataset crawled
from WikiArt (using the pipeline in [44]), with the caption generated by PaLI
[8]. Note that the goal of art generation is to not only learn the alignment
with content description, but also learn the alignment between art style
description. Figure 15 presents the examples from both datasets, which are
augmented with a sampled text instruction that summarize the goal of the
generation (whether it is natural image or art generation).

Control-to-Image Generation. For control-related tasks (Figure 16), we use the
widely-adopted conditions – mask, edge, and depth. This allows the trained the
model to control the outputs based on the aforementioned conditions.
Specifically, we use MiDas [34] for depth estimation, HED [46] for edge
extraction, and salient object [30] for mask extraction. We also employed
edge-to-image data from a sketch dataset [23] as additional edge signals.
Since edge is a very loose definition and can present at many different
granularity, we perform the edge augmentation during the training.
Particularly, we applied edge extraction on the original image, the depth map,
and the mask, to obtain both coarse-grained and fine-grained contour images.
Additionally, we perform image dilation (with random configurations) on the
edge map to simulate the edge image data with different thickness. Finally,
for different control signals, we add different text intructions as prefixes
to hint the model about the scope of the task to the text description of the
image content.

Subject-driven Generation. As aforementioned, we employ two subject-driven
datasets for general objects and face generation. Particularly, we use the
subject-driven dataset introduced in SuTI [7] for general object learning, and
the celebrity face datasets [25, 19] to learn face rendering. For face
rendering, we group the faces of the same person and caption them with PaLI
[8], then we use one sampled (image, text) example as the input text and
target image, and using the rest as multi-modal context. Both datasets then
join the instruction templates, with reference markers inserted to refer the
multi-modal context. Figure 17 provides a qualitative example of these two
constructed datasets.

Styled Generation. We apply the recipe of StyleDrop [42] to fine-tune our
backbone cascaded diffusion model (500 steps on the $128\times 128$ model) and
create data for styled image generation. The outcome model are used to sample
with a styled image for a set of text prompts, which gives as the triplet of
(style image, text prompts, and styled image) in return for Instruct-Imagen
training. Note that the text prompts used here are sampled from the manual
prompts of the aforementioned internal natural image dataset, and the style
images used for fine-tuning is sampled from WikiArt. We employ a CLIP model to
filter out examples that fails the alignment with either style image or text
content, which provides a total of 100K data in total. Then we create the
multi-modal instructions via combining the instruction template with style
image and the manual caption, such that the style image is correctly referred.
Figure 18 (a) presents an example of the style-to-image generation data.

Style Transfer. Similarly, we construct the style transfer dataset via
combining style images from our WikiArt crawl and content images from the
internal dataset (with the captions discarded). We use a style transfer model
[13] based on the backbone cascaded diffusion model, which allows fast and
large-scale generation, to blend the style image with the content image. Note
that in the style transfer task, language is not providing any information
about the content of the target image, so the model needs to referring fully
to the content image to extract semantic information of the target image
output. Figure 18 (b) presents an example of the style transfer data.

Instruction Template Generation. As aforementioned, we prompted the GPT-4 [27]
to generate 100 rephrased instruction templates with high variation, and
validated the semantic correctness of them manually. During the instruction
creation, we use the placeholders in the place where multi-modal contexts are
suppose to be inserted, and populate the reference marker (and its associative
short prompt) when the instruction is going to be added to each particular
data. For example, in subject driven generation, one template could be
“Generate an image of [placeholder], using the caption:”, where the
placeholder would be substituted with the subject prompt and reference
“[ref#1] a dog”. Note that the reference marker corresponds to a special
tokens in the language embedding model.

###  A.4 Details on the Evaluation Datasets

As aforementioned, we now describe the details of both adopted and constructed
evaluation benchmark, with more details. To facilitate the comparison with
future methods, we make the dataset construction scripts for our evaluation
datasets publicly available.

####  A.4.1 In-domain Evaluation Datasets

For in-domain tasks, we tried to re-use existing datasets and evaluation
protocols to evaluate different approaches. For tasks that does not have a
standardized evaluation, we construct our own evaluation data, based on the
CustomConcept101 [21] dataset. The details of these evaluation are described
as follows.

Text2Image Generation. We adopt the text prompts used in ImagenHub [20] for
its comprehensiveness quality evaluation on text-to-image generation (a total
of 197 prompts).

Control2Image Generation. We randomly sample images from the CustomConcept101
[52] and use those as source images to extract control signals as condition,
and generated the text prompts using PaLM2 [2] for evaluating control2image
generation. This process is repeated for each control signal (_i.e_., edge,
mask, depth), to produce 100 (control image, prompt text) pairs per control
signal, which adds up to a total of 300 pairs of examples.

Styled Image Generation. We adopt the style images and text prompts from a
subset of the evaluation dataset used in the StyleDrop [42]. The dataset
consists of 12 text prompts that describe the image content and 10 acquired
style images, which leads to a total of 120 pairs of examples.

Subject-driven Image Generation. We adopt a subset of the DreamBench v1 and v2
datasets [37, 7] to serve as evaluation for subject-driven image generation,
which consists of a total of 220 pairs of (subject images, prompt text).

In-context Face Generation. We use the hold-out people images from the
validation split of the Celeb-A [25] and Celeb-HQ [19] dataset for in-context
face generation. The resulting dataset consists of 100 samples.

Style Transfer. We use the hold-out painting images from the WikiArt website
(re-crawled using the tool by [44], see dataset description in main text) and
the content images from the CustomConcept101 [21] dataset to form the (style
image, content image) pairs. The resulting dataset consists of 455 samples.

####  A.4.2 Zero-shot Compositional Evaluation Datasets

For zero-shot tasks, we either adopt the existing evaluation (_i.e_., multi-
subject evaluation using a subset of 100 examples on CustomConcept101 [21]) or
construct the evaluation ourselves (_e.g_., subject + control, style +
control, style + subject) by adopting images from corresponding datasets. The
details are described as what follows.

Style & Subject Conditioned Image Generation. We adopt the 100 subjects in the
CustomConcept101 [21] and construct the corresponding stylization text prompts
(using PaLM2 [2] model) based on the selected subjects. A total of 20 style
images from aforementioned datasets are adopted, 10 from the StyleDrop dataset
[42] and 10 from the hold-out WikiArt dataset. A total of 132 triplets of such
(subject images, style image, text prompt) for evaluation.

Multi-Subject Conditioned Image Generation. We adopt a subset of the (subject
A, subject B, text prompt) triplets from the CustomConcept101 [21] multi-
subject dataset, for the task multi-subject generation. The resulting dataset
consists of a total of 120 such triplets.

Subject & Control Conditioned Image Generation. We adopt the subject images in
the CustomConcept101 [21] dataset. Specifically, We select one of the
reference images for computing the control signals, and use the remaining
images as the reference images for the subjects. We used a mixture of three
controls in this dataset – edge, mask, and depth, supplemented with PaLM2
generated text prompts, with the goal of re-creating those subjects in new
context. The resulting dataset contains a total of 99 (control image, subject
images, text prompt) triplets.

Style & Control Conditioned Image Generation. Similarly, we adopt the 100
subject images in the CustomConcept101 [37] dataset as the source images for
generating control signals. We then construct text prompts to describe the
selected subjects in new context, and in the visual style of corresponding
style images (using the PaLM2 [2]). Particularly, we re-used the 20 style
images in the style & subject evaluation. The resulting dataset contains a
total of 100 (control image, style image, text prompt) triplets.


# Constraints on pseudo-Dirac neutrinos using high-energy neutrinos from NGC
1068

Thomas Rink<EMAIL_ADDRESS>Max-Planck-Institut für Kernphysik,
Saupfercheckweg 1, 69117 Heidelberg, Germany  Manibrata Sen  manibrata@mpi-
hd.mpg.de Max-Planck-Institut für Kernphysik, Saupfercheckweg 1, 69117
Heidelberg, Germany

###### Abstract

Neutrinos can be pseudo-Dirac in Nature - they can be Majorana fermions while
behaving effectively as Dirac fermions. Such scenarios predict active-sterile
neutrino oscillation driven by a tiny mass-squared difference $(\delta
m^{2})$, which is an outcome of soft-lepton number violation. Oscillations due
to tiny $\delta m^{2}$ can take place only over astrophysical baselines and
hence are not accessible in terrestrial neutrino oscillation experiments. This
implies that high-energy neutrinos coming from large distances can naturally
be used to test this scenario. We use the recent observation of high-energy
neutrinos from the active galactic nuclei NGC 1068 by the IceCube
collaboration to constrain $\delta m^{2}\leq 1.1\times 10^{-18}{\rm eV}^{2}$
at more than $90\%$ confidence level - one of the strongest limits to date on
the values of $\delta m^{2}$.

_Introduction –_ The advent of multi-messenger astronomy in the last decade
has been one of the biggest success stories of particle physics Bartos and
Kowalski (2017). The observation of high-energy neutrino events, in direct
correlation with the gamma-ray flares from the blazer TXS $0506+056$,
identified blazars as powerful sources of these neutrinos Aartsen  _et al._
(2018a, b). Since then, additional searches have been performed to locate more
of these sources. The observations of these high-energy neutrinos, which can
point back to their sources, indicate the presence of some of the most
powerful cosmic accelerators in the Universe.

Very recently, the IceCube collaboration released a search result for neutrino
emission from a list of 110 gamma-ray sources during the period 2011-2020
Abbasi  _et al._ (2022). An excess of $79^{+22}_{-20}$ neutrino events, with
energies of few TeV, was reported to be coming from the direction of a nearby
active galactic nucleus (AGN) – the NGC 1068 – with a global significance of
$4.2\sigma$, which was a major improvement over a previous result of
$2.9\sigma$. The neutrino spectra can be fit using a power-law,
$\Phi_{\nu_{\mu}+\bar{\nu}_{\mu}}=\Phi_{0}\left(E_{\nu}/\mathrm{1\,TeV}\right)^{-\gamma}$,
with $\Phi_{0}$ being the overall flux normalization at neutrino energy
$E_{\nu}=\mathrm{1\,TeV}$ and $\gamma$ the spectral index. Using this, the
collaboration reported $\Phi_{0}^{\mathrm{1TeV}}=(5.0\pm
1.5_{\mathrm{stat.}}\pm 0.6_{\mathrm{sys.}})\times
10^{-11}\,\mathrm{TeV}^{-1}\mathrm{cm}^{-2}\mathrm{s}^{-1}$, with
$\widehat{\gamma}=3.2\pm 0.2_{\mathrm{stat.}}\pm 0.07_{\mathrm{sys.}}$. The
distance to the AGN is estimated to be $d=14.4\,{\rm Mpc}$, although it has
been quoted previously in the literature to lie between $10.3\pm 3\,{\rm Mpc}$
199 (1992) and $d=16.5\,{\rm Mpc}$ Traianou  _et al._ (2020).

The discovery of these high-energy neutrinos has opened up new avenues of
testing exotic physics, which was otherwise inaccessible in terrestrial
laboratories. The arrival of these neutrinos clearly indicates that the
Universe is not opaque to neutrinos at this energy range. This can be used to
put tight constraints on different kinds of non-standard physics in the new
sector (see Kelly and Machado (2018) and references therein for a detailed
discussion).

The naturally long baseline offered by these neutrinos allows us to test the
violation of lepton number in the Standard Model (SM). If lepton number is not
considered a symmetry of the SM, and we know that it is already broken by
quantum effects even within the SM, neutrinos can be Majorana fermions. The
extent of lepton number violation in the SM is quantified by the Majorana mass
term of neutrinos in the Lagrangian. However, if lepton number is violated
_softly_ \- measurable through a tiny Majorana mass term - neutrinos can be
pseudo-Dirac, where they behave effectively as Dirac neutrinos Wolfenstein
(1981); Petcov (1982); Bilenky and Pontecorvo (1983); Kobayashi and Lim
(2001); Anamiati  _et al._ (2018); de Gouvea  _et al._ (2009); Vissani and
Boeltzig (2015). In this case, the neutrino mass-eigenstates corresponding to
the active and sterile states develop a _tiny_ mass-squared difference,
proportional to the extent of lepton number violation. Oscillations driven by
this tiny mass-squared difference $(\delta m^{2})$ will only be relevant for
long baselines, inversely proportional to $\delta m^{2}$.

Clearly, for testing these oscillations due to tiny $\delta m^{2}$, one needs
access to long baselines. Hence, constraints on the pseudo-Dirac hypothesis
from terrestrial experiments are weak Das  _et al._ (2014); Hernández  _et
al._ (2019); Anamiati  _et al._ (2019). Stronger constraints are available
from atmospheric neutrinos $\delta m^{2}\lesssim 10^{-4}{\rm eV}^{2}$ Beacom
_et al._ (2004), solar neutrinos $\delta m^{2}\lesssim 10^{-12}{\rm eV}^{2}$
de Gouvea  _et al._ (2009), as well as supernova neutrinos $\delta
m^{2}\lesssim 10^{-20}{\rm eV}^{2}$ De Gouvêa  _et al._ (2020); Martinez-Soler
_et al._ (2022), which naturally provide a longer baselines for neutrinos.
This naturally opens up the possibility of using high-energy neutrinos coming
from the far reaches of the Universe to probe even smaller values of $\delta
m^{2}$. Using models of gamma-ray bursts as sources for the astrophysical
neutrino flux observed by IceCube, limits have been placed on values of
$10^{-18}{\rm eV}^{2}\lesssim\delta m^{2}\lesssim 10^{-12}{\rm eV}^{2}$ Beacom
_et al._ (2004); Keranen  _et al._ (2003); Esmaili (2010); Esmaili and Farzan
(2012); Joshipura  _et al._ (2014); Brdar and Hansen (2019).

In this _letter_ , we utilize, for the first time, the high-energy neutrino
events observed from NGC 1068 to probe neutrino oscillations due to a small
$\delta m^{2}$. The advantages are two-fold: firstly, there exists a precise
measurement of the location of the source $d\sim 14.4\,{\rm Mpc}$, RA (right
ascension) and DEC (declination) $=(40.667,-0.0069)$; secondly, the neutrino
energies are well measured to be around a few TeV to tens of TeV. The
combination of these allows us to precisely pin down the oscillation length,
and study the sensitivity of neutrino oscillations to tiny mass-squared
differences. This allows us to rule out $\delta m^{2}\geq 1.1\times
10^{-18}{\rm eV}^{2}$ at more than $90\%$ confidence level. We emphasize that
this is the smallest value of $\delta m^{2}$ constrained with such a high
significance. An earlier work by one of the authors used data from SN1987A to
probe even tinier values of $\delta m^{2}\lesssim 10^{-20}{\rm eV}^{2}$
Martinez-Soler  _et al._ (2022), however, the significance was lower due to
the small number of events observed. This observation by IceCube, along with
the data from SN1987A, provide two of the tightest constraints on the strength
of lepton number violation in the SM.

_Pseudo-Dirac neutrinos –_ To explain the masses of the neutrinos, the SM can
be expanded to include three additional sterile neutrinos with at least two of
them having Majorana masses $M_{N}$, which characterize the extent of lepton
number violation. Apart from the Majorana mass term, the neutrino mass matrix
also consists of Dirac-type masses given by $M_{D}=Yv$, where $Y$ is the
Yukawa coupling and $v$ denotes the vacuum expectation value of the Higgs
field. In the limit of soft lepton number violation, $|M_{N}|\ll|Yv|$, the
generic neutrino mass matrix can be diagonalized by a $6\times 6$ block-
diagonal unitary matrix $U$, consisting of the PMNS matrix, and another
$3\times 3$ unitary matrix Kobayashi and Lim (2001). A generalization using
the spectral density function was recently worked out in Banks  _et al._
(2022).

For a tiny $|M_{N}|$, the mixing between the active and sterile states becomes
maximal. In this case, a neutrino flavor state can be written as a
superposition of two almost degenerate mass states,

$\displaystyle\nu_{\alpha L}$ $\displaystyle=U_{\alpha
k}\frac{(\nu_{k}^{+}+i\,\nu_{k}^{-})}{\sqrt{2}}\,,$ (1)

with masses $m_{k,\pm}^{2}=m_{k}^{2}\pm\delta m_{k}^{2}/2$. Here
$\nu_{k}^{\pm}$ correspond to the mass eigenstates associated with the flavor
state. This scenario leads to active-sterile oscillations driven by $\delta
m_{k}^{2}/2$, in addition to oscillations due to solar $(\Delta m^{2}_{\rm
sol})$ and atmospheric $(\Delta m^{2}_{\rm atm})$ mass-squared differences
Tanabashi  _et al._ (2018). For simplicity, we will assume that $\delta m^{2}$
is the same for all states, and hence drop the index $k$.

For neutrinos travelling over astrophysical baselines, flavor oscillations
induced by $(\Delta m^{2}_{\rm sol,atm})$ average out due to rapid
oscillations, causing decoherence Tanabashi  _et al._ (2018). However, for
small enough $\delta m^{2}$, active-sterile oscillations can persist. The
corresponding active neutrino survival probability can be computed as De
Gouvêa  _et al._ (2020)

$\displaystyle
P_{\alpha\alpha}(E_{\nu})=\frac{1}{2}\left(1+e^{-\left(\frac{L}{L_{\rm
coh}}\right)^{2}}\cos\left(\frac{2\pi L}{L_{\rm osc}}\right)\right)\,.$ (2)

Here $L_{\rm osc}$ denotes the oscillation length and can be computed for a
given neutrino energy $E$ and $\delta m^{2}$ as,

$L_{\rm osc}=\frac{4\pi E_{\nu}}{\delta m^{2}}\approx 15\,{\rm
Mpc}\left(\frac{E_{\nu}}{1\,{\rm TeV}}\right)\left(\frac{5\times
10^{-18}\,{\rm eV}^{2}}{\delta m^{2}}\right)\,.$ (3)

However, even these oscillations driven by $\delta m^{2}$ can get washed out
due to the separation of wave packets over long distances, leading to
decoherence. This is measured through the coherence length $L_{\rm coh}$,
which gives the length scale up to which flavor coherence is expected to be
maintained. This depends also on the width of the neutrino wave-packet
$\sigma_{x}$, which is usually determined from the process of neutrino
production Kersten and Smirnov (2016). The coherence length is estimated as,

$\displaystyle L_{\rm coh}$ $\displaystyle=\frac{4\sqrt{2}E_{\nu}^{2}}{|\delta
m^{2}|}\sigma_{x}$ $\displaystyle\approx 10^{6}\,{\rm
Gpc}\left(\frac{E_{\nu}}{1\,{\rm TeV}}\right)^{2}\left(\frac{5\times
10^{-18}\,{\rm eV}^{2}}{\delta
m^{2}}\right)\left(\frac{\sigma_{x}}{10^{-10}\,{\rm\ m}}\right)\,.$ (4)

Taking into account the active sterile oscillation probability, and the flavor
averaging due to the solar and atmospheric terms, the probability of obtaining
a neutrino flavor $\nu_{\beta}$, starting from $\nu_{\alpha}$ is given by

$\displaystyle P_{\alpha\beta}=P_{\alpha\alpha}(E_{\nu};L,\delta
m^{2})\sum_{k}\left|U_{\alpha k}\right|^{2}\left|U_{\beta k}\right|^{2}\,.$
(5)

This is the new physics that we will constrain through our analysis in the
next section. The long baseline $(d=14.4\,{\rm Mpc})$ offered by the high-
energy neutrinos observed by IceCube can be used to constrain tiny values of
$\delta m^{2}\sim 10^{-18}{\rm eV}^{2}$, which are otherwise difficult to
access in other sources.

_Analysis –_ The investigation performed in this work closely follows the
procedure applied in Ref. Abbasi  _et al._ (2022). Thus, we use an unbinned
likelihood ratio method with a corresponding likelihood function defined as
Braun  _et al._ (2008)

$\displaystyle\log\mathcal{L}=\sum_{i}\log\left(\frac{n_{s}}{N}\mathcal{S}_{i}+\left(1-\frac{n_{s}}{N}\mathcal{B}_{i}\right)\right)\,,$
(6)

with $\mathcal{S}_{i}$ and $\mathcal{B}_{i}$ being the probability density
functions for signal and background neutrino events, respectively. Further,
$n_{s}$ denotes the number of signal events and $N$ is the total number of
recorded events. The analysis routines provided by the IceCube collaboration
IceCube Collaboration (2022) have been modified in order to incorporate the
oscillation probability of pseudo-Dirac neutrinos, cf. Eq. 5. For the latter,
we took the latest global fit results Esteban  _et al._ (2020). We set
$\sigma_{x}\sim 10^{-10}\,{\rm m}$ \- this number is just representative of
the width of the neutrino wave packet. From Eq. Constraints on pseudo-Dirac
neutrinos using high-energy neutrinos from NGC 1068, it is clear that this
choice does not lead to additional decoherence due to wave-package separation
on the Mpc-scale.

The neutrino flux of NGC 1068 in muon and the anti-muon flavor is assumed to
follow a power-law of the form

$\displaystyle\Phi_{\nu_{\mu}+\bar{\nu}_{\mu}}=\Phi_{0}\left(\frac{E_{\nu}}{\mathrm{1\,TeV}}\right)^{-\gamma}\,,$
(7)

with $\Phi_{0}$ being the overall flux normalization associated with an energy
of $1\,{\rm TeV}$ and $\gamma$ is the spectral index. The corresponding number
of signal events is then given by Abbasi  _et al._ (2021)

$\displaystyle n_{s}=t\int d\Omega\int_{0}^{\infty}$ $\displaystyle dE_{\nu}\
A_{\mathrm{eff}}(E_{\nu},\Omega)$ (8) $\displaystyle P(E_{\nu};\delta
m^{2})\Phi_{\nu}(E_{\nu};\Phi_{0},\gamma)\,,$

with the detector’s lifetime $t$ and its effective area $A_{\mathrm{eff}}$
being a function of the neutrino energy $E_{\nu}$ and the solid angle
$\Omega$.

Hence, in our analysis the number of signal events depends on the three
parameters: $n_{s}\equiv n_{s}(\delta m^{2},\Phi_{0},\gamma)$. Existing
correlations between $\Phi_{0}$ and $\gamma$ are accounted for by reproducing
the covariance matrix of the original IceCube analysis and incorporating it
with a two-dimensional Gaussian pull term to the likelihood function in Eq. 6.
In doing so, we also ensure that the fit preserves the original best-fit
values of $\Phi_{0}$ and $\gamma$, taking into account a normalization
correction due to the introduced oscillation probability in Eq. 5. Note that
for tiny mass-squared differences $\delta m^{2}\lesssim 10^{-22}$eV2, the
effects of active-sterile mixing become negligible for the energy region of
interest, such that we coincide with the usual no additional oscillation case.
In order to determine a limit on the mass-squared difference of pseudo-Dirac
neutrinos, we determine a profile likelihood ratio as Cowan  _et al._ (2011)

$\displaystyle q_{\delta m^{2}}=-2\log\left(\frac{\mathcal{L}(\delta
m^{2},\widehat{\widehat{\Phi}}_{0},\widehat{\widehat{\gamma}})}{\mathcal{L}(\widehat{\delta
m^{2}},\widehat{\Phi}_{0},\widehat{\gamma})}\right)\,,$ (9)

where the numerator represents the likelihood function with a fixed $\delta
m^{2}$, while in the denominator all three quantities are tuned in the
minimization procedure. In particular, the parameters
$\widehat{\widehat{\Phi}}_{0}$ and $\widehat{\widehat{\gamma}}$ are the
conditional (maximum likelihood) estimators that depend on the chosen value of
$\delta m^{2}$, whereas the parameters occurring in the denominator are the
estimators that maximize the unconstrained likelihood function.

For each $\delta m^{2}$ value under study, we determine the likelihood ratio
in Eq. 6 and with asymptotic sampling distributions given in Ref. Cowan  _et
al._ (2011) we can assign each so-determined ratio a corresponding p-value,
i.e. $p_{\delta m^{2}}=\int_{q_{\delta m^{2},\mathrm{obs}}}^{\infty}dq_{\delta
m^{2}}f(q_{\delta m^{2}}|\delta m^{2})$, which we use as an exclusion
criterion for the tested $\delta m^{2}$ values. Since the introduced
oscillation probability in Eq. 5 affects the expected signal only in one
direction, i.e. a decrease of the expected neutrino flux, we have to perform a
one-sided statistical test. Following the IceCube collaboration, we assume
that the test statistic $f(q_{\delta m^{2}}|\delta m^{2})$ used has
approximately $\chi^{2}$-like behavior IceCube Collaboration (2022), except
for modifications due to the one-sided test case. For details about the
modified profile likelihood ratio and the corresponding sampling distribution,
we refer to Ref. Cowan  _et al._ (2011). In this work, we are interested in a
limit on $\delta m^{2}$ at 90% confidence level (C.L.) ($1-p_{\delta m^{2}}$).
Thus, our results is determined by finding the value of $\delta m^{2}$, which
yields $p_{\delta m^{2}}\leq 0.1$. The whole analysis has been performed
within the Anaconda/SciPy framework Ana (2022); Virtanen  _et al._ (2020),
while the iminuit package Dembinski and et al. (2020) was used to minimize the
likelihood function in Eq. 6.

_Results –_

Figure 1: Neutrino flux variations due to oscillations driven by $\delta
m^{2}$. We fixed the neutrino flux parameters to the values determined by
IceCube Collaboration Abbasi  _et al._ (2022), i.e. $\widehat{\Phi}_{0}\sim
5.0\times 10^{-11}\,\mathrm{TeV}^{-1}\mathrm{cm}^{-2}\mathrm{s}^{-1}$ and
$\widehat{\gamma}=3.2$, and varied the pseudo-Dirac mass splitting $\delta
m^{2}$ in a region that is relevant for this analysis.

The fact that IceCube observed these neutrinos from NGC1068 implies that these
did not oscillate into their sterile counterparts. In the presence of active-
sterile oscillations, the expected power-law flux will undergo spectral
distortions, as shown in Fig. 1 for different values of $\delta
m^{2}=\\{10^{-17},10^{-18},10^{-19}\\}\,{\rm eV}^{2}$. As a result, a profile
likelihood analysis can be used to set constraints on possible values of
$\delta m^{2}$. In our analysis, we assume that only muon (anti)neutrinos
produced at the source are detected as muon (anti)neutrinos i.e., we set
$\alpha=\beta=\mu$ in Eq. 5. This implies that we only detect $\sim$ 41% of
all neutrinos. This leads to a slightly different flux normalization from the
original bf values observed by IceCube: $\widehat{\Phi}_{0}\sim 1.2\times
10^{-10}\,\mathrm{TeV}^{-1}\mathrm{cm}^{-2}\mathrm{s}^{-1}$, while $\gamma$
basically remains the same. In principle, the contribution of a $\nu_{e}$
produced at source and detected as $\nu_{\mu}$ is expected to be few tens of
percentage of that of the original $\nu_{\mu}$ contribution. However, our
results do not change much if we include this contribution.

Figure 2: Exclusion for the mass-squared difference $\delta m^{2}$ of pseudo-
Dirac neutrinos. Confidence levels are shown for different $\delta m^{2}$
values. With the observation of neutrinos from NGC 1068, we can exclude
$\delta m^{2}>1.1\times 10^{-18}$ eV2 (@ 90% C.L.).

Our main result, depicted in Fig. 2, shows that the data from IceCube can be
used to rule out $\delta m^{2}\geq 1.1\times 10^{-18}{\rm eV}^{2}$ at more
than $90\%$ confidence level ($p_{\delta m^{2}{}}\leq 0.1$). There is some
sensitivity to even lower values of $\delta m^{2}$, albeit at lower
confidence. This agrees with the analytical estimate in Eq. 3, which shows the
mass-squared difference sensitive to this oscillation length for TeV energy
neutrinos. To the best of our knowledge, this is the _strongest_ constraint to
date on the smallness of $\delta m^{2}$ with such a high significance. For
smaller values of $\delta m^{2}$, the oscillation length becomes larger than
the distance to the source, hence sensitivity is lost.

Fig. 3 shows the correlation between $\delta m^{2}$ and $\gamma$ for a
profiled flux normalization $\Phi_{0}$. Analogous to our determined limits, we
indicate the 90% C.L. contour for both parameters. Correlations between
$\Phi_{0}$ and $\gamma$ are considered via an appropriate pull-term in the
applied likelihood function. The figure indicates that the data is clearly
consistent with the “no-oscillation” hypothesis, which corresponds physically
to oscillations with longer baselines. This shows that the addition of these
oscillations does not help improve the fit, rather the data can be used to put
constraints on these oscillations.

A similar argument can be used to put bounds on $\delta m^{2}$ from
observations of a neutrino event of energy $E_{\nu}\sim 290\,{\rm TeV}$ from
the blazar TXS 0506 + 056 at a redshift of $z=0.33\pm 0.0010$, corresponding
to a distance $d=1.3\,{\rm Gpc}$ Aartsen  _et al._ (2018a, b). A naive
estimate yields

$L_{\rm osc}\approx 1.3\,{\rm Gpc}\left(\frac{E_{\nu}}{290\,{\rm
TeV}}\right)\left(\frac{10^{-17}\,{\rm eV}^{2}}{\delta m^{2}}\right)\,,$ (10)

which is stronger due to the larger energy of the neutrino. Additionally,
IceCube later reported a series of $13\pm 5$ events from the same direction.
These observations can be used, in principle, for a similar study to constrain
values of $\delta m^{2}$.

Figure 3: Parameter correlation between $\delta m^{2}$ and the spectral index
for a profiled flux normalization $\Phi_{0}$. The black lines denote the 90%
CL contours for these parameters.

_Conclusion –_ Detection of high-energy neutrino events by IceCube has opened
up new frontiers in multi-messenger astronomy. In particular, the naturally
long baseline available to the neutrinos, coupled with their high energies,
allows them to be the harbingers of new exotic physics, otherwise inaccessible
to mankind. In this work, we used the IceCube observation of $\sim 79$ TeV-sh
neutrino events coming from the direction of the active galactic nuclei NGC
1068 to probe the extent of lepton-number violation in the Standard Model.

A soft lepton-number violation can be manifested through active-sterile
neutrino oscillations over astronomical baselines inversely proportional to
the mass-squared difference $(\delta m^{2})$ between active and sterile
neutrinos. These oscillations, if present, will lead to a distortion of the
event spectra, and a reduction in the number of neutrinos observed.

This simple yet powerful idea can be utilized to probe the strength of lepton
number violation, characterised by the magnitude of $\delta m^{2}$. The
identification of the source of these neutrinos, and precise measurement of
their energies, allowed us to precisely calculate the oscillation lengths
associated with $\delta m^{2}$, and the corresponding spectral modifications.
We used the analysis of the IceCube collaboration, outlined in Ref. Abbasi
_et al._ (2022), to rule out $\delta m^{2}\geq 1.1\times 10^{-18}{\rm eV}^{2}$
at more than $90\%$ confidence level. The result obtained is one of the
strongest bounds on the extent of lepton-number violation from experiments.

These extreme new physics scenarios are clearly inaccessible to any
terrestrial experiments. The observations of these point-sources of
astronomical neutrinos like NGC 1068 and TXS 0506+056 clearly pave way for new
laboratories for neutrino physics.

_Acknowledgements –_ We thank Yuber Perez-Gonzalez for discussions and
collaborations in the early phase of the project, and Sergio Palomares Ruiz
for useful comments on the draft. We would also like to thank the IceCube
Collaboration for providing their datasets and analysis routines to the
public, which set the foundation for the analysis performed in this work.

## References

  * Bartos and Kowalski (2017) I. Bartos and M. Kowalski,  _Multimessenger Astronomy_ (2017).
  * Aartsen  _et al._ (2018a) M. G. Aartsen _et al._ (IceCube), Science 361, 147 (2018a), arXiv:1807.08794 [astro-ph.HE] .
  * Aartsen  _et al._ (2018b) M. G. Aartsen _et al._ (IceCube, Fermi-LAT, MAGIC, AGILE, ASAS-SN, HAWC, H.E.S.S., INTEGRAL, Kanata, Kiso, Kapteyn, Liverpool Telescope, Subaru, Swift NuSTAR, VERITAS, VLA/17B-403), Science 361, eaat1378 (2018b), arXiv:1807.08816 [astro-ph.HE] .
  * Abbasi  _et al._ (2022) R. Abbasi _et al._ (IceCube), Science 378, 538 (2022).
  * 199 (1992) _Physics of Nearby Galaxies: Nature or Nurture?_ (1992).
  * Traianou  _et al._ (2020) E. Traianou _et al._ , Astron. Astrophys. 634, A112 (2020), arXiv:1912.00776 [astro-ph.GA] .
  * Kelly and Machado (2018) K. J. Kelly and P. A. N. Machado, JCAP 10, 048 (2018), arXiv:1808.02889 [hep-ph] .
  * Wolfenstein (1981) L. Wolfenstein, Nucl. Phys. B186, 147 (1981).
  * Petcov (1982) S. T. Petcov, Phys. Lett. 110B, 245 (1982).
  * Bilenky and Pontecorvo (1983) S. M. Bilenky and B. Pontecorvo, Sov. J. Nucl. Phys. 38, 248 (1983), [Lett. Nuovo Cim.37,467(1983); Yad. Fiz.38,415(1983)].
  * Kobayashi and Lim (2001) M. Kobayashi and C. S. Lim, Phys. Rev. D64, 013003 (2001), arXiv:hep-ph/0012266 [hep-ph] .
  * Anamiati  _et al._ (2018) G. Anamiati, R. M. Fonseca, and M. Hirsch, Phys. Rev. D 97, 095008 (2018), arXiv:1710.06249 [hep-ph] .
  * de Gouvea  _et al._ (2009) A. de Gouvea, W.-C. Huang, and J. Jenkins, Phys. Rev. D80, 073007 (2009), arXiv:0906.1611 [hep-ph] .
  * Vissani and Boeltzig (2015) F. Vissani and A. Boeltzig, PoS NEUTEL2015, 008 (2015).
  * Das  _et al._ (2014) A. Das, P. S. Bhupal Dev, and N. Okada, Phys. Lett. B 735, 364 (2014), arXiv:1405.0177 [hep-ph] .
  * Hernández  _et al._ (2019) P. Hernández, J. Jones-Pérez, and O. Suarez-Navarro, Eur. Phys. J. C 79, 220 (2019), arXiv:1810.07210 [hep-ph] .
  * Anamiati  _et al._ (2019) G. Anamiati, V. De Romeri, M. Hirsch, C. A. Ternes, and M. Tórtola, Phys. Rev. D 100, 035032 (2019), arXiv:1907.00980 [hep-ph] .
  * Beacom  _et al._ (2004) J. F. Beacom, N. F. Bell, D. Hooper, J. G. Learned, S. Pakvasa, and T. J. Weiler, Phys. Rev. Lett. 92, 011101 (2004), arXiv:hep-ph/0307151 [hep-ph] .
  * De Gouvêa  _et al._ (2020) A. De Gouvêa, I. Martinez-Soler, Y. F. Perez-Gonzalez, and M. Sen, Phys. Rev. D 102, 123012 (2020), arXiv:2007.13748 [hep-ph] .
  * Martinez-Soler  _et al._ (2022) I. Martinez-Soler, Y. F. Perez-Gonzalez, and M. Sen, Phys. Rev. D 105, 095019 (2022), arXiv:2105.12736 [hep-ph] .
  * Keranen  _et al._ (2003) P. Keranen, J. Maalampi, M. Myyrylainen, and J. Riittinen, Phys. Lett. B 574, 162 (2003), arXiv:hep-ph/0307041 .
  * Esmaili (2010) A. Esmaili, Phys. Rev. D 81, 013006 (2010), arXiv:0909.5410 [hep-ph] .
  * Esmaili and Farzan (2012) A. Esmaili and Y. Farzan, JCAP 1212, 014 (2012), arXiv:1208.6012 [hep-ph] .
  * Joshipura  _et al._ (2014) A. S. Joshipura, S. Mohanty, and S. Pakvasa, Phys. Rev. D 89, 033003 (2014), arXiv:1307.5712 [hep-ph] .
  * Brdar and Hansen (2019) V. Brdar and R. S. L. Hansen, JCAP 02, 023 (2019), arXiv:1812.05541 [hep-ph] .
  * Banks  _et al._ (2022) H. Banks, K. J. Kelly, and M. McCullough, (2022), arXiv:2209.11270 [hep-ph] .
  * Tanabashi  _et al._ (2018) M. Tanabashi _et al._ (Particle Data Group), Phys. Rev. D98, 030001 (2018).
  * Kersten and Smirnov (2016) J. Kersten and A. Y. Smirnov, Eur. Phys. J. C 76, 339 (2016), arXiv:1512.09068 [hep-ph] .
  * Braun  _et al._ (2008) J. Braun, J. Dumm, F. De Palma, C. Finley, A. Karle, and T. Montaruli, Astropart. Phys. 29, 299 (2008), arXiv:0801.1604 [astro-ph] .
  * IceCube Collaboration (2022) IceCube Collaboration, “Evidence for neutrino emission from the nearby active galaxy ngc 1068,”  (2022).
  * Esteban  _et al._ (2020) I. Esteban, M. C. Gonzalez-Garcia, M. Maltoni, T. Schwetz, and A. Zhou, JHEP 09, 178 (2020), arXiv:2007.14792 [hep-ph] .
  * Abbasi  _et al._ (2021) R. Abbasi _et al._ (IceCube),  (2021), 10.21234/CPKQ-K003, arXiv:2101.09836 [astro-ph.HE] .
  * Cowan  _et al._ (2011) G. Cowan, K. Cranmer, E. Gross, and O. Vitells, Eur. Phys. J. C 71, 1554 (2011), [Erratum: Eur.Phys.J.C 73, 2501 (2013)], arXiv:1007.1727 [physics.data-an] .
  * Ana (2022) “Anaconda software distribution,”  (2022).
  * Virtanen  _et al._ (2020) P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, İ. Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, Nature Methods 17, 261 (2020).
  * Dembinski and et al. (2020) H. Dembinski and P. O. et al.,  (2020), 10.5281/zenodo.3949207.