# StableSemantics: A Synthetic Language-Vision Dataset of Semantic
Representations in Naturalistic Images

###### Abstract

Understanding the semantics of visual scenes is a fundamental challenge in
Computer Vision. A key aspect of this challenge is that objects sharing
similar semantic meanings or functions can exhibit striking visual
differences, making accurate identification and categorization difficult.
Recent advancements in text-to-image frameworks have led to models that
implicitly capture natural scene statistics. These frameworks account for the
visual variability of objects, as well as complex object co-occurrences and
sources of noise such as diverse lighting conditions. By leveraging large-
scale datasets and cross-attention conditioning, these models generate
detailed and contextually rich scene representations. This capability opens
new avenues for improving object recognition and scene understanding in varied
and challenging environments. Our work presents StableSemantics, a dataset
comprising 224 thousand human-curated prompts, processed natural language
captions, over 2 million synthetic images, and 10 million attention maps
corresponding to individual noun chunks. We explicitly leverage human-
generated prompts that correspond to visually interesting stable diffusion
generations, provide 10 generations per phrase, and extract cross-attention
maps for each image. We explore the semantic distribution of generated images,
examine the distribution of objects within images, and benchmark captioning
and open vocabulary segmentation methods on our data. To the best of our
knowledge, we are the first to release a diffusion dataset with semantic
attributions. We expect our proposed dataset to catalyze advances in visual
semantic understanding and provide a foundation for developing more
sophisticated and effective visual models.
https://stablesemantics.github.io/StableSemantics/

Rushikesh Zawar*
Carnegie Mellon University<EMAIL_ADDRESS>Shaurya Dewan*
Carnegie Mellon University<EMAIL_ADDRESS>Andrew F. Luo
Carnegie Mellon University<EMAIL_ADDRESS>
Margaret M. Henderson
Carnegie Mellon University<EMAIL_ADDRESS>Michael J. Tarr Carnegie Mellon
University<EMAIL_ADDRESS>Leila Wehbe
Carnegie Mellon University<EMAIL_ADDRESS>
##  1 Introduction

Visual scene understanding is a complex task that requires the integration of
cues, context, and prior knowledge to navigate the inherent variability and
complexity of the visual world. This complexity is particularly evident when
considering the diversity of visual appearances that can correspond to a
single semantic concept. For instance, entities that correspond to "man-made
structures" can have vastly different visual appearances, ranging from sleek
skyscrapers to rustic cottages. Similarly, objects that serve the same
purpose, such as "containers," can have diverse shapes, sizes, and materials.
This disconnect between semantic meaning and visual appearance poses a
significant challenge for Computer Vision systems [1, 2, 3, 4], requiring the
disentanglement of the underlying semantic structure from visual differences
[5, 6, 7, 8, 9, 10]. To overcome this challenge, recent advances in Computer
Vision have adopted data-driven approaches, which learn to recognize patterns
and relationships in large datasets of images and annotations. However, the
reliance on large datasets of images and annotations poses a significant
challenge in the development of Computer Vision systems. Acquiring and
annotating such datasets can be a time-consuming and resource-intensive
process, requiring careful consideration of data quality and diversity.

This limitation has sparked interest in exploring alternative approaches that
can reduce the need for large human-annotated datasets. One promising
direction is the use of generative models, which have shown impressive results
in translating between semantic meaning and visual appearance [11, 12, 13,
14]. In particular, diffusion-based text-to-image synthesis models have
demonstrated an impressive ability to generate highly realistic images from
textual descriptions, suggesting that these models must possess an implicit
understanding of the semantic structure of the visual world, and have learned
to associate words and phrases with specific visual concepts. By leveraging
cross-attention mechanisms, these models learn to link textual input to visual
representations and enable the generation of images that are grounded in the
semantic content of the input text [15].

Figure 1: Images and maps corresponding to select noun chunks from
StableSemantics. Images are generated using natural language captions derived
from human generated and curated prompts. For reproducibility, seeds are
recorded for each generation. Noun chunks are extracted by performing
dependency parsing the natural language captions. Semantic maps corresponding
to each noun chunk is computed using the cross-attention maps with the DAAM
[15] method. Only a single attention map is shown here for each image, please
see below for additional examples. Yellow indicates high relevance, black
indicates low relevance.

In this work, we introduce StableSemantics, a dataset that consists of human-
generated and curated prompts, natural language captions, images generated
from the captions, and attention attribution maps corresponding to objects in
the captions. Unlike prior work which sourced unfiltered human-generated
prompts, we source our prompts from a pool of images that have been evaluated
by humans for their visual appeal and interest, resulting in a dataset that
mirrors the types of images people find engaging. As the original prompts may
not always reflect natural language, we employ a large language model to
paraphrase and refine them into fluent and natural-sounding captions, thereby
bridging the gap between human-generated prompts filtered for visual appeal
and naturalistic language. Each natural language caption is provided to a
Stable Diffusion XL model to generate high-resolution and reproducible images.
Finally, we explicitly record the dense text-to-image cross-attention maps
used to condition the image generation process. We visualize the distribution
of semantics across images, evaluate the spatial distribution of semantic
classes within images, and evaluate the alignment of current captioning and
open-set segmentation models on our dataset. To our knowledge, our dataset is
the first to systematically record the spatial distribution of cross-attention
activations corresponding to individual noun chunks. We hope that
StableSemantics will inspire future research on the visual distribution of
semantic concepts and the development of more interpretable text-to-image
synthesis models.

##  2 Related work

#### Natural Scene Statistics.

Natural image statistics have been a long-standing area of research in
computer vision and neuroscience. The human visual system is thought to be
adapted to the statistical properties of natural images, which are
characterized by complex dependencies between pixels [16, 17]. The power law
distribution of gradient magnitude statistics is thought to be a result of the
hierarchical, self-similar structure of natural images, which arises from the
presence of edges, textures, and other features at multiple scales.
Understanding natural image statistics has important implications for image
recognition tasks [18, 19, 20], and has inspired the development of a range of
algorithms and models that are tailored to the statistical properties of
natural images [21, 22, 23, 24, 25, 26]. Other work has also explored the
semantic structure of visual data, seeking to understand how higher-level
categories and concepts are reflected in the statistical patterns present in
images. This work has shown that different categories of images, such as
scenes and objects, exhibit distinct statistical patterns [27, 28]. These
semantic statistics have important implications for the development of models
that can effectively represent and analyze visual data.

#### Deep Image Generative Models.

Recent progress on generative models has enabled the generation of images,
video, text, and audio [11, 12, 13, 14, 29, 30, 31, 31, 32]. Models rely on a
variety of different mathematical assumptions and architectures. Variational
autoencoders [33, 34, 35, 36, 37] and flow-based models [38, 39, 40, 41],
while highly efficient, tend to produce lower-quality samples. Generative
Adversarial Networks (GANs) [42, 43, 44, 45, 46, 47, 48] can yield high-
fidelity samples but may neglect modes in the data and can exhibit unstable
training dynamics. Auto-regressive methods [49, 50, 51, 52, 53], although
capable of producing high-quality samples, typically experience slow sampling.
Recent progress in energy/score/diffusion models [11, 53] has given us methods
that are simultaneously stable during training and yield high-quality samples.

#### Visual Datasets.

Deep learning models have achieved remarkable results by leveraging vast
amounts of data. There has been a significant push to collect large-scale
datasets. Earlier works such as LAION-5B [54], Flickr Caption [55] and
YFCC100M [56] scrape real-world data of image-caption pairs from web sources.
COCO [57] goes a step further to also provide pixel-level segmentation masks
on top of the image-caption pairs. [58, 59, 60, 61, 62] introduce datasets
specifically for the task of VQA. Given the difficulty of collecting real
data, recently there has been a shift towards synthetic datasets. StableRep
[63] also demonstrated the usefulness of Stable Diffusion images in training
contrastive image models. Pick-a-Pic [64] provides a dataset of image-caption
pairs where each sample contains a pair of diffusion-generated images and the
human preference between those images. JourneyDB [65] and DiffusionDB [66] are
the closest works to ours that release large-scale datasets of synthetic
image-caption pairs.

##  3 Data collection

In this section, we provide details on the collection and creation process of
our dataset. Our data originates from human-generated and curated prompts
submitted publicly by users online for Stable Diffusion XL. We describe our
prompt collection process in section 3.1. The prompts are filtered and
transformed into natural language captions, and we describe our procedure in
section 3.2. Finally, we generate images and compute noun-chunk to image
saliency maps via cross-attention in section 3.

Figure 2: Data collection and generation process. (1) We collect our data from
Stable Diffusion Discord, specifically the showdown and pantheon channels
which are derived from user rankings of images generated from public prompt
submissions. (2) The prompts are cleaned using regex to remove common errors,
and further processed using an LLM to generate natural language captions. (3)
The natural language captions are provided to a Stable Diffusion XL model,
while we record the attention attribution maps corresponding to noun chunks.

###  3.1 Collecting human curated prompts

Our dataset is collected from the Stable Diffusion discord server, where users
can publicly submit prompts to generate images using a discord bot. After
users submitted prompts using the /dream command, the bot would return images
corresponding to a prompt. Beyond accepting a prompt, users could also submit
negative prompts, and image styles which were achieved via a prefix/affix
pattern of text to the original prompt. These style patterns were not visible
to the users.

We started our data collection after the Stable Diffusion XL 1.0 [12]
candidate was made available via bots. The data was continuously collected
from July 11, 2023 (a day after SDXL 1.0 candidate bots were launched) until
Feb 07, 2024 (SDXL bot shutdown). Users were allowed to submit prompts to
bot-# channels where # corresponds to a number. We observed that the number of
channels varied over time, and generally remained at slightly over $10$. For
each prompt, the bot would return $2$ images. Users were asked to select which
image was better by clicking on a button corresponding to an image, without
explicit guidance on what "better" meant. Our understanding from discussions
with members of staff was that these prompts and image pairs were used for
fine-tuning the SDXL candidates using RLHF/DPO [67, 68], selection of model
candidates, and selection of generation hyperparameters.

Prior work has also collected user-generated prompts from discord servers for
MidJourney and Stable Diffusion [65, 66]. Our work goes further by only
collecting prompts that were human-curated. The Stable Diffusion discord
followed a three-tier hierarchy for prompts, where users first submit and rate
images in bot-#, with highly rated images from all bot channels going into a
single showdown channel every $15$ minutes. The showdown channel was reset
every $30$ minutes and had the history wiped. In the showdown channel, 2
images and their respective prompts were placed side-by-side. Users again were
asked to select the images that were more visually appealing. Every $30$
minutes, the top-ranked images and prompts would go into the pantheon channel.
The pantheon had history going back to inception May 02, 2023. We note that
strictly speaking the showdown to pantheon selection process was not fair to
images that came in at the second $15$ minute slice, as they were given less
time to be voted upon. Due to this, we do not further distinguish between
prompts collected from these two sources. Our data collection process ran
every $14$ minutes on the showdown channels and ran once on the pantheon
channel. This was sufficient as after the initial collection date, new
pantheon entries were a strict subset of showdown prompts. Visual inspection
of the generations from showdown and pantheon suggest that these images were
generally more artistic and contained more interesting visual compositions
than the bot-# channel. We collect a total of 235k unique user-generated
prompts, which is further filtered according to NSFW ratings and caption
length.

###  3.2 Obtaining natural language captions

As shown in Figure 4, the user-submitted prompts generally took a tag-like
format, with descriptors being separated by commas. Such prompts are
convenient for users to specify and likely achieve good results due to the use
of CLIP text networks for conditioning, which can operate like bag-of-words
models [69, 70]. However, such prompts generally perform poorly when typical
NLP pipelines are used for analysis. These prompts further may not explicitly
specify needed visual relationships in the text, and instead excessively rely
on the prior learned by the diffusion model to disambiguate relationships. In
order to mitigate this issue, we utilize an LLM model to clean up the original
raw user-generated prompts. We use Gemini 1.0 Pro for this task, as it
performed competitively against other models at the time of our work [71] and
offered a free API. The model was instructed to take the user-generated
prompts and transform them into natural language captions. To enhance the
results, we augment the prompt via in-context learning from GPT-4 input/output
pairs. To remove NSFW prompts, we record the Gemini API safety ratings for
each input prompt, and remove the prompts where a 4 out of 4 rating was given
on the axes of sexuality/hate speech/harassment, or if the model itself
produced a refusal, or if the prompt was repeatedly returned with an error
(blocked by Google). Please see Figure 2 for a visualization of the pipeline.

###  3.3 Image generation and semantic attribution

Dataset | Total Images | Total Captions | Caption Source | Human Preferred | Open-set Semantics
---|---|---|---|---|---
COCO 2017 [57] |  123k | 617k | H | N | N
LAION-COCO [54] | 600M | 600M | M | N | N
DiffusionDB [66] | 14M | 1.8M | H | N | N
JourneyDB [65] | 4.7M | 1.7M | H+M | N | N
StableSemantics | 2M | 224k | H+M | Y | 10.8M

Table 1: Size of the different components of StableSemantics. Our captions are
selected by humans to correspond to visually interesting images. We are the
only dataset to provide dense open-set spatial semantic maps. Our maps are
derived from the cross-attention maps in Stable prompts. Note that 235k unique
captions are collected, 224k remain after NSFW filtering and only 200k
captions are used for image generation after filtering for length. Figure 3:
Histogram of dataset statistics. (a) We visualize the cosine CLIP similarities
between generated images and original captions. (b) Number of tokens in the
captions. (c) NSFW scores of the captions after LLM filtering. Scores measured
by LLaMA Guard 2 for sexuality and hate.

To provide a fully reproducible pipeline for the images and maximize the
usability of our dataset, we generate the images ourselves using open weights
and record the random seed for each generation. Images are generated using
sdxl lightning 4step unet [72], a few-step distilled version of Stable
Diffusion XL. For each prompt, we perform parsing using spaCy en_core_web_lg
to extract noun chunks. To obtain mappings from noun chunks to spatial
attributions, we use Diffusion Attentive Attribution Maps which measures the
cross-attention from tokens in the language condition to the UNet.
Specifically, we used the improved DAAM-i2i guided heatmap variant [15, 73]
which improves object localization. We observe that unrelated articles like
"a", and "the" and possessive determiners like "his", "her", "our", "their"
are not typically localized to a specific object, but rather have attribution
maps diffuse over the background or various objects in the image. While
similar phenomena has been noted in ViTs [74] and pure text LLMs [75, 76, 77],
our observation is novel in that text-to-image diffusion models are encoding
contextual information in these "filler" words. For effective localization, we
remove articles and possessive determiners if they are the first word of a
noun chunk.

##  4 Dataset exploration

Figure 4: Example of SDXL generated images from the captions, raw user prompts
and LLM processed captions. Raw prompts from users often contain typos or take
the form a non-natural language tag-like format. We instruct an LLM to
transform the prompts into a natural language caption. Noun chunks (bolded and
underlined) are derived from dependency parsing. Images are generated from the
captions, with diffusion attribution maps recorded for the noun chunks.

We first evaluate the CLIP similarity between the generated images and the
captions, and further characterize the safety and length of the captions. We
then explore the semantic distribution of the images and the captions using
CLIP, and visualize the spatial distribution of objects in a scene using
diffusion attribution maps. Finally, we evaluate the performance of captioning
and open-set segmentation models on our dataset. These characterizations
demonstrate how StableSemantics can be a promising dataset for advancing
visual semantic understanding. The data will be released under a CC0 1.0
license.

###  4.1 Dataset characterization

After deduplication and LLM NSFW filtering, we have 224 thousand natural
language captions. We evaluate the similarity of the SDXL-lightning generated
images and the captions in Figure 3a using OpenAI’s CLIP ViT-B/16 [78]. We
find that the CLIP similarity peaks at $0.34$, which is similar to CLIP scores
achieved using SDXL. These scores typically range from $0.2$ to $0.5$ which
means that our prompts and images can be interpreted to be semantically very
similar given the higher range of scores. We visualize the token length of the
captions in Figure 3b. Note that we do not generate images for captions
exceeding $77$ tokens post-padding. This yields a total of 200k captions which
are used for image generation. In Figure 3c, we plot the NSFW scores of the
captions used for image generation, as evaluated using the state-of-the-art
Meta Llama Guard 2 model. We define the unsafe categories to the 3 official
categories relating to sexual content, and the 1 official category related to
hate speech. The scores are the "unsafe" softmax outputs between the "safe"
and "unsafe" tokens. We find that the captions used for generation are
overwhelmingly safe. In Figure 4, we provide examples of the images, the
original human generated prompts which may often contain typos or tags, and
the LLM output natural language prompts. Likely due to human preference, we
observe a higher ratio of images with visually interesting compositions.

Figure 5: Visualization of the dataset. We show example captions used for
image generation, images generated from the captions, and select noun chunks
and their corresponding attention attribution maps. We find that our dataset
contains accurate localizations for different semantic concepts. Figure 6:
UMAP visualization of dataset CLIP embeddings. We use OpenAI CLIP ViT-B/16 to
compute embeddings for both the generated (a) images and the (b) text. UMAP
with the cosine metric is used to perform dimensionality reduction. We observe
that images describing people, scenes, text, and animals occur with high
frequency.

###  4.2 Semantic exploration of the dataset

In Figure 6 we visualize the semantic distribution of whole images and the
captions used to generate the images. We utilize UMAP [79] with a cosine
metric applied to CLIP embeddings for this visualization with wizmap [80]. We
find that the distribution of both images and text exhibit peaks in concepts
such as people, scenes, text, and animals (cats and dogs). These peaks likely
reflect the effect of human preference on visually interesting images.

The semantic maps we provide in our dataset help localize image regions
corresponding to specific noun chunks from the prompts. In Figure 5 we
visualize the captions used from image generation, the generated RGB image,
and attention attribution masks corresponding to noun chunks shown in bold. We
find that our dataset can provide semantic attributions that are well aligned
to objects in the scene. This is likely due to the nature of Stable Diffusion,
which leverages cross-attention guidance to generate complex compositional
images.

Figure 7: Spatial distribution of semantic concepts. For each concept, we
visualize an example image containing the concept, as well as the spatial
distribution averaged over occurrences. We utilize CLIP text similarity to
select the top-100 most similar noun chunks and average those occurrences.

We use these maps to analyze whether our images exhibit a trend of certain
concepts being generated in specific regions on average. In Figure 7 we
aggregate the masks of the top 100 noun chunks that have the highest CLIP
similarity scores with concepts of interest. We apply this similarity based
matching to allow for inexact matching. Figure 7 clearly shows that the
spatial distribution of concepts can be highly non-uniform. This bias likely
reflects the distribution of concepts in natural images [27, 81]. For
instance, it makes sense for the sunset to always be on the top, the walls
towards the sides, and the floor towards the bottom. It is also very common to
find human beings as the primary subjects in images which explains the
placement of hair surrounding a central region. Finally, many images exhibit
food on top of a table and cars on roads. In these scenarios, these semantic
concepts typically occupy the bottom half of the visual field. We visualize
[82] the frequency distribution of nouns grouped by wordnet hierarchy in
Figure 8. Our dataset could be used to understand the spatial and visual bias
present in natural images.

Figure 8: Frequency of nouns visualized with wordnet hierarchy. We parse the
sentences and extract the nouns. The hierarchy is from wordnet [83]. The
circle size corresponds to frequency.

###  4.3 Evaluation of models

In this section we evaluate the performance of state-of-the-art open-
vocabulary image segmentation and captioning models on our dataset. For open-
vocabulary segmentation methods, we evaluate the standard mean Intersection
over Union (mIOU), where discrete masks are computed by taking the argmax over
all noun chunks’ continuous masks for a given prompt. As these methods also
produce soft masks, we also evaluate the pearson correlation of the
attribution maps from our datasets. In Table 3, We find that recent open-
vocabulary segmentation models which modify CLIP (LSeg, SCLIP) or leverage
text-to-image diffusion models (ODISE) perform better than their peers.

We also evaluate recent vision-language models for image captioning in Table
3. We evaluate the generated captions against the original captions using
state-of-the-art E5-Mistral model to evaluate cosine similarity ($\times 100$
for clarity), BLEU-4, and CIDEr scores. These results suggest that while the
captions predicted by models may use different wording from the original
caption, they can be semantically very similar.

Method mIoU $\uparrow$ Pearson $\uparrow$ MaskCLIP [84] 0.015 0.199 SCLIP [85]
0.109 0.236 LSeg [86] 0.164 0.032 CLIPSeg [87] 0.133 0.143 ODISE [6] 0.096
0.300 OVSeg [88] 0.035 0.181 Table 2: Performance comparison of open-set
segmentation models. We evaluate the intersection over union and pearson
correlation for noun chunks against model outputs. Methods E5-Mistral[89]
$\uparrow$ BLEU [90] $\uparrow$ CIDEr [91] $\uparrow$ LLaVA [92] 67.9 1.2 3.1
BLIP-2 [93] 70.9 1.9 10.2 GIT [94] 63.3 1.0 6.8 CoCa [95] 66.8 1.7 9.7 Table
3: Performance comparison of captioning models. We apply captioning models and
evaluate the alignment of the outputs against the captions used to generate
the images.

##  5 Discussion

Limitations and Future Work. Our work relies on human-submitted prompts, which
may exhibit non-natural semantic co-occurrences. During the data collection
process, we also observed a strong shift in the semantic distribution of
prompts and images around holidays (Thanksgiving, Christmas). This suggests
that continual data collection is required to mitigate bias.
Conclusion. We introduce StableSemantics, the first large-scale dataset that
combines natural language captions, synthetic images, and diffusion
attribution maps. Our work goes beyond prior datasets by providing spatially
localized noun chunk to image region mappings. We explore the semantic
distribution of whole images and objects within an image. The availability of
this dataset will allow for the use of synthetic visual data in additional
domains.

## References

  * Brust and Denzler [2018] Clemens-Alexander Brust and Joachim Denzler.  Not just a matter of semantics: the relationship between visual similarity and semantic similarity.  _arXiv preprint arXiv:1811.07120_ , 2018.
  * Duan and Kuo [2021] Jiali Duan and C.-C. Jay Kuo.  Bridging gap between image pixels and semantics via supervision: A survey.  _ArXiv_ , abs/2107.13757, 2021.  URL https://api.semanticscholar.org/CorpusID:236493168.
  * Alqasrawi [2016] Yousef Alqasrawi.  Bridging the gap between local semantic concepts and bag of visual words for natural scene image retrieval.  _ArXiv_ , abs/2210.08875, 2016.  URL https://api.semanticscholar.org/CorpusID:63956645.
  * Barz and Denzler [2020] Björn Barz and Joachim Denzler.  Content-based image retrieval and the semantic gap in the deep learning era.  _ArXiv_ , abs/2011.06490, 2020.  URL https://api.semanticscholar.org/CorpusID:226306702.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9650–9660, 2021.
  * Xu et al. [2023] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello.  Open-vocabulary panoptic segmentation with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2955–2966, 2023.
  * Elharrouss et al. [2021] Omar Elharrouss, Somaya Al-Maadeed, Nandhini Subramanian, Najmath Ottakath, Noor Almaadeed, and Yassine Himeur.  Panoptic segmentation: A review.  _arXiv preprint arXiv:2111.10250_ , 2021.
  * VS et al. [2024] Vibashan VS, Shubhankar Borse, Hyojin Park, Debasmit Das, Vishal Patel, Munawar Hayat, and Fatih Porikli.  Possam: Panoptic open-vocabulary segment anything.  _arXiv preprint arXiv:2403.09620_ , 2024.
  * Hu et al. [2023] Jie Hu, Linyan Huang, Tianhe Ren, Shengchuan Zhang, Rongrong Ji, and Liujuan Cao.  You only segment once: Towards real-time panoptic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17819–17829, 2023.
  * Quinn et al. [2017] Max H. Quinn, Erik Conser, Jordan M. Witte, and Melanie Mitchell.  Semantic image retrieval via active grounding of visual situations.  _2018 IEEE 12th International Conference on Semantic Computing (ICSC)_ , pages 172–179, 2017.  URL https://api.semanticscholar.org/CorpusID:4871106.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Podell et al. [2023] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ , 2023.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020.
  * Ho et al. [2022] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Tang et al. [2022] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture.  What the daam: Interpreting stable diffusion using cross attention.  _arXiv preprint arXiv:2210.04885_ , 2022.
  * Girshick et al. [2011] Ahna R Girshick, Michael S Landy, and Eero P Simoncelli.  Cardinal rules: visual orientation perception reflects knowledge of environmental statistics.  _Nature Neuroscience_ , 14:926–932, 7 2011.  ISSN 1097-6256.  doi: 10.1038/nn.2831.  URL http://www.nature.com/articles/nn.2831.
  * van der Schaaf and van Hateren [1996] A. van der Schaaf and J.H. van Hateren.  Modelling the power spectra of natural images: Statistics and information.  _Vision Research_ , 36(17):2759–2770, 1996.  ISSN 0042-6989.  doi: https://doi.org/10.1016/0042-6989(96)00002-8.  URL https://www.sciencedirect.com/science/article/pii/0042698996000028.
  * Zoran [2013] Daniel Zoran.  _Natural Image Statistics for Human and Computer Vision_.  PhD thesis, Citeseer, 2013.
  * Heiler and Schnörr [2005] Matthias Heiler and Christoph Schnörr.  Natural image statistics for natural image segmentation.  _International Journal of Computer Vision_ , 63:5–19, 2005.
  * Fang et al. [2012] Hui Fang, Gary Kwok-Leung Tam, Rita Borgo, Andrew J Aubrey, Philip W Grant, Paul L Rosin, Christian Wallraven, Douglas Cunningham, David Marshall, and Min Chen.  Visualizing natural image statistics.  _IEEE Transactions on Visualization and Computer Graphics_ , 19(7):1228–1241, 2012.
  * Mechrez et al. [2019] Roey Mechrez, Itamar Talmi, Firas Shama, and Lihi Zelnik-Manor.  Maintaining natural image statistics with the contextual loss.  In _Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part III 14_ , pages 427–443. Springer, 2019.
  * Kleinlein et al. [2022] Ricardo Kleinlein, Alexander Hepburn, Raúl Santos-Rodríguez, and Fernando Fernández-Martínez.  Sampling based on natural image statistics improves local surrogate explainers.  _arXiv preprint arXiv:2208.03961_ , 2022.
  * Hepburn et al. [2023] Alexander Hepburn, Valero Laparra, Raul Santos-Rodriguez, and Jesús Malo.  Disentangling the link between image statistics and human perception.  _arXiv preprint arXiv:2303.09874_ , 2023.
  * Xiang et al. [2024] Ao Xiang, Jingyu Zhang, Qin Yang, Liyang Wang, and Yu Cheng.  Research on splicing image detection algorithms based on natural image statistical characteristics.  _arXiv preprint arXiv:2404.16296_ , 2024.
  * Hepburn et al. [2021] Alexander Hepburn, Valero Laparra, Raul Santos-Rodriguez, Johannes Ballé, and Jesús Malo.  On the relation between statistical learning and perceptual distances.  _arXiv preprint arXiv:2106.04427_ , 2021.
  * Talbot et al. [2023] Morgan B Talbot, Rushikesh Zawar, Rohil Badkundri, Mengmi Zhang, and Gabriel Kreiman.  Tuned compositional feature replays for efficient stream learning.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2023.
  * Torralba and Oliva [2003] Antonio Torralba and Aude Oliva.  Statistics of natural image categories.  2003\.
  * Henderson et al. [2023] Margaret M. Henderson, Michael J. Tarr, and Leila Wehbe.  Low-level tuning biases in higher visual cortex reflect the semantic informativeness of visual features.  _Journal of Vision_ , 23(4):8–8, 04 2023.  ISSN 1534-7362.  doi: 10.1167/jov.23.4.8.  URL https://doi.org/10.1167/jov.23.4.8.
  * Bao et al. [2024] Fan Bao, Chendong Xiang, Gang Yue, Guande He, Hongzhou Zhu, Kaiwen Zheng, Min Zhao, Shilong Liu, Yaole Wang, and Jun Zhu.  Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models.  _arXiv preprint arXiv:2405.04233_ , 2024.
  * Gupta et al. [2023] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama.  Photorealistic video generation with diffusion models.  _arXiv preprint arXiv:2312.06662_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Evans et al. [2024] Zach Evans, Julian D Parker, CJ Carr, Zack Zukowski, Josiah Taylor, and Jordi Pons.  Long-form music generation with latent diffusion.  _arXiv preprint arXiv:2404.10301_ , 2024.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Luhman and Luhman [2023] Troy Luhman and Eric Luhman.  High fidelity image synthesis with deep vaes in latent space.  _arXiv preprint arXiv:2303.13714_ , 2023.
  * Harvey et al. [2021] William Harvey, Saeid Naderiparizi, and Frank Wood.  Conditional image generation by conditioning variational auto-encoders.  In _International Conference on Learning Representations_ , 2021.  URL https://api.semanticscholar.org/CorpusID:249191864.
  * Razavi et al. [2019] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2.  _Advances in neural information processing systems_ , 32, 2019.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Rezende and Mohamed [2015] Danilo Rezende and Shakir Mohamed.  Variational inference with normalizing flows.  In _International conference on machine learning_ , pages 1530–1538. PMLR, 2015.
  * Tong et al. [2023] Alexander Tong, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector-Brooks, Kilian Fatras, Guy Wolf, and Yoshua Bengio.  Improving and generalizing flow-based generative models with minibatch optimal transport.  _arXiv preprint arXiv:2302.00482_ , 2023.
  * Dinh et al. [2014] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  _CoRR_ , abs/1410.8516, 2014.  URL https://api.semanticscholar.org/CorpusID:13995862.
  * Kingma and Dhariwal [2018] Diederik P. Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  _ArXiv_ , abs/1807.03039, 2018.  URL https://api.semanticscholar.org/CorpusID:49657329.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Arjovsky et al. [2017] Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein generative adversarial networks.  In _International conference on machine learning_ , pages 214–223. PMLR, 2017.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Kang et al. [2023] Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park.  Scaling up gans for text-to-image synthesis.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10124–10134, 2023.  URL https://api.semanticscholar.org/CorpusID:257427461.
  * Mirza and Osindero [2014] Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _ArXiv_ , abs/1411.1784, 2014.  URL https://api.semanticscholar.org/CorpusID:12803511.
  * Zawar et al. [2022] Rushikesh Zawar, Krupa Bhayani, Neelanjan Bhowmik, Kamlesh Tiwari, and Dhiraj Sangwan.  Detecting anomalies using generative adversarial networks on images.  _arXiv preprint arXiv:2211.13808_ , 2022.
  * Huang et al. [2023a] Mengqi Huang, Zhendong Mao, Quang Wang, and Yongdong Zhang.  Not all image regions matter: Masked vector quantization for autoregressive image generation.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 2002–2011, 2023a.  URL https://api.semanticscholar.org/CorpusID:258841408.
  * Parmar et al. [2018] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam M. Shazeer, Alexander Ku, and Dustin Tran.  Image transformer.  In _International Conference on Machine Learning_ , 2018.  URL https://api.semanticscholar.org/CorpusID:3353110.
  * Lee et al. [2022] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han.  Autoregressive image generation using residual quantization.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11513–11522, 2022.  URL https://api.semanticscholar.org/CorpusID:247244535.
  * Huang et al. [2023b] Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang.  Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization.  _2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 22596–22605, 2023b.  URL https://api.semanticscholar.org/CorpusID:258823089.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _ArXiv_ , abs/2204.06125, 2022.  URL https://api.semanticscholar.org/CorpusID:248097655.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=M3Y74vmsMcY.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and J. Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78, 2014.  URL https://api.semanticscholar.org/CorpusID:3104920.
  * Thomee et al. [2016] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: the new data in multimedia research.  _Commun. ACM_ , 59(2):64–73, jan 2016.  ISSN 0001-0782.  doi: 10.1145/2812802.  URL https://doi.org/10.1145/2812802.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European Conference on Computer Vision_ , 2014.  URL https://api.semanticscholar.org/CorpusID:14113767.
  * Agrawal et al. [2015] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra.  Vqa: Visual question answering.  _International Journal of Computer Vision_ , 123:4 – 31, 2015.  URL https://api.semanticscholar.org/CorpusID:3180429.
  * Goyal et al. [2016] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  _International Journal of Computer Vision_ , 127:398 – 414, 2016.  URL https://api.semanticscholar.org/CorpusID:8081284.
  * Marino et al. [2019] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 3190–3199, 2019.  URL https://api.semanticscholar.org/CorpusID:173991173.
  * Wang et al. [2018] Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton van den Hengel.  Fvqa: Fact-based visual question answering.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 40(10), 2018.  doi: 10.1109/TPAMI.2017.2754246.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _Int. J. Comput. Vision_ , 123(1), may 2017.  ISSN 0920-5691.  doi: 10.1007/s11263-016-0981-7.  URL https://doi.org/10.1007/s11263-016-0981-7.
  * Tian et al. [2024] Yonglong Tian, Lijie Fan, Phillip Isola, Huiwen Chang, and Dilip Krishnan.  Stablerep: Synthetic images from text-to-image models make strong visual representation learners.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Kirstain et al. [2023] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=G5RwHpBUv0.
  * Sun et al. [2023] Keqiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li.  JourneyDB: A benchmark for generative image understanding.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=vfzXDRTcF4.
  * Wang et al. [2022a] Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau.  DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models.  _arXiv:2210.14896 [cs]_ , 2022a.  URL https://arxiv.org/abs/2210.14896.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Rafailov et al. [2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Thrush et al. [2022] Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5238–5248, 2022.
  * Yuksekgonul et al. [2022] Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Lin et al. [2024] Shanchuan Lin, Anran Wang, and Xiao Yang.  Sdxl-lightning: Progressive adversarial diffusion distillation.  _arXiv preprint arXiv:2402.13929_ , 2024.
  * Chowdhury [2024] Rishi Dey Chowdhury.  Daam-image2image, 2024.  URL https://github.com/RishiDarkDevil/daam-i2i.
  * Darcet et al. [2023] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski.  Vision transformers need registers.  _arXiv preprint arXiv:2309.16588_ , 2023.
  * Clark et al. [2019] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning.  What does bert look at? an analysis of bert’s attention.  _arXiv preprint arXiv:1906.04341_ , 2019.
  * Kovaleva et al. [2019] Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky.  Revealing the dark secrets of bert.  _arXiv preprint arXiv:1908.08593_ , 2019.
  * Xiao et al. [2023] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.  Efficient streaming language models with attention sinks.  _arXiv preprint arXiv:2309.17453_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2021.
  * McInnes et al. [2018] Leland McInnes, John Healy, and James Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_ , 2018.
  * Wang et al. [2023a] Zijie J Wang, Fred Hohman, and Duen Horng Chau.  Wizmap: Scalable interactive visualization for exploring large machine learning embeddings.  _arXiv preprint arXiv:2306.09328_ , 2023a.
  * Greene [2013] Michelle R. Greene.  Statistics of high-level scene context.  _Frontiers in Psychology_ , 4, 2013.  doi: 10.3389/fpsyg.2013.00777.
  * [82]
  * Fellbaum [2010] Christiane Fellbaum.  Wordnet.  In _Theory and applications of ontology: computer applications_ , pages 231–243. Springer, 2010.
  * Dong et al. [2023] Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al.  Maskclip: Masked self-distillation advances contrastive language-image pretraining.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10995–11005, 2023.
  * Wang et al. [2023b] Feng Wang, Jieru Mei, and Alan Yuille.  Sclip: Rethinking self-attention for dense vision-language inference.  _arXiv preprint arXiv:2312.01597_ , 2023b.
  * Li et al. [2022] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation.  _arXiv preprint arXiv:2201.03546_ , 2022.
  * Lüddecke and Ecker [2022] Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 7086–7096, June 2022.
  * Liang et al. [2023] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7061–7070, 2023.
  * Wang et al. [2023c] Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei.  Improving text embeddings with large language models.  _arXiv preprint arXiv:2401.00368_ , 2023c.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.  doi: 10.3115/1073083.1073135.  URL https://aclanthology.org/P02-1040.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575, 2015.
  * Liu et al. [2024] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _Advances in neural information processing systems_ , 36, 2024.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _International conference on machine learning_ , pages 19730–19742. PMLR, 2023.
  * Wang et al. [2022b] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  _arXiv preprint arXiv:2205.14100_ , 2022b.
  * Yu et al. [2022] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Gebru et al. [2021] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford.  Datasheets for datasets.  _Communications of the ACM_ , 64(12):86–92, 2021.

Supplementary Material:
StableSemantics: A Synthetic Language-Vision Dataset of Semantic
Representations in Naturalistic Images

Sections

  1. 1.

Additional dataset visualization (section A.1)

  2. 2.

Visualization of object distributions (section A.2)

  3. 3.

Comparison of open vocabulary segmentation methods (section A.3)

  4. 4.

Prompt used for language model cleanup of raw prompts (section A.4)

  5. 5.

Dataset documentation (section A.5)

  6. 6.

Broader impacts (section A.6)

  7. 7.

Dataset website (section A.7)

  8. 8.

Statement of responsibility (section A.8)

  9. 9.

Dataset license (section A.9)

  10. 10.

Hosting plan and dataset format (section A.10)

##  A.1 Additional dataset visualization

In this section, we provide additional visualizations of the natural language
captions generated using a large language model from the raw user prompts, the
RGB image, and semantic masks corresponding to select noun chunks in the image
in Figure S.1 and Figure S.2.

Figure S.1: Visualization of additional dataset examples. We show the natural
language caption used for the image generation, the image, and masks
corresponding to select noun chunks. Figure S.2: Visualization of additional
dataset examples. We show the natural language caption used for the image
generation, the image, and masks corresponding to select noun chunks.

##  A.2 Visualization of object distributions

In this section, we visualize the spatial distribution of various noun chunks
in Figure S.3. We note that several types of objects exhibit highly non-
uniform spatial distributions.

Figure S.3: Additional examples on the spatial distribution of concepts. We
provide additional examples of images containing a concept, and the average
distribution of the top-100 images containing the most similar noun chunks as
evaluated using the CLIP text model.

##  A.3 Comparison of open vocabulary segmentation methods

In this section, we provide additional visualizations of the semantic maps
from our dataset, and segmentation outputs in Figure S.4 and Figure S.5. We
note that in general, the semantic masks in our dataset can accurately
localize objects. However, there are specific cases (Lone Man) where the
attention maps corresponding to noun chunks can include other contextual
objects. We believe this occurs when the diffusion model tries to generate co-
occurring scene and image parts that are not explicitly mentioned in the
caption.

Figure S.4: Comparison of semantic maps and open vocabulary segmentation
methods. We visualize the semantic attribution maps corresponding to noun
chunks from our dataset, and the segmentation maps produced by various state-
of-the-art methods. Figure S.5: Comparison of semantic maps and open
vocabulary segmentation methods. We visualize the semantic attribution maps
corresponding to noun chunks from our dataset, and the segmentation maps
produced by various state-of-the-art methods. Note that the Lone Man
illustrates how the semantic attribution maps can be imperfect. In this case
it includes additional background.

##  A.4 Prompt used for language model cleanup of raw prompts

We utilize the following prompt followed by the raw user prompt to obtain a
natural language caption. Note that our raw prompts undergo simple regex based
processing to remove some obvious errors before being provided to the language
model.

You are going to be provided with the description of an image. You will
transform and edit the description as needed into a cohesive natural language
sentence or sentences without elaborating. If the original is mixed language,
then your output should also be mixed language. Do not elaborate, do not
provide information about people mentioned in the description.

Make a best effort to use ALL WORDS AND DETAILS from the original description.
DO NOT MAKE UP DETAILS unless absolutely necessary. BE AS CONCISE AS POSSIBLE,
WHILE ATTEMPTING TO INCLUDE ALL WORDS AND DETAILS FROM THE ORIGINAL
DESCRIPTION. You may only omit details or words if they are nonsensical or
form a contradiction. Attempt to fix typos and remove invalid punctuation.
Omit emojis in your output.

When you output, use [START] before the output, and include [END] after the
output. You may retain hash tags in the output only if hash tags were used in
the original prompt. Here is the original description:

##  A.5 Dataset documentation

This document is based on Datasheets for Datasets by Gebru et al. [96].

##  MOTIVATION

For what purpose was the dataset created?  Was there a specific task in mind?
Was there a specific gap that needed to be filled? Please provide a
description.
This dataset consists of raw human-generated prompts that underwent human
curation (via human voting to the showdown and pantheon channels), natural
language captions generated using a large-language model on these prompts,
multiple images generated using the natural langauge captions, and the
semantic maps from diffusion cross-attention maps for the noun chunks parsed
from the captions. To our knowledge, no other dataset has systematically
recorded the cross-attention maps from a text-to-image generative model. Our
dataset provides a basis for improving open vocabulary segmentation methods
and for contrastive learning by leveraging the multiple images generated for
each caption.

Who created this dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?
This dataset was created by a team from Carnegie Mellon University and
leveraged CMU computational resources for the image generation.

What support was needed to make this dataset?  (e.g.who funded the creation of
the dataset? If there is an associated grant, provide the name of the grantor
and the grant name and number, or if it was supported by a company or
government agency, give those details.)
No specific grant was used. We utilized GPU computational resources available
to CMU students for image generation.

##  COMPOSITION

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)?  Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.
The dataset is composed of the raw prompts generated by humans, the natural
language captions derived from those prompts using a large language model, the
ten images for each caption generated using a Stable Diffusion XL based model,
and the individual semantic maps corresponding to the noun chunks for each
caption for each generated image.

How many instances are there in total (of each type, if appropriate)?
There are 10 instances of generated images for each caption. Each of these
captions has an average of 5 noun chunks and thus 5 semantic maps for a
generated image. Thus, on average, there are about 10 generated images and 50
semantic maps for each prompt. After NSFW and caption length filtering, 200
thousand such captions are used for image generation. Hence, about 2 million
generated images and about 10 million semantic maps. We also share the raw
versions of the prompts, thus there are also 224 thousand raw prompts (post
NSFW filtering).

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set?  If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).
We process the dataset to remove NSFW prompts and captions that exceed the
Stable Diffusion XL 77 token limit from the image generation process.

What data does each instance consist of?  “Raw” data (e.g., unprocessed text
or images) or features? In either case, please provide a description.
The generated image instances and the semantic maps are of "jpg" type. Both
the raw prompts and natural language prompts are stored in a "pickle" file.

Is there a label or target associated with each instance?  If so, please
provide a description.
The generated image, corresponding semantic maps, the corresponding natural
language text prompt, the raw prompt are associated with a unique hash key.
This key helps identify and extract the corresponding instances. There are
also "pickle" files present for each image that help identify the semantic map
for a given noun chunk in the prompt. The seed used for image generation is
also saved.

Is any information missing from individual instances?  If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.
We perform some simple heuristic-based filtering to remove certain noun chunks
corresponding to non-physical objects. We only save the semantic maps for the
remaining noun chunks. However, we will also provide the original captions so
all noun chunks can be extracted if needed.

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)?  If so, please describe how these
relationships are made explicit.
Yes, the relations are made explicit. The instances of generated images,
prompts, and semantic maps are linked by a unique hash key. The relation
between noun chunks and their corresponding semantic maps can be identified
with the provided "offset" pickle file.

Are there recommended data splits (e.g., training, development/validation,
testing)?  If so, please provide a description of these splits, explaining the
rationale behind them.
No, there are no recommended splits. The user is free to split the data as per
their need.

Are there any errors, sources of noise, or redundancies in the dataset?  If
so, please provide a description.
We observe that some raw prompts are highly similar, which seems to happen as
a user quickly iterates on a prompt and creates prompts with a similar theme.
We do not explicitly filter or remove such conceptually similar prompts. We
also note that large language models are not perfect, and may create errors or
deviations of the generated caption from the raw user prompts.

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)?  If it links to
or relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a future user? Please provide descriptions of all external resources and
any restrictions associated with them, as well as links or other access
points, as appropriate.
It is self-contained and doesn’t depend on any other resource. The resources
used to create/modify the dataset have been mentioned above, however, they are
no longer needed to use the dataset.

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor-patient
confidentiality, data that includes the content of individuals’ non-public
communications)?  If so, please provide a description.
No, the data was created by using publicly available information and doesn’t
involve any confidential or non-public information.

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety?  If so, please
describe why.
We have used Gemini 1.0 Pro to process the dataset to ensure that there are no
harmful or offensive prompts. The prompts were then used to generate images
from Stable Diffusion XL, which has an NSFW filter. The data is however
subject to the failure of these models.

Does the dataset relate to people?  If not, you may skip the remaining
questions in this section.
As the prompts in the dataset were extracted from human written prompts in a
discord server, the images generated and the prompts in the data may contain
information about people.

Does the dataset identify any subpopulations (e.g., by age, gender)?  If so,
please describe how these subpopulations are identified and provide a
description of their respective distributions within the dataset.
The dataset is not targeted towards any subpopulation. However, it is subject
to the distribution and preferences of the users of the discord server.

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset?  If so, please describe how.
The dataset is not targeted towards any individual, and we remove the
information of the person submitting the prompt from the public release. The
prompts themselves are subject to the distribution and preferences of the
users of the discord server.

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)?  If so,
please provide a description.
The prompts used are extracted from users’ input in the discord server. We
have used Gemini 1.0 Pro to process the dataset to ensure that there are no
harmful or offensive prompts. The prompts were then used to generate images
from Stable Diffusion XL, which has an NSFW filter. The data is however
subject to the failure of these models and the bias/preferences of the humans.

##  COLLECTION

How was the data associated with each instance acquired?  Was the data
directly observable (e.g., raw text, movie ratings), reported by subjects
(e.g., survey responses), or indirectly inferred/derived from other data
(e.g., part-of-speech tags, model-based guesses for age or language)? If data
was reported by subjects or indirectly inferred/derived from other data, was
the data validated/verified? If so, please describe how.
The data was collected from the showdown and pantheon channels of the official
Stability.ai discord server. In the discord server, users publicly posted
prompts to obtain image generations and competed against other users for the
chance to have prompts and images featured in the pantheon channel. We used
DiscordChatExporter software to obtain the public prompts.

Over what timeframe was the data collected?  Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created. Finally, list when the
dataset was first published.
This data was collected from the showdown channel from July 11, 2023, which is
1 day after the SDXL 1.0 candidates were made available via bots on discord;
until Feb 07, 2024 which is when the SDXL bots were shut down. As pantheon was
derived from showdown, and maintained the history from the start of channel
creation, we scraped the pantheon channel only once, which collected posts
from May 02, 2023 onwards.

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?
How were these mechanisms or procedures validated?
We used DiscordChatExporter executed on a Linux machine on a $14$ minute cron
job to capture all posts from showdown. The showdown was updated every $15$
minutes, with history cleared every $30$ minutes.

What was the resource cost of collecting the data?  (e.g. what were the
required computational resources, and the associated financial costs, and
energy consumption - estimate the carbon footprint.)
We executed the data collection from Discord on a machine that was on
regardless of usage, we do not believe this imposed significant CPU usage or
energy cost.

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?
We filtered the dataset using a large language model, which caused around
$4.6\%$ of the 235k prompts to be removed due to safety reasons (not
transformed to natural language captions), yielding 224k natural language
captions. Furthermore, captions longer than $77$ tokens were not used for
image generation, and roughly 200k captions were used for images.

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)?
The data collection was performed by students as part of their research.

Were any ethical review processes conducted (e.g., by an institutional review
board)?  If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.
No IRB was required as this data collection did not directly involve human
subjects.

Does the dataset relate to people?  If not, you may skip the remainder of the
questions in this section.
Not directly, however, the prompts were submitted by users.

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)?
They were collected from the chat messages publicly submitted by users to
Discord.

Were the individuals in question notified about the data collection?  If so,
please describe (or show with screenshots or other information) how notice was
provided, and provide a link or other access point to, or otherwise reproduce,
the exact language of the notification itself.
No notification was provided as the messages were publicly shared, and
expected to be visible and voted upon by other users. We notified moderators
of the Stability Discord server that we were performing this data collection
effort and did not receive any objection.

Did the individuals in question consent to the collection and use of their
data?  If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.
Users consented to their prompts being publicly visible by submitting prompts
for image generation. Other users were explicitly encouraged to view and vote
on the prompts and images.

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis)been conducted?  If so,
please provide a description of this analysis, including the outcomes, as well
as a link or other access point to any supporting documentation.
We performed an NSFW evaluation using Llama Guard 2 and found that our
captions post-filtering were overwhelmingly safe. Visual examination of over a
thousand randomly selected captions also confirmed that our captions were
largely safe.

##  PREPROCESSING / CLEANING / LABELING

Was any preprocessing/cleaning/labeling of the data done(e.g.,discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)?  If so, please provide a
description. If not, you may skip the remainder of the questions in this
section.
We performed our prompt cleaning in three steps. (1) We use a set of
handcrafted grep filters to remove common prompt corruptions, mispunctuation,
extra spaces, and non-text Unicode. 2 We deduplicate the prompts after grep
filtering. (3) These prompts are provided to a large language model, where
they are filtered and transformed into natural language captions.

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)?  If so, please provide a link or
other access point to the “raw” data.
Yes, the raw prompts will be shared.

Is the software used to preprocess/clean/label the instances available?  If
so, please provide a link or other access point.
Yes, the code will be publicly provided after paper acceptance.

##  USES

Has the dataset been used for any tasks already?  If so, please provide a
description.
We use our dataset to evaluate captioning and open vocabulary segmentation
models. However, the dataset has not been used to train new models as of
writing.

Is there a repository that links to any or all papers or systems that use the
dataset?  If so, please provide a link or other access point.
The link to the dataset will contain papers that use our dataset:
https://stablesemantics.github.io/StableSemantics/

What (other) tasks could the dataset be used for?
The dataset can be used for training/finetuning and evaluating scene
understanding, semantic segmentation, and object detection/classification
models. It can also be used for developing models for tasks like Visual
Question Answering (VQA), visual grounding, inpainting, etc. It can also be
used for understanding the distribution of natural language prompts and the
biases present in them.

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses?  For
example, is there anything that a future user might need to know to avoid uses
that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?
The raw prompts collected are human written prompts and are then processed by
Gemini 1.0 Pro. These prompts are then used to generate the images from Stable
Diffusion XL based model. These models were used following their license
terms.

Are there tasks for which the dataset should not be used?  If so, please
provide a description.
Preventative steps have been taken to ensure there is no harmful content in
the data as explained in the data processing part. However, we still recommend
the data should not be used for any harmful or illegal purpose.

##  DISTRIBUTION

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created?  If so, please provide a description.
The dataset will be under CC0 1.0 Universal Public Domain Dedication license
and thus will distributed openly without any restriction.

How will the dataset be distributed (e.g., tarball on website, API, GitHub)?
Does the dataset have a digital object identifier (DOI)?
The dataset will be shared via a torrent and hosted via http(s). The links to
access the dataset will be available on our website which will be always
maintained.

When will the dataset be distributed?
The complete dataset will be made publicly available by the end of June 2024.

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)?  If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.
It will not be under any copyright or intellectual property (IP) license. The
dataset will be released under the CC0 1.0 Universal Public Domain Dedication
license.

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances?  If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.
No.

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances?  If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.
No.

##  MAINTENANCE

Who is supporting/hosting/maintaining the dataset?
The authors will be hosting and maintaining the dataset.

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?
The authors should be contacted via email for any questions/problems regarding
the dataset.

Is there an erratum?  If so, please provide a link or other access point.
There is no identified erratum with the dataset so far. However, the user must
consider any possible errors associated with the LLM and Diffusion models
used.

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)?  If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)?
The dataset will be updated by the authors if there is any error detected. The
information will be announced on our website from where people can access the
dataset.

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were individuals in question
told that their data would be retained for a fixed period of time and then
deleted)?  If so, please describe these limits and explain how they will be
enforced.
The dataset doesn’t collect any individual’s personal information.

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to users.
The older versions of the dataset will continue to be supported. Any updates
made in the dataset will be hosted as separate instances so that users of the
previous version can still access it.

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so?  If so, please provide a description. Will
these contributions be validated/verified? If so, please describe how. If not,
why not? Is there a process for communicating/distributing these contributions
to other users? If so, please provide a description.
We have no restriction on others augmenting this dataset. However, we don’t
aim to officially merge the extended/augmented data to avoid any possible
inclusion of offensive/harmful or illegal instances.

##  A.6 Broader Impacts

StableSemantics consists of raw prompts, natural language captions derived
from the prompts, $10$ images per caption generated using a Stable Diffusion
XL based model, and semantic maps corresponding to the noun chunks in the
caption for each image generation. The availability of this dataset is
expected to improve visual semantic learning, open vocabulary segmentation
methods, and improve the characterization of text-to-image generative models.
We expect StableSemantics to help develop models that can understand complex
visual scenes. This has the potential to increase the robustness of deep
learning-based visual systems.

##  A.7 Dataset website

Note that the dataset will be soon released. For now, we have released a
sample consisting of images corresponding to $100$ samples of $5$ images each
on the website. https://stablesemantics.github.io/StableSemantics/

##  A.8 Statement of responsibility

Authors accept all responsibility for potential violation of rights.

##  A.9 Dataset license

We release our dataset under a CC0 1.0 Universal Public Domain Dedication
license following Stable Diffusion’s discord policy and other work collecting
data from Stable Diffusion’s discord [66].

##  A.10 Hosting plan and dataset format

We plan to host our dataset via an http(s) link, and simultaneously distribute
the data via a torrent where we permanently seed the files. The captions are
stored in a Python pickle file, while images and semantic maps are jpg files.