only if/.style args=entry of #1 is #2
/pgfplots/boxplot/data filter/.code=

only if both/.style args=entry of #1 is #2 and #3 is #4
/pgfplots/boxplot/data filter/.code=

Text-to-image models can often generate some relations, i.e., “astronaut riding horse”, but fail to generate other relations composed of the same basic parts, i.e., “horse riding astronaut”. These failures are often taken as evidence that models rely on training priors rather than constructing novel images compositionally. This paper tests this intuition on the stablediffusion 2.1 text-to-image model. By looking at the subject-verb-object (SVO) triads that underlie these prompts (e.g., “astronaut’’, “ride’’, “horse’’), we find that the more often an SVO triad appears in the training data, the better the model can generate an image aligned with that triad. Here, by aligned we mean that each of the terms appears in the generated image in the proper relation to each other. Surprisingly, this increased frequency also diminishes how well the model can generate an image aligned with the flipped triad. For example, if “astronaut riding horse” appears frequently in the training data, the image for “horse riding astronaut’’ will tend to be poorly aligned. Our results thus show that current models are biased to generate images with relations seen in training, and provide  new data to the ongoing debate on whether these text-to-image models employ abstract compositional structure in a traditional sense, or rather, interpolate between relations explicitly seen in the training data.

§ INTRODUCTION

Whether neural networks extrapolate beyond their training data is an open question. Part of the debate hinges on the mechanism that supports models' generalization: is it an abstract combinatorial structure of the type that characterized good-old-fashioned-AI [Fodor, 1998], or rather a “mix-and-match” strategy born out of patterns seen in training [Lake and Baroni, 2018, Chaabouni et al., 2020]?

Whichever mechanism models are using, both text [OpenAI, 2023, Ouyang et al., 2022]  and image [Ramesh et al., 2022, Rombach et al., 2021, Saharia et al., 2022, Yu et al., 2022] generation models push far beyond the bounds of what was assumed possible only a few years ago. Anecdotal examples of models writing working code, authoring poetry, and generating images [Bubeck et al., 2023] make it hard to deny that they possess some mechanism for compositionality. However, failures of such models consistently crop up [Marcus, 2022, McKenzie et al., 2022, McKenzie et al., 2022, McKenzie et al., 2023]. Moreover, the increasing scale of these models’ training datasets and costs makes it difficult to run controlled experiments, and thus most debates about how models achieve their apparent compositional behavior rely on speculation or in-principle arguments.

In this work, we analyze the relationship between items seen in training and a model's ability to generalize, in order to better understand what types of mechanisms might underlie the observed behavior.

We focus on the stablediffusion 2.1 text-to-image model. Such models can often generate many relations, i.e., “astronaut riding horse”, but fail to generate atypical relations composed of the same basic parts, i.e., “horse riding astronaut”. These failures are often taken as evidence that the models rely on training priors rather than constructing novel images using more systematic mechanisms. By and large, our results support this intuition. Looking at the subject-verb-object relations that underlie these prompts, we find that the increased frequency of a subject-verb-object relation in the training data improves how well the model can generate an image aligned with that relation, but  this increased frequency diminishes how well the model can generate an image aligned with the flipped relation (object-verb-subject). For example, the more common a relation like “astronaut riding horse” is in the training data, the worse “horse riding astronaut’’ is generated. These results demonstrate that the relations seen during training have a significant impact on what the model is able to successfully generate. We also find that frequencies of individual terms, like how often the subject has been seen in any context, have a significant impact on alignment. Together, these results strongly suggest the model uses an underlying “mix-and-match” mechanism to support generalization, rather than using a more abstract combinatorial process of the type traditionally assumed to be necessary for models with the observed capabilities.

§ EXPERIMENTAL DESIGN

In this section, we formulate our primary questions as hypotheses about how the estimated counts of subject-verb-objects in the training data impact text-to-image prompt and image alignments.

Example generations. Left: “dog chasing a ball”; Right: “ball chasing a dog.” These examples were generated by stablediffusion 2.1 (and are cherry-picked out of 4 generations.)

§.§ The Problem

Our aim is to characterize when text-to-image models successfully generate relations like in <ref>:

A dog chasing a ball.     $\langle \textrm{dog, chase, ball} \rangle$

A ball chasing a dog.     $\langle \textrm{ball, chase, dog} \rangle$

Each prompt has an underlying triad: a subject $s$, a verb $v$, and an object $o$.[Note that in the paper, our relations are such that the subject is always the semantic agent and the object is the semantic patient. Thus, throughout, we use the terms “subject” and “agent” interchangeably, and likewise use “object” and “patient” interchangeably.] For example, for the sentence “A dog chasing a ball,” $s = $“dog,” $v = $“chase,” and $o = $“ball”. For shorthand, we denote this triad  as $\langle s, v, o \rangle$. We construct our dataset to include both $\langle s, v, o \rangle$ and $\langle o, v, s \rangle$– we call the more frequently appearing triad the default triad, and the other, the flipped triad.

Notation To help track various counts, we introduce some notation. This notation is all with respect to a given prompt formatted from a triad $\langle s=S, v=V, o=O \rangle$.

SVO      Est. count of the relation

OVS     Est. count of the flipped relation

Xxx, xXx $\dots$     Est. count of X in the given role

Here, Xxx, xXx $\dots$ refer to the counts of individual entities that appear in that slot in any context except for the counts covered by SVO or OVS.[That is, the triads that contribute to the count Sxx are disjoint from the triads that contribute to the count of SVO. This is done so that the terms are not algebraically tied, violating the independence assumptions needed in our regressions.] For example, Sxx refers to the count of triads with the subject S and any verb and object. Analogously, xxS refers to the count of triads with the term S as the object; Sxx, Oxx, xVx, xxS, xxO are all similarly defined. These terms are all estimated (parsed) counts from the training data of stablediffusion 2.1.[In <ref> we do a deeper dive checking that the (noisy) parsed counts we use reflect the actual counts.]

For each frequency, we use a log-transformed value in our regressions and plots. For example, for SVO, we use $\log_{10}(\textrm{SVO} + 1)$ s.t. the frequencies range from $[0, \infty]$.

§.§ Hypotheses

We test two primary hypotheses, below.

Increased SVO causes better alignment for $\langle S, V, O \rangle$.

Increased OVS causes worse alignment for $\langle S, V, O \rangle$.

§.§ Statistical Methods

Though we present results across different views of our dataset, we focus on a single multiple regression to test our hypotheses.

Alignment $\sim$ SVO + OVS +
Sxx + xVx + xxO + Sxx + xxO[This R syntax is meant to show that we see how the alignment of the prompts can be fit by the counts SVO and OVS while being controlled by the individual term counts.]

Specifically, we are interested in which of the above counts have a significant effect on alignment, controlling for the other counts. We detail how we measure alignment below in <ref>. In short, we crowdsource $N=5$ ratings per image-prompt pair. There are a number of different options with different tradeoffs on how to handle the multiple ratings per image. If we consider the raters as random effects, we control for raters' variation, but we are no longer able to report $p-$values. Happily, this decision did not meaningfully impact the results, so the regressions use disaggregated ratings (no averaging) without random effects. See alternate formulations in  <ref> and  <ref> in the Appendix. For visual acuity, our plots use average alignments.

Our data and data collection break independence assumptions. The individual terms (the S, V, and O) are all individually frequent. Furthermore, we bootstrapped our search to find more relations with high-frequency counts. Both of these decisions mean the relations in our dataset are not randomly sampled from among the population of possible relations. Furthermore, as part of the curation, we manually edit and select prompts that are “drawable”–i.e., one could in principle draw it, even if the relation is not natural. For example, we filter out relations like “I love it” and “we help child.”

Lastly, the default and flipped versions of a prompt are not independent. The symmetry across the diagonals of <ref> highlights this point. Splitting the dataset into two partitions, as we do in <ref>, removes this problem as we run the regressions separately.[<ref> plots highlight that the triad frequencies (SVO and OVS) are correlated with the individual frequencies (Sxx, Oxx, …). Moreover, the SVO and OVS frequencies between a default and a flipped triad are inverted. To control these dependencies, when we analyze the full dataset in <ref>, we split default and flipped pairs into different subsets.]

§ DATASET

We use the text-to-image model stablediffusion 2.1 because its training data, code, and model weights are public. We parse its training set, LAION [Schuhmann et al., 2022][We use the subset with an aesthetic score 4.5+ as this is what stablediffusion 2.1 was trained on.], and create a curated dataset of about 769 triads, counts, prompts, generated images, and alignment ratings.
Later sections (<ref>) use different views of this dataset. The dataset is released in the linked url.[Dataset: <https://docs.google.com/spreadsheets/d/1hcMEMqSX_0regA-s0UplvzrNX2Ho0jHNeIIq-iVoQPI/edit?usp=sharing>]

group style=
group size=2 by 2,
horizontal sep=5pt,
vertical sep=25pt,
x descriptions at=edge bottom,
y descriptions at=edge left,

point meta=explicit,
mark=square*, mark size=1pt,
only marks,

] table tikz/dataset.dat;
point meta=explicit,
mark=square*, mark size=1pt,
only marks,
] table tikz/dataset.dat

legend style=at=(1.05, 0.0),anchor=south west,
legend columns=1,
reverse legend
point meta=explicit,
mark=square*, mark size=1pt,
only marks,
] table tikz/dataset.dat;

$0$, $10^1_-$,$10^2_-$,$10^3_-$,$10^4_-$,$10^5_+$;

Dataset Samples Plotted by SVO (x-axis) and OVS (y-axis).
These plots each show the (same) dataset colored by different frequency counts: Sxx, xVx, xxO. The legend details the coloring. $10^5_+$ examples are seen that many times or more. $10^4_-$ examples are seen less than $10^4$ but more than $10^3$ times. The other colors are analogously defined.
As SVO increases along the x-axis, the individual frequencies Sxx and xxO decrease; (the colors match the gradient of the legend.) It seems that words that appear most frequently in specific relations appear less frequently in others. xVx decreases as both SVO and OVS increase. Thus, the individual terms and the triad frequencies are (negatively) correlated. Each plot is symmetric across the diagonal (not by color) because for each relation we have the default and flipped prompt.

Desiderata Testing our hypotheses requires collecting relations from the training data, creating a prompt for each relation, generating an image for each prompt, and then measuring how well each prompt and image aligns. Because our dataset is only for evaluation a large dataset is unnecessary. Moreover, because we want to have multiple ratings for each prompt-image pair, a smaller dataset is best. The primary difficulty in constructing such a dataset is finding relations with varying count statistics. These constraints guide some of the decisions we made when collecting our dataset.

Measuring Alignment We measure the performance of the text-to-image models by asking people how well the prompts and the generated images align. We use SurgeAI [https://www.surgehq.ai/] to run these experiments; <ref> in <ref> is a screenshot of the interface. The alignment scores were collected across a 5-point Likert scale from “Strongly Disagree” to “Strongly Agree.” In our plots and results, we standardize the five ratings from 0 to 1. For the sake of analysis, we consider an average alignment score above 0.75 as a successful generation. (This threshold is chosen because 0.75 and 1 correspond to affirmative responses “Agree” and “Strongly Agree,” as defined by the rating task.)

Dataset Collection and Measuring Frequency
Plots for Oxx and xxS are not shown, but, because of the structure of the dataset, they would be colored symmetrically across the diagonal from the default counterpart (Sxx and Oxx) (xxO and xxS).
We parse LAION [Schuhmann et al., 2022][We sample from the subset with an aesthetic score 4.5+ because this is what stablediffusion 2.1 was trained upon.], a dataset of paired text and image using Spacy for subject-verb-object relations. The parsing is noisy: Our search finds subject-verb-object relations in $10\%$ of (about 1.3 billion) instances.[This is admittedly a small fraction. We performed significant manual analysis in order to verify that our 10% sample is not unduly biased, and thus that our conclusions are reliable despite the small sample. See <ref> for a detailed discussion.] Many instances are fragments, e.g., “homes with wrap around porches…”, with no subject-verb-object relation. After we collect the counts of subject-verb-objects, we sample relations from different frequencies. Next, we manually create prompts for relations that are “drawable”. <ref> further details the initial collection process and sanity checks a number of assumptions that we outline in <ref>. <ref> shows the distribution of examples across the different frequencies defined above; Appendix <ref> reports alignment score statistics.

Formatting Prompts We format the prompts such that the subject is always the agent (the entity doing the action) and the object is always the patient (the entity on the receiving end of the action). Specifically, we format all relations into prompts using the structure:  A photograph of a {subject} {verb} a {object}. “A photograph of ” is preprended to each example because in pilot studies it narrowed the type of generation to be photographic, avoiding a lot of trivial errors. After adding the prefix, we edit each example to ensure it is grammatical. For instance, for mass nouns there is no “a” included: $\langle \textrm{girl, bring, water} \rangle$ yields “A photograph of a girl bringing water.” Though we could generate many reasonable prompt variations for each relation, we have a single prompt per relation (and we generate a single image per prompt). Future work could easily expand our dataset along both of these axes.

§ EXPERIMENTS

We find broad support for both hypotheses (<ref>), but the strength of the results depends on the relative size of the priors (SVO vs OVS). By partitioning the dataset,[In the appendix, <ref> visually shows how the dataset is split across the various experiments across <ref>, <ref>, <ref>. <ref> shows the average alignment scores.] we get a more complete picture of how the frequencies impact the generations.[Appendix <ref> reports the results for the full dataset.]

Summary of Takeaways of <ref>
For relations that have been observed during training, we find that higher SVO counts result in better alignment. For relations that are entirely unseen, we find that OVS has little effect on alignment. This is surprising, as we initially expected that higher OVS (i.e., a stronger prior for the competing relation) would make it harder for the model to generate the unseen relation–e.g., it should be harder to generate “horse riding astronaut” if you have seen “astronaut riding horse” often. When SVO is greater than OVS, the effects of both SVO and OVS strongly support the  and  hypotheses. However, for flipped examples, cases like   with the more infrequent ordering, the effects are weaker. That is, when a model is trying to generate a relation that goes against its training prior, we see less strong effects across the board. The increased difficulty (and lower overall alignment scores, Appendix <ref>) of such examples may explain the differences across partitions.
Lastly, individual terms Oxx and xxS also impact the results: That is, we find that words appear to have typical roles–they tend to be used either as agents or patients [Mahowald et al., 2022], and models show some resistance to generating them in their less-typical roles.

§.§  Increased SVO
Increases Alignment

First, we want to establish whether or not the increasing frequency of relations in the training data increases the alignment scores, e.g., the  hypothesis. The relations in this experiment are filtered so that the inverse relation is always unseen, e.g., OVS is 0, isolating the effect of SVO. For example, if $\langle \textrm{dog, chase, ball} \rangle$ is in this subset, then $\langle \textrm{ball, chase, dog} \rangle$ is unseen.

§.§.§ Results

The results, see <ref> and <ref>, support the  hypothesis. More frequent prompts are better aligned with their outputs. SVO has a $+0.31$ effect over a log scale meaning that for every order of magnitude more, say,   has been seen in the training data (holding all else equal) there is an increase of $0.31$ in alignment score. This is substantial, as recall that alignment ranges from 0 to 1.

2cSVO Isolated <ref>

Term     Effect     p-value

SVO             0.31       0.00

Sxx             0.04       0.27

xVx             0.12       0.01

xxO             0.07       0.24

Oxx            -0.18       0.00

xxS            -0.23       0.00

Regression Coefficients for Predicting Alignment (when OVS count is 0).

After $\textrm{SVO} > 10^2$ examples, a  majority of generations are successful (specifically, 18 of 26 prompts were successfully generated with average scores above 0.75.) Several individual terms also impact the results. The effect of Oxx is $-0.18$, meaning the more frequently that the object of relation takes the role of the subject in the training data, the worse the alignment. The same is analogously true for subjects (the effect of xxS is $-0.23$). These results don't discount the  hypothesis but do suggest that a number of overlapping training priors play a part in the generations, not just the frequency of OVS. The correlation between the frequency and the alignment is 0.421 ($p < 1e-7$).

legend style=at=(1.0,1.025),anchor=south east,

[no markers, thick, black, table/x=FrequencyLog] table [
y=create col/linear regression=x=FrequencyLog,y=Alignment,
] tikz/exp1scatter.dat;
[mark=none, thick, dashed, black] coordinates (0,0.75) (3.6 , 0.75);
(axis cs:-0.1,0.775) rectangle (0.95, 1) [orange];
(axis cs:-0.1,0.1) rectangle (0.4, 0.28) [blue];
(axis cs:1.65,0.1) rectangle (3.15, 0.32) [red];
[scatter, table/y=Alignment,
mark=o,only marks, blue] table tikz/exp1scatter.dat

Alignment improves with Frequency.  Each alignment score is the mean across 5 ratings. Top left samples in <ref>; Bot left samples in <ref>; Bot right samples in <ref>.

§.§.§ Qualitative Examples

Firstly, there are some counterexamples to the  hypothesis: the orange rectangle in <ref> highlights success cases at low-frequencies.

Secondly, the text-to-image model appears to fail in different ways at low versus high frequency. In a sample of low-frequency samples, highlighted by the blue rectangle in <ref> and shown in <ref>, either the subject or the object is generated but not both. Failed high-frequency, highlighted by the red rectangle in <ref> and shown in <ref> differs. The subjects and objects are generated, but the verb is not properly shown. While failing, the high-frequency samples are closer to being aligned with the prompt. They appear to express a more common relation/verb than the one requested by the prompt, again pointing towards a training prior.

Successful Low-Frequency Generations that Run Against the  Hypothesis. These relations (from the orange rectangle in <ref>) were present at most 10 times in the training data.

Failed Low-Frequency Generations that Support the  Hypothesis. These examples appeared at most three times in the training data, sampled from the blue rectangle in <ref>.
In five of six examples, only the subject or the object is present, not both.

Failed High-Frequency Generations that Run Against the  Hypothesis. These examples were all present in the training data with high frequency, sampled from the red rectangle in <ref>. In all six examples, the subject is present; In four of six, the object is present; In two of six, the model appropriately generates the relationship between the entities.

§.§  Increased OVS Decreases Alignment

We established the frequency of a relation, say , improves the generation performance, given that the flipped relation, i.e., , was unseen. Now, we look at the opposite case. We generate images for unseen triads, say . Here, the inverse triads () are seen to varying degrees. The question this section addresses is how does the text-to-image model behave when it is forced to fight a training prior? I.e., this section evaluates our  hypothesis.

2cOVS Isolated <ref>

Term     Effect     p-value

OVS      -0.06       0.38

Sxx           0.07     0.01

xVx            0.13     0.00

xxO             0.19     0.00

Oxx            0.10     0.01

xxS             -0.19     0.00

Regression Coefficients for Predicting Alignment (when SVO count is 0).

§.§.§ Results

Our results are shown in <ref> and <ref>. It is worth highlighting, first, that alignment overall is poor. That is, when the model generates an unseen relation, the generated image is rarely judged to reflect the prompt. Moreover, the frequency of the flipped relation (OVS) does not have a significant effect on alignment. There is a slight (insignificant) negative effect for OVS, but this drop appears to be better explained by xxS: The more common the subject specifically is seen in the object role, the worse the generation.  For example, that   is generated poorly seems more because a “ball” is commonly seen as the object (patient), not because the inverse triad as a whole () is common.

The individual terms xVx and xxO positively impact the alignment: The more often words are used in a given slot, the better the generation. This result is intuitive, especially
given that we filter the data such that SVO could not impact the alignment.

legend style=at=(1.0,1.025),anchor=south east,

[no markers, thick, black, table/x=InverseFrequencyLog] table [
y=create col/linear regression=x=InverseFrequencyLog,y=Alignment,
] tikz/exp2scatter.dat;
[mark=none, thick, dashed, black] coordinates (0,0.75) (2.35, 0.75);
(axis cs:1.05,0.62) rectangle (1.55, 0.69) [blue];
(axis cs:2.05,0.025) rectangle (2.4, 0.125) [red];
[scatter, table/y=Alignment,
mark=o,only marks, blue] table tikz/exp2scatter.dat;

Inverse Frequency Degrades Performance.  The results weakly support the  hypothesis that increased OVS decreases performance. We show the samples middle in <ref> and   right in <ref>.

Examples Receive Ambiguous Ratings Decrease as OVS Increases. Here we look at a set of examples highlighted in <ref> by the blue rectangle. The average scores are below  success (0.75), but some raters considered the generations successful.

Examples Receive Low Ratings when OVS is High. Here we look at a set of examples highlighted in <ref> by the red rectangle.

§.§ Interaction between  and  Hypotheses

The above results focused on cases when either SVO or OVS was zero–i.e., prompts for relations that are only attested in one direction. In this section, we focus our analysis on cases where the counts of SVO and OVS are both nonzero.

§.§.§ Results

The results, shown in <ref>, largely support both hypotheses, but there are  differences between default and flipped examples.[When pooling all the examples (not shown) – which introduces dependencies between the default and flipped examples – the results are effectively dampened (slightly lowering the high effects we see for the default examples.)] Again, default cases are prompts where (say, ) the relation is more common than the inverse relation (e.g., SVO $>$ OVS). Flipped is the opposite (say, ): OVS $>$ SVO.

For default examples, e.g., , effects for both SVO ($+0.25$) and OVS ($-0.40$) are large. Again, as in <ref>, Oxx ($-0.13$) and xxS ($-0.18$) also have a negative impact. For intuition, consider a specific example: . The more that “ball” has been seen in the subject role (and the more “dog” has been seen in the object role) in any context the worse the alignment.

For the flipped examples, e.g., , the effects are weaker. We expected a stronger result here because for these examples the model had to directly fight against strong priors. The overall lower alignment scores for this partition possibly impacted the results (see Appendix <ref>).
The most notable result is that SVO ($+0.14$)  still impacts the generations positively and that OVS ($-0.11$) is directionally aligned with the  hypothesis (though the result is not significant.)

§.§ Takeaways

The frequency effects were stronger for default examples, but there is evidence for both hypotheses. This was unexpected (especially for OVS) because flipped examples are the cases where the model has the strongest prior to fight against. Therefore, our original motivation about why the text-to-image model fails to generate some relations is not straightforwardly true: We did not find strong evidence that is poorly generated because is common. However, in sum, our results do suggest that training priors impact the alignments in a number of ways. In Appendix <ref>, we show that the regression coefficients across the full dataset match the individual patterns: Both primary hypotheses are supported, and the individual effects directionally align with the triads.

Individual effects: term frequency in a given role  impacts the alignment. Though we did not foresee this result, the logic runs in tandem with our primary hypotheses: Across the board, some of the alignment is explained by the triad frequencies–SVO and OVS–and some by the individual effects–Sxx, xVx, xxO, Oxx, and xxS. In <ref> and <ref>, this makes sense as the triads are prevented from impacting the results. However, these same strong individual effects are found for default examples in <ref>, suggesting that training priors on individual terms also impact generations. This mirrors some human results [Mahowald et al., 2022] which show that agent/patient roles can be predicted from the words alone (no ordering) in a large majority of naturally occurring sentences.

Semantic similarity between infrequent and frequent relations may further explain some of the results. There are cases where low-frequency (low SVO) prompts were well generated; See <ref>. A possible explanation is that if a relation is similar to another relation with high frequency in the training data, then that first relation may be better generated: For example, if  $\langle \textrm{dog, bite, bone} \rangle$ is  high frequency, then perhaps  $\langle \textrm{hound, chew, stick}\rangle$ is better generated.  More simply, how do the statistics of “dog” impact the generations of “hound”? We leave the exploration of this idea to future work.

2c|Default <ref>     2cFlipped <ref>

Term      Effect     p-value       Effect     p-value

SVO             0.25       0.00      0.14      0.04

OVS     -0.40     0.00      -0.11       0.11

Sxx            0.12       0.00     0.03     0.27

xVx             -0.03        0.34      -0.05      0.12

xxO             0.07      0.16     0.08      0.11

Oxx            -0.13       0.00      -0.06     0.01

xxS            -0.18      0.00      0.04     0.36

Regression Coefficients for Predicting Alignment from OVS and SVO (and other factors).

§ RELATED WORK

Prompt Challenge Sets
Often released with a new text-to-image model, there are a number of prompt challenge sets that look at the capabilities of text-to-image models. These challenge sets, like this work, look at how and where text-to-image models fall short over different types of prompts: For example, DrawBench [Saharia et al., 2022] [<https://docs.google.com/spreadsheets/d/1y7nAbmR4FREi6npB1u-Bo3GFdwdOPYJc617rBOxIRHY/edit#gid=0>] has prompts across a range of categories like “Conflicting” and “Colors”. See some examples in <ref>. However, in contrast to our work, these prompts are constructed largely by intuition rather than sourced by parsing the training data of the model. Another benchmark [Yu et al., 2022][<https://github.com/google-research/parti/blob/main/PartiPrompts.tsv>] test a broader set of phenomena beyond the scope of this work. Finally, WinoGround [Thrush et al., 2022], gets at the same ideas we are interested in: how and whether a vision+language model captures small differences in captions.

Improving Diffusion Models Establishing how good (or not) current text-to-image models are at generating different types of relations provides context for future (and ongoing) work. The success of RLHF (Reinforcement Learning with Human Feedback) in improving large language models [Ziegler et al., 2019, Stiennon et al., 2020, Ouyang et al., 2022, Gao et al., 2022, Bai et al., 2022] suggests that this type of training could also improve diffusion models. Already, we see some success: Hao et al., 2022 used reinforcement learning to suggest additional tags for prompts to improve the generations (like “high definition”, “8k”); Lee et al., 2023 use RLHF to improve stablediffusion (although the weights are not public.) There is a growing literature in this area that experiment with guiding the diffusion process [Yang et al., 2022, Feng et al., 2023, Po and Wetzstein, 2023, Wang et al., 2023, Liu et al., 2023, Chen et al., 2023].

§ DISCUSSION

The marked success of recent text-to-image models [Ramesh et al., 2022, Saharia et al., 2022] may undermine prior intuitions about the compositional capacity of neural network models. This raises questions about what type of mechanism supports such generalization–are neural models using something that resembles abstract combinatorial structure, or some form of clever mix-and-match strategy? In this work, we found that the  counts of the underlying relations in prompts significantly impact the generations of a stablediffusion 2.1. Although not definitive, such results suggest the latter strategy (mix-and-match). Despite the important caveat that our results only reflect the behavior of one model,

they still generate several interesting points worthy of discussion and further exploration.

Implications for compositionality of neural networks. The primary question motivating this work is: can neural networks reason compositionally, and if so, how? The results we present only truly weigh on the second part of this question–i.e., our results strongly suggest that models employ some form of content-specific, data-dependent representations rather than using abstract systematic processes. A similar observation was suggested by [Chaabouni et al., 2020]. This result is both surprising and obvious. It is obvious because, in the history of debates about neural networks and compositionality, it has always been asserted that they would use such idiosyncratic strategies to solve tasks. However, it is surprising because such methods appear to bring us much further in producing behavior that has all the signatures of compositionality than previously thought. That is–behavior of the type we currently see in image-, language-, and code-generation models were once thought to, in and of itself, be a diagnostic for abstract combinatorial structure. Increasingly, it seems to be the case that such structure is not a prerequisite for compositional (appearing) behavior [Chaabouni et al., 2020]. In our analysis, for example, there exist examples of low-frequency triads with high alignment scores. A very important question for future work is to revisit what, exactly, we seek when we seek models which are compositional: are we placing requirements on the behavior, the mechanism, or some combination of the two?

§ LIMITATIONS

We focus on a particular type of construction: subject, verb, object. Furthermore, we do not nest these types of prompts, e.g., “the dog chased the man holding the child.” We also do not focus on adjectives, like color. These types of ideas have been touched upon by previous work [Saharia et al., 2022]. Many sentences in the dataset are not typical sentences and end up being unattested to in our dataset counts. Instead, many are descriptive fragments like “Suki Round Coffee Table” or “columbia valley wine collections chateau the best cabernet sauvignon 20 the wine snob picks.” Our counts do not capture how often terms are used in these types of sentences.

Assumptions of our experimental design: (1) The relations pertain to the meaning of the sentences. (2) The sentences pertain to the content of the images. (3) The true statistics of relations in the dataset are proportional to the statistics we parsed.

We investigate each of these concerns in <ref>. Though the counts we provide are estimates, for a sample of triads, a deeper dive through the dataset does not yield significantly different counts. Importantly: Unseen triads remained unseen and seen triads had similar counts as originally found. Finally, there are a couple of notes that we consider out of scope and leave to follow-up work: First, our focus is on the training dataset of the diffusion model, not the text model. Second, our experiments assume that the counts of triads is the causal factor leading to deltas in alignment scores. Ultimately, the reason that certain triads are frequent and others are not will impact more than the frequency counts alone.

Our experimental design requires access to the training data, and then we search and parse the training data. First, this is a slow process. Second, this precludes testing commercially available, but closed, models like DALLE under our framework.

§ ETHICS STATEMENT

Our work aims to help us understand how text-to-image models work, and enable us to better understand the types of things models are currently capable of. Though the triads we find are generic, this paper makes clear that this type of text-to-image model will tend to reflect the relations seen in the wild.

[Bai et al., 2022]
Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain,
S. Fort, D. Ganguli, T. Henighan, et al.
Training a helpful and harmless assistant with reinforcement learning
from human feedback.
arXiv preprint arXiv:2204.05862, 2022.

[Bubeck et al., 2023]
S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar,
P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro,
and Y. Zhang.
Sparks of artificial general intelligence: Early experiments with
gpt-4, 2023.

[Chaabouni et al., 2020]
R. Chaabouni, E. Kharitonov, D. Bouchacourt, E. Dupoux, and M. Baroni.
Compositionality and generalization in emergent languages.
In Proceedings of the 58th Annual Meeting of the Association
for Computational Linguistics, pages 4427–4442, Online, July 2020.
Association for Computational Linguistics.
URL <https://aclanthology.org/2020.acl-main.407>.

[Chen et al., 2023]
M. Chen, I. Laina, and A. Vedaldi.
Training-free layout control with cross-attention guidance, 2023.

[Feng et al., 2023]
W. Feng, X. He, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E.
Wang, and W. Y. Wang.
Training-free structured diffusion guidance for compositional
text-to-image synthesis, 2023.

[Fodor, 1998]
J. A. Fodor.
Concepts: Where Cognitive Science Went Wrong.
Oxford University Press, 02 1998.
ISBN 9780198236368.
URL <https://doi.org/10.1093/0198236360.001.0001>.

[Gao et al., 2022]
L. Gao, J. Schulman, and J. Hilton.
Scaling laws for reward model overoptimization, 2022.
URL <https://arxiv.org/abs/2210.10760>.

[Hao et al., 2022]
Y. Hao, Z. Chi, L. Dong, and F. Wei.
Optimizing prompts for text-to-image generation.
arXiv preprint arXiv:2212.09611, 2022.

[Lake and Baroni, 2018]
B. Lake and M. Baroni.
Generalization without systematicity: On the compositional skills of
sequence-to-sequence recurrent networks.
In International conference on machine learning, pages
2873–2882. PMLR, 2018.

[Lee et al., 2023]
K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel,
M. Ghavamzadeh, and S. S. Gu.
Aligning text-to-image models using human feedback.
arXiv preprint arXiv:2302.12192, 2023.

[Liu et al., 2023]
N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum.
Compositional visual generation with composable diffusion models,

[Mahowald et al., 2022]
K. Mahowald, E. Diachek, E. Gibson, E. Fedorenko, and R. Futrell.
Experimentally measuring the redundancy of grammatical cues in
transitive clauses, 2022.

[Marcus, 2022]
G. Marcus.
Horse rides astronaut.
<https://garymarcus.substack.com/p/horse-rides-astronaut>, 2022.
Accessed: 2023-02-28.

[McKenzie et al., 2022]
I. McKenzie, A. Lyzhov, A. Parrish, A. Prabhu, A. Mueller, N. Kim, S. Bowman,
and E. Perez.
The inverse scaling prize, 2022a.
URL <https://github.com/inverse-scaling/prize>.

[McKenzie et al., 2022]
I. McKenzie, A. Lyzhov, A. Parrish, A. Prabhu, A. Mueller, N. Kim, S. Bowman,
and E. Perez.
Inverse scaling prize: First round winners, 2022b.
URL <https://irmckenzie.co.uk/round1>.

[McKenzie et al., 2023]
I. McKenzie, A. Lyzhov, A. Parrish, A. Prabhu, A. Mueller, N. Kim, S. Bowman,
and E. Perez.
Inverse scaling prize: Second round winners, 2023.
URL <https://irmckenzie.co.uk/round2>.

[OpenAI, 2023]
Gpt-4 technical report, 2023.

[Ouyang et al., 2022]
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang,
S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller,
M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe.
Training language models to follow instructions with human feedback,
URL <https://arxiv.org/abs/2203.02155>.

[Ouyang et al., 2022]
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang,
S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller,
M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe.
Training language models to follow instructions with human feedback,

[Po and Wetzstein, 2023]
R. Po and G. Wetzstein.
Compositional 3d scene generation using locally conditioned
diffusion, 2023.

[Ramesh et al., 2022]
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.
Hierarchical text-conditional image generation with clip latents.
arXiv preprint arXiv:2204.06125, 2022.

[Rombach et al., 2021]
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.
High-resolution image synthesis with latent diffusion models, 2021.

[Saharia et al., 2022]
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S.
Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, et al.
Photorealistic text-to-image diffusion models with deep language
arXiv preprint arXiv:2205.11487, 2022.

[Schuhmann et al., 2022]
C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti,
T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al.
Laion-5b: An open large-scale dataset for training next generation
image-text models.
arXiv preprint arXiv:2210.08402, 2022.

[Stiennon et al., 2020]
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford,
D. Amodei, and P. F. Christiano.
Learning to summarize with human feedback.
In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin,
editors, Advances in Neural Information Processing Systems, volume 33,
pages 3008–3021. Curran Associates, Inc., 2020.

[Thrush et al., 2022]
T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross.
Winoground: Probing vision and language models for visio-linguistic
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 5238–5248, 2022.

[Wang et al., 2023]
C. Wang, X. Yang, J. Huang, and K. Ferreira.
A diffusion-based method for multi-turn compositional image
generation, 2023.

[Yang et al., 2022]
L. Yang, Z. Huang, Y. Song, S. Hong, G. Li, W. Zhang, B. Cui, B. Ghanem, and
M.-H. Yang.
Diffusion-based scene graph to image generation with masked
contrastive pre-training, 2022.

[Yu et al., 2022]
J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku,
Y. Yang, B. K. Ayan, et al.
Scaling autoregressive models for content-rich text-to-image
arXiv preprint arXiv:2206.10789, 2022.

[Ziegler et al., 2019]
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei,
P. Christiano, and G. Irving.
Fine-tuning language models from human preferences, 2019.
URL <https://arxiv.org/abs/1909.08593>.

§ DATASET COLLECTION

§.§ Details

We parse sentences from LAION with aesthetic scores above 4.5 because this is the training data of stable diffusion 2.1. (Stablediffusion 2.1 also filtered out images that have a probablity of being unsafe above 0.1. In our process, we did not do this additional filtering.)  We access the dataset via HuggingFace at <https://huggingface.co/datasets/ChristophSchuhmann/improved_aesthetics_4.5plus> which has 1372 million rows. We successfully parse 134 million sentences with 50 million unique subject-verb-object triads (with 1,687,675 unique subjects, 669,858 unique verbs, and 1,553,343 unique objects.) As detailed below, we looked for common individual terms which appeared in many contexts, significantly reducing the pool of sentences.

<ref> visually show how the dataset is split across the various experiments across <ref>, <ref>, <ref>. <ref> shows the average alignment scores.

group style=
group size=2 by 2,
horizontal sep=5pt,
vertical sep=5pt,
x descriptions at=edge bottom,
y descriptions at=edge left,

point meta=explicit,
mark=square*, mark size=1pt,
only marks,

] table tikzdataset/dataset_svo_isolated.dat;
,SVO Isolated;

point meta=explicit,
mark=square*, mark size=1pt,
only marks,

] table tikzdataset/dataset_ovs_isolated.dat;

,OVS Isolated;

point meta=explicit,
mark=square*, mark size=1pt,
only marks,

] table tikzdataset/dataset_default.dat;

point meta=explicit,
mark=square*, mark size=1pt,
only marks,

] table tikzdataset/dataset_flipped.dat;

These plots show how the full dataset is split up the various view across <ref>, <ref>, <ref>.

SVO Iso.     OVS Iso.     Default     Flipped     All

count     755     835     1635     1635     4323

mean        0.56     0.44     0.50     0.46     0.49

std         0.34     0.31    0.33     0.33     0.33

min         0.00     0.00     0.00     0.00     0.00

25%         0.25     0.25     0.25     0.25     0.25

50%         0.75     0.25     0.50     0.25     0.5

75%         0.75     0.75     0.75     0.75     0.75

max         1.00     1.00      1.00     1.00     1.00

Dataset Partition Stats for Predicting Alignment from OVS and SVO (and other factors). There are 769 total unique prompts/relations. The difference in mean and median (50%) values between the first and third vs second and fourth highlights the relative difficulty of the second pair.

§.§ Evaluating Our Assumptions

Seen Triads are Seen
We randomly sampled the following seen tuples: <ref>. (1) We search the training dataset for sentences that contain all three words. (2) We re-parse these found sentences to get a parse-rate for each triad. (3) We manually label $2 * 4 * 25 = 200$ sentences (25 per triad) (parsed and not) looking for a parse-intent-possible. Here, we're informally (liberally) looking to find sentences where the searched terms could plausibly be the subject, verb and object. The sampled sentences: <https://docs.google.com/spreadsheets/d/1tViq4rYLXvjzA4vqhEFB-TqXvo4PzRt2ezAlEZrOvVI/edit?usp=sharing>. (4) We estimate the total number examples for each of the triads by summing the number of parsed (and not parsed) examples by the parse-intent-possible, respectively. We find that the estimates are in general larger than the number of sentences we originally found, but not beyond 2x. We do not compute estimates of but instead use the raw counts from the collected data.

In the sampled triads, the OVS counts were low. We found no examples of the flipped terms when the sentence was parsed (in part because they are much more rare.) The number of actual flipped examples amond the very few parsed examples we did find was extremely high (near perfect).

Triad     SVO     OVS     Estimated SVO     Estimated OVS

$\langle \textrm{girl, play, dog} \rangle$     1389     11     3228     11

$\langle \textrm{woman, walk, dog} \rangle$     1444     10     2652     8

$\langle \textrm{woman, carry, child} \rangle$     1536     2     1076     5

$\langle \textrm{woman, play, dog} \rangle$     3207     2     7444     3

Seen Triads are Seen.

Unseen Triads are Unseen
Like with the seen triads, we sampled the following unseen tuples (<ref>) and follow the same procedure as above. Somewhat surprisingly, we found no examples of the unseen tuples, although ultimately, because we are making use of sampling, it is quite possible that there are some examples. To be clear: There were still no cases of the unseen triads being parsed and no sampled examples of these triads.  Of note: The rate with which even one of the three words took a subject, object, or verb role was below 3% of the sentences. In comparison, for the seen triads, the individual terms matched on average for subjects 23%, for verbs 17%, and 24% for objects. (The sampled sentences are in the same document as above, <https://docs.google.com/spreadsheets/d/1tViq4rYLXvjzA4vqhEFB-TqXvo4PzRt2ezAlEZrOvVI/edit?usp=sharing>.)

Triad     SVO     OVS

$\langle \textrm{girl, move, dog} \rangle$     0     0

$\langle \textrm{girl, fall, family} \rangle$     0     0

$\langle \textrm{man, fall, family} \rangle$     0     0

$\langle \textrm{man, hit, kid} \rangle$     0     0

Unseen Triads are Unseen.

Triads Pertain to Sentence Meaning; Images and Sentences Align We take 100 parsed and 100 not-parsed sentences sampled from the training data: <https://docs.google.com/spreadsheets/d/1hIj-PvQsHM6WnRjRVhK1OKp3fOVp2wwjVjdKLRZwmIs/edit?usp=sharing>. (Note: The distribution of these sentences differs from the high-frequency triads and terms, but provides a baseline window into the parsing process.) We find:

* 5 of 100 not-parsed are false negatives (failed to be parsed when they could have been.)

* 36 of 100 parsed examples are false positives (should not have been parsed)

* 39 of 100 parsed examples had the clear potential for being successfully parsed (called possible parses below).

* 36 of the 39 possible parses, the parsed triad was the correct triad.

* 35 of the 39 possible parses, the parsed triad captured the meaning of the sentence.

* 34 of the 39 possible parses, the sentence captured the meaning of the photo.

In sum, the process is noisy, but we are finding reasonable success: For most examples that are possible parses, the parsed triad matches the meaning of the sentence and the paired photo matches the sentence. Detailed in the subsection below, we select individual terms that are common, with the aim of handling some of the noisy parses and finding some safety in the higher counts.

§.§ Protocol for SVO Triads

We will make the source code for this process public in future release of this paper after peer review.

* Parse dataset for SVO triads using spacy small english parser.

* Filter for alpha characters only.

* Get the number of instances each triad appears. For example, we build an index: S → Count[triads S appears in].

* Threshold such that each individual word (S, V, O) has been seen N = 5000 times.

* Find the intersection of S and O.

* Choose the top 100 for each S, V, O (using the intersection for S and O).

* Build index (triad → count)

* Determine attestation for each triad in a crossed dataset. Attestation is one of (FF, TT, FT, TF) where FF is neither direction of the triad has been seen, TT is both directions of the triad are seen, FT is the backwards direction has been seen, and TF is the forward direction has been seen.

* For each split of attestation (FF, TT, FT, TF) sample 15 random triads per verb.

* Manually label each triad for whether it is drawable in both directions (SVO, OVS). This step is subjective. This step could be automated using online workers. Because the drawable images are relatively sparse, we found it overall easier to do the labelling ourselves and avoid the overhead.

* Using this approach found a limited number of high-frequency tuples. We expanded the pool of candidate triads by bootstrapping off the successful cases and finding more examples of high-frequency tuples.

§ LABELLING TASK INTERFACE

<ref> is the interface we used to collect human judgements on the text/caption alignment. We provided examples to the SurgeAI workers are the following link:

Interface for Human Preferences.

§ ADDITIONAL RESULTS

§.§ Alternate Regressions

The results across all the partitions combined are in <ref>.
The results from <ref> are reproduced but with random controls in <ref> and averaged scores in <ref>.

Term      Effect     p-value

SVO             0.25       0.00

OVS     -0.20      0.00

Sxx            0.04       0.001

xVx             0.01        0.79

xxO              0.13      0.00

Oxx            -0.08       0.00

xxS            -0.08      0.001

Regression Coefficients for All Partitions Combined for Predicting Alignment from OVS and SVO (and other factors).

2c|Default     2cFlipped

Term      Effect     t-value       Effect     t-value

SVO             0.24       3.79      0.12      2.00

OVS     -0.41      -6.59      -0.14       -2.15

Sxx            0.11       4.93     0.03     1.16

xVx             -0.02        -0.86      -0.06      -2.06

xxO             0.06      1.32     0.08     1.63

Oxx            -0.13       -4.95      -0.06     -2.66

xxS            -0.19      -3.96      0.03     0.65

Regression Coefficients with Raters as Random Controls for Predicting Alignment from OVS and SVO (and other factors).

2c|Default     2cFlipped

Term      Effect     p-value       Effect     p-value

SVO             0.25      0.04      0.14      0.23

OVS            -0.10      0.00     -0.11       0.35

Sxx            0.12       0.01     0.03     0.52

xVx             -0.03       0.58     -0.05      0.36

xxO             0.08        0.42     0.08      0.35

Oxx            -0.06      0.01     -0.06     0.15

xxS             0.04      0.03     0.04      0.60

Regression Coefficients with Averaged Scores for Predicting Alignment from OVS and SVO (and other factors).

§.§ Controls

We re-run our experiments in <ref> but present to the raters images from the flipped prompt. For example, in Control Default, we present the raters the prompt from   and the image generated from . (In Control Flipped, we show the raters the prompt from   and the image generated from .)

Predicted Outcomes: A positive correlation between SVO and the swapped images suggests that the increased count of the SVO lead the flipped prompt to still generate the relation described by . A negative correlation between OVS and the swapped images suggests that the more frequent the OVS count is the better the generation of the flipped prompt leading to the flipped relation. Both of these are direct predictions of the  and  hypotheses we outline in <ref>.

Alternate Outcomes: If the reverse of the above held–if SVO had a negative correlation or OVS had a positive correlation–this would provide evidence counter to our hypotheses.

Results: The results largely match the predicted outcomes: <ref>. We see that the Control Default has a stronger effect on OVS whereas the Control Flipped has strong effects on both SVO and OVS. We don't take this evidence as bringing much more to bear on the problem than what we found in the original experiments.

2c|Control Default     2cControl Flipped

Term      Effect     p-value       Effect     p-value

SVO            0.04       0.53     0.20     0.00

OVS     -0.27     0.00      -0.14       0.02

Sxx            0.12       0.00     0.01     0.70

xVx             -0.08        0.02     -0.00     0.94

xxO             -0.05      0.27     0.14      0.00

Oxx            -0.01       0.62      -0.05     0.02

xxS            -0.13      0.01      -0.07     0.10

Regression Coefficients for Control Experiments. Significant ($p < 0.05$) effects over 0.1 are bolded.

§.§ Prior Work

We evaluate how well stablediffusion 2.1 generates images aligned with prompts from prior work. We report alignment scores on prior work in <ref>, <ref>, and <ref>. Unlike in the main body of the work, the scores  are from 0 to 4.

Our results on Appendix <ref> suggest that WinoGround is also a difficult dataset for text-to-image models. (Vision-and-language models were unable to perform much beyond chance upon it when set to disambiguate paired captions). <ref> shows some successes and failures. stablediffusion 2.1 was successful on the Noun-based prompts but failed on a descriptive prompt that required understanding the weight of different types of balls. (Presumably, the desired generations involved things like first a bowling ball and a beach ball, and then a bowling ball and a tennis ball.)

DrawBench example generations. Top left: “horse riding an astronaut”; Top right: “astronaut riding a horse”; Bot left: “blue colored dog”; Bot right: “red colored dog”. The first two images, belonging to the “Conflicting” category show how the model was biased towards the typical relation. The second two images show the model behaving more flexibly. These examples were generated by stablediffusion 2.1.

WinoGround example generations. Top: “There is a [fire] [truck]” vs “There is a [truck] [fire.]”; Bottom: “a larger ball is lighter and a smaller one is heavier” vs “a larger ball is heavier and a smaller one is lighter”. The top examples belong to the Noun category; the bottom examples belong to Adjective-Weight. These examples were generated by stablediffusion 2.1.

Category     Count      Median      0.25      0.75

Rare Words          7         0.0       0.00       1.50

Misspellings         10         0.5       0.00       2.75

Gary Marcus et al.          10         1.0       0.25       3.00

Positional         20         1.0       1.00       2.00

Reddit         38         1.0       1.00       3.00

Text         21         1.0       0.00       2.00

Counting         19         2.0       1.00       4.00

DALL-E         20         2.5       0.75       3.00

Colors         25         3.0       3.00       4.00

Conflicting         10         3.0       1.50       4.00

Descriptions         20         3.0       1.75       3.25

Both         52         1.0        1.0        3.0

Relation        466         1.0        1.0        3.0

Object        282         2.0        1.0        3.0

Alignment Scores on Prior Work. The alignment columns reports the median and quartiles. A fine-grained breakdown of categories in WinoGround can be found in <ref>. The alignment scores are scaled from 0 to 4.

Category     Count      Median      0.25      0.75

Noun Phrase, Adjective-Animate          2         0.0       0.00       0.00

Determiner-Possessive         12         2.0       2.00       2.00

Verb-Transitive Phrase, Verb-Intransitive, Preposition Phrase          2         2.0       2.00       2.00

Determiner-Numeral Phrase          2         2.0       2.00       2.00

Determiner-Numeral         36         2.0       2.00       3.75

Altered POS, Determiner-Numeral          2         2.0       2.00       2.00

Verb-Intransitive Phrase, Adverb-Animate          2         2.0       2.00       2.00

Noun, Adjective-Size          2         2.0       2.00       2.00

Noun, Preposition Phrase, Scope          2         2.0       2.00       2.00

Adverb-Spatial          2         2.0       2.00       2.00

Adjective-Weight          6         2.0       2.00       4.00

Preposition         42         2.0       2.00       4.00

Adjective-Age          4         2.0       2.00       3.50

Adjective-Speed Phrase, Verb-Intransitive          2         2.0       2.00       2.00

Verb-Intransitive, Noun          4         2.0       2.00       2.25

Adjective-Size         24         2.0       2.00       4.00

Adjective-Shape         12         2.0       2.00       2.50

Scope, Adjective-Manner          8         2.0       2.00       2.00

Verb-Transitive          8         2.0       2.00       4.00

Verb-Transitive Phrase          2         2.0       2.00       2.00

Adjective-Color (3-way swap)          2         2.0       2.00       2.00

Adjective-Animate         16         2.0       2.00       4.00

Preposition Phrase         12         2.0       2.00       4.00

Verb-Intransitive, Determiner-Numeral          2         2.0       2.00       2.00

Noun Phrase, Adjective-Color          2         2.5       2.25       2.75

Noun, Verb-Intransitive          2         2.5       2.25       2.75

Pronoun, Verb-Intransitive          2         2.5       2.25       2.75

Scope, Altered POS, Verb-Intransitive, Verb-Transitive          2         2.5       2.25       2.75

Noun Phrase, Determiner-Possessive          4         2.5       2.00       3.50

Negation, Noun Phrase, Preposition Phrase          2         2.5       2.25       2.75

Noun        214         3.0       2.00       4.00

Scope, Noun, Preposition          2         3.0       2.50       3.50

Scope, Conjunction Phrase          4         3.0       2.50       3.50

Scope, Conjunction          2         3.0       2.50       3.50

Adjective-Color         88         3.0       2.00       4.00

Scope, Adjective-Texture          2         3.0       3.00       3.00

Verb-Intransitive, Verb-Transitive Phrase          4         3.0       2.50       3.00

Sentence         10         3.0       2.00       3.50

Adjective-Spatial          4         3.0       2.50       3.50

Noun Phrase         44         3.0       2.00       4.00

Pronoun, Noun Phrase          2         3.0       3.00       3.00

Adjective-Temperature         12         3.0       2.00       4.00

Adjective-Temporal          2         3.0       3.00       3.00

Adverb-Animate          2         3.0       2.50       3.50

Verb-Intransitive, Adjective-Manner          2         3.0       2.50       3.50

Verb-Intransitive Phrase, Preposition          2         3.0       2.50       3.50

Verb-Intransitive Phrase          2         3.0       2.50       3.50

Scope, Preposition          4         3.0       2.50       3.50

Negation, Scope         22         3.0       2.00       4.00

Alignment Scores for WinoGround. The alignment columns reports the median and quartiles. The WinoGround results are the subset of groups with at least 10 examples. Plot continues on the next page.

Category     Count      Median      0.25      0.75

Preposition Phrase, Scope         14         3.0       2.00       4.00

Scope, Preposition Phrase, Adjective-Color          2         3.0       2.50       3.50

Scope, Relative Clause          2         3.5       2.75       4.25

Verb-Transitive, Noun          2         3.5       3.25       3.75

Scope          6         3.5       2.25       4.00

Adverb-Temporal          4         3.5       2.75       4.00

Adverb-Spatial Phrase          2         3.5       3.25       3.75

Adjective-Speed          4         3.5       2.75       4.25

Scope, Preposition, Verb-Intransitive          2         4.0       3.50       4.50

Relative Clause, Scope          2         4.0       4.00       4.00

Determiner-Numeral, Noun Phrase         10         4.0       4.00       5.00

Altered POS         36         4.0       2.00       4.00

Adjective-Texture         14         4.0       2.75       4.25

Adjective-Manner Phrase          2         4.0       4.00       4.00

Adjective-Manner         18         4.0       3.00       4.00

Scope, Preposition Phrase          4         4.0       3.00       4.00

Noun, Adjective-Color          2         4.0       4.00       4.00

Scope, Verb-Transitive          2         4.5       4.25       4.75

Noun Phrase, Determiner-Numeral          2         5.0       5.00       5.00

Adjective-Height          2         5.0       5.00       5.00