# Evaluating Numerical Reasoning in Text-to-Image Models

Ivana Kajić
Google DeepMind
Olivia Wiles
Google DeepMind
Isabela Albuquerque
Google DeepMind
Matthias Bauer
Google DeepMind
Su Wang
Google DeepMind
Jordi Pont-Tuset
Google DeepMind
Aida Nematzadeh
Google DeepMind
Corresponding author<EMAIL_ADDRESS>
###### Abstract

Text-to-image generative models are capable of producing high-quality images
that often faithfully depict concepts described using natural language. In
this work, we comprehensively evaluate a range of text-to-image models on
numerical reasoning tasks of varying difficulty, and show that even the most
advanced models have only rudimentary numerical skills. Specifically, their
ability to correctly generate an exact number of objects in an image is
limited to small numbers, it is highly dependent on the context the number
term appears in, and it deteriorates quickly with each successive number. We
also demonstrate that models have poor understanding of linguistic quantifiers
(such as “a few” or “as many as”), the concept of zero, and struggle with more
advanced concepts such as partial quantities and fractional representations.
We bundle prompts, generated images and human annotations into GeckoNum, a
novel benchmark for evaluation of numerical reasoning.

##  1 Introduction

Recent generative text-to-image models can produce images of impressive
quality in a variety of styles and following the text descriptions provided by
users [4, 20, 27]. However, they may still fail to accurately generate images
where the given descriptions contain numbers and quantities (such as “7
pistachios”, as shown in Figure 1) [18, 19]. While recent work has focused on
designing metrics, benchmarks and methods for evaluating specific capabilities
of text-to-image generative models, such as alignment [29, 13, 7],
compositionality [14], or spatial reasoning [11], there is no comprehensive
benchmark for evaluation of numerical reasoning.

We address this gap by proposing GeckoNum, a comprehensive and controlled
benchmark of text prompts aimed at evaluating different aspects of numerical
reasoning in text-to-image models. We formalize evaluation as three tasks:
exact number generation, approximate number generation, and reasoning about
partial quantities. For each task, we design various template types to control
for different variables such as sentence structure, the context in which the
number words occur in, and the number of attributes/entities in a prompt.
Table 1 shows examples of numerical tasks and the associated prompt types.

Using GeckoNum, we evaluate seven models chosen from three different model
families (DALL·E 3, Imagen and Muse): we generate images for these models and
collect human annotations to measure whether the images correctly match the
prompts with respect to numerical reasoning. Our benchmark consists of $1386$
text prompts, $52721$ generated images, and a total of $479570$ human
annotations that we release: https://github.com/google-
deepmind/geckonum_benchmark_t2i.

Overall, our results demonstrate that the recent generative text-to-image
models have rudimentary numerical reasoning skills, and are most accurate when
tested for generation of small exact quantities. We highlight the utility of
GeckoNum as an evaluation benchmark: it can discriminate between models, even
the powerful ones, such as Imagen-D [27] and DALL·E 3 [2], that are similar in
terms of high image quality. Finally, we demonstrate that our benchmark can be
used to drive progress in related research areas, such as the development of
automatic evaluation metrics and evaluation and improvement of pretrained
vision–language models on counting.

##  2 Related Work

Figure 1: Examples of images generated by selected models: DALL·E 3, Imagen-C
and Muse-B. Correctly generated images are marked with a check mark “✓”, and
incorrect with a cross mark “✗”.

Text-to-image benchmarking. Benchmarks used for evaluation of text-to-image
models typically consist of a set of text prompts that target a specific
capability. Some benchmarks, such as DALL·E 3 Eval [2] are more general and
aim to capture real world use cases, while others contain a variety of
challenges, such as prompts in DrawBench [21], PartiPrompts [32], and HEIM
[15]. Others are developed to interrogate models on more specific capabilities
such as alignment (e.g. TIFA [13], DSG [7] and Gecko [29]) or compositionality
(e.g. T2I-CompBench [14]).

Often, prompts in such benchmarks are curated from the data harvested from the
Web, and they may include numbers or other numerical concepts relevant to
evaluation of numerical reasoning. However, the number of such prompts in
existing benchmarks is often limited, and the complexity of prompts may not be
appropriate to evaluate models on numerical reasoning as a correct machine
interpretation of such prompts often requires a combination of several
different reasoning skills. For example, the understanding of the number “2”
in a simple prompt such as _“Two zebras in Cape Town”_ (from CountBench [18])
also requires an interpretation of a geographical location. For the correct
interpretation of _“Two dogs in a grassy field with one dog holding an orange
disc.”_ (from MS COCO [5]) the model would need to correctly generate several
objects (i.e. _“dog”, “disc”_), relations between them (_“holding”_), and
correctly bind attributes (_“orange”_) to objects. While the complexity of
such prompts may be representative of the complexity observed in natural
language, it may hinder our ability to accurately evaluate numerical reasoning
in text-to-image models.

The prompt set in GeckoNum is vastly more comprehensive and it systematically
covers various dimensions of evaluation that affect numerical reasoning, such
as different number ranges, types of nouns, different ways of representing
numbers, approximations based on linguistic quantifiers, and reasoning about
partial quantities that are missing in other datasets. We show that to
evaluate a specific capability thoroughly requires an extensive, comprehensive
dataset. Most similar to our work is [19], which investigates number
generation for text-to-image models. We go beyond this by evaluating a
comprehensive set of model families, considering other properties of numerical
reasoning such as estimation and conceptual quantitative reasoning, and a more
comprehensive breakdown of performance based on how commonplace an entity is.

Measuring counting in image-to-text models. While numerical reasoning has no
standardized benchmark for text-to-image generation, there are some widely
accepted benchmarks within the visual question answering domain. For example,
TallyQA [1] and CountBench [18] both measure the counting ability of question-
answering models. CountBench is a small dataset of only 540 images whereas
TallyQA contains approximately 20K images for evaluation. However, despite
TallyQA’s size, the distribution of numbers is highly skewed towards small
numbers (e.g. 1, 2); and the quality of the images and associated labels is
mixed.

##  3 Tasks to Examine Numerical Reasoning

Table 1: Twelve prompt types, example prompts and templates to probe different aspects of numerical reasoning in text-to-image models. Task | Prompt Type | Prompt Template / Example Prompts
---|---|---
Exact | numeric-simple |  | <num> <noun>
---
_One pistachio._
_7 kangaroos._
numeric-sentence |  | There <verb> <num> <noun> in this image.
---
_There is 1 fish._
_An image showing 1 fish._
2-additive |  | <num1> <noun1> and <num2> <noun2>.
---
_3 durians and three bonsais._
_Four axolotls and three cookies._
2-additive-color |  | <num1> <color> <noun1> and <num2> <color> <noun2>.
---
_Four red cats and four red mushrooms._
_One black cat and 1 black apple._
3-additive |  | <num1> <noun1>, <num2> <noun2> and <num3> <noun3>.
---
_2 parsnips, 3 coconuts and one seahorse._
_Two manatees, 2 burgers and 1 okra._
attribute-color |  | <num> <color> <noun>.
---
_One black koala._
_4 green bottles._
attribute-spatial |  | There <verb> <num1> <noun1> <rel> <num2> <noun2>.
---
_There are three cats above one manatee._
_There are three coconuts to the left of 5 burgers._
Approx. | approx-1-entity |  | An image of <noun>. There <verb> <quant> <loc>.
---
_An image of a shelf. There are many books on the shelf._
_An image of a watermelon. There are no seeds in the watermelon._
approx-2-entity |  |  An image with some <noun1> and <noun2>. There <verb> <quant> <noun1> <quant> <noun2>.
---
_An image with some tables and some pencils. There are fewer tables_ _than
pencils._
_An image with some bottles and some apples. There are as many bottles_ _as
apples._
Quantitative | fractional-simple |  | There <verb> <num> <noun> and <frac> of another <noun>.
---
_A pizza cut into 3 slices._
_There are three apples and quarter of another apple on the table._
fractional-complex |  | A <noun> is cut into <num> pieces. One piece is twice the size of the other.
---
_An image of a pencil where one half of it is red and the other half is blue._
_A banana is cut into two pieces. One piece is twice the size of the other._
part-whole |  | There <verb> <num> <noun>, but one <noun> is broken into two pieces.
---
_There are 2 forks on the table, but one fork is broken into two pieces._
_There are 5 pencils on the table, but one pencil is broken into two pieces._

We evaluate text-to-image models on different aspects of numerical reasoning
formulated as three numerical tasks and a set of text prompt types for each
task, spanning different levels of difficulty. Our working definition of
numerical reasoning draws inspiration from literature in numerical cognition,
and concerns both the ability to reason in abstract terms and the ability to
manipulate such abstractions across different contexts. Specifically, one
fundamental aspect of abstract reasoning is the ability to form a
representation of a set size independently of the identity of objects, known
as _The abstraction principle_ [9]. For example, understanding that “two”
refers to the same quantity in “Two apples” as it does in “Two letters” even
though apples and letters are different objects. We first define the tasks and
describe what aspect of numerical reasoning they intend to evaluate, followed
by description of prompts that aim to require those aspects of reasoning.
These prompts are generated using intentionally designed 12 templates where we
sweep over combinations of numbers and selected word nouns, resulting in
$1386$ prompts in total. The 12 prompt types, including prompts and example
templates used to generate the prompts are shown in Table 1, with further
details available in Table A2.

###  3.1 Task 1: Exact Number Generation

Task 1 examines a model’s ability to correctly generate an exact number of
objects. We define number generation as the correct visual depiction of an
entity specified in a prompt and its associated quantity (e.g.,“2 red cats.”
or “One mushroom and 3 koalas.”).

To probe how well models capture The abstraction principle, we vary the
context of numerical terms appear in. Then, we investigate whether and how
image generation accuracy changes depending on the prompt structure (e.g.,
attributes, and compositionality) enumerated below to give seven different
prompt types listed together with examples in rows “Exact” in Table 1.

Prompt structure: The simplest setting consists of phrases with an object and
a number (_numeric-simple_). We explore different prompt structure of
sentences (_numeric-sentence_), and also consider prompts which contain two or
three number-noun combinations (_2-additive_ and _3-additive_). We then
include color adjectives for both simple prompts with one entity (_attribute-
color)_ , and two entities with associated colors (_2-additive-color_).
Finally, we combine number terms with spatial relationships such as “There are
three cats above one manatee.” (_attribute-spatial_).

Exact number generation: For the simplest setting (_numeric-simple_), we
additionally examine the role of three other factors: (i) _Number magnitude_
for which we generate text prompts with numbers ranging from 1 to 10. (ii)
_Number representation_ where we consider prompts that represent digits with
both Arabic numerals (e.g., 1, 2, 3) or words (e.g., “one”, “two”, “three”).
(iii) _Noun frequency_ in English for which we select nouns (i.e., entities)
to cover both frequent and rare words.111We use the wordfreq Python library
[24] to determine word frequency. Our vocabulary in this task consists of
approx. $40$ words from four common categories: food, animals, nature, and
objects. See Appendix A for more details.

###  3.2 Task 2: Approximate Number Generation and Zero

In Task 2, we evaluate models on their ability to correctly depict entities
with quantities expressed in _approximate_ terms by means of linguistic
quantifiers (e.g., “many”, “a few”, or “more”). Such quantity terms are
ubiquitous in ordinary language use and may denote a range of numbers thus
carrying a more fuzzy interpretation of quantities. However, people tend to be
relatively consistent when using such terms, as shown in Figure A17. We expect
that models that correctly depict approximate quantities align with human
perception of these quantities. We also examine how well models interpret the
concept of zero, which we evaluate separately as it represents a challenging
milestone in number learning in children [28].

Prompt types. We design two prompt templates for this purpose:
_approx-1-entity_ and _approx-2-entity_ , as shown in Table 1. The former
tests for approximate generation of one entity in the prompt (e.g., a few
_candles_); the latter includes two entities with quantities expressed in
relation to each other (e.g., more _books_ than _cats_), as existing studies
indicate poor performance on this prompt type [16].

###  3.3 Task 3: Conceptual Quantitative Reasoning

In this task we evaluate models on prompts that require a conceptual
understanding of objects and their parts and are thus more challenging
compared to the previous tasks. Notions of “parts”, “proportion” and
“fractions” tend to be concepts that are harder to acquire for both children
and animals [22, 30]. For example, when a fork is broken into two parts,
three- to four-year-old children count each discrete physical object as a
separate fork [22].

Prompt types. Three prompt types are used in this task (the row “Quantitative”
in Table 1): the _fractional-simple_ category includes notions of a one whole,
and basic fractions such as ½ and ¼ (e.g., “A cake cut into quarters.”); the
_fractional-complex_ prompts include notion of fractions in relation to
another attribute such as color or size (e.g., “one piece is twice the size of
the other”). Finally, inspired by the fork experiment with children [22], we
include _part-whole_ prompt type where an object such as a fork, pencil or a
plate is split into pieces.

##  4 Human Annotations of Images

Figure 2: Three types of annotation templates used to collect data for the
evaluation of text-to-image models on three numerical reasoning tasks.

We evaluate text-to-image models based on annotations collected from study
participants who completed three different tasks, where each task corresponds
to one of the three numerical reasoning tasks discussed in Section 3. An
example template used for each annotation task is shown in Figure 2.

To evaluate models on Task 1, we ask participants to count the number of
objects in a generated image. Specifically, we pair each generated image with
an automatically generated question How many <obj> are in the image?, where
<obj> refers to the noun in the source prompt used to generate the image. We
ask one question for each noun in the prompt. Participants responded with a
number representing the count of objects entered in free-form text format;
where relevant, they were instructed to provide number ranges (e.g. 1–2, 2.5)
or “10+” labels (see Appendix C.1 for detailed annotation instructions).

For Task 2, an image is paired with 3–5 lines of text describing the image.
The number of lines depends on the original prompt, for _approx-1-entity_
there were three lines, and for _approx-2-entity_ there were 5 lines.
Participants were asked to select the line that describes the image the best.
The suggested lines were derived from the text prompt used to generate the
image (see Figure 2 for an example).

Finally, to evaluate models on Task 3, we generate a series of short questions
covering the words in the prompt words using automatic question generation
approach based on Davidsonian Scene Graph (DSG) [7]. The questions appeared
next to the image. DSG generates prompt-specific questions so that the answer
to all of them should be “yes” if the model accurately depicted contents of
the prompt. In analyses, we excluded all numerically irrelevant questions
(i.e. Is this an image?, Is there a table?, Are/is <obj> on the table?). There
are between two and six generated questions per each prompt.

We do not show the original prompt (used to generate a given image) to
participants in any of the tasks, as we want to collect unbiased estimates of
generated quantities. Example screenshots showing the web interface
participants were seeing for each annotation task are available in Figures
A8-A8. Twenty-five participants have been recruited through a crowd-sourcing
platform and provided informed consent to participate in the study. Our
institution’s independent ethical review committee reviewed and approved
details of the study design, including working conditions and minimum hourly
compensation of £15. Each image was annotated by five participants, and we
observe a high level of inter-annotator agreement as in over 96% of cases at
least 3/5 annotators provided the same rating (see Appendix C.2 for details).

Processing human annotations. Number annotations for object counts in Task 1
are first preprocessed by removing typos and standardizing the format (see
Appendix C.2 for details on text preprocessing). The mode of the five numbers
(i.e. the most frequent number) for each image–question pair is considered as
the numerical “label” (i.e. model response) used to calculate model accuracy.
When calculating accuracy, the ground-truth number is the original number in
the text prompt used to generate the image. If the two numbers match, the
accuracy is “1”, otherwise it is “0”.

In Task 2 each original prompt template is encoded as a number (e.g., “no <X>”
would be encoded as “0”, “many <X>” as “4” etc., see Appendix A.3) taken to be
the ground-truth number. Each participant response, a radio-button selection,
is also encoded as a number on the same scale. If the two numbers match, the
accuracy is “1”, otherwise it is “0”.

In Task 3 the “yes”/“no” responses given by participants are coded as binary
values 1/0 representing accuracy on the question. We aggregate all binary
responses and to obtain the average accuracy for a given prompt–image pair.

##  5 Evaluating Text-to-Image Models

Text-to-image models. We study seven different text-to-image models across
three different model families: DALL·E 3 [2], Imagen-based models [21, 27],
and Muse [4]. DALL·E 3 and Imagen models are pixel-based diffusion models,
while the Muse family uses a different generative approach based on predicting
masked image patches. Models within each family may differ in their size,
architecture and training data. For models that have been trained on internal
data sources we use letters of the alphabet to refer to earlier versions of
models (i.e. Imagen-A is an earlier version of Imagen-B, and Imagen-D is the
most recent version [27]). For each model we generate five images using five
different seeds.

Table 2: Per task accuracy and the standard error of the mean, with percentage point difference from the baseline in brackets. Accuracy of the best model on a task is highlighted in bold, and that of the best model within a family is underlined. |  Task 1 Exact Number Generation |  Task 2 Approximate Number Generation and Zero |  Task 3 Conceptual Quantitative Reasoning
---|---|---|---
DALL·E 3 | $\bm{45.2}\pm 0.5\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+35.2\%})$ | $\bm{48.7}\pm 2.7\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+24.1\%})$ | $\underline{48.8}\pm 1.1\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-1.2\%})$
Imagen-A | $26.3\pm 0.4\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+16.3\%})$ | $20.0\pm 2.2\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-4.6\%})$ | $41.1\pm 1.3\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-8.9\%})$
Imagen-B | $27.0\pm 0.4\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+17.0\%})$ | $24.6\pm 2.3\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+0.0\%})$ | $42.9\pm 1.4\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-7.1\%})$
Imagen-C | $\underline{34.8}\pm 0.4\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+24.9\%})$ | $27.0\pm 2.4\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+2.4\%})$ | $\bm{50.6}\pm 1.2\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+0.6\%})$
Imagen-D | $28.5\pm 0.4\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+18.5\%})$ | $\underline{28.7}\pm 2.4\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+4.0\%})$ | $43.8\pm 1.3\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-6.2\%})$
Muse-A | $34.8\pm 0.4\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+24.8\%})$ | $21.0\pm 2.2\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-3.6\%})$ | $45.1\pm 1.2\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-4.9\%})$
Muse-B | $\underline{39.8}\pm 0.5\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+29.8\%})$ | $\underline{24.6}\pm 2.3\,({\color[rgb]{0,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{0,.5,.5}+0.0\%})$ | $\underline{46.2}\pm 1.2\,({\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}-3.8\%})$
Random | $10.0$ | $24.6$ | $50.0$

Overall performance. Table 2 shows average performance of each model on all
tasks, with the baselines providing a reference on what would the accuracy be
assuming an annotator randomly responded to the question. Overall, looking at
how far model accuracies are from the baseline, we find that Task 1 is easiest
for all models, while Task 3 is the hardest as performance of all models on it
is close to or worse than that baseline. On Tasks 1 and 2, DALL·E 3
outperforms other models by a large margin, followed by Muse-B on Task 1 and
Imagen-D on Task 2. However, even for Task 1, performance of the best model is
still below 50%. We also find that more recent models do not necessarily
perform better than earlier models (e.g. Imagen-D $<$ Imagen-C on Tasks 1 and
3).

Per prompt template breakdown of accuracy for each task in Figure 4 reveals
further variability in difficulty, even within a single task.222For Task 1 we
only use prompts with the same number range (i.e. 1–4) and vocabulary (words
“apple”, “cat”, “koala”, “mushroom” and “bottle”) to dissociate the effect of
the prompt template from other factors. For example, within Task 1, when we
added another number term in the text prompt (c.f. _numeric-simple_ and
_2-additive_ rows in Figure 4) the accuracy on exact number generation for all
models dropped substantially. One of the most difficult prompt categories in
this task was _attribute-spatial_. In Task 3, the _fractional-simple_ prompts
appear easier than the _fractional-complex_ prompts.

Finally, we find that while DALL·E 3 is the overall best performing model,
there are certain cases where other models perform better. For example, Muse-B
appears stronger at additive prompts (e.g. _2-additive, 2-additive-color_ and
_3-additive_), while Imagen-C appears stronger at _numeric-simple_. These
models might indeed generate images that are better aligned with those prompt
types, or they might be more accurate at the specific words present in the
prompts under question. In the following sections, we further dissociate the
effect of such variables while focusing on the strongest models. Results for
all models, including the results of statistical significance tests used for
model comparisons that follow are included in Appendix B.333Unless otherwise
specified, we use the significance level $\alpha=.05$ when rejecting the null
hypothesis based on the p-value of the Chi-squared test.

Figure 3: Accuracy of models on each prompt type. Task 1 prompts contain only
small numbers 1–4 and a smaller subset of words.

Figure 4:  Top: The confusion matrices for A) DALL·E 3 and B) Imagen-A on
_numeric-simple_ prompts. Bottom: The effect of C) number representation and
D) word frequencies in Task 1. 95% bootstrap confidence intervals are shown.

###  5.1 Task 1: Exact Number Generation

Number magnitude. We find that number magnitude strongly affects model
performance in number generation, consistent with existing work in the
multimodal and generative models [19, 18]. Figure 4 shows normalized confusion
matrices for DALL·E 3 (A) and Imagen-A (B). The accuracy of the strongest
model (DALL·E 3) drops substantially with each successive number (18 p.p.
decrease for 1 $\rightarrow$ 2, 9 p.p. for 2 $\rightarrow$ 3, and 23 p.p. for
3 $\rightarrow$ 4). All models have a tendency to overestimate numbers (i.e.
they tend to depict a higher number of entities than what is specified in the
prompt), as shown by the non-zero entries over the “10+” label on the x-axis
in the figure. With the exception of DALL·E 3, we also see underestimation,
including instances where models fail entirely to generate the entity in the
prompt (non-zero entries over the “0” label).

Number representation (digits vs. words). Based on _numeric-simple_ prompts
and the small number range where models have higher accuracy (1–4), we find
that six out of seven models, with DALL·E 3 being the exception, are
significantly more accurate when numbers in text prompts are represented with
words as opposed to digits. Figure 4C) shows accuracy for a subset of best
performing models in each family for Task 1.

Noun frequency (frequent vs. rare words). Within _numeric-simple_ prompts
type, we find significant differences in accuracy between prompts that include
frequent nouns and prompts that include rare nouns for five out of seven
models, with DALL·E 3 and Imagen-D being the exceptions. A subset of models is
shown in Figure 4Ḋ). While this finding may indicate that these two models are
better at abstracting the notion of a number as opposed to memorizing
frequently occurring number-noun combinations, we note that this finding is
based on a smaller subset of words $(N=40)$ that may or may not generalize for
a greater sample size.

Prompt structure. Prompts types with more than one number-noun combination,
such as _2-additive_ and _3-additive_ , were on average more difficult
compared to prompts with a single noun and a number (i.e. _numeric-simple_)
(c.f. Table 4). Specifically, when numbers 2+ co-occurred with other numbers
in the prompt, the accuracy of all models on these numbers dropped
substantially compared to when those numbers appeared standalone, and the drop
was larger for _3-additive_ than for _2-additive_ (see Section B.1.3 in the
Appendix). With the exception of DALL·E 3, performance of all models was also
significantly lower when numbers appeared in the same context with spatial
relationships as in _attribute-spatial_ prompts, as those prompts were
significantly harder when compared to _2-additive_ prompts that did not
include spatial relationships. Adding color terms to numbers significantly
reduced accuracy only in some cases for some Imagen models and Muse-B. These
results indicate that prompt complexity, such as including additional
prepositions or numbers in the text, can dramatically impair the ability of
models to correctly generate images containing even a small numbers of
entities. In Section B.2 in the Appendix we discuss different types of model
failures in Task 1, and in Section C.3.1 we discuss the occasional cases of
disagreements in human annotations with several qualitative examples.

###  5.2 Task 2: Approximate Number Generation and Zero

Figure 5: Accuracy for _approx-1-entity_ and _approx-2-entity_ prompts.

When testing for approximate number generation, similarly to Task 1, we
observe that prompts with fewer entities (i.e. _approx-1-entity_) are on
average easier than prompts with two entities (i.e. _approx-2-entity_), as
seen in Figure 4. Figure 5 shows a further breakdown of accuracy per specific
prompt template. For _approx-1-entity_ (first 3 rows containing only “X”), all
seven models were the most accurate in generating “many” objects, and all
seven models were least accurate in generating images with zero objects (i.e.
“There is/are no X.”) For _approx-2-entity_ (last 3 rows, with both “X” and
“Y”), eight out of 10 models have highest accuracy when generating images from
prompts of the “more X than Y”-type and lowest accuracy when generating “as
many X as Y”. Our results highlight that understanding of linguistic
quantifiers, even as simple as “no” is still challenging for generative
models. Section C.3.2 in the Appendix shows qualitative examples including
images and annotations in Task 2.

###  5.3 Task 3: Conceptual Quantitative Reasoning

Task 3 is the hardest, as most models perform close to or below the baseline
(c.f. Table 2). Six out of seven models had the highest accuracy on
_fractional-simple_ prompts, followed by _part-whole_ and _fractional-complex_
, consistent with intuitions regarding the prompt difficulty. For DALL·E 3
differences in accuracies between the three prompt types were not significant.
The last column in Figure 1 shows images for a prompt in the _fractional-
simple_ category, where many models failed to correctly depict “one apple and
half of another apple”. Further examples, including images and annotations,
are shown in Table A15.

However, evaluating models on Task 3 prompts was more challenging due to the
lack of clear ground truth as in Task 1 (number in the prompt) and Task 2
(linguistic quantifier)—for some prompts, the DSG method generates only two
relevant questions out of which one asks about the presence of an object in
the image (e.g. _Are there apples?_) and thus answering “yes” yields the
baseline of 50%, without necessarily capturing the nuance between whole
apples, sliced apples or thirds/quarters of an apple. Further methodological
challenges related to evaluation of models on this task are discussed in
Appendix C.4.

##  6 Measuring What Counts: Challenges in Evaluation of Numerical Reasoning

Table 3:  The ability of auto-metrics to discriminate between correct and
incorrect image generations. ($\checkmark$ indicates $p<0.05$ using Mann-
Whitney U test)

| CLIP | TIFA | Gecko | DSG | VNLI
---|---|---|---|---|---
Imagen-C | = | ✓ | ✓ | ✓ | ✓
Muse-B | ✓ | = | = | = | ✓
DALL·E 3 | = | = | = | = | ✓

We explore the utility of GeckoNum for two related areas of research,
developing automatic evaluation metrics and evaluating vision–language models
(VLMs) with respect to numerical reasoning

Evaluating auto-eval metrics. Developing metrics that can reliably replace
human evaluation is an active area of research. In particular, a recent line
of work proposes _auto-eval metrics_ for measuring text-to-image alignment by
using pretrained language and/or VLMs [12, 13, 29, 7, 31]. Here, we evaluate
such auto-eval metrics on their ability to capture exact number generation by
examining how well they distinguish between correctly and incorrectly
generated images for prompts in Task 1.

We consider five different auto-eval metrics: CLIPscore [12], a metric based
on computing similarity between the text and an image, question-answering (QA)
based metrics such as TIFA [13], Gecko [29], DSG [7], as well as VNLI [31], a
metric fine-tuned to predict alignment between images and text. We divide
images generated by Imagen-C, Muse-B, and DALL·E 3 into two groups each based
on whether the image was generated correctly, and test whether the
distributions of scores in the two groups is identical. Results in Table 3
show that only VNLI was able to reliably distinguish between correct and
incorrect image generations, while all QA-based methods were able to do so
only some models.

We also investigated whether auto-metrics are able to rank models based on
accuracy. Following the approach in [29], we use a Wilcoxon signed-rank test
to test whether there is a significant difference in average scores between
models for the same metric, for all three combinations of model pairs. If such
a difference exists, models are ranked based on the average score. We find
that only Gecko, TIFA, and VNLI reliably rank models based on model accuracy
(i.e. DALL·E 3 > Muse-B > Imagen-C). Overall, VNLI seems to be the most
reliable auto-metric in evaluation of numerical reasoning.

##### Evaluating counting in vision-language models (VLMs).

While counting is known to be challenging for VLMs, only a few datasets and
benchmarks exist to train and evaluate VLMs on counting [1, 25]. We explore if
GeckoNum, which VLMs have never been explicitly trained on, can be used as an
evaluation task by curating a VQA benchmark. We evaluate the accuracy of
PaLIGemma [3], a state-of-the-art open-source VLM, on counting. Technical
details about the setup and more detailed results of experiments in this
section are available in Appendix D.

Comparing the base PaLIGemma model to a checkpoint that has been fine-tuned
for counting on TallyQA (train) [1], we find that the fine-tuned model
performs better ($73.3{\scriptstyle\pm 0.4}\%$) on GeckoNum when compared to
the base model ($68.4{\scriptstyle\pm 0.4}\%$). We observe that the base model
already performs well on small numbers (up to $4$), but fine-tuning on another
counting dataset (i.e., TallyQA) improves accuracy on higher counts (${}\geq
5$), a pattern we also observe with GeckoNum. We also briefly investigate the
utility of GeckoNum as synthetic training data: we fine-tune base PaLIGemma on
a mixture of TallyQA (train) and images from all Imagen models, and evaluate
it on TallyQA (test) and Muse-B images (see Appendix D for setup details). We
find that including Imagen data does not significantly change performance on
TallyQA (test) but vastly improves performance on Muse-B (by more than $20$
p.p. in some cases). This is still true when we test on a set of held-out
Muse-B classes that we removed from the Imagen fine-tuning data. These
preliminary results highlight that: 1) training VLMs with synthetic data can
improve results on other benchmarks; 2) there is great need for more public
high-quality datasets and benchmarks that evaluate counting and numerical
reasoning.

##  7 Discussion and Conclusion

Most work on evaluation of text-to-image models uses generic prompts—manually
written or harvested form the Web—as a starting point to generate images. For
example, recent research has focused on designing comprehensive prompt sets
that measure text-to-image alignment [29, 13, 7, 14, 26, 11].

In this work, we propose GeckoNum, a benchmark that specifically focuses on
measuring numerical reasoning capacity through three tasks: (i) exact, (ii)
approximate number generation, and (iii) conceptual reasoning about
quantities. We define various prompt types within these tasks to better
control for the effect of factors such as the context of the number in the
sentence and the sentence structure. We use GeckoNum to evaluate seven
different text-to-image models from three different model families (DALL·E 3,
Imagen, and Muse) by collecting human annotations for the generated images. We
find that DALL·E 3 has the highest overall accuracy on exact and approximate
number generation, while being the least impacted by some prompt manipulations
we investigate (e.g. numbers represented as words vs. digits, frequent vs rare
words); however, its performance on these tasks is still close to or under
50%. Depiction of approximate quantities and the concept of zero was also a
weak point in all models. For example, even DALL·E 3 would consistently fail
to correctly generate “a watermelon with no seeds” or “a cake with no
candles”. We also find that the task requiring reasoning about parts and
fractions was challenging for all models as their performance was close to the
baseline.

We used GeckoNum to highlight that many auto-eval metrics do not reliably
differentiate between correct and incorrect images on numerical prompts—among
metrics we tested, our preliminary experiments show that VNLI [31] is the only
auto-eval metric capable of such distinction. Our results may suggest that
current auto-eval metrics may require a specific adaptation for evaluation of
numerical reasoning, either through fine-tuning, as done for VNLI, or by
incorporating number-focused methods that require object detection [8]. We
also demonstrated that our benchmark could be used to study and potentially
improve the performance of pretrained vision–language models on counting.

Overall, our results indicate that the current text-to-image models do not
form abstract representations of numbers as their ability to reason about
numbers is rudimentary: it is limited to depiction of exact, small quantities
in images, and models frequently fail to generate approximate quantities and
zero. We highlight the effectiveness of numerical reasoning as an open
challenge in evaluation, since even in the large-scale training regime, due to
the large combinatorial space of numbers, it is challenging to mitigate
models’ weaknesses with more training data. Future modeling innovations—and
not only better training data—might be needed to improve model performance on
numerical reasoning. Our results show that most models are sensitive to the
number representation in a given prompt (words vs digits). This showcases
another interesting challenge involved in evaluating for numbers: they can
occur in various formats (e.g., dates, phone numbers, or expressions such as
“4k”) and representations. Also, effective ways of tokenizing numbers, again
due to the vast space of possible numbers, is an open problem in language
modeling [23] and an interesting direction for future research.

##### Acknowledgments.

We are grateful to our colleagues at Google DeepMind for advice and feedback
that helped improve this work. We thank Cristina Vasconcelos for a detailed
review of the manuscript, Jason Baldridge for support and useful feedback,
Richard Tucker and Aayush Upadhyay for their help and technical support
essential in realization of this work, and to Cyrus Rashtchian and Matko
Bošnjak for valuable feedback.

## References

  * Acharya et al. [2019] M. Acharya, K. Kafle, and C. Kanan.  Tallyqa: Answering complex counting questions.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 8076–8084, 2019.
  * Betker et al. [2023] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang, L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, et al.  Improving image generation with better captions.  _Computer Science. https://cdn. openai. com/papers/dall-e-3.pdf_ , 2(3):8, 2023.
  * Beyer* et al. [2024] L. Beyer*, A. Steiner*, A. Susano Pinto*, A. Kolesnikov*, X. Wang*, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, A. Gritsenko, X. Chen, S. Koppula, A. Grycner, M. Bauer, M. Bošnjak, F. Liu, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Papalampidi, O. Henaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai*.  PaliGemma: A versatile 3B VLM for transfer, 2024.  To appear.
  * Chang et al. [2023] H. Chang, H. Zhang, J. Barber, A. Maschinot, J. Lezama, L. Jiang, M.-H. Yang, K. P. Murphy, W. T. Freeman, M. Rubinstein, Y. Li, and D. Krishnan.  Muse: Text-to-image generation via masked generative transformers.  In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 4055–4075. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/chang23b.html.
  * Chen et al. [2015] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Chen et al. [2023] X. Chen, X. Wang, L. Beyer, A. Kolesnikov, J. Wu, P. Voigtlaender, B. Mustafa, S. Goodman, I. Alabdulmohsin, P. Padlewski, D. Salz, X. Xiong, D. Vlasic, F. Pavetic, K. Rong, T. Yu, D. Keysers, X. Zhai, and R. Soricut.  Pali-3 vision language models: Smaller, faster, stronger, 2023.
  * Cho et al. [2023a] J. Cho, Y. Hu, J. M. Baldridge, R. Garg, P. Anderson, R. Krishna, M. Bansal, J. Pont-Tuset, and S. Wang.  Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-image generation.  In _The Twelfth International Conference on Learning Representations_ , 2023a.
  * Cho et al. [2023b] J. Cho, A. Zala, and M. Bansal.  Visual programming for text-to-image generation and evaluation.  _arXiv preprint arXiv:2305.15328_ , 2023b.
  * Gelman and Gallistel [1986] R. Gelman and C. R. Gallistel.  _The child’s understanding of number_.  Harvard University Press, 1986.
  * Gemma Team et al. [2024] T. M. Gemma Team, C. Hardin, R. Dadashi, S. Bhupatiraju, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, and et al.  Gemma.  2024\.  doi: 10.34740/KAGGLE/M/3301.  URL https://www.kaggle.com/m/3301.
  * Gokhale et al. [2022] T. Gokhale, H. Palangi, B. Nushi, V. Vineet, E. Horvitz, E. Kamar, C. Baral, and Y. Yang.  Benchmarking spatial relationships in text-to-image generation.  _arXiv preprint arXiv:2212.10015_ , 2022.
  * Hessel et al. [2021] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi.  Clipscore: A reference-free evaluation metric for image captioning.  _arXiv preprint arXiv:2104.08718_ , 2021.
  * Hu et al. [2023] Y. Hu, B. Liu, J. Kasai, Y. Wang, M. Ostendorf, R. Krishna, and N. A. Smith.  Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 20406–20417, 2023.
  * Huang et al. [2024] K. Huang, K. Sun, E. Xie, Z. Li, and X. Liu.  T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation.  36, 2024.
  * Lee et al. [2024] T. Lee, M. Yasunaga, C. Meng, Y. Mai, J. S. Park, A. Gupta, Y. Zhang, D. Narayanan, H. Teufel, M. Bellagente, et al.  Holistic evaluation of text-to-image models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Leivada et al. [2023] E. Leivada, E. Murphy, and G. Marcus.  Dall· e 2 fails to reliably capture common syntactic processes.  _Social Sciences & Humanities Open_, 8(1):100648, 2023.
  * Lison and Tiedemann [2016] P. Lison and J. Tiedemann.  Opensubtitles2016: Extracting large parallel corpora from movie and tv subtitles.  2016\.
  * Paiss et al. [2023] R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel.  Teaching clip to count to ten.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3170–3180, 2023.
  * [19] S. Rane, A. Ku, J. M. Baldridge, I. Tenney, T. L. Griffiths, and B. Kim.  Can generative multimodal models count to ten?  In _ICLR 2024 Workshop on Reliable and Responsible Foundation Models_.
  * Rombach et al. [2022] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695, June 2022.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  35:36479–36494, 2022.
  * Shipley and Shepperson [1990] E. F. Shipley and B. Shepperson.  Countable entities: Developmental changes.  _Cognition_ , 34(2):109–136, 1990.  ISSN 0010-0277.  doi: https://doi.org/10.1016/0010-0277(90)90041-H.  URL https://www.sciencedirect.com/science/article/pii/001002779090041H.
  * Singh and Strouse [2024] A. K. Singh and D. Strouse.  Tokenization counts: the impact of tokenization on arithmetic in frontier llms.  _arXiv preprint arXiv:2402.14903_ , 2024.
  * Speer [2022] R. Speer.  rspeer/wordfreq: v3.0, Sept. 2022.  URL https://doi.org/10.5281/zenodo.7199437.
  * Trott et al. [2018] A. Trott, C. Xiong, and R. Socher.  Interpretable counting for visual question answering.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=S1J2ZyZ0Z.
  * Tuo et al. [2023] Y. Tuo, W. Xiang, J.-Y. He, Y. Geng, and X. Xie.  Anytext: Multilingual visual text generation and editing.  _arXiv preprint arXiv:2311.03054_ , 2023.
  * Vasconcelos et al. [2024] C. N. Vasconcelos, A. R. A. Waters, T. Walker, K. Xu, J. Yan, R. Qian, S. Luo, Z. Parekh, A. Bunner, H. Fei, et al.  Greedy growing enables high-resolution pixel-based diffusion models.  _arXiv preprint arXiv:2405.16759_ , 2024.
  * Wellman and Miller [1986] H. M. Wellman and K. F. Miller.  Thinking about nothing: Development of concepts of zero.  _British Journal of Developmental Psychology_ , 4(1):31–42, 1986.
  * Wiles et al. [2024] O. Wiles, C. Zhang, I. Albuquerque, I. Kajić, S. Wang, E. Bugliarello, Y. Onoe, C. Knutsen, C. Rashtchian, J. Pont-Tuset, et al.  Revisiting text-to-image evaluation with gecko: On metrics, prompts, and human ratings.  _arXiv preprint arXiv:2404.16820_ , 2024.
  * Woodruff and Premack [1981] G. Woodruff and D. Premack.  Primative mathematical concepts in the chimpanzee: proportionality and numerosity.  _Nature_ , 293(5833):568–570, 1981.
  * Yarom et al. [2024] M. Yarom, Y. Bitton, S. Changpinyo, R. Aharoni, J. Herzig, O. Lang, E. Ofek, and I. Szpektor.  What you see is what you read? improving text-image alignment evaluation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yu et al. [2022] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022\.
  * Zhai et al. [2023] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer.  Sigmoid loss for language image pre-training.  _International Conference on Computer Vision (ICCV)_ , 2023.

[appendices]

## Appendices

[appendices]l1

##  Appendix A Additional Information on Prompts

Table A1: Nouns used in text prompts in Task 1. Frequent words | Rare words
---|---
apple | lychee
egg | parsnip
burger | samosa
cookie | cinnamon stick
pizza | axolotl
dog | seahorse
fish | kangaroo
cat | koala
ant | manatee
fly | mushroom
tree | durian
leaf | bonsai
flower | pistachio
coconut | okra
olive | crib
table | paperclip
book | flute
bottle | trowel
spoon | corkscrew
pencil |
chocolate |
Figure A1: The distribution of frequencies for the nouns used in text prompts
in Task 1.

###  A.1 Words and Word Frequencies

Nouns used in text prompts in Task 1 are selected from the list of 40 words,
listed in Table A1. The words cover different semantic categories such as
everyday objects, food, nature and animals. Approximately the half of them
($N=21$) are labelled as frequent, and the rest ($N=19$) as non-frequent (i.e.
rare), as word frequency was a variable we investigated in Section 5.1. The
distribution of frequencies for those words is shown in Figure A1, and
frequencies are obtained using the Python wordfreq library [24] that gathers
frequency data based on multiple sources of data such as Wikipedia, OPUS
OpenSubtitles 2018 [17] and Google Books Ngram, among many others. For the
word ‘cinnamon stick‘ we used the word ‘cinnamon‘ which was present in the
database. In Tasks 2 and 3 we mostly used the same words as in Task 1, but in
Task 3 also added a few additional objects/foods such as “fork”, “a loaf of
bread” or “cake”, where we investigated how well text-to-image models can
generate parts of objects or fractions.

###  A.2 The Distribution of Prompt Types in the Benchmark

Table A2: The distribution of prompt types in the benchmark ($N=1386$) Task | Prompt Type | Number of prompts | Numbers
---|---|---|---
Task 1 | numeric-simple | 600 | 1, 2, 3, 4, 5, 6, 7, 8, 9, 10
attribute-color | 160 | 1, 2, 3, 4
numeric-sentence | 100 | 1, 2, 3, 4, 5
2-additive | 100 | 1, 2, 3, 4, 5
2-additive-color | 100 | 1, 2, 3, 4, 5, 6, 7, 8
3-additive | 100 | 1, 2, 3, 4, 5
attribute-spatial | 100 | 1, 2, 3, 4, 5
Task 2 | approx-1-entity | 24 | no, few, many
approx-2-entity | 45 | fewer, as many as, more
Task 3 | fractional-simple | 36 | 1, 2, 3, 1/2, 1/3, 1/4, 1/5
part-whole | 15 | 1/2
fractional-complex | 6 | 1/3 + 2/3, 1/2

###  A.3 The Encoding Scheme for Task 2 Answers

  * •

0: An image with no X and no Y. // There are no X.

  * •

1: An image with some X or some Y, but not with both X and Y.

  * •

2: There are fewer X than Y. // There are only a few X.

  * •

3: There are as many X as Y.

  * •

4: There are more X than Y. // There are many X.

##  Appendix B Additional and Detailed Experimental Results

In this section we show results for all models for experiments in Section 5.
In significance tests we use $\alpha=.05$ when rejecting the null hypothesis
based on the p-value of a test. When Chi-squared test is used to compare
accuracies between two groups representing two different conditions (i.e.
“digits” vs “numerals”, or “frequent” vs “rare” words), we build a contingency
table based on binary accuracy counts for prompt–image pairs in different
groups. In those cases, the null hypothesis is that there is no significant
difference in accuracy between the two groups. We report accuracies as
percentages in tables for easier comparison with other values in the
manuscript. As a control condition, i.e. to confirm that the variable we
investigate indeed explains the differences in results, we also conducted
permutation tests where we randomly permuted labels between the two groups
under comparison. Unless indicated differently in the text in those conditions
we found that all differences in accuracies were not significant.

###  B.1 Task 1: Exact Number Generation

####  B.1.1 Number magnitude

Figure A2: The confusion matrices for all models for _numeric-simple_ prompt
type. The generated number of objects in an image (as annotated by humans) is
on the x-axis, and the ground truth number in the original text prompt is
shown on the y-axis. Empty cells denote 0.

####  B.1.2 Number representation and word frequency

Figure A3: Number representation (A) and word frequency (B) accuracy for
_numeric-simple_ prompts and all models. Number range: 1–4. Lines on top of
bars show 95% bootstrapped confidence intervals.

.

Table A3: Chi-squared test results for comparison of accuracy between prompts
with digits ($Acc_{D}$) and prompts with word numerals ($Acc_{W}$). N: Number
of samples, $\chi^{2}$: Chi-squared statistic value, Diff?: whether the
difference between two accuracies is significant, $\checkmark$: significant
difference, =: insignificant difference.

| p-value | $Acc_{D}$ | $Acc_{W}$ | N | $\chi^{2}$ | Diff?
---|---|---|---|---|---|---
DALL·E 3 | 0.827 | 69.8 | 70.4 | 1600 | 0.05 | =
Imagen-A | 0.001 | 42.2 | 50.4 | 1592 | 10.60 | $\checkmark$
Imagen-B | 0.000 | 35.1 | 45.7 | 1595 | 17.95 | $\checkmark$
Imagen-C | 0.000 | 54.5 | 67.6 | 1592 | 27.94 | $\checkmark$
Imagen-D | 0.000 | 31.4 | 45.1 | 1596 | 31.09 | $\checkmark$
Muse-A | 0.018 | 47.4 | 53.4 | 1594 | 5.55 | $\checkmark$
Muse-B | 0.000 | 44.4 | 56.4 | 1600 | 22.56 | $\checkmark$

Table A4: Chi-squared test results for comparison of accuracy between prompts
with rare words ($Acc_{R}$) and prompts with frequent words ($Acc_{F}$). N:
Number of samples, $\chi^{2}$: Chi-squared statistic value, Diff?: whether the
difference between two accuracies is significant, $\checkmark$: significant
difference, =: insignificant difference.

| p-value | $Acc_{R}$ | $Acc_{F}$ | N | $\chi^{2}$ | Diff?
---|---|---|---|---|---|---
DALL·E 3 | 0.189 | 71.7 | 68.6 | 1600 | 1.73 | =
Imagen-A | 0.000 | 39.4 | 52.5 | 1592 | 26.94 | $\checkmark$
Imagen-B | 0.000 | 32.3 | 47.6 | 1595 | 38.04 | $\checkmark$
Imagen-C | 0.001 | 56.7 | 64.8 | 1592 | 10.69 | $\checkmark$
Imagen-D | 0.621 | 38.9 | 37.6 | 1596 | 0.24 | =
Muse-A | 0.000 | 41.2 | 58.6 | 1594 | 47.71 | $\checkmark$
Muse-B | 0.000 | 44.9 | 55.4 | 1600 | 17.14 | $\checkmark$

####  B.1.3 Prompt structure: Additive prompts

Table A5: Percentage point drop in accuracy for a specific number (1–3) present in different prompt types. | Dalle 3 | Imagen-C | Muse-B
---|---|---|---
Ground-truth number | 1 | 2 | 3 | 1 | 2 | 3 | 1 | 2 | 3
_numeric-simple_ $\rightarrow$_2-additive_ | -7.3 | -41.3 | -40.3 | -30.9 | -46.8 | -23.6 | -2.2 | -22.4 | -10.3
_2-additive_ $\rightarrow$_3-additive_ | -7.3 | -9.8 | -4.6 | 1.3 | -6.6 | -14.8 | -11.4 | -7.8 | -7.8
_numeric-simple_ $\rightarrow$_3-additive_ | -14.5 | -51.1 | -45.0 | -29.6 | -53.4 | -38.4 | -13.6 | -30.1 | -18.1

Figure A4: Accuracy per ground truth number in the prompt for different
additive prompt types. Lines on top of bars show 95% bootstrapped confidence
intervals.

.

Figure A4 shows drops in accuracy when a specific number (indicated on the
x-axis) occurs in different prompt types. In addition, Table A5 shows the
exact drops in accuracy for selected models for the numbers: “1”, “2” and “3”.
While for some models the drop in accuracy for the number “1” (c.f. the three
bars above the x-tick “1”) is much smaller (e.g. DALL·E 3) or non-significant
(Muse-B) compared to accuracy drops for some other models (e.g. Imagen-A,
Imagen-C), all models show a substantial and significant drop when “2” is
present in the prompt (c.f. the three bars above the x-tick “2”).

####  B.1.4 Prompt structure: Colors and spatial relationships

Figure A5: Accuracy in exact number generation when: A) color terms are added
to number terms in the prompt, B) color terms are added in 2-additive prompts,
and C) spatial relationships are introduced. Lines on top of bars show 95%
bootstrapped confidence intervals.

.

To investigate how the addition of color terms and spatial relationships
affects accuracy in number generation we ask the following three questions:

  1. 1.

Does adding color adjectives to numbers (such as “1 red koala”) affect the
number generation accuracy?

  2. 2.

Does adding color adjectives to numbers in 2-additive prompts (such as “1 red
koala and two green cats”) affect the number generation accuracy?

  3. 3.

Does the introduction of spatial relationships between nouns associated with
numbers affect the number generation accuracy?

To answer the first question, we split the data into the two groups: the first
group contains the subset of data for _numeric-simple_ prompts, and the second
group the subset of data for _attribute-color_ prompts. We only consider
prompts in both groups that contain the same numbers (1–4) and the same words
(“cat”, “apple”, “koala”, “bottle”, “mushroom”), to isolate the effect of
adding the color term as opposed to potential confounding factors. For example
a confounding factor might be the word identity, as a model might be more
accurate in generating correct images when the prompt contains the word “dog”,
and if this word exists only in the first prompt type and not in the second
then responses in the first prompt type will on average have higher accuracy
that may or may not depend on color terms. We split the data in a similar way
for the other two questions. To answer the second question we split the data
based on _2-additive_ and _2-additive-color_ prompts, and to answer the third
question we split the data into _2-additive_ and _attribute-spatial_ prompts.

We show the average accuracy between groups for each of these conditions in
Figure A5, and significance test results based on Chi-squared test in Tables
A8-A8. We see that most differences were not significant when the color term
was added, with the exception of Muse-A in Figure A5A) where the accuracy was
actually higher when the color term was added, and Imagen-B and Imagen-D in
Figure A5B). In both Figures A5A) and B) we highlight relatively large
confidence intervals for _numeric-simple_ and _2-additive_ groups. Those two
types of prompts were associated with fewer datapoints, compared to the groups
they were compared to (i.e. _attribute-color_ and _2-additive-color_ ,
respectively). While the average accuracies are much closer in Figure A5A), we
speculate that with more samples in Figure A5B) we may see more significant
differences for some other models. In contrast, when the prompt contained a
spatial relationship, we see significant differences for all models except for
DALL·E 3 (Figure A5C).

Table A6: Chi-squared test results for comparison of accuracy between prompts
with no color terms ($Acc_{NC}$, i.e. _numeric-simple_) and prompts with color
terms ($Acc_{C}$, i.e. _attribute-color_). N: Number of samples, $\chi^{2}$:
Chi-squared statistic value, Diff?: whether the difference between two
accuracies is significant, $\checkmark$: significant difference, =:
insignificant difference.

| p-value | $Acc_{NC}$ | $Acc_{C}$ | N | $\chi^{2}$ | Diff?
---|---|---|---|---|---|---
DALL·E 3 | 0.470 | 0.77 | 0.79 | 1800 | 0.52 | =
Imagen-A | 0.994 | 0.69 | 0.69 | 1785 | 0.00 | =
Imagen-B | 0.499 | 0.59 | 0.62 | 1800 | 0.46 | =
Imagen-C | 0.071 | 0.82 | 0.76 | 1800 | 3.25 | =
Imagen-D | 0.717 | 0.55 | 0.54 | 1795 | 0.13 | =
Muse-A | 0.020 | 0.64 | 0.72 | 1797 | 5.37 | $\checkmark$
Muse-B | 0.298 | 0.73 | 0.69 | 1800 | 1.08 | =

Table A7: Chi-squared test results for comparison of accuracy between
2-additive prompts with no color terms ($Acc_{2A}$) and 2-additive prompts
with color terms ($Acc_{2AC}$). N: Number of samples, $\chi^{2}$: Chi-squared
statistic value, Diff?: whether the difference between two accuracies is
significant, $\checkmark$: significant difference, =: insignificant
difference.

| p-value | $Acc_{2A}$ | $Acc_{2AC}$ | N | $\chi^{2}$ | Diff?
---|---|---|---|---|---|---
DALL·E 3 | 1.000 | 0.48 | 0.48 | 2015 | 0.00 | =
Imagen-A | 0.883 | 0.26 | 0.25 | 2010 | 0.02 | =
Imagen-B | 0.036 | 0.42 | 0.31 | 2006 | 4.42 | $\checkmark$
Imagen-C | 0.861 | 0.41 | 0.43 | 2006 | 0.03 | =
Imagen-D | 0.023 | 0.51 | 0.38 | 2014 | 5.16 | $\checkmark$
Muse-A | 0.212 | 0.53 | 0.45 | 2006 | 1.56 | =
Muse-B | 0.061 | 0.67 | 0.56 | 2015 | 3.50 | =

Table A8: Chi-squared test results for comparison of accuracy between prompts
without spatial relationships ($Acc_{NS}$, i.e. _2-additive_) and prompts with
spatial relationships ($Acc_{S}$ i.e. _attribute-spatial_). N: Number of
samples, $\chi^{2}$: Chi-squared statistic value, Diff?: whether the
difference between two accuracies is significant, $\checkmark$: significant
difference, =: insignificant difference.

| p-value | $Acc_{NS}$ | $Acc_{S}$ | N | $\chi^{2}$ | Diff?
---|---|---|---|---|---|---
DALL·E 3 | 0.095 | 0.40 | 0.37 | 2350 | 2.79 | =
Imagen-A | 0.000 | 0.19 | 0.07 | 2313 | 74.82 | $\checkmark$
Imagen-B | 0.000 | 0.23 | 0.12 | 2315 | 50.00 | $\checkmark$
Imagen-C | 0.000 | 0.29 | 0.16 | 2311 | 55.31 | $\checkmark$
Imagen-D | 0.000 | 0.31 | 0.17 | 2341 | 56.94 | $\checkmark$
Muse-A | 0.000 | 0.28 | 0.17 | 2315 | 37.80 | $\checkmark$
Muse-B | 0.000 | 0.37 | 0.27 | 2350 | 27.00 | $\checkmark$

###  B.2 Qualitative Analysis of Model Failures

Table A9: Examples of images where models failed to generate one object as specified in the prompt. See the text for details. | A | B | C | D | E
---|---|---|---|---|---
1 |  |  |  |  |
2 |  |  |  |  |

We manually inspected a subset of images for _numeric-simple_ prompts with the
ground truth “1” (i.e. the prompts of the format: 1 <noun>., or One <noun>.)
to understand different types of model failures in this most basic case. One
trivial kind of failure, especially observed with some earlier models such as
Imagen-A and Muse-A, is not generating the object at all (column A in Table
A9). For example, in A1) the model failed entirely to generate a _spoon_ ,
while in A2) it failed to generate a _corkscrew_. In other instances, the
generated object may have some characteristics of the object in the prompt,
such as a silhouette of a _pencil_ in B1), or the same material, such as metal
of a _corkscrew_ in B2).

In many other instances, we found that models generate more than one object,
such as two _okras_ in C1) or many _chocolates_ in C2). Among models we
evaluated, Imagen models were most susceptible to generating more than one
object when that object was a pistachio, an olive or a cookie. There were also
some more nuanced cases of model failures, subject to interpretation and
details of our methodology. For example, for the _cinnamon stick_ in D1),
annotators counted 2, 2, 0, 1.5, 1.5, while for the _manatee_ in D2) the
counts were: 1.5, 1.5, 1, 2, 1. In both of these examples, rounding up the
numbers and taking the most frequent response results in the number “2” as the
label for that image, although the raw annotations reveal potential ambiguity
in the image.

Finally, and as discussed in Appendix C.3, there were certain “edge cases” in
counting such as objects that are shown in the background (column E) or shown
only partially, where it is possible to count objects in different ways. For
example, in E1), it seems that all annotators assumed that the object in the
background is another _samosa_ , while in E2) they counted all _kangaroos_
that appeared in the image. A few cases similar to E2) were observed with
DALL·E 3, where the model would generate one dominant entity in the
foreground, and several other, much smaller ones in the background. While our
instructions specify that such objects should be counted separately (see
Appendix C.1 and Figure A9), but we found that sometimes they were not counted
separately or that this was difficult to do as the boundary between foreground
and background may not be apparent in all such images.

We also inspected failure cases in other prompt types. For _additive_ prompts
we found that DALL·E 3 and Muse-B often correctly depicted two or three
entities as specified in the prompt while failing to depict the correct number
of those entities, while some smaller models such as Muse-A and Imagen-A
frequently omitted one of the entities entirely. For _attribute-spatial_
prompts, which were generally one of the hardest prompts, we notice that
DALL·E 3 frequently generates a correct (or close to correct) number of
entities, but not necessarily in the required spatial arrangement, in contrast
to some models that fail to generate those objects at all.

###  B.3 Technical Resources

To generate images for all Imagen models and Muse-A, we used a cluster system
containing NVIDIA L4 Tensor Core GPUs. For Muse-B, and training experiments in
Section 6 we used internal hardware accelerators. We estimate that we used 140
GPU hours to generate all images.

##  Appendix C Collecting Human Annotations

###  C.1 Annotation Instructions

Figure A6: A screenshot of the user interface for one annotation example for
Annotation Task 1. The response to the question is given in a free-form text
format.

Figure A7: A screenshot of the user interface for one annotation example for
Annotation Task 2. The response to the question is a single radio-button
choice.

Figure A8: A screenshot of the user interface for one annotation example for
Annotation Task 3. The response to the question is a single radio-button
choice with yes/no options. In this example, there were three questions for
this image, and this was the third question.

All human annotators received training before completing the annotation tasks.
They were instructed on how to use the web interface, were given task-specific
instructions and were shown several examples of expected annotations with
explanations. After the training, we conducted a pilot study with a small
subset of images to confirm that task instructions were understood and
followed. All images shown to annotators were passed through safety filters,
and flagged images were manually inspected to ensure that no offensive,
harmful or otherwise problematic content would be present in images. We did
not collect any personally identifiable information.

##### Annotation Task 1 Instructions.

Annotation instructions, including a few examples of edge cases in counting
are shown in Figure A9. The goal of including a few examples of edge cases was
to establish a guideline on how to count objects in cases where several
interpretations are possible. For example, if it is possible that two shown
halves of an object come from the same object we instructed annotators to
count it as one object. If it is unlikely that they come from the same object,
we instructed them to count the two parts as two different objects. Here, our
aim was to provide guidance to reasonably standardize responses in such cases,
but we did not aim to exhaustively cover all possible cases as we also wanted
annotations to reflect individual differences in object perception in such
less precise or ambiguous cases. If there were more than 10 objects present in
the image, the instruction was to respond with “10+”. In this task, responses
were provided as free-form text. An example screenshot from this task is shown
in Figure A8.

Figure A9: Annotation instructions including examples of edge cases in
counting in Task 1.

##### Annotation Task 2 Instructions.

Figure A10: Annotation instructions including an example Task 2.

Instructions shown to annotators in the second task are shown in Figure A10.
Annotators were told to provide a response quickly while being accurate, to
encourage them to estimate quantities on this task, as opposed to counting
each object in the image separately. We wanted to get a response that best
reflects judgement of approximate quantities, as opposed to exact quantities
that were the subject of interest in Task 1. An example screenshot from this
task is shown in Figure A8.

##### Annotation Task 3 Instructions.

Figure A11: Annotation instructions including examples in Task 3.

Figure A11 shows instructions and an example annotation in Task 3, as well as
an additional example screenshot in Figure A8.

###  C.2 Quantitative Analysis

Figure A12: Distribution of numbers in ground-truth prompts used to generate
images, and numbers given as counts of objects in annotated images.

Table A10: Percentage of annotations where n out of 5 annotators provided the
same response in Annotation Task 1.

| 1/5 | 2/5 | 3/5 | 4/5 | 5/5
---|---|---|---|---|---
Dalle 3 | 0.0 | 2.3 | 6.5 | 8.5 | 82.6
Imagen-A | 0.1 | 2.3 | 6.6 | 8.7 | 82.3
Imagen-B | 0.1 | 2.7 | 7.4 | 9.4 | 80.4
Imagen-C | 0.1 | 2.8 | 8.0 | 10.4 | 78.7
Imagen-D | 0.1 | 3.8 | 7.6 | 9.5 | 79.1
Muse-A | 0.2 | 3.2 | 8.6 | 11.0 | 77.0
Muse-B | 0.1 | 3.3 | 10.1 | 12.0 | 74.5

In total, in Annotation Task 1 we collected $432035$ annotations, ranging from
approximately $58-59$K annotations per model, except for Imagen-C where we
collected $91$K annotations as five additional images per prompt were
generated for evaluation and fine-tuning experiments in Section 6 (for a total
of 10 seeds for Imagen-C). In Annotations Tasks 2 and 3, we collected $12045$
and $36100$ annotations, respectively. The analyses and statistics below will
be based on using five seeds for each model, although the data we release also
includes additional images and annotations. As well, released data will not
contain any images showing faces (approximately 122 images), due to potential
privacy concerns, but analyses below were done including annotations collected
for those images.

##### Response processing.

From raw responses we removed spaces and all non-numeric characters, except
for the letters “o”/“O” which we replaced with the number 0, “10+” entries
were replaced 11 to get a numerical format, and ranges (i.e. 2–4) were
converted into an integer by rounding up the average of the two values. When
two comma-separated numbers were given, we took only the first number. If
after these steps the response was an empty string, we removed it from the
dataset. In total, only 10 such responses were removed.

##### Response aggregation.

Each image and each question was shown to five annotators, collecting five
annotations in total per image–question pair. In Task 1, _additive_ prompts
had several questions, one for each entity in the image, while in Task 3 there
were several questions derived from the original text prompt. To get a single
number representing the number of objects in an image we used the mode of five
numbers in Tasks 1 and 2. In Task 2 those numbers were numerically encoded
single-choice radio button options. In total, after aggregating responses
there were $80450$ labels in Task 1, and $2409$ labels in Task 2. In Task 3,
we averaged all responses across all questions to get a score and refer to it
as “accuracy” for consistency in analyses. Figure A12 shows normalized
distributions of numbers in the original text prompts (green) and the
distribution of numbers in annotated data (blue).

##### Annotator agreement.

We observe high level of agreement among annotators. In Task 1, for each
image–question pair, all 5/5 raters gave the same response in 79% of cases, at
least 4/5 raters gave the same response in 89% of cases and at least 3/5 in
97%. In Task 2, the analogous percentages were as follows: 79%, 92% and 99%,
and in Task 3 where we only had binary responses all 5/5 raters agreed in 85%
of cases, while at least 4/5 agreed in 95%. Expressed as Krippendorff’s alpha
coefficient of inter-annotator agreement, $\alpha=0.865$ for Task1,
$\alpha=0.863$ for Task 2, and $\alpha=0.903$ for Task 3. A breakdown per
model annotator agreements in Task 1 is shown in Table A10.

Example # | Prompt | Image | Questions | Answers
---|---|---|---|---
|  |  |  |
1 | 10 coconuts. | How many coconuts are in the image? | 3.5, 3.5, 4, 4, 3.5
|  |  |  |
2 | 5 cinnamon sticks. | How many cinnamon sticks are in the image? | 0, 0, 5.5, 0, 6
|  |  |  |
3 | 5 trees. | How many trees are in the image? | 7.5, 7.5, 8, 8.5, 11
|  |  |  |
4 | 8 paperclips. | How many paperclips are in the image? | 8, 6.5, 9, 7, 6
|  |  |  |
5 | Two ants. | How many ants are in the image? | 1.5, 2, 1.5, 2, 2.5
|  |  |  |
6 | 9 koalas. | How many koalas are in the image? | 3.5, 4, 3.5, 2.5, 4
|  |  |  |
7 | 8 eggs. | How many eggs are in the image? | 4, 0, 4, 4, 0
|  |  | How many apples are in the image? | 3.5, 4, 3.5, 4, 3-4
8 | Four white apples. | How many white apples are in the image? | 3.5, 0, 0, 0, 3.5
|  |  | How many chocolates are in the image? | 6, 6, 6, 6, 6
| There are 4 chocolates | How many fish are in the image? | 1, 1, 1, 1, 1
9 | to the left of one fish. | How many chocolates are to the left of fish? | 4, 0, 0, 2, 0
|  | How many fish are to the right of chocolates? | 1, 1, 1, 1, 1
|  |  |  |
10 | 1 leaf. | How many leaves are in the image? | 7, 7, 7, 7, 1

Figure A13: Examples of images and annotations in Annotation Task 1. Only
images and questions were shown to annotators when completing the task.

Example # | Prompt | Image | Answer (Best line match)
---|---|---|---
|  |  | some trowels or some manatees, but not with both
|  | no trowels and no manatees
1 |  An image with some trowels and some manatees. There are as many trowels as manatees. | as many trowels as manatees
|  | some trowels or some manatees, but not with both
|  | more trowels than manatees
|  |  | more fish than seahorses
|  | fewer fish than seahorses
2 |  An image with some fish and some seahorses. There are more fish than seahorses. | as many fish as seahorses
|  | some fish or some seahorses, but not with both
|  | as many fish as seahorses
|  |  | some kangaroos or some cribs, but not with both
|  | some kangaroos or some cribs, but not with both
3 |  An image with some kangaroos and some cribs. There are as many kangaroos as cribs. | more kangaroos than cribs
|  | some kangaroos or some cribs, but not with both
|  | some kangaroos or some cribs, but not with both
|  |  | many flowers in the vase
|  | many flowers in the vase
4 |  An image of a vase. There are no flowers in the vase. | no flowers in the vase
|  | only a few flowers in the vase
|  | only a few flowers in the vase
|  |  | some eggs or some burgers, but not with both
|  | some eggs or some burgers, but not with both
5 |  An image with some eggs and some burgers. There are as many eggs as burgers. | some eggs or some burgers, but not with both
|  | as many eggs as burgers
|  | some eggs or some burgers, but not with both
|  |  | no seeds in the watermelon
|  | many seeds in the watermelon
6 |  An image of a watermelon. There are many seeds in the watermelon. | no seeds in the watermelon
|  | no seeds in the watermelon
|  | no seeds in the watermelon

Figure A14: Examples of images and annotations in Annotation Task 2. Only
images, the question “Which line describes the image the best?” and 3–5 lines
of text were shown to annotators when completing the task. The exact answers
annotators selected are shortened for brevity.

Example # | Prompt | Image | Questions and Answers
---|---|---|---
|  |  | Is there a plate? 1, 1, 1, 1, 1
|  | Is there one plate? 0, 0, 0, 1, 1
1 |  There is 1 plate on the table, but it is broken into two pieces. | Are there pieces of a plate? 1, 1, 1, 0, 0
|  | Are there two pieces of a plate? 0, 0, 0, 0, 0
|  | Is the plate broken? 1, 1, 1, 0, 0
|  |  | Are there pencils? 1, 1, 1, 1, 1
|  | Are there 2 pencils? 0, 0, 0, 0, 0
2 |  There are 2 pencils on the table, but one pencil is broken into two pieces. | Is one pencil broken? 0, 1, 1, 0, 0
|  | Is the pencil broken into 2 pieces? 0, 0, 0, 0, 0
|  |  |
|  | Is there a loaf of bread? 1, 1, 1, 1, 1
3 | A loaf of bread cut into thirds. | Is the loaf of bread cut into thirds? 0, 1, 0, 0, 1
|  |  | Is there an apple? 1, 1, 1, 1, 1
|  | Is the apple cut into pieces? 1, 1, 1, 1, 1
4 | An apple cut into 5 pieces. | Are there 5 apple pieces? 1, 1, 1, 1, 1
|  | Are the pieces inside the apple? 0, 1, 1, 1, 0
|  |  | Is there a pencil? 1, 1, 1, 1, 1
|  | Is there half of a pencil? 0, 0, 0, 0, 0
5 |  An image of a pencil where one half of it is red and the other half is blue. | Is the pencil in two halves? 0, 0, 0, 0, 0
|  | Is one half of the pencil red? 0, 0, 1, 1, 0
|  | Is the other half of the pencil blue? 0, 0, 1, 1, 0
|  |  | Is there a pencil? 1, 1, 1, 1, 1
|  | Is there half of a pencil? 0, 0, 0, 0, 0
6 |  An image of a pencil where one half of it is red and the other half is blue. | Is the pencil in two halves? 0, 0, 0, 0, 0
|  | Is one half of the pencil red? 1, 1, 1, 1, 1
|  | Is the other half of the pencil blue? 1, 1, 1, 1, 1

Figure A15: Examples of images and annotations in Annotation Task 3. Only
images and questions were shown to annotators when completing the task. The
answers are encoded as numbers: “1” denotes “yes”, and “0” denotes “no”.

###  C.3 Qualitative Analysis

While the overall level of disagreement among annotators was low, as shown in
Table A10, we manually inspected annotations to understand potential reasons
for annotator disagreement and ensure the overall quality of the annotations.
Examples for each task are shown in Figures A13–A15, and below we discuss
specific cases and the overall observed trends.

####  C.3.1 Task 1: Exact Number Generation

Figure A13 shows 10 selected examples of annotations with a varying level of
disagreement in Task 1. We selected these examples as a representative subset
of images, and discuss some common reasons why we believe there was some level
of disagreement related to the perception of objects or quantities in an
image. Some of these observations, such as that most people only count objects
when those objects are clearly identifiable in an image, also held for the
analysis of disagreement in Tasks 2 and 3.

First, we find that some disagreement may arise from a different
interpretation of instructions, such as two annotators counting one half of a
coconut as a whole, while three annotators counting it as a half, as per
instructions (Example 1 in Table A13). In other cases, where there is some
ambiguity regarding the identity of objects, it appears that annotators only
counted objects when they were certain that shown objects matched the identity
of objects mentioned in the question (Example 2 in the table). There were
images where it was difficult to count objects based on how salient such
objects were in an image, such as the trees shown in Example 3. We attempted
to account for some such cases in instructions (such as dividing objects
between those in the foreground and those in the background), but even then
the boundary between foreground and background was not always apparent in
generated images. Sometimes objects appeared distorted (Example 4), or there
were morphed objects in an image and it was not possible to distinguish
between individual entities (Examples 5 and 6).

There was a noticeable discrepancy in labels for images containing eggs, such
as in Example 7. Specifically, we noticed that all annotators counted eggs
when those appeared in shells or they were boiled, but if eggs were cracked
(in the form of an egg whites and yolk) or fried, they would frequently not be
counted as eggs. When we asked our annotators about the reasons for counting
eggs in this way, we were told that some annotators perceived fried eggs as a
dish, distinct from the concept of a “raw egg”.

We also found that some level of disagreement stems from individual
differences in color perception, such as “white apples” in Example 8. Finally,
we also found that there were rare instances of incorrect annotations. For
example in Example 9 we expect the correct answer to be “4” for the third
question “How many chocolates are to the left of the fish?”, but only one
response matched our expectation. As well, Example 10 shows another instance
where there is some ambiguity in interpretation, as the expected answer here
is “1” and only one response matched the expectation.

####  C.3.2 Task 2: Approximate Number Generation and Zero

Figure A14 shows six examples of selected images and corresponding annotations
where there was some level of disagreement among annotators. We highlight
these examples as representative of the types of disagreements we see in the
data. When objects in images are not clearly identifiable people tend to
indicate that one of the two objects is not present, such as Examples 1–3 in
the figure.

In Examples 1 and 2, the model generated objects that were morphed forms of
the two entities in the prompt. In Example 1, many of the objects appear to be
approximately shaped like trowels, with one in the lower left corner being
shaped like a manatee, but all objects have the color of a manatee. In this
example, 3/5 people indicated that they do not recognize manatees, trowels or
either. Somewhat similar trend is observed in Example 2, where generated
objects appear to be a morphed combination of the two entities mentioned in
the prompt–fish and seahorses. However, in this image more people were able to
recognize objects, though from the data we have it is unclear how ambiguous
objects were perceived.

In Example 3 we see another instance where the majority of annotators did not
perceive an object when that object was not clearly identifiable. Specifically
we see guard rails, but it is not clear that those rails belong to a crib, and
so only one annotator indicated that there are more kangaroos, while the
remaining four indicated that there are “no kangaroos or no cribs“.

In Example 4, we see some individual differences in perception of quantity,
where some people perceive the quantity of flowers as “many” while other as
“only a few”. In Example 5, and as in images in Task 1, we again observe that
4/5 annotators did not consider “fried eggs” as eggs, with the exception of
one annotator. Finally, Example 6 is interesting because it highlights that
people naturally expect to see watermelon seeds in a watermelon, even though
the objects in the watermelon do appear as some kind of seeds.

Figure A16: The proportion of “no”, “few” and “many” annotated labels
associated with a specific count of objects in an image.

Figure A17: The proportion of “no”, “few” and “many” labels from the ground
truth prompts associated with a specific count of objects in an image.

As in this task we used linguistic quantifiers to study approximate
quantities, there is some inherent subjectivity involved when interpreting the
quantities. Specifically, as seen in Example 4, some people perceive that
there are “many” flowers in the vase, while others perceive “only a few”
flowers.

To further understand these individual differences, we collected additional
annotations for images generated for _approx-1-entity_ prompts in Task 2.
These are the prompts that only contain one entity and the quantity associated
with the entity is either “no”, “[only a] few” or “many”. Additional
annotations were counts of objects in those images, as done in Task 1. Then,
we paired annotations collected in Task 2, namely labels such as “no”, “few”
and “many” extracted from the lines annotators selected when completing the
task with those counts, and plot the proportion of labels for each count in
Figure A17.444For each image we had five annotations, but we take the most
frequent annotation as the label.

Despite differences in individual interpretations of linguistic quantifiers,
we find that annotators were highly consistent in their interpretations of
such quantities–the overwhelming majority of them selected “no” when they
counted zero objects in an image (the dark blue bar over “0” in Fig. A17), and
similarly, when there are more than 10 objects annotators labelled those
images as “many”. The notion of “few” had a more distributed span ranging from
the numbers 1 to 10, but for counts in the range 3–7 100% of labels were
“few”.

We performed a similar analysis for labels extracted from the ground truth
prompts, instead of annotations, for a comparison on how generated quantities
in images correspond to labels. The results are shown in Figure A17. We indeed
see a trend where the bars for “no” are more skewed towards the left, meaning
more models generated fewer items when the word “no” was in the prompt, and
analogously we see more “many” items toward the right.

####  C.3.3 Task 3: Conceptual Quantitative Reasoning

Figure A15 shows six examples of images and corresponding annotations in Task
3. We generally see some ambiguity in responses when images are supposed to
depict broken objects. For instance, in Example 1, 2/5 annotators responded
that there is one plate in the image, while the other three annotators
responded that there are pieces of a plate and that the plate was broken.
Similarly in Example 2, 2/5 annotators perceive that a pencil is broken when
pencil lead was cracked.

Some disagreement was present when talking about parts, as seen in Examples
3–5. While the loaf of bread in Example 3 is cut into three parts, the parts
are not equal, but 2/5 annotators responded that the loaf of bread is cut into
thirds. Examples 4 and 5 highlight the difficulty of model evaluation in this
task. In Example 4, the question where annotators gave different responses
(“Are the pieces inside the apple?”) is neither meaningful nor relevant for
evaluation of numerical reasoning, and some level of disagreement in this case
might be expected.

In Example 5, we speculate that there were individual differences in
perception of “half of a certain color”—2/5 annotators did indicate that one
half of the pencil is red and the other half is blue, while 3/5 did not. We
manually inspected images and annotations for this specific prompt to see what
would an image look like if all annotators agreed. One such example is shown
in the last row of the table. We speculate that in answering this question,
the annotators might have considered the color of pencil lead as well as the
wood when answering the question.

Based on these and other similar examples, we conclude that disagreement in
this task may come from ambiguity present in images, questions or both. The
ambiguity in images may arise because objects and their parts are not clearly
shown, as observed in Tasks 1 and 2, but also because it may not be clear
whether objects are broken/sliced etc. The ambiguity in questions may arise
because questions are not sufficiently specific, or in some cases, not
meaningful.

###  C.4 Methodological Challenges in Model Evaluation

In Tasks 1 and 2 we could directly express model accuracy by comparing whether
the annotated count (e.g. “3”) or quantity (e.g. “many”) matched the
corresponding count or quantity in the ground truth prompt used to generate
the image. This was more difficult in Task 3, where prompts included objects
that were shown in parts or pieces, and these parts or pieces might be
associated with different properties, such as different sizes or colors. Text-
to-image models which generate images that better depict more of such
properties should be given a higher score, compared to those that match only
some. For example, let us consider the following text prompt: _A pizza cut
into 3 slices._

Given this prompt, if a model A produces an image of a pizza that is not
sliced at all, and a model B produces an image with a pizza cut into quarters,
we would expect that model B is better aligned with the prompt as it correctly
captured the notion of “cut” and “slice”. For this reason, we decided to ask
several questions that are grounded in the text prompt for each image in
evaluation of models on this task. To obtain such questions, we use a recent
automatic method based on the Davidsonian Scene Graph (DSG) [7]. This method
generates questions based on words in the prompt in such a way that the
expected answer to the question is “yes”. For the example prompt above, such
questions could be: _Is there a pizza?_ , _Is the pizza cut?_ and _Is the
pizza cut into 3 slices?_ Model B would score higher as it generated an image
that can be answered with “yes” in 2/3 questions, while for model A this would
only be the case for 1/3 questions.

While the majority of automatically generated questions are relevant in the
context of evaluation of numerical reasoning, there are some limitations to
this approach, as hinted in Section 5.3 and in Appendix C.3.3. First, some
questions might be less discriminative for more capable text-to-image models
that are able to clearly depict the object in the prompt. Specifically, it may
be the case that for all models the answer to the question _Is there a pizza?_
will be “yes”. Second, occasionally the answer to some questions might be
“no”, as seen in Examples 5 and 6 in Table A15 where questions imply that the
pencil might be split into halves (i.e. _Is there half of a pencil?_ and _Is
the pencil in two halves?_). We also noticed that some questions were not
informative or were confusing, such as _Are the pieces inside the apple?_
(Example 4 in Table A15) or _Is the quarter of a loaf of bread on the loaf of
bread?_ Based on manual inspection of the data, such questions accounted for a
small proportion of all questions and we expect that as automatic methods
advance their number will be even smaller in the future.

##  Appendix D VQA experiments

###  D.1 The VQA Setup

In our VQA experiments we use PaLIGemma [3]. PaLIGemma is an open 3B vision-
language model (VLM) inspired by PaLI-3 [6], built with open components, such
as the SigLIP [33] vision model and the Gemma-2B language model [10]. We use
the public checkpoint for an input resolution of $448\times 448$ pixels as
counting requires fine-grained visual information available at
https://www.kaggle.com/models/google/paligemma/jax/paligemma-3b-pt-448.

To turn GeckoNum into a VQA dataset/benchmark, we added questions of the form
How many <noun>s are in the image? to each data point. As the ground truth
answer we used the mode of the answers provided by the annotators, as we did
in the rest of the work.

For the evaluation benchmark we only included high-quality images on which at
least four out of the five annotators gave the same response; in this case,
the mode corresponds to the majority vote. This yields $59,582$ high-agreement
question-answer pairs from Imagen-A+B+C+D and Muse-A+B. For this set of
experiments only, we also generated additional images with Imagen-C, as it was
one of the best performing models in this family. In total, we had 10
different images for each prompt for Imagen-C. For the fine-tuning experiments
we included all images even when three or more annotators disagreed in their
responses. We only trained on Imagen-A+B+C+D data, which corresponds to
$53,053$ “noisy” question-answer pairs. In experiments where we split the
dataset into “frequent” and “rare” classes, we use the filter that filters
images based on the division of prompts as described in Appendix A.

To fine-tune PaLIGemma we found that fine-tuning for two epochs with a
learning-rate of $10^{-5}$, a weight-decay strength of $10^{-6}$, and a batch-
size of $256$ worked well across different training mixtures (otherwise we
used the default hyper parameter settings). Due to the exploratory nature of
these experiments, we did not re-tune hyper parameters for each experiment and
setting separately, but we used a single setting for all fine-tuning
experiments.

###  D.2 Additional Results

####  D.2.1 GeckoNum as a VQA benchmark

Table A11: Accuracy (in %) when evaluating PaLIGemma checkpoints on TallyQA (test) and on the GeckoNum VQA benchmark. PaLIGemma checkpoint | TallyQA test | GeckoNum
---|---|---
simple | complex |
Base | $35.4$ | $37.8$ | $68.4$
Fine-tuned (TallyQA train) | $84.9$ | $72.1$ | $73.3$

(a) Accuracy on the TallyQA test set

(b) Accuracy on GeckoNum (high agreement)

Figure A18: Accuracy of PaLIGemma (base model or fine-tuned on TallyQA
(train)) when evaluated on the TallyQA test set (left) and GeckoNum (right) by
ground truth number.

Here we first provide additional results on using GeckoNum as a VQA benchmark.
We evaluate the base model and a model fine-tuned on TallyQA (train) on both
TallyQA (test) as well as on GeckoNum, see Table A11 and Figure A18 for
results.

We observe that GeckoNum seems to be an easier benchmark as the base model is
already able to answer many questions correctly, especially for lower counts.
This is likely because while TallyQA includes complex and cluttered scenes,
GeckoNum images focus on one or very few object classes.

Moreover, we find that the fine-tuned model particularly improves on higher
counts ($\geq 5$), as shown in Figure A18. We observe the same trend for both
TallyQA (test) and GeckoNum; this indicates that fine-tuning on TallyQA
transfers to other benchmarks and datasets and that PaLIGemma fine-tuned has
indeed improved in counting. We note that TallyQA (test) only includes very
few examples with high numbers ($\geq 9$), leading to larger uncertainties in
the reported results. As we discuss in this work, all evaluated text-to-image
models already struggle with much smaller numbers ($\geq 4$), and similar
issues are observed with VLMs. We speculate that as models continue to improve
rapidly, benchmarks with high-quality images containing higher counts of
objects will become even more important in model development and evaluation.

####  D.2.2 Fine-tuning PaLIGemma on GeckoNum

We briefly investigate the utility of GeckoNum as synthetic training data. For
this we fine-tune PaLIGemma on different mixtures of TallyQA (train) and
Imagen A+B+C+D, and evaluate these on TallyQA (test) and the split of Muse-B
images, see Tables A12 and A13. We have $53.053$ Imagen-A+B+C+D images in
total, which we can use for fine-tuning, while the size of TallyQA (train) is
$249,318$. We perform two sets of experiments:

  * •

Experiment 1, in which we match GeckoNum data roughly 1:1 with TallyQA data to
explore the utility of the $53$k synthetic data points compared to $53$k real
data points.

  * •

Experiment 2, in which we train on all available TallyQA data and add GeckoNum
data to explore the benefits of enlarging an established training set with
synthetic data.

In Experiment 1 we control for fine-tuning set sizes and pair the $53$k Imagen
images with $53$k randomly sampled TallyQA train images (for a total of $106$k
examples) and compare them to fine-tuning on only $106$k randomly sampled
TallyQA train images. We also consider paring them with half the amount of
TallyQA images ($26$k) as well as only training on the Imagen images. To
control for the fact that Imagen and Muse share the same object classes, we
also consider a split of Imagen and Muse into two disjoint object sets of
“frequent” and “rare” as described above and only train on the “frequent”
subset and treating the “rare” subset as a held-out set.

The results in Table A12 can be summarized as:

  * •

Including GeckoNum Imagen images _in addition to_ TallyQA images neither
boosts nor deteriorates performance on TallyQA (test) significantly; there is
a slight improvement on TallyQA (test) “simple” and a slight deterioration on
TallyQA (test) “complex”.

  * •

However, _replacing_ some or all of the TallyQA (train) images with Imagen
images does deteriorate performance on TallyQA (test) by up to a few
percentage points.

  * •

Including Imagen images always markedly improves performance on Muse-B (by
more than $25$ p.p.). This is still true when only training on “frequent”
classes while evaluating only on “rare” classes, though the improvement is
slightly smaller in this case (by about $20$ p.p.). There is clear
generalization from frequent to rare classes.

  * •

Fine-tuning on Imagen only already leads to strong performance on TallyQA
(test) “simple” (about $70\%$ accuracy compared to $35\%$ for the base model
and $82\%$ when training on TallyQA) but not “complex” (only an increase to
$44.9\%$ from $37.8\%$ compared to $67.8\%$ when training on TallyQA). The
latter makes intuitive sense as the GeckoNum VQA questions resemble “simple”
TallyQA questions (How many <object> are there in the image?) but not
“complex” ones (How many <object>s have <property>?, e.g. “How many giraffes
are lying down?”).

Thus, as noted in Section 6, adding Imagen images to the training mix does not
hurt performance on TallyQA (test), but it allows us to achieve much better
results on other data splits not covered by the TallyQA training data as well
(the Muse-B split in this case). We also found that during fine-tuning on
TallyQA (train) the model slightly overfits to the TallyQA dataset in that the
best TallyQA (test) performance is achieved at the end of fine-tuning while
the best performance on Muse-B is achieved partway through fine-tuning and
performance reduces slightly as fine-tuning continues.

In Experiment 2, we use the full TallyQA (train) set and add the $53$k Imagen
images in addition, and compare this mixture to slight variations where we
account for the number of training examples (by removing $53$k examples from
TallyQA) and the number of training steps (by training on TallyQA for longer).
The results are shown in Table A13. We replicate previous results and observe
the same trends; the changes on TallyQA (test) performance are very small
($<1\%$ in the worst case but usually much smaller) when including Imagen
images while performance on Muse-B is drastically improved even in the held-
out case.

Table A12: Accuracy of PaLIGemma fine-tuned on different data mixtures (at resolution $448\times 448$ pixels). Here, we only train on a subset of TallyQA (train), which in total has approximately 250k examples. Imagen here means Imagen A+B+C+D. We provide the approximate number of unique data points in brackets. Fine-tuning data | TallyQA test | Muse-B
---|---|---
simple | complex | all |  freq. | rare
TallyQA (106k) | $83.1$ | $69.4$ | $70.3$ | $76.5$ | $63.2$
TallyQA (53k) | $82.0$ | $67.8$ | $68.3$ | $75.0$ | $60.5$
TallyQA (53k) + Imagen (freq & rare, 53k) | $82.4$ | $67.8$ | $89.4$ | $90.0$ | $88.8$
TallyQA (53k) + Imagen (only freq, 28k) | $82.2$ | $68.3$ | $86.0$ | $89.5$ | $82.0$
TallyQA (26k) | $81.1$ | $66.5$ | $68.7$ | $75.2$ | $61.3$
TallyQA (26k) + Imagen (freq & rare, 53k) | $81.5$ | $66.4$ | $89.6$ | $90.0$ | $89.2$
Imagen (freq & rare, 53k) | $70.7$ | $44.9$ | $90.0$ | $90.1$ | $89.7$
Imagen (only freq, 28k) | $70.1$ | $44.7$ | $85.0$ | $88.6$ | $80.7$
Base model (no fine-tuning) | $35.4$ | $37.8$ | $67.0$ | $71.0$ | $62.4$
Table A13: Accuracy (in $\%$) of PaLIGemma fine-tuned on different data mixtures (at resolution $448\times 448$ pixels). Here, we train on all TallyQA (train) images. Imagen here means Imagen-A+B+C+D and “all” refers to freq. and rareuent objects. Fine-tuning data | data | TallyQA test | Muse-B
---|---|---|---
points | simple | complex | all | only freq. | rare
TallyQA | 250k | $84.7$ | $72.8$ | $71.1$ | $76.6$ | $64.8$
TallyQA (only 200k) + Imagen (all) | 250k | $84.7$ | $71.9$ | $90.6$ | $90.9$ | $90.3$
TallyQA + Imagen (all) | 300k | $84.9$ | $72.7$ | $90.5$ | $91.2$ | $89.8$
TallyQA + Imagen (only freq) | 275k | $84.9$ | $72.5$ | $87.3$ | $90.4$ | $83.6$
TallyQA (longer training) | 250k | $84.7$ | $72.4$ | $69.8$ | $76.9$ | $61.7$
Base model (no fine-tuning) | – | $35.4$ | $37.8$ | $67.0$ | $71.0$ | $62.4$