#  Cultural and Linguistic Diversity Improves Visual Representations

Andre Ye†, Sebastin Santy†, Jena D. Hwang‡, Amy X. Zhang†, Ranjay Krishna†
†University of Washington, ‡Allen Institute for Artificial Intelligence
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Computer vision often treats perception as objective, and this assumption gets
reflected in the way that datasets are collected and models are trained. For
instance, image descriptions in different languages are typically assumed to
be translations of the same semantic content. However, work in cross-cultural
psychology and linguistics has shown that individuals differ in their visual
perception depending on their cultural background and the language they speak.
In this paper, we demonstrate significant differences in semantic content
across languages in both dataset and model-produced captions. When data is
multilingual as opposed to monolingual, captions have higher semantic coverage
on average, as measured by scene graph, embedding, and linguistic complexity.
For example, multilingual captions have on average 21.8$\%$ more objects,
24.5$\%$ more relations, and 27.1$\%$ more attributes than a set of
monolingual captions. Moreover, models trained on content from different
languages perform best against test data from those languages, while those
trained on multilingual content perform consistently well across all
evaluation data compositions. Our research provides implications for how
diverse modes of perception can improve image understanding.

##  1 Introduction

There’s an old adage that goes, “a picture is worth a thousand words.” These
“thousands words” represent the myriad ways different people can interpret an
image. Differing interpretations result from a diverse set of informational
cues: emotional cues (Gallup et al., 2014), social cues (Watson & Platt, 2012;
Gallup et al., 2012), visual elements (e.g., optical blur; (Sprague et al.,
2016)) and, of course, factual details. People also incorporate prior
experiences, communicative intent (Joo et al., 2014; Park et al., 2019), and
contextual information (Yorzinski & Platt, 2014). Moreover, previous work
suggests that the way someone perceives and describes the world is also
influenced by their culture (Oyserman & Lee, 2008; Masuda et al., 2008;
Nisbett & Masuda, 2003) as well as the language they speak (Langacker, 1993;
Boroditsky et al., 2003). All of this indicates that different people can look
at the same image and yet focus on completely different aspects.

Computer vision models are currently not trained in a way that reflects or
aligns with this human condition (Stretcu et al., 2023). Today, the
predominant paradigm for training vision models relies on large-scale image-
description pairs mined from the internet (Radford et al., 2021; Li et al.,
2022). These pairs are scraped without attention to the underlying humans’
visual preferences or biases (Röttger et al., 2021). Even models finetuned for
applications like image captioning rely on human annotated datasets curated
without explicitly considering how annotators perceive scenes (Berg et al.,
2012; Hutchinson et al., 2022).

Figure 1: A Japanese speaker and an English speaker in our human evaluation
caption the images differently based on several factors such as linguistic
constraints and background context. Red and blue highlights in text and image
indicate content that is uniquely captured by each speaker.

The implicit assumption made in this paradigm is that perception is objective.
This suggests that regardless of who describes an image, the semantic concepts
covered in the description should highlight roughly the same salient parts of
the image. However, this stands against decades of research in cross-cultural
psychology (Hall, 1976). Human studies find evidence that perception varies
among individuals, especially between cultures (Nisbett & Masuda, 2003). These
factors alter the way individuals attend to different parts of an image and
consequently which components they mention when talking about what they see.
Since culture is difficult to quantify, these studies frequently use language
as a proxy. Many cultural artifacts are also emergent in language; for
example, Japanese has several more levels of formality (e.g., honorifics
(Harada, 1976)), Mandarin infuses Confucian values such as “tranquility” not
present in other languages (Wen et al., 2023), and German’s complex
morphosyntactic system provides events with nuanced understanding of spatial
relationships (Prange & Schneider, 2021).

In this paper, drawing from cultural and linguistic factors, we investigate
variations in visual perception by examining differences in the semantic
coverage of image captions across multiple languages. We investigate these
differences across all the ways that image descriptions are produced in
computer vision: human-annotated, model-produced, and commercial API-produced.
We define and evaluate $4$ metrics for semantic coverage over $2$ datasets,
$3$ models, and $1$ commercial API across $7$ languages.

We show—against the current prevailing assumption in computer vision—that
people who speak different languages mention different semantic content when
describing images, and that these differences are internalized by models. In
general, multilingual captions cover significantly more semantic concepts than
monolingual ones. When represented as a joint scene graph, a set of
multilingual captions have on average $21.8\%$ more objects, $24.5\%$ more
relations, and $27.1\%$ more attributes than a set of monolingual captions
(Table 2). As text, multilingual captions expand the range of object
abstraction/concreteness by $12.2\%$, analytic expression by $45.0\%$, and
color by $37.5\%$ (Table 5). Providing different caption as training datasets
for models lead the model to learn and reproduce the levels of abstraction
present in the data—image captioning models finetuned on monolingual datasets
perform best when evaluated on test sets with descriptions collected in that
monolingual language. On the other hand, those trained on multilingual data
perform consistently well across all test sets. As support for the cultural
basis for observed differences, our results corroborate findings in cross-
cultural psychology (Nisbett & Masuda, 2003; Koo et al., 2018). In a human
image captioning study, Japanese individuals tended to mention background
objects, whereas American individuals tended to mention foreground ones. In
general, across multiple experiments, we observe that captions in Western
languages mention more similar concepts to other Western languages than to
Eastern languages (Tables 3, 7). Our work contributes a computer vision
perspective to the long line of research on the subjectivity of perception.

##  2 Related work

Psychologists, anthropologists, and philosophers provide strong evidence that
perception is subjective. Studies have ranged broadly from organizational
principles of perception: from Gestalt psychology (Wertheimer, 1912) and
Universal Grammar (Chomsky, 1986), to cognitive realities of one’s perceptual
experience (Wittgenstein, 1953). Even the perception of length (Segall et al.,
1967), geometrical intuition (Pedersen & Wheeler, 1983; Dawson et al., 1973),
and depth (Jahoda & McGurk, 1974) vary across people.

Cross-cultural psychology observes that many features of perception vary
between individuals from different cultures (Ceněk & Sasinka, 2020). To
highlight these differences, anthropologists (Hall, 1976) have introduced the
concept of “analytical” versus “holistic” societies. This concept can be used
to compare “Westerners” with “Easterners”. The Western self is analytical: it
is composed of fixed attributes and can move between settings without an
alteration. Meanwhile, the Eastern self is holistic: the self is connected to
others and to their surrounding context (Hall, 1976). Philosophers posit that
the Eastern self understands him or herself “in terms of their relation to the
whole, such as the family, society, Tao Principle, or Pure Consciousness”
(Munro, 1985).

Evidence suggests that the differences in cultural selves can alter the way
individuals attend to and understand visual content and how they talk about
what they see (Nisbett & Masuda, 2003). For instance, when looking an image,
Westerners tend to concentrate on a focal object and focus on analyzing its
attributes and categorization. Attributions tend to focus exclusively on the
object (Masuda & Nisbett, 2001); for example, “the tree is blooming tall” or
“that person looks angry”. Meanwhile, East Asians are more likely to attend to
a broader perceptual and conceptual field, noticing relationships and changes
and grouping objects based on family resemblance rather than category
membership (Koo et al., 2018). For example, an American person might describe
an image as “Two old people on a grey park bench”, focusing on the objects
(“people”, “bench”) and their attributes (“old”, “grey”). While viewing the
same image, a Japanese person would see “Two friends laughing over a joke”,
focusing instead on the relationships (“friends”) and actions (“laughing”).
Since culture is a difficult component to quantify and isolate, our work uses
language as a proxy to study differences between people describing the same
image (Goldstein, 1957; Laesser et al., 2014). In addition, language also
introduces expressive constraints that incentivize speakers to structure
information in a certain way as to avoid sounding unnatural.

In computer vision, recent models have demonstrated the ability to create
stunning images (Ramesh et al., 2022) and write convincing descriptions of
visual events (Ouyang et al., 2022). Most of these capabilities are learned by
vision models from large swathes of vision-language data scraped from the
internet (Radford et al., 2021). The shift from class labels to image-text
pairs (Schuhmann et al., 2022; Thomee et al., 2016) opens up computer vision
to greater expression of perceptual subjectivity. As such, disregarding the
cultural background of language produced to describe images can unduly
overrepresent a set of perceptual behaviors over others. Troublingly, vision-
language datasets remain largely unrepresentative of most non-Western
populations, or are too small or poor-quality for the large scale training of
today’s complex models. Datasets without perceptual diversity risk
contributing towards the homogenization of dominant perceptual modes
(Bommasani et al., 2021) into an “algorithmic monoculture” (Kleinberg &
Raghavan, 2021). This harms progress towards important human-centric (Berg et
al., 2012; Wilchek et al., 2023), decolonization (Bird, 2020), diversity, and
equity goals (Hershcovich et al., 2022) for AI models.

Computer vision has begun its first steps towards understanding human
perception in a machine learning context, such as in object saliency (Spain &
Perona, 2008; Elazary & Itti, 2008; Berg et al., 2012; Yun et al., 2013;
Tavakoli et al., 2017), user-centric vision modeling (Stretcu et al., 2023),
and task ambiguity (Hutchinson et al., 2022). There have also been attempts to
build datasets consisting of geographically diverse images (Atwood et al.,
2020; Schumann et al., 2021). However, our work posits that it is also
important to diversely represent the perceptual styles underlying the
descriptions of those images.

##  3 Analysis of datasets, models, and applications

Our paper investigates the following key research question: Are there
differences in the semantic concepts described by vision-language datasets,
models, and applications across languages? The answer to this question could
inform how we sample pre-training data for large-scale vision-language models,
as well as who we recruit to annotate images with descriptions when finetuning
for applications like image captioning. This question challenges the
prevailing assumption in computer vision that the contents of an image are
objective. Popular datasets such as CIFAR, ImageNet, COCO, PASCAL-VOC have
traditionally treated supervised labels as objective (Gordon et al., 2021). If
this is so, then monolingual semantic content across languages should be
roughly similar and multilingual datasets should not be significantly
different from their particular monolingual counterparts. Our analysis shows
that this “objectivity assumption” does not hold.

In order to test the “objectivity assumption”, we analyze three stages of
today’s computer vision modeling pipeline: from data to model to application.
We analyze the descriptions generated by human annotators in datasets,
annotations generated by models, and by available commercial APIs. We measure
the semantic coverage of concepts mentioned in these descriptions to quantify
differences across languages, in particular between monolingual descriptions
versus multilingual ones. We limit our experiments to $7$ languages.

###  3.1 Sources of image-language data

Datasets. We examine the Crossmodal (‘XM’) dataset (Thapliyal et al., 2022),
which contains image descriptions in 36 languages over 3.6k geographically
diverse images. Moreover, to verify that observed trends occur in humans, we
conduct an image captioning human evaluation to collect descriptions for $30$
Visual Genome (Krishna et al., 2016) images from $10$ English speakers in the
United States and $10$ Japanese speakers in Japan, recruited from Prolific.

Models. We analyze how models trained with multilingual data describe images
across different languages. Specifically, we study LLaVA (Liu et al., 2023) to
produce multilingual captions for images. Although LLaVA is trained with
English data, it inherits probable limited multilingual capabilities from its
large language model (LLM) component, LLaMA (Touvron et al., 2023). In total,
we generate 64.8k captions for 3.6k Crossmodal images in 6 languages with 3
captions each. LLaVA allows us to study language factors across descriptions
because the vision representations are constant across text prompts in
different languages (see Appendix D.1 for details on probing multilingual
behavior). To provide an English-only baseline, we also compare against
descriptions generated by BLIP2 (Li et al., 2023a) and GIT (Wang et al.,
2022).

APIs. We use the Google Vertex API to generate multilingual captions for
images – 75.6k captions for 3.6k Crossmodal images in 7 languages, and 42k
captions for 2k Visual Genome images. These outputs reflect established
standards and have gone through extensive quality checks. This is our main
focus because of its stability and control across languages.

Languages. We focus on English, French, German, Russian, Chinese, Japanese,
and Korean.111We do not include LLaVA’s Korean captions because they are
nearly always incoherent and inaccurate. Together, these 7 languages encompass
a wide range of typologically diverse linguistic families. Moreover, the
speakers of these languages originate from a large variety of cultures and
experiences. To conduct our experiment, we require the consistent application
of linguistic tools such as parsers, which are more mature for English than
other languages. Hence, for our analysis, we translate all descriptions, human
or model produced, into English with GPT-4 (OpenAI, 2023).

To verify that the translations preserve salient conceptual information, we
conduct a human evaluation where we recruit 15 users to rate 30 translations
across each of the $6$ non-English languages. Participants mark the
translations with an average score of $4.68\pm 0.60$ on a 1-to-5 Likert scale,
where 1 is “entirely inaccurate”, 5 is ‘entirely accurate / perfect
translation”, and 4 is “most of the information is preserved (in the
translation)”. In other words, participants marked $98.42\%$ of important
information in a visual scene (objects, spatial relations, color, etc.) as
faithfully preserved across translation (see Appendix C for more details on
the translation prompt and evaluation).

###  3.2 Measuring semantic coverage

Figure 2: Captions of an image in different languages and their scene graphs,
when unioned together produce more varied and complex scene graphs for
multilingual distributions than monolingual ones.

We measure semantic coverage through four mechanisms: A complexity of a scene
graph, B representational diversity, C linguistic diversity, and D ground
truth coverage.

A

Complexity of a scene graph. A scene graph $\mathcal{G}$ is defined as a list
of tuples of the form $(\text{object},\text{attribute})$ or
$(\text{subject},\text{predicate},\text{object})$. Scene graphs are a common
representation to encapsulate the first-order relations between objects in a
scene (Johnson et al., 2015b; Choi et al., 2022; Ji et al., 2020).

We use scene graphs to measure the semantic coverage of each individual
description. First, we parse the descriptions in our data sources into scene
graphs using FLAN-T5 (Chung et al., 2022) model fine-tuned on the FACTUAL-MR
dataset.222The finetuned model is already available online (Li et al., 2023b).
We found that LLM-based parsers were better able to resolve complex semantic
relationships which often arose in the descriptions than conventional syntax-
based parsers (Schuster et al., 2015). Next, we use the following metrics to
measure semantic coverage in the scene: object count (number of objects
mentioned), relationship count (number of pairwise objects connected by a
relationship), and attribute count (number of attributes mentioned).

We can also leverage scene graphs to understand the overlap between two
descriptions. The union of two scene graphs
$\mathcal{G}_{1}\cup\mathcal{G}_{2}$ represents the concepts shared between
them: $\mathcal{G}_{1}$ and $\mathcal{G}_{2}$. Often, descriptions use
different words (e.g. “person” and “man”) when referring to the same object.
To avoid double counting, we canonicalize the concepts using WordNet path
similarity (Miller, 1995) and cosine similarity between concept embeddings
(Reimers & Gurevych, 2019) to merge concepts which have different text but
ground to the same concept. Multiple relationships between two entities are
similarly merged into a single edge using the same process.

We define the information gained from a scene graph $\mathcal{G}$ to
$\mathcal{G}_{+}$ across metric $M$ as
$M(\mathcal{G}\cup\mathcal{G}_{+})-M(\mathcal{G})$. The metric of an
intersection can be computed using inclusion-exclusion as follows:
$M(\mathcal{G}_{1})+M(\mathcal{G}_{2})-M(\mathcal{G}_{1}\cup\mathcal{G}_{2})=M(\mathcal{G}_{1}\cap\mathcal{G}_{2})$.

B

Representational diversity. The second measure of semantic coverage we use is
representational diversity in the embedding space occupied by a description.
Let $E$ be a text embedding model (Reimers & Gurevych, 2019) and $\mathbb{P}$
be a set of natural language text. Let
$\mathbb{Q}=\\{E(\bm{p}):\bm{p}\in\mathbb{P}\\}$. We measure its coverage as
the maximum pairwise cosine distance,
$coverage(\mathbb{Q})=\text{max}\left(\left\\{1-\frac{\bm{a}\cdot\bm{b}}{\|\bm{a}\|\|\bm{b}\|}:\\{\bm{a},\bm{b}\\}\in\mathbb{Q}\times\mathbb{Q}\right\\}\right)$.
Consider $\mathbb{P}_{A}\sim D_{A}$ and $\mathbb{P}_{A\cup B}\sim D_{A}\cup
D_{B}$, text samples drawn from the monolingual language A text distribution
$D_{A}$ and a multilingual distribution $D_{A}\cup D_{B}$ comprising jointly
of language A and B, respectively. If
$\mathbb{E}\left[coverage(\mathbb{Q}_{A\cup
B})\right]>\mathbb{E}\left[coverage(\mathbb{Q}_{A})\right]$, then the
multilingual distribution $D_{A}\cup D_{B}$ has a wider semantic coverage than
the monolingual distribution $D_{A}$, as measured by $E$.

C

Linguistic diversity. Linguistic measures allow us to examine specific
dimensions of text content according to linguistic principles. For some metric
$M$ which maps text to a real number and some set of text $\mathbb{T}$, we
define the semantic coverage of the set as measured by $M$ to be
$coverage_{M}(\mathbb{T})=\max(\\{M(i):i\in\mathbb{T}\\})-\min(\\{M(i):i\in\mathbb{T}\\})$.
If $\mathbb{E}[coverage_{M}(\mathbb{A})]>\mathbb{E}[coverage_{M}(\mathbb{B})]$
for $\mathbb{A}$ sampled from a multilingual distribution and $\mathbb{B}$
sampled from a monolingual distribution, then we claim that the multilingual
distribution is more diverse/rich than the monolingual one as measured by $M$.
We use several metrics $M$. Concreteness is computed across noun objects and
indicates how much a word refers to a perceptible entity (Brysbaert et al.,
2014). For instance, “purple” has a high concreteness rating (4.04 on a
5-point scale), whereas “beautiful” has a low one (2.16). Analytic, Clout,
Authentic, and Tone are psychological measures of logical and hierarchal
thinking patterns, social status and confidence, spontaneity, and emotional
tone in text, from the Linguistic Inquiry and Word Counts (LIWC) package
(Tausczik & Pennebaker, 2010). Color is computed as the maximum pairwise
distance between colors (using the $\Delta$E algorithm (Sharma et al., 2005),
which ensures perceptual uniformity) mentioned across text.

D

Ground truth coverage. Our final metric for semantic coverage uses ground
truth data as reference. The Visual Genome dataset ((Krishna et al., 2016))
provides dense annotations: 50 descriptions per image with their associated
scene graphs. Given a ground truth scene graph, we measure how many objects
are represented in the Visual Genome annotations. Formally, given the ground
truth object set $\mathbb{G}$ and the objects mentioned in a caption
$\mathbb{C}$, semantic coverage is
$\frac{|\mathbb{C}\cap\mathbb{G}|}{|\mathbb{G}|}$.

###  3.3 Results

Across all semantic coverage mechanisms outlined above, we find that
multilingual data improves semantic coverage compared to monolingual data. The
increases are more pronounced for model generated descriptions than human
annotated captions but are visible in both.

A

Complexity of a scene graph.

Table 1: Scene graph metrics across human evaluation captions shows that unioning from 5 English and 5 Japanese scene graphs is richer than ones from 10 English or 10 Japanese. |  | en | ja | en,ja
---|---|---|---|---
| Objects | 9.13 | 8.97 | 9.90
| Relations | 9.63 | 9.30 | 10.37
Metric | Attributes | 9.17 | 8.93 | 9.77

We run LLaVA and the Google Vertex API inference on the Crossmodal dataset
(3.6k images). For the model and the API, sampling an equal number of
descriptions from multilingual data yields higher semantic coverage than
sampling from a monolingual distribution (Table 2). In fact, despite an
expected diminishing-returns trajectory, continuously unioning even a well-
developed existing scene graph with a new language’s scene graph expands it
(Figure 3). Since image captions are typically short and targeted to focus on
the more salient parts of an image, this continual increase in the unique
objects suggests that the salient objects are not self-evident across
languages, as assumed by prior work (Berg et al., 2012), but rather vary
widely. We see the same pattern in scene graphs constructed from our human
evaluation (Table 1) in which sampling captions from across Japanese and
English annotations increases scene graph size over sampling individually from
Japanese or English annotations. Moreover, a manual inspection of the captions
(Figure A) suggests that the captions roughly echo the predictions from cross-
cultural perceptual psychology – Japanese captions tend to mention background
objects and information more than English ones. The identification of such
patterns across multiple domains (humans and models) provides support against
the prevailing “objectivity” assumption.

Table 2: Scene graph metrics across Vertex and LLaVA captions in different languages show that multilingual scene graph unions are richer than monolingual ones. “Monolingual” represents scene graph unions from 3 captions within the same language. “Multilingual” represents scene graph unions from 3 captions in the given languages. “mm” refers to the multimodel baseline. Increases are relative to the English average. A |  | Monolingual |  | Multilingual |
---|---|---|---|---|---
|  | en | de | fr | ru | zh | ja | ko |  | en,fr,zh | fr,de,ru | all | mm
| Objects | 3.65 | 3.51 | 3.60 | 3.86 | 3.46 | 3.13 | 3.18 |  | 4.31 | 4.25 | 5.93 | 4.63
| Relations | 2.96 | 2.83 | 2.89 | 3.20 | 2.68 | 2.37 | 2.47 |  | 3.60 | 3.56 | 6.08 | 3.64
Vertex | Attributes | 1.67 | 1.67 | 1.79 | 1.86 | 1.66 | 1.59 | 1.62 |  | 2.13 | 2.15 | 3.34 | 2.19
| Objects | 4.54 | 5.05 | 5.26 | 4.52 | 4.54 | 3.32 |  |  | 5.87 | 6.02 | 8.44 | 6.65
| Relations | 3.79 | 4.21 | 4.42 | 3.67 | 3.66 | 2.40 |  |  | 4.84 | 4.97 | 7.92 | 3.42
LLaVA | Attributes | 2.75 | 3.47 | 3.50 | 2.76 | 3.25 | 2.70 |  |  | 4.10 | 4.07 | 6.98 | 2.88

(a) # objects in scene graph

(b) # relations in scene graph

(c) # attributes in scene graph

Figure 3: Scene graphs of captions unioned cumulatively from different
languages lead to more coverage in objects, relations, and attributes. For
more graphs, see Appendix D.3. A

We also compare these values against an English baseline consisting of the
union between a Vertex/LLaVA, GIT, and BLIP2 parsed scene graph. The baseline
is ‘ambitious’ in the sense that the Vertex/LLaVA, GIT, and BLIP2 do not share
the same architecture or training data. The only thing they have in common is
that their training data uses English data. Therefore, looking at Figure 3, it
is noteworthy that adding additional Vertex captions in different languages
continues to increase the coverage of objects, attributes, and relationships.
This increase is approximately the same as if we had started with a baseline
using just Vertex/LLaVA’s English descriptions.

The concepts mentioned by GIT and BLIP2 have a surprisingly low content
intersection with all of the monolingual Vertex scene graphs (Table 3). This
suggests that although additional models increase the number of concepts by
about as much as additional languages, the content of these contributed
concepts are very different. We even find low content intersection between any
two languages, measured using the scene graphs extracted from Vertex
monolingual scene graphs (Table 12), demonstrating that content distributions
across languages are meaningfully separate. Interestingly, we observe
“Western” languages (English, German, French, Russian) tend to cover more
similar concepts than to “Eastern” languages (Chinese, Japanese, Korean)
(Tables 3 and 12).

Table 3: Intersection sizes between 3 unioned monolingual Vertex captions and an English multimodel baseline (a unioned BLIP2 $\cup$ GIT scene graph, held constant across all languages) are both relatively small and smaller for Asian than European languages. All relationships between European languages in red and Asian languages cells in blue are statistically significant with Bonferroni correction. The ‘mm’ column includes the size of the unioned GIT and BLIP model for reference. A |  | Language |
---|---|---|---
|  | en | de | fr | ru | zh | ja | ko | mm
| Objects | 1.96 | 1.92 | 1.93 | 1.97 | 1.85 | 1.73 | 1.76 | 3.59
| Relations | 0.79 | 0.76 | 0.74 | 0.78 | 0.70 | 0.62 | 0.64 | 2.51
Metric | Attributes | 0.44 | 0.36 | 0.37 | 0.42 | 0.37 | 0.37 | 0.33 | 1.45

B

Representational diversity. When measuring semantic coverage in embedding
space, we find that groups of multilingual captions have larger embedding
space coverage than monolingual ones. The mean coverage point cloud of three
embedded English Vertex captions, for instance, almost doubles from $0.19$ to
$0.37$ when replacing two of them with embedded French and Chinese Vertex
captions (Table 4).

Table 4: Embeddings from multilingual captions have a larger mean coverage than embeddings from monolingual captions. Crossmodal provides at minimum two captions per language per image, so we adjust the multilingual compositions from 3 to 2 accordingly. B | Monolingual |  | Multilingual |
---|---|---|---|---
| en | de | fr | ru | zh | ja | ko |  | en,fr,zh | fr,zh,ru | zh,ja,ko | all
Vertex | 0.19 | 0.18 | 0.19 | 0.22 | 0.20 | 0.19 | 0.21 |  | 0.37 | 0.37 | 0.38 | 0.51
LLaVA | 0.22 | 0.29 | 0.28 | 0.29 | 0.36 | 0.40 |  |  | 0.45 | 0.47 | 0.46 | 0.57
|  |  |  |  |  |  |  |  | en,fr | en,zh | fr,zh |
XM | 0.38 | 0.40 | 0.38 | 0.40 | 0.45 | 0.42 | 0.41 |  | 0.41 | 0.46 | 0.45 | 0.65

We observe similar patterns in the human-labeled Crossmodal dataset, although
expectedly with weaker signals (Table 13). For European languages, using a
multilingual set over a monolingual one noticeably increases the mean
coverage, from $0.38$ for monolingual English to $0.46$ for English and
Chinese. This effect is less prominent for Asian languages.

C

Linguistic diversity. We find, across all linguistic metrics but to varying
degrees, that multilingual descriptions increase the linguistic diversity
coverage over monolingual data for Vertex (Table 5), and generally for LLaVA
(Table 14) and Crossmodal (Table 15). This suggests that multilingual data
vary more in the form of their descriptions, and also interestingly
perceive/mention a wider range of color.

Table 5: Mean coverage across different linguistic measures. All values represented as percentages of the total metric range. ‘en’, ‘de’, etc. represent the metrics as calculated across a monolingual distribution. ‘multi‘ represents metrics across a multilingual distribution of the same size. ‘all’ represents metrics across the entire multilingual distribution. Computed across Vertex captions. C |  | Monolingual |  |
---|---|---|---|---
|  | en | de | fr | ru | zh | ja | ko | multi | all
| Concreteness | 32.80 | 32.60 | 33.40 | 33.20 | 30.20 | 30.00 | 31.20 | 35.80 | 46.60
| Analytic | 0.84 | 0.43 | 0.62 | 1.08 | 2.30 | 2.60 | 2.17 | 2.08 | 7.95
| Clout | 4.94 | 5.05 | 5.4 | 7.14 | 6.92 | 6.49 | 5.56 | 11.16 | 27.15
| Authentic | 23.21 | 22.8 | 21.68 | 23.07 | 23.51 | 25.01 | 21.67 | 39.5 | 76.47
Metric $M$ | Tone | 1.85 | 1.84 | 2.03 | 2.63 | 2.05 | 1.96 | 2.05 | 4.33 | 10.18
| Color | 18.20 | 23.57 | 21.24 | 22.60 | 17.92 | 16.49 | 21.19 | 27.74 | 51.28
Table 6: Ground truth coverage ($|\mathbb{C}\cap\mathbb{G}|/|\mathbb{G}|$) increases when sampling multilingual captions. $\mathbb{C}$ refers to the caption concept set and $\mathbb{G}$ refers to the ‘ground truth’ Visual Genome concept set, per earlier notation. Relationships between monolingual and multilingual distributions are statistically significant with correction. As a baseline, $\mathbb{E}\left[|\mathbb{G}|\right]=21.40$. $|\mathbb{C}\cap\mathbb{G}|$ (unnormalized intersection size, number of objects shared) and $|\mathbb{C}|$ are provided for reference. D |  | Monolingual |  | Multilingual
---|---|---|---|---
|  | en | de | fr | ru | zh | ja | ko |  | en,fr,zh | en,de,ja | de,fr,ru
Vertex | $|\mathbb{C}\cap\mathbb{G}|/|\mathbb{G}|$ | 0.183 | 0.171 | 0.177 | 0.185 | 0.170 | 0.162 | 0.156 |  | 0.200 | 0.197 | 0.225
$|\mathbb{C}\cap\mathbb{G}|$ | 3.49 | 3.28 | 3.38 | 3.53 | 3.23 | 3.08 | 2.96 |  | 3.84 | 3.79 | 3.64
$|\mathbb{C}|$ | 3.70 | 3.56 | 3.60 | 3.83 | 3.48 | 3.09 | 3.03 |  | 4.31 | 4.24 | 4.10
LLaVA | $|\mathbb{C}\cap\mathbb{G}|/|\mathbb{G}|$ | 0.195 | 0.186 | 0.200 | 0.170 | 0.155 |  |  |  | 0.214 |  | 0.215
$|\mathbb{C}\cap\mathbb{G}|$ | 3.79 | 3.60 | 3.90 | 3.29 | 3.01 |  |  |  | 4.21 |  | 4.19
$|\mathbb{C}|$ | 4.58 | 5.32 | 5.66 | 4.83 | 4.78 |  |  |  | 6.28 |  | 6.48

D

Ground truth coverage. Sampling the same number of multilingual descriptions
compared to monolingual data increases the mean semantic coverage of Visual
Genome-annotated objects (Table 6), a pattern echoing the results in A.
Interestingly, when English, French, and Chinese descriptions are merged,
their coverage increases when compared to using only English descriptions from
$3.70$ to $4.31$ objects per image. However, their intersection with Visual
Genome ground truth objects only increases from $3.49$ to $3.84$. This
suggests that the Visual Genome dataset itself is ‘perceptually limited’,
despite being a canonical densely annotated dataset. This unaccounted
$4.31-3.84=0.47$ objects likely refer to objects which have not been
documented in Visual Genome. We provide several examples to support this
hypothesis in Appendix D.7 and discuss the implications in Appendix A.
Multilingual captions identify undocumented objects at a higher rate than
monolingual captions (on average, $10.9\%$ vs $5.3\%$).

##  4 Finetuning on multilingual versus monolingual data

Our analysis thus far highlights two important findings. First, vision-
language datasets across different languages cover widely varying concepts.
Second, models amplify these differences. In this section, we identify what
these findings mean for when we finetune vision-language models with
multilingual versus monolingual data.

Method. We finetune a pretrained GIT model (Wang et al., 2022) on 8 image
captioning datasets. These finetuning datasets use the same set of 1.8k
training images from the Crossmodal dataset. We automatically create these
eight training datasets by translating captions generated from Vertex, LLaVa,
and XM multilingual captions generated in the our analysis above into
English.333Translating all multilingual captions into English allows us to
vary the description’s semantic content without simultaneously changing the
language and, thus, token distribution used by the model. and use the original
Crossmodal’s human generated data as finetuning data. The first seven training
sets contain translations from a single language (i.e. we have a finetuning
dataset for French, for German etc.). The eighth dataset is constructed with
equal proportions of captions across all 7 languages. We choose GIT because of
its simple architecture and robust performance across benchmarks.

We evaluate the finetuned models across a held out 1.8k images from the
Crossmodal dataset. Validating each of the 8 fine-tuned models on each of the
8 holdout caption sets yields 64 evaluations. We use SPICE F1-score as our
evaluation metric (Anderson et al., 2016), which measures conceptual overlap
over low-level syntax adherence. We generate “ground truth” captions across
the different languages using Google’s Vertex commercial API, which again, we
deem to be the highest quality given its commercial application and quality
checks.

We interpret our results using the following logic. Let $\mathcal{L}(A;B)$
represent the error of a model fine-tuned on the data from language $B$ on the
evaluation data from language $A$. If the scores across all evaluations $A$
are approximately equal, then the prevailing assumption that the content of
captions across languages are approximately equal holds. On the other hand, if
$\forall_{A\in\mathbb{L}}\forall_{B\in\mathbb{L}}\mathcal{L}(A;A)>\mathcal{L}(B;A)$,
this is evidence against the prevailing assumption: models learn particular
concepts when finetuned using data from a specific language, and performs
better when evaluated against captions from that language.

Results. We find that a model fine-tuned on language $A$ performs best on
language $A$ (Table 7), again against the prevailing assumption. We find the
same results across models finetuned on data from LLaVA (Table 17), and from
the human annotated Crossmodal dataset (Table 18). Moreover, finetuning
multilingual data from Vertex yields a model which performs consistently well
across all evaluation data compositions. Together, these suggest that models
trained with multilingual data, representing diverse perceptual modes learn
more about the visual world. One possible confounder is syntactic artifacts
introduced during translation. For instance, text translated from German into
English might have a unique syntactic structure which distinguishes it from
text originally written in English. If this is the case, then it should be
possible to identify translated text from one language versus another. To test
this limitation, we embed all translated captions using a BERT-based model
(Reimers & Gurevych, 2019). We fit a logistic regression model to predict a
sample’s original language from these features, and find near-random chance
performance at $16.43\%$.

Table 7: SPICE F-scores when evaluating a model fine-tuned on the training set from the language on the left against the validation set from the language on the top. Vertex captions. For instance, a model fine-tuned on English Vertex captions obtains 0.219 SPICE score on Russian Vertex captions. ‘multi’ refers to an even split across all languages. Red indicates best performance on a split, yellow highlights model fine-tuned on ‘multi’. |  | Evaluated on
---|---|---
|  | en | de | fr | ru | zh | ja | ko | multi
| en | 0.271 | 0.225 | 0.229 | 0.219 | 0.218 | 0.229 | 0.232 | 0.230
| de | 0.213 | 0.245 | 0.219 | 0.217 | 0.215 | 0.210 | 0.226 | 0.219
| fr | 0.248 | 0.240 | 0.259 | 0.234 | 0.236 | 0.239 | 0.253 | 0.246
| ru | 0.226 | 0.234 | 0.228 | 0.254 | 0.231 | 0.236 | 0.237 | 0.239
| zh | 0.199 | 0.202 | 0.199 | 0.207 | 0.247 | 0.220 | 0.224 | 0.216
Fine-tuned on | ja | 0.212 | 0.212 | 0.215 | 0.212 | 0.226 | 0.266 | 0.245 | 0.223
| ko | 0.218 | 0.222 | 0.224 | 0.217 | 0.242 | 0.239 | 0.271 | 0.235
| multi | 0.239 | 0.233 | 0.234 | 0.233 | 0.235 | 0.243 | 0.252 | 0.244

##  5 Conclusion

Similar to previous theories and empirical findings in social science that
studies the subjectivity of human perception, we uncover differences between
the content of captions from different languages in datasets, model behavior,
and model training. Our work poses an “inversion to the curse of
multilinguality”: language-accessibility and model performance in computer
vision may not necessarily be in conflict with each other, as previous work
has suggested (Conneau et al., 2020), but has the potential to improve
information coverage in visual scenes because of diverse perception modes. In
general, we turn from a constant and objective conception of the “ground
truth” towards the human subject, which selectively attends to certain objects
and qualities in the nearly-infinite field of raw visual features. Previous
works have alluded to image captioning being a subjective process involving
emotion and aesthetic factors (Mohamed et al., 2022) or suggested that
language is fluid and can represent the same set of concepts in many different
ways (Luong et al., 2021). In our work, we make the stronger observation that
even the “objective concepts” within a scene are, at their root, observed by
human subjects with particular perceptual tendencies, rather than
independently existing. Our findings have broader implications for machine
vision beyond just image captioning, such as open-vocabulary segmentation (Sun
et al., 2023; Kirillov et al., 2023) and image generation (Ramesh et al.,
2021).

## Acknowledgements

We thank Katharina Reinecke for the many insightful discussions that inspired
this project; Quan Ze (Jim) Chen, Matthew Wallingford, Thao Nguyen, Hyunwoo
Kim, Wei-Chiu Ma, Chris Lin, Lisa Orii, Nino Migineishvili, Ashish Sharma, Jae
Sung (James) Park and members of RAIVN lab for the multiple rounds of
discussion, feedback and reviewing; Cheng-Yu Hsieh and Ashish Sharma for
code/package assistance; Allen AI for running the surveys; and Chun-Liang Li
at Google Cloud for running the Vertex API. We also thank participants of our
study for carefully captioning the images.

## References

  * Anderson et al. (2016) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould.  Spice: Semantic propositional image caption evaluation.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), _Computer Vision – ECCV 2016_ , pp. 382–398, Cham, 2016. Springer International Publishing.  ISBN 978-3-319-46454-1.
  * Atwood et al. (2020) James Atwood, Yoni Halpern, Pallavi Baljekar, Eric Breck, D Sculley, Pavel Ostyakov, Sergey I Nikolenko, Igor Ivanov, Roman Solovyev, Weimin Wang, et al.  The inclusive images competition.  In _The NeurIPS’18 Competition: From Machine Learning to Intelligent Conversations_ , pp. 155–186. Springer, 2020.
  * Banerjee & Lavie (2005) Satanjeev Banerjee and Alon Lavie.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pp. 65–72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.  URL https://aclanthology.org/W05-0909.
  * Berg et al. (2012) Alexander C. Berg, Tamara L. Berg, Hal Daumé, Jesse Dodge, Amit Goyal, Xufeng Han, Alyssa C. Mensch, Margaret Mitchell, Aneesh Sood, Karl Stratos, and Kota Yamaguchi.  Understanding and predicting importance in images.  _2012 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3562–3569, 2012.
  * Bird (2020) Steven Bird.  Decolonising speech and language technology.  In _International Conference on Computational Linguistics_ , 2020.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel J. Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang.  On the opportunities and risks of foundation models.  _ArXiv_ , abs/2108.07258, 2021.
  * Boroditsky (2006) Lera Boroditsky.  Linguistic relativity.  _Encyclopedia of cognitive science_ , 2006.
  * Boroditsky et al. (2003) Lera Boroditsky, Lauren A Schmidt, and Webb Phillips.  Sex, syntax, and semantics.  _Language in mind: Advances in the study of language and thought_ , 22:61–79, 2003.
  * Brysbaert et al. (2014) Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman.  Concreteness ratings for 40 thousand generally known english word lemmas.  _Behavior Research Methods_ , 46:904–911, 2014.  URL https://link.springer.com/article/10.3758/s13428-013-0403-5.
  * Buolamwini & Gebru (2018) Joy Buolamwini and Timnit Gebru.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In Sorelle A. Friedler and Christo Wilson (eds.), _Proceedings of the 1st Conference on Fairness, Accountability and Transparency_ , volume 81 of _Proceedings of Machine Learning Research_ , pp. 77–91. PMLR, 23–24 Feb 2018.  URL https://proceedings.mlr.press/v81/buolamwini18a.html.
  * Caliskan et al. (2022) Aylin Caliskan, Pimparkar Parth Ajay, Tessa Charlesworth, Robert Wolfe, and Mahzarin R. Banaji.  Gender bias in word embeddings: A comprehensive analysis of frequency, syntax, and semantics.  In _Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society_ , AIES ’22, pp. 156–170, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392471.  doi: 10.1145/3514094.3534162.  URL https://doi.org/10.1145/3514094.3534162.
  * Ceněk & Sasinka (2020) Jiři Ceněk and Ceněk Sasinka.  Cross-cultural differences in visual perception.  _The Journal of Education, Culture, and Society_ , 2015:187–206, 2020.
  * Chen et al. (2022) Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel M. Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut.  Pali: A jointly-scaled multilingual language-image model.  _ArXiv_ , abs/2209.06794, 2022.  URL https://arxiv.org/abs/2209.06794.
  * Choi et al. (2022) Woo Suk Choi, Yu-Jung Heo, Dharani Punithan, and Byoung-Tak Zhang.  Scene graph parsing via Abstract Meaning Representation in pre-trained language models.  In _Proceedings of the 2nd Workshop on Deep Learning on Graphs for Natural Language Processing (DLG4NLP 2022)_ , pp. 30–35, Seattle, Washington, July 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.dlg4nlp-1.4.  URL https://aclanthology.org/2022.dlg4nlp-1.4.
  * Chomsky (1986) Noam Chomsky.  _Knowledge of language: Its nature, origin, and use_.  Greenwood Publishing Group, 1986.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models.  _ArXiv_ , abs/2210.11416, 2022.  URL https://arxiv.org/abs/2210.11416.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.747.  URL https://aclanthology.org/2020.acl-main.747.
  * Dawson et al. (1973) John L. M. Dawson, Brian M. Young, and Peter P. C. Choi.  Developmental influences on geometric illusion susceptibility among hong kong chinese children.  _Journal of Cross-Cultural Psychology_ , 4:49 – 74, 1973.
  * de Saussure (1960) Ferdinand de Saussure.  Course in general linguistics.  1960\.
  * Divjak (2019) Dagmar Divjak.  _Frequency in Language: Memory, Attention and Learning_.  Cambridge University Press, 2019.  doi: 10.1017/9781316084410.  URL https://doi.org/10.1017/9781316084410.
  * Elazary & Itti (2008) Lior Elazary and Laurent Itti.  Interesting objects are visually salient.  _Journal of vision_ , 8 3:3.1–15, 2008.
  * Gallup et al. (2012) Andrew C Gallup, Joseph J Hale, David JT Sumpter, Simon Garnier, Alex Kacelnik, John R Krebs, and Iain D Couzin.  Visual attention and the acquisition of information in human crowds.  _Proceedings of the National Academy of Sciences_ , 109(19):7245–7250, 2012.
  * Gallup et al. (2014) Andrew C Gallup, Andrew Chong, Alex Kacelnik, John R Krebs, and Iain D Couzin.  The influence of emotional facial expressions on gaze-following in grouped and solitary pedestrians.  _Scientific reports_ , 4(1):5794, 2014.
  * Goldstein (1957) Leon J Goldstein.  On defining culture.  _American Anthropologist_ , 59(6):1075–1081, 1957.
  * Gordon et al. (2021) Mitchell L Gordon, Kaitlyn Zhou, Kayur Patel, Tatsunori Hashimoto, and Michael S Bernstein.  The disagreement deconvolution: Bringing machine learning performance metrics in line with reality.  In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_ , pp. 1–14, 2021.
  * Hall (1976) Edward T Hall.  _Beyond culture_.  Anchor, 1976.
  * Harada (1976) Shin-Ichi Harada.  Honorifics.  In _Japanese generative grammar_ , pp. 499–561. Brill, 1976.
  * Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard.  Challenges and strategies in cross-cultural nlp.  _ArXiv_ , abs/2203.10020, 2022.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.  Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20. JMLR.org, 2020.
  * Hu et al. (2023) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith.  Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.  _ArXiv_ , abs/2303.11897, 2023.  URL https://arxiv.org/abs/2303.11897.
  * Hutchinson et al. (2022) Ben Hutchinson, Jason Baldridge, and Vinodkumar Prabhakaran.  Underspecification in scene description-to-depiction tasks.  _arXiv preprint arXiv:2210.05815_ , 2022.
  * Jahoda & McGurk (1974) Gustav Jahoda and Harry McGurk.  Pictorial depth perception: a developmental study.  _British journal of psychology_ , 65 1:141–9, 1974.
  * Ji et al. (2020) Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles.  Action genome: Actions as compositions of spatio-temporal scene graphs.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10236–10247, 2020.
  * Johnson et al. (2015a) Justin Johnson, Andrej Karpathy, and Li Fei-Fei.  Densecap: Fully convolutional localization networks for dense captioning.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 4565–4574, 2015a.  URL https://arxiv.org/abs/1511.07571.
  * Johnson et al. (2015b) Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Image retrieval using scene graphs.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 3668–3678, 2015b.  doi: 10.1109/CVPR.2015.7298990.
  * Joo et al. (2014) Jungseock Joo, Weixin Li, Francis F Steen, and Song-Chun Zhu.  Visual persuasion: Inferring communicative intents of images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 216–223, 2014.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick.  Segment anything.  _ArXiv_ , abs/2304.02643, 2023.  URL https://arxiv.org/abs/2304.02643.
  * Kleinberg & Raghavan (2021) Jon M. Kleinberg and Manish Raghavan.  Algorithmic monoculture and social welfare.  _Proceedings of the National Academy of Sciences_ , 118, 2021.
  * Koo et al. (2018) Minkyung Koo, Jong An Choi, and Incheol Choi.  Analytic versus holistic cognition: Constructs and measurement.  2018\.
  * Krishna et al. (2016) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_ , 123:32 – 73, 2016.  URL https://arxiv.org/abs/1602.07332.
  * Laesser et al. (2014) Christian Laesser, Pietro Beritelli, and Samuel Heer.  Different native languages as proxy for cultural differences in travel behaviour: insights from multilingual switzerland.  _International Journal of Culture, Tourism and Hospitality Research_ , 8(2):140–152, 2014.
  * Lai et al. (2023) Zhengfeng Lai, Haotian Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, et al.  From scarcity to efficiency: Improving clip training via visual-enriched captions.  _arXiv preprint arXiv:2310.07699_ , 2023.
  * Lakoff (1987) George Lakoff.  _Fire, Women, and Dangerous Things_.  University of Chicago Press, 1987.
  * Langacker (1993) Ronald W Langacker.  Universals of construal.  In _Annual Meeting of the Berkeley Linguistics Society_ , volume 19, pp. 447–463, 1993.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pp. 12888–12900. PMLR, 2022.
  * Li et al. (2023a) Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _ArXiv_ , abs/2301.12597, 2023a.  URL https://arxiv.org/abs/2301.12597.
  * Li et al. (2023b) Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran.  FACTUAL: A benchmark for faithful and consistent textual scene graph parsing.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 6377–6390, Toronto, Canada, July 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.398.  URL https://aclanthology.org/2023.findings-acl.398.
  * Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  _ArXiv_ , abs/2304.08485, 2023.  URL https://arxiv.org/abs/2304.08485.
  * Luong et al. (2021) Quoc-An Luong, Duc Minh Vo, and Akihiro Sugimoto.  Saliency based subject selection for diverse image captioning.  In _2021 17th International Conference on Machine Vision and Applications (MVA)_ , pp. 1–5. MVA Organization, 2021.  ISBN 4901122207.
  * Masuda & Nisbett (2001) Takahiko Masuda and Richard E. Nisbett.  Attending holistically versus analytically: comparing the context sensitivity of japanese and americans.  _Journal of personality and social psychology_ , 81 5:922–34, 2001.
  * Masuda et al. (2008) Takahiko Masuda, Mikako Akase, Mark HB Radford, and Huaitang Wang.  Effect of contextual factors on patterns of eye-movement: Comparing sensitivity to background information between japanese and westerners.  _Japanese Journal of Psychology_ , 2008.
  * Miller (1995) George A. Miller.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38(11):39–41, nov 1995.  ISSN 0001-0782.  doi: 10.1145/219717.219748.  URL https://doi.org/10.1145/219717.219748.
  * Mishra et al. (2021) Santosh Kumar Mishra, Rijul Dhir, Sriparna Saha, and Pushpak Bhattacharyya.  A hindi image caption generation framework using deep learning.  _ACM Trans. Asian Low-Resour. Lang. Inf. Process._ , 20(2), mar 2021.  ISSN 2375-4699.  doi: 10.1145/3432246.  URL https://doi.org/10.1145/3432246.
  * Mohamed et al. (2022) Youssef Mohamed, Mohamed AbdelFattah, Shyma Alhuwaider, Li Fei-Fei, Xiangliang Zhang, Kenneth Ward Church, and Mohamed Elhoseiny.  Artelingo: A million emotion annotations of wikiart with emphasis on diversity over language and culture.  _ArXiv_ , abs/2211.10780, 2022.  URL https://arxiv.org/abs/2211.10780.
  * Munro (1985) Donald J Munro.  Individualism and holism: Studies in confucian and taoist values.  1985\.
  * Nguyen et al. (2023) Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt.  Improving multimodal datasets with image captioning.  _arXiv preprint arXiv:2307.10350_ , 2023.
  * Nisbett & Masuda (2003) Richard E. Nisbett and Takahiko Masuda.  Culture and point of view.  _Proceedings of the National Academy of Sciences of the United States of America_ , 100:11163 – 11170, 2003.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023.  URL https://cdn.openai.com/papers/gpt-4.pdf.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Oyserman & Lee (2008) Daphna Oyserman and Spike W. S. Lee.  Does culture influence what and how we think? effects of priming individualism and collectivism.  _Psychological bulletin_ , 134 2:311–42, 2008.  URL https://pubmed.ncbi.nlm.nih.gov/18298274/.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, pp. 311–318, USA, 2002. Association for Computational Linguistics.  doi: 10.3115/1073083.1073135.  URL https://doi.org/10.3115/1073083.1073135.
  * Park et al. (2019) Junwon Park, Ranjay Krishna, Pranav Khadpe, Li Fei-Fei, and Michael Bernstein.  Ai-based request augmentation to increase crowdsourcing participation.  In _Proceedings of the AAAI Conference on Human Computation and Crowdsourcing_ , volume 7, pp. 115–124, 2019.
  * Pedersen & Wheeler (1983) Darhl M. Pedersen and Jay Wheeler.  The müller-lyer illusion among navajos.  _The Journal of social psychology_ , 121 1st Half:3–6, 1983.
  * Prange & Schneider (2021) Jakob Prange and Nathan Schneider.  Draw mir a sheep: A supersense-based analysis of german case and adposition semantics.  _KI-Künstliche Intelligenz_ , 35(3-4):291–306, 2021.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 8821–8831. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/ramesh21a.html.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019.  URL https://arxiv.org/abs/1908.10084.
  * Röttger et al. (2021) Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet B Pierrehumbert.  Two contrasting data annotation paradigms for subjective nlp tasks.  _arXiv preprint arXiv:2112.07475_ , 2021.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _ArXiv_ , abs/2210.08402, 2022.  URL https://arxiv.org/abs/2210.08402.
  * Schumann et al. (2021) Candice Schumann, Susanna Ricco, Utsav Prabhu, Vittorio Ferrari, and Caroline Pantofaru.  A step toward more inclusive people annotations for fairness.  _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , 2021.
  * Schuster et al. (2015) Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning.  Generating semantically precise scene graphs from textual descriptions for improved image retrieval.  In _Proceedings of the fourth workshop on vision and language_ , pp. 70–80, 2015.
  * Segall et al. (1967) Marshall H. Segall, Donald T. Campbell, and Melville J. Herskovits.  The influence of culture on visual perception.  1967\.
  * Sharma et al. (2005) Gaurav Sharma, Wencheng Wu, and Edul N. Dalal.  The ciede2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations.  _Color Research and Application_ , 30:21–30, 2005.  URL https://doi.org/10.1002/col.20070.
  * Skeem & Lowenkamp (2016) Jennifer L Skeem and Christopher T. Lowenkamp.  Risk, race, & recidivism: Predictive bias and disparate impact.  _Political Economy: Structure & Scope of Government eJournal_, 2016.  URL https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2687339.
  * Spain & Perona (2008) Merrielle Spain and Pietro Perona.  Some objects are more equal than others: Measuring and predicting importance.  In _European Conference on Computer Vision_ , 2008.
  * Sprague et al. (2016) William W Sprague, Emily A Cooper, Sylvain Reissier, Baladitya Yellapragada, and Martin S Banks.  The natural statistics of blur.  _Journal of vision_ , 16(10):23–23, 2016.
  * Stretcu et al. (2023) Otilia Stretcu, Edward Vendrow, Kenji Hata, Krishnamurthy Viswanathan, Vittorio Ferrari, Sasan Tavakkol, Wenlei Zhou, Aditya Avinash, Enming Luo, Neil Gordon Alldrin, MohammadHossein Bateni, Gabriel Berger, Andrew Bunner, Chun-Ta Lu, Javier A Rey, Giulia DeSalvo, Ranjay Krishna, and Ariel Fuxman.  Agile modeling: From concept to classifier in minutes.  In _International Conference on Computer Vision_ , 2023.  URL https://openaccess.thecvf.com/content/ICCV2023/papers/Stretcu_Agile_Modeling_From_Concept_to_Classifier_in_Minutes_ICCV_2023_paper.pdf.
  * Sun et al. (2023) Pei Sun, Shoufa Chen, Chenchen Zhu, Fanyi Xiao, Ping Luo, Saining Xie, and Zhicheng Yan.  Going denser with open-vocabulary part segmentation.  _ArXiv_ , abs/2305.11173, 2023.  URL https://arxiv.org/abs/2305.11173.
  * Tahaei et al. (2023) Mohammad Tahaei, Marios Constantinides, Daniele Quercia, and Michael Muller.  A systematic literature review of human-centered, ethical, and responsible ai.  2023\.  URL https://arxiv.org/pdf/2302.05284.pdf.
  * Talmy (1988) Leonard Talmy.  Force dynamics in language and cognition.  _Cognitive Science_ , 12(1):49–100, 1988.  ISSN 0364-0213.  doi: https://doi.org/10.1016/0364-0213(88)90008-0.  URL https://www.sciencedirect.com/science/article/pii/0364021388900080.
  * Tausczik & Pennebaker (2010) Yla R. Tausczik and James W. Pennebaker.  The psychological meaning of words: Liwc and computerized text analysis methods.  _Journal of Language and Social Psychology_ , 29(1):24–54, 2010.  doi: 10.1177/0261927X09351676.  URL https://doi.org/10.1177/0261927X09351676.
  * Tavakoli et al. (2017) Hamed Rezazadegan Tavakoli, Rakshith Shetty, Ali Borji, and Jorma T. Laaksonen.  Paying attention to descriptions generated by image captioning models.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pp. 2506–2515, 2017.
  * Taylor et al. (2023) Randon R. Taylor, Bessie O’Dell, and John W. Murphy.  Human-centric ai: philosophical and community-centric considerations.  _AI & SOCIETY_, 2023.  URL https://doi.org/10.1007/s00146-023-01694-1.
  * Thapliyal et al. (2022) Ashish V. Thapliyal, Jordi Pont-Tuset, Xi Chen, and Radu Soricut.  Crossmodal-3600: A massively multilingual multimodal evaluation dataset.  In _Conference on Empirical Methods in Natural Language Processing_ , 2022.
  * Thomee et al. (2016) Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  _Commun. ACM_ , 59(2):64–73, jan 2016.  ISSN 0001-0782.  doi: 10.1145/2812802.  URL https://doi.org/10.1145/2812802.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _ArXiv_ , abs/2302.13971, 2023.  URL https://arxiv.org/abs/2302.13971.
  * Wang et al. (2022) Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang.  Git: A generative image-to-text transformer for vision and language.  _ArXiv_ , abs/2205.14100, 2022.  URL https://arxiv.org/abs/2205.14100.
  * Watson & Platt (2012) Karli K Watson and Michael L Platt.  Social signals in primate orbitofrontal cortex.  _Current Biology_ , 22(23):2268–2273, 2012.
  * Wen et al. (2023) Xiaochen Wen, Yaping Zhou, Yinan Li, Yue Lv, Siyi Han, and Junshu Zhao.  A randomized controlled trial examining a tranquil sitting intervention compatible with confucian values.  _Frontiers in Psychology_ , 14, 2023.
  * Wertheimer (1912) Max Wertheimer.  Experimentelle studien uber das sehen von bewegung.  _Zeitschrift fur psychologie_ , 61:161–165, 1912.
  * Wilchek et al. (2023) Matthew Wilchek, Will Hanley, Jude Lim, Kurt Luther, and Feras A. Batarseh.  Human-in-the-loop for computer vision assurance: A survey.  _Engineering Applications of Artificial Intelligence_ , 2023.
  * Wittgenstein (1953) Ludwig Wittgenstein.  _Philosophical Investigations_.  Wiley-Blackwell, New York, NY, USA, 1953.
  * Yorzinski & Platt (2014) Jessica L Yorzinski and Michael L Platt.  Selective attention in peacocks during predator detection.  _Animal cognition_ , 17:767–777, 2014.
  * Yun et al. (2013) Kiwon Yun, Yifan Peng, Dimitris Samaras, Gregory J. Zelinsky, and Tamara L. Berg.  Studying relationships between human gaze, description, and computer vision.  _2013 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 739–746, 2013.

##  Appendix A Further Discussion

Table 8: A few examples of captions collected from the human study across
English and Japanese speakers show differences in the observed content for
each image. Japanese captions tend to include more context (e.g., background
objects, added details). Samples are selected but representative of broader
trends.

| English | I. |  Two very small boats on a river
---|---|---|---
| II. |  toy boats in the water
| III. |  A yellow boat and a red boat that appear to be models.
2-4[1pt/2pt] | Japanese | I. |  Two boats on the water and a building in the back
| II. |  close-up of a model of a boat and people on the waterfront
| III. |  Two boats floating on the river and a model of the town in the distance
| English | I. |  Luggage left unattended at a table.
| II. |  Luggage lined up next to tables with jackets resting on the tables.
| III. |  luggage sitting next to tables
2-4[1pt/2pt] | Japanese | I. |  A man sitting in a lobby with lots of suitcases and bags
| II. |  A man is sitting in a room, and there are several tables filled with luggage nearby.
| III. |  Man waiting with a lot of luggage
| English | I. |  Cat laying down in an arm chair.
| II. |  A Siamese cat laying on its back on a couch next to a pillow.
| III. |  A cat stretched out and upside down on a chair
2-4[1pt/2pt] | Japanese | I. |  A cat stretches out on a blue chair and a pillow with an embroidered owl next to it.
| II. |  A cat is relaxing next to a cushion with a picture of an owl on it.
| III. |  Cat sitting on his back in an armchair with an owl-patterned cushion

From Object to (Human) Subject: Relativizing Ground Truth.

Computer Vision often adopts an object-centric approach, in which a set of
salient objects in visual scenes constitute a singular ground truth. This
assumption is written into the very optimization objectives and metrics used
in image captioning, which require the model to minimize the difference
between predicted lexical units and the ‘true’ lexical units (BLEU Papineni et
al. (2002), METEOR Banerjee & Lavie (2005)) or between predicted concepts and
‘true’ concepts (SPICE Anderson et al. (2016)). Even models such as DenseCap
(Johnson et al. (2015a)) and dataset efforts such as Visual Genome (Krishna et
al. (2016)) fail to cover all semantic information in a visual scene (see D).
Neither is it the objective of image captioning to do so in the first place:
the task of image captioning contains an irreducible subjective core.

Therefore, we must take into account patterns in subjects’ attention towards
certain object sets rather than simply upholding some finite object set as the
ground truth. For instance, in our evaluations for the distillation/fine-
tuning experiments A, we observe that image captioning models fine-tuned on
the same images but captions following different content distributions behave
differently when validated on differently distributed ‘ground truth labels’.

We are not only suggesting that image captioning involves subjective processes
such as emotion and aesthetic judgement (Mohamed et al. (2022)) or that
language is fluid and can represent the same set of concepts in many different
ways (Luong et al. (2021)). As we reiterate, even the “objective concepts”
within a scene are, at their root, observed by human subjects with particular
perceptual tendencies.

Human-Centered AI and Social Biases

Our work’s turn from simply existing objects to the human subject fits within
the broader research direction of human-centered AI (HAI). HAI strives to
build and evaluate AI models in alignment with human values and behavior
(Tahaei et al. (2023); Taylor et al. (2023)). Importantly, blindly building AI
for “humans” in the abstract can substantively neglect populations which are
not represented under that signifier, and possibly harm algorithmic equity and
fairness goals (Skeem & Lowenkamp (2016); Buolamwini & Gebru (2018);
Hershcovich et al. (2022)). For instance, Berg et al. (2012) argue that
adopting a human-centric view of recognition is essential for developing
practical image and video search applications: “in response to an image search
for “tree”, returning an image with a tree that no person would ever mention
is not desirable.” However, failing to understand diversity in perception
might lead an AI system designer to falsely claim that “no person” would
mention some tree, when in fact the persons who would have simply not been
considered. The English-dominated language modeling landscape may not only
introduce accessibility issues for non-English speakers but also possibly
perceptual bias / hegemony (Kleinberg & Raghavan (2021)). In some ways, our
contribution is similar to that of Caliskan et al. (2022) in showing that
models (and datasets) encode and internalize social stereotypes (in their
case) or content distributions (in our case).

Reframing Language Accessibility

Arguments in favor of directing attention towards multilingual NLP and CV are
often made from an accessibility perspective of expanding information and
technology access to individuals (Hu et al. (2020)) However, models
empirically encounter the “curse of multilinguality” (Conneau et al. (2020)),
in which increasing the number of languages a model handles degrades
monolingual performance after a basic performance threshold. Therefore,
expanding language accessibility is often seen as a trade-off against model
performance and/or resource cost.

This appears to be even more so the case in CV than in NLP. If one accepts the
previously discussed prevailing assumption in CV, then the visual data serves
as an objective field of information which language – even with small
deviations – must be constrained by. Therefore, many efforts to build vision-
language systems for a non-English language $X$ train on translations from
English to $X$ (Mishra et al. (2021); Chen et al. (2022)). Although these
systems are usable in $X$, our investigation suggests that they will still
reflect an English content distribution rather than an “organically $X$”
content distribution.

Our work demonstrates a substantive system benefit to including multilingual
data: increased conceptual richness, as measured both formally and
empirically. Using organically produced multilingual data introduces shifts in
the conceptual distribution which translation cannot. This is an “inversion of
the curse of multilinguality”: rather than expanding image captioning to
multiple languages for accessibility goals at the expense of performance,
incorporating the diverse content distribution may improve model robustness
and grasp of diverse concepts.

Linguistic Structure and Information

What explains the observed differences between the content of monolingual
distributions? There are some potential factors associated with ML landscape
artifacts which cannot be eliminated, including the availability, quality, and
context of data in different languages; the model training or data gathering
procedures across languages; etc. Accounting for these factors, we can still
examine patterns across languages. Particularly, we see the pattern that
European languages (English, German, French, and to some extent Russian) are
distributed more closely to each other than towards Asian languages (Chinese,
Japanese, Korean), as demonstrated in A in multilingual and multimodel
intersections and elsewhere. Interestingly, besides the geographical
proximity, the group similarities fall along shared linguistic typological
(family) features. More research would be required to qualify their
similarities in a more meaningful manner. However, the current observations
aligns with the studies that show the linguistic, cultural, and historical
influence the way we perceive the world (Boroditsky, 2006; Boroditsky et al.,
2003; Lakoff, 1987) and that those influences have a tangible effect on how
the people express through language their perceived world (Langacker, 1993;
Talmy, 1988; Divjak, 2019).

Our experiments may point towards further dialogue on linguistic differences
(‘relativism’) and its relevance in vision-language tasks. Unlike NLP tasks,
which are often open-ended and not strictly tethered to reference material,
vision-language tasks require bridging an open-ended language field with a
constant visual reference field. Indeed, the very structure of language –
grammar, convention, etc. – may incentivize the production of certain
information over others.

In a strictly linguistic sense, translation approximates but does not directly
represent meaning across languages, as word acquires its particular meaning
within a language by its relations to other words within the language (de
Saussure (1960)). Therefore, in this strict sense, the notion of a ‘content
distribution’ without the language confounder is impossible. However,
importantly, we are not as interested in being entirely faithful to the
specific meaning determinations in a specific language as we are in
understanding what can come out of representing uncommon or novel concepts in
another language, even if they do not have exact correspondence with the
original text.

##  Appendix B Limitations and Ethical Concerns

While our paper sheds light on the relationship between language, culture, and
visual perception, it is crucial to recognize and acknowledge its limitations
as well as potential ethical ramifications.

###  B.1 Limitations

Overemphasis on Language. While our study advocates for accounting of diverse
sources of captioning, it relies on and possibly overemphasizes language as
the major axis of diverse sourcing. Our experiments also found that captions
generated by different models showed promise in increased information coverage
and further research in this area can inform how to effectively source diverse
concepts beyond using language as a proxy and a constraint. Multilingual
sources are a sometimes readily available, but underutilized resource that not
only contribute to inclusivity but has the potential to improve vision
systems.

Languages chosen. We used only 7 languages in our study and hence our findings
may not generalize beyond that. Languages were chosen to represent those
spoken in “the West” and “the East” generally, but is still not representative
of global linguistic diversity.

Measurements. Measuring semantic content is subjective, and hence most
measures are merely proxies of actual information coverage. The choice of
metrics we work with (i.e. scene graph, embedding, linguistic complexity) may
also have inherent biases. There could be other ways to evaluate semantic
content that were not considered in this study. Furthermore, the assumption
that an increase in objects, relations, or attributes correlates with better
understanding or improved perception can be questioned.

Scale. Following on from the preceding point, our findings are based on small-
scale experiments. These findings may not scale to large scale models (e.g.,
CLIP), which are often considered the gold standard for image understanding.
We resort to small-scale experiments for several reasons, including 1)
focusing on breadth instead of depth, i.e. curating and analyzing human-
annotated / model-generated image captions in seven languages. 2) focusing on
the science underlying perceptual differences as documented in the literature
of cross-cultural psychology and linguistics. In this paper, we study if the
reported perceptual differences exist for computer vision tasks and how we
might use them to improve information coverage in visual representations.
Future studies may forego a controlled setup like ours to determine the
validity of our findings on a larger scale, i.e. whether CLIP-scale models
differ in their image understanding when trained with captions from different
languages. Previous works have shown promise in this direction by showing how
better / denser (albeit synthetic) captions can help with better image
understanding. (Nguyen et al., 2023; Lai et al., 2023)

Content Translation. The paper relies on translating content (using GPT-4)
from the original language to English in order to use the same robust tools
that are generally only available in English. Moreover, such translation helps
eliminate confounders such as scripts and their issues with tokenization.
However, even with careful human studies to ensure acceptable quality, the
translation process can be noisy, with the potential for errors to propagate.
The translation process is also often considered to be more expensive than
synthetic data, and so may not be economically viable to scale.

Experimental design. Our choices for selection of experiments, especially the
model architecture and training process could influence some of the reported
results. While we made effort to adhere to standard practices in computer
vision, the ideas we propose and experiments we conduct are relatively new to
the field, necessitating new forms of experiments that might be prone to our
inherent biases. Moreeover, we were limited by the small number of currently
accessible multilingual vision-language models to evaluate on and dense
multilingual image captioning datasets to examine. With attention being
increasingly being paid towards building multilingual vision-language models,
such as PALI (Chen et al. (2022)), and high-quality multilingual vision-
language datasets, we anticipate patterns to solidify and new problems to
emerge.

Cultural Interpretation & Relativism The paper makes an assumption that some
of linguistic differences are symptoms of cultural differences, which may
oversimplify the intricate link between language and culture. Furthermore,
several of the findings in the paper allude to language and culture
influencing how people think and function. This proposition is often subject
to heated debates, with linguists and anthropologists opposing determinism
(the stronger form) i.e. ‘Language _shapes_ the way we think’ and supporting
relativism (the lesser version) i.e. ‘Language _affects_ the way we think’.
Linguistic relativism is frequently referred to as the ‘Sapir-Whorf
Hypothesis’. Boroditsky (2006)

###  B.2 Ethical Concerns

Cultural Essentialism. By categorizing languages or cultures based on
differences in image perception and captioning, there’s a risk of
essentializing or stereotyping cultures, suggesting that all members of a
certain culture perceive things a certain way. This may also increase the risk
of researchers seeing languages as merely objects for increasing the
advancement of AI systems, rather than engaging with these languages and
developing multilingual systems that work for people speaking a variety of
languages.

##  Appendix C Translation

We prompted GPT-4 (OpenAI (2023)) to translate text with: “Return the
translation (and only the translation) of the following text from `[SRC_LANG]`
into `[TGT_LANG]` exactly with all details: `[TEXT]`”. We find that this
prompt produces translations which are especially focused on representing the
conceptual details of the original text.

Although language-specific meanings will inevitably lost in any translation
between languages, we ensure that our English translations are as faithful as
possible to the concepts expressed in the original language by conducting a
human evaluation. We recruit 2-3 speakers for each of the six non-English
languages (French, German, Russian, Chinese, Japanese, Korean), fluent in both
the original image and English, from Prolific. Each subject evaluates 30 pairs
of original and translated text. Of these 30 pairs, 10 are Vertex captions on
Crossmodal images, 10 are LLaVA captions on Crossmodal images, and 10 are
Vertex captions on Visual Genome images. This composition ensures wide
coverage across image domains and caption format. Each translation evaluation
has two parts. Firstly, subjects annotate the overall translation quality on a
1 to 5 scale, in which 1 is “entirely inaccurate”, 2 is “some of the
information is preserved”, 3 is “only the most important information is
preserved”, 4 is “most of the information is preserved (the translation is
adequate but not perfect)”, and 5 is “entirely accurate”. Secondly, subjects
examine 11 general categories of concepts in natural visual scenes, provided
by TIFA (Hu et al. (2023)): objects, animals/humans, attributes, activities,
spatial relations, counting, food, materials, shapes, locations and colors.
Subjects mark each category either as “Good” (the concept was present in the
original text and faithfully represented in the translation), “Missed” (the
concept was present in the original text but absent or not faithfully
represented in the translation), or “N/A“ (the concept was not present in the
original text). Table 9 demonstrates that the translations are nearly entirely
accurate, especially for European languages, and preserve nearly all of the
salient content categories for understanding visual scenes.

Table 9: Human evaluations for translation quality using GPT-4 on multilingual captions. TIFA categories represent the mean proportion of non-N/A responses which are marked “Good” (as opposed to “Missed”). Metric | de | fr | ru | zh | ja | ko
---|---|---|---|---|---|---
Quality Ratings | Mean | 4.95 | 4.76 | 4.82 | 4.63 | 4.48 | 4.48
Median | 5.00 | 5.00 | 5.00 | 5.00 | 5.00 | 5.00
25th Percentile | 5.00 | 5.00 | 5.00 | 4.00 | 4.00 | 4.00
TIFA Categories | Objects | 1.00 | 0.99 | 1.0 | 0.97 | 0.98 | 0.90
Animals/Humans | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Attributes | 1.00 | 0.89 | 1.00 | 0.93 | 1.00 | 1.00
Activities | 1.00 | 1.00 | 1.00 | 0.98 | 0.91 | 0.96
Spatial Relations | 1.00 | 1.00 | 1.00 | 0.92 | 0.94 | 0.96
Counting | 1.00 | 1.00 | 1.00 | 0.99 | 1.00 | 0.90
Food | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Material | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Shape | 1.00 | 1.00 | 1.00 | 1.00 | 1.00 | 1.00
Location | 1.00 | 0.98 | 1.00 | 0.98 | 0.96 | 0.89
Color | 1.00 | 0.96 | 1.00 | 0.97 | 1.00 | 1.00

Annotators were allowed to provide free-text explanations for areas in which
the translation was inadequate. We provide a random sampling of comments, to
provide a holistic idea of the translation weaknesses.

Table 10: Example comments providing corrections to translations. Comment
---
$\hookrightarrow$ Should use “above” instead of “on”
$\hookrightarrow$ More appropriate to use ”memories” instead of ”impressions”
$\hookrightarrow$ should be ‘small’ balls (remove ‘round’, add ‘small’)
$\hookrightarrow$ Particle suggests that numbers are written “using” sheet of
paper, not “on” it.
$\hookrightarrow$ “On the side” is translated as “next to”.
$\hookrightarrow$ “toile d’ araignée” can be directly translated to “cobweb”,
it’s a bit excessive to say spider $\&$ spider web in the translation

##  Appendix D Miscellaneous Experiment Details

###  D.1 Probing Multilingual Capabilities in LLaVA

Models like LLaVA which are trained/fine-tuned/aligned with English data but
which include multilingual LLM components can retain some of these
multilingual capabilities. In order to request LLaVA generate captions in a
target language, we change the prompt at all levels to correspond to that
language language, displayed in Table 11.

Table 11: Prompt information for probing multilingual behavior in LLaVA. Prompt Type | Language | Prompt
---|---|---
Roles | English |  (user, assistant)
French |  (utilisateur, assistant)
German |  (Benutzer, Assistent)
… |  …
System | English |  A conversation between a user and an LLM-based AI assistant. The assistant gives helpful and honest answers.
French |  Une conversation entre un utilisateur et un assistant IA basé sur LLM. L’assistant donne des réponses utiles et honnêtes.
German |  Ein Gespräch zwischen einem Benutzer und einem auf LLM basierenden KI-Assistenten. Der Assistent gibt hilfreiche und ehrliche Antworten.
… |  …
User Prompt | English |  What is in this image? Answer in English.
French |  Qu’est-ce qu’il y a dans cette image? Répondez en français.
German |  Was ist auf diesem Bild? Antwort auf Deutsch.
… |  …

We unsuccessfully attempted to probe multilingual capabilities in other models
in a similar fashion, such as the BLIP family (Li et al. (2023a)).

###  D.2 Image Captioning User Study

We recruited 10 English speakers from the US and 10 Japanese speakers from
Japan from Prolific. The instructions given to them are presented in Figures
3(d) and 3(e). A sample of the produced captions is given in Table A.

(d) English instructions.

(e) Japanese instructions.

Figure 4: Instructions and examples presented to human evaluation participants
for image captioning.

###  D.3 Supplementary Normalized Continual Graph Expansion Charts

The continual scene graph expansion as measured by number of objects,
attributes, and relations in Figure 3. This pattern also holds when measuring
by object-standardized measures: number of attributes per object (Figure 4(a))
and number of relations per object (Figure 4(b)).

(a) $\\#$ attributes per object

(b) $\\#$ relations per object

Figure 5: Growth in scene graphs by successively unioning with monolingual
scene graphs in each language.

###  D.4 Pairwise Monolingual Scene Graph Intersections A

Table 12 displays the sizes of intersections between monolingual scene graphs
as measured by the number of objects and relations, using the formula
$M(A)+M(B)-M(A\cup B)=M(A\cap B)$.

Table 12: Sizes of intersections between monolingual unioned scene graphs, listed in the form “number of objects / number of relations”. Sizes listed along the diagonal arc of monolingual graphs and can be used for reference. | en | de | fr | ru | zh | ja | ko
---|---|---|---|---|---|---|---
en | 3.65 / 2.96 | 2.34 / 1.20 | 2.41 / 1.26 | 2.39 / 1.24 | 2.15 / 0.97 | 2.04 / 0.91 | 2.02 / 0.89
de |  | 3.51 / 2.83 | 2.37 / 1.24 | 2.47 / 1.32 | 2.14 / 0.98 | 2.04 / 0.91 | 2.06 / 0.94
fr |  |  | 3.60 / 2.89 | 2.44 / 1.27 | 2.16 / 0.97 | 2.07 / 0.91 | 2.09 / 0.95
ru |  |  |  | 3.86 / 3.2 | 2.25 / 1.06 | 2.13 / 0.98 | 2.12 / 0.96
zh |  |  |  |  | 3.46 / 2.68 | 2.08 / 0.95 | 2.04 / 0.92
ja |  |  |  |  |  | 3.13 / 2.37 | 2.10 / 1.02
ko |  |  |  |  |  |  | 3.18 / 2.47

###  D.5 Representational Diversity for Crossmodal B

The Crossmodal dataset provides at least two captions per language for each
image. We compute the mean coverage as defined in B between pairwise
languages.

Table 13: Language-pairwise mean coverage of embedding space, computed across the Crossmodal dataset. | en | de | fr | ru | zh | ja | ko
---|---|---|---|---|---|---|---
en | 0.38 | 0.43 | 0.41 | 0.44 | 0.46 | 0.44 | 0.45
de | 0.43 | 0.40 | 0.40 | 0.42 | 0.46 | 0.42 | 0.44
fr | 0.41 | 0.40 | 0.38 | 0.42 | 0.45 | 0.42 | 0.42
ru | 0.44 | 0.42 | 0.42 | 0.40 | 0.47 | 0.44 | 0.45
zh | 0.46 | 0.46 | 0.45 | 0.47 | 0.45 | 0.46 | 0.46
ja | 0.44 | 0.42 | 0.42 | 0.44 | 0.46 | 0.42 | 0.42
ko | 0.45 | 0.44 | 0.42 | 0.45 | 0.46 | 0.42 | 0.41

###  D.6 Additional Linguistic Metrics C

Table 14 shows linguistic metrics in the same format as Table 5 for LLaVA
captions, and Table 15 for XM captions. Observe that, in most cases,
resampling from a multilingual distribution increases the mean coverage of
each linguistic measure.

Table 14: Mean coverage across different linguistic measures, as in Table 5. Computed across LLaVA captions.  |  | Monolingual |  |
---|---|---|---|---
|  | en | de | fr | ru | zh | ja | multi | all
| Concreteness | 2.17 | 2.44 | 2.53 | 2.27 | 2.33 | 2.21 | 2.57 | 3.19
| Analytic | 2.85 | 4.64 | 14.72 | 7.53 | 23.05 | 20.86 | 18.09 | 44.48
| Clout | 14.19 | 19.96 | 15.81 | 23.15 | 21.3 | 29.57 | 29.14 | 67.1
| Authentic | 35.97 | 38.01 | 34.33 | 37.20 | 44.63 | 42.92 | 52.59 | 90.58
Metric $M$ | Tone | 6.79 | 9.53 | 16.64 | 16.48 | 11.56 | 9.55 | 16.56 | 45.97
| Color | 18.8 | 26.5 | 17.46 | 12.63 | 17.98 | 13.55 | 24.97 | 69.21
Table 15: Mean coverage across different linguistic measures, as in Table 5. Computed across XM captions.  |  | Monolingual |  |
---|---|---|---|---
|  | en | de | fr | ru | zh | ja | ko | multi | all
| Concreteness | 36.06 | 44.74 | 42.16 | 40.51 | 45.13 | 44.90 | 42.70 | 43.28 | 59.65
| Analytic | 1.88 | 2.31 | 1.62 | 0.76 | 5.02 | 5.28 | 5.13 | 4.66 | 16.56
| Clout | 9.66 | 14.77 | 15.00 | 14.14 | 13.32 | 13.10 | 12.24 | 19.66 | 47.16
| Authentic | 33.21 | 40.09 | 42.22 | 32.35 | 34.02 | 33.83 | 35.28 | 51.87 | 89.05
Metric $M$ | Tone | 8.62 | 12.16 | 9.74 | 10.90 | 10.59 | 9.18 | 9.07 | 13.88 | 40.24
| Color | 11.73 | 34.59 | 24.90 | 27.74 | 48.98 | 36.30 | 16.37 | 40.28 | 94.01

###  D.7 Examples, Multilingual Distributions Exceed Visual Genome D

Table 16 provides examples of Visual Genome images in which multilingual scene
graphs mention objects which are not even documented in Visual Genome, despite
it being a very densely annotated dataset.

Table 16: Examples in which multilingual distributions identify visual features which are not documented in the Visual Genome dataset. Image |  Visual Genome, Documented Objects |  Comments
---|---|---
|  car, street, motorcycle, plate, men, sign, building, hotel, shirt, road, woman, tires, chairs, van, motorbike |  Multilingual scene graphs mention “alleyway”. An alleyway is a negative space formed by walls and the street; while Visual genome mentions both “building”s and the “road”, it does not mark this middle space as an “alleyway“.
|  pane, door, shirt, box, cd, glass pane |  Multilingual scene graphs will often mention a “wheelchair”. This is not recorded in the Visual Genome attributes list.
|  tablet, chair, cushions, laptop bag, sepia triptych, drawstring, screen, room, computer, keyboard, computer keyboard, computer mouse, computer desk, cushion, office chair, window, key, curtain |  Multilingual scene graphs distinguish between and include both “computer”s and “monitor”s – both of which are in the image – whereas there is no such distinction recorded in the Visual Genome dataset.

###  D.8 Finetuning/Distillation Experiments A

Table 17 and Table 18 display evaluations of models finetuned on one language
composition against test data of another language composition, using LLaVA and
XM captions, respectively. Observe that the patterns are similar to those in
Table 7, albeit weaker as expected due to less stability in LLaVA and a
dataset like XM.

Table 17: Evaluations for models fine-tuned on LLaVA captions. Generally speaking, a model fine-tuning on a particular language performs best on that language. |  | Evaluated on
---|---|---
|  | en | de | fr | ru | zh | multi
| en | 0.271 | 0.225 | 0.229 | 0.219 | 0.218 | 0.230
| de | 0.213 | 0.245 | 0.219 | 0.217 | 0.215 | 0.219
| fr | 0.248 | 0.240 | 0.259 | 0.234 | 0.236 | 0.246
| ru | 0.226 | 0.234 | 0.228 | 0.254 | 0.231 | 0.239
Fine-tuned on | zh | 0.199 | 0.202 | 0.199 | 0.207 | 0.247 | 0.216
| multi | 0.239 | 0.233 | 0.234 | 0.233 | 0.235 | 0.244
Table 18: Evaluations for models fine-tuned on XM captions. Generally speaking, a model fine-tuning on a particular language performs best on that language. |  | Evaluated on
---|---|---
|  | en | de | fr | ru | zh | ja | ko | multi
| en | 0.254 | 0.124 | 0.1421 | 0.120 | 0.114 | 0.129 | 0.130 | 0.148
| de | 0.158 | 0.153 | 0.152 | 0.143 | 0.124 | 0.140 | 0.146 | 0.149
| fr | 0.182 | 0.142 | 0.181 | 0.143 | 0.130 | 0.146 | 0.150 | 0.154
| ru | 0.172 | 0.136 | 0.152 | 0.159 | 0.125 | 0.137 | 0.142 | 0.148
| zh | 0.144 | 0.116 | 0.129 | 0.120 | 0.124 | 0.130 | 0.142 | 0.130
Fine-tuned on | ja | 0.144 | 0.128 | 0.137 | 0.125 | 0.124 | 0.154 | 0.144 | 0.135
| ko | 0.151 | 0.116 | 0.131 | 0.116 | 0.115 | 0.134 | 0.159 | 0.134
| multi | 0.179 | 0.140 | 0.153 | 0.145 | 0.131 | 0.149 | 0.151 | 0.151