# mOSCAR: A Large-scale Multilingual
and Multimodal Document-level Corpus

Matthieu Futeral  1,2 Armel Zebaze1,4 Pedro Ortiz Suarez5 Julien Abadji1 Rémi
Lacroix3, 6 Cordelia Schmid1,2 Rachel Bawden1 Benoît Sagot1

1Inria 2Département d’informatique de l’ENS, CNRS, PSL Research University
3Institut du développement et des ressources en informatique scientifique,
CNRS
4Sorbonne Université, Paris, France 5Common Crawl Foundation 6Université
Paris-Saclay
Correspondence to<EMAIL_ADDRESS>
###### Abstract

Multimodal Large Language Models (mLLMs) are trained on a large amount of
text-image data. While most mLLMs are trained on caption-like data only,
Alayrac et al. (2022) showed that additionally training them on interleaved
sequences of text and images can lead to the emergence of in-context learning
capabilities. However, the dataset they used, M3W, is not public and is only
in English. There have been attempts to reproduce their results but the
released datasets are English-only. In contrast, current multilingual and
multimodal datasets are either composed of caption-like only or medium-scale
or fully private data. This limits mLLM research for the 7,000 other languages
spoken in the world. We therefore introduce mOSCAR, to the best of our
knowledge the first large-scale multilingual and multimodal document corpus
crawled from the web. It covers 163 languages, 315M documents, 214B tokens and
1.2B images. We carefully conduct a set of filtering and evaluation steps to
make sure mOSCAR is sufficiently safe, diverse and of good quality. We
additionally train two types of multilingual model to prove the benefits of
mOSCAR: (1) a model trained on a subset of mOSCAR and captioning data and (2)
a model train on captioning data only. The model additionally trained on
mOSCAR shows a strong boost in few-shot learning performance across various
multilingual image-text tasks and benchmarks, confirming previous findings for
English-only mLLMs. The dataset can be accessed here.111https://oscar-
project.github.io/documentation/versions/mOSCAR/

##  1 Introduction

Multimodal large language models (mLLMs) are trained on large amounts of text-
image data (Radford et al., 2021; Yu et al., 2022; Li et al., 2023; Wang et
al., 2023; OpenAI, 2023; Gemini Team et al., 2023; Chameleon Team, 2024). The
main paradigm until recently was to train a model from a large collection of
web-crawled images and their captions (Li et al., 2021; Wang et al., 2022;
Chen et al., 2023b). Models such as Flamingo (Alayrac et al., 2022) challenged
this paradigm by being additionally trained on interleaved sequences of text
and images from web documents, showing state-of-the-art results on various
tasks and in-context learning capabilities that are not present in models
trained on caption-like data only. Additionally, McKinzie et al. (2024)
recently proved that including interleaved text-image data during training was
necessary to get good few-shot learning performance. However, the datasets
used to train mLLMs are either private (Alayrac et al., 2022), monolingual or
multilingual but only medium-scale (Srinivasan et al., 2021). Some attempts
have been made to reproduce these datasets (Zhu et al., 2023; Laurençon et
al., 2023) but the resulting datasets are only available in English.

Few image-text datasets are multilingual and most of them are obtained by
translating English caption-like datasets, such as multilingual Conceptual
Captions (Sharma et al., 2018), into multiple languages using neural machine
translation (NMT) systems (Surís et al., 2022; Maaz et al., 2024). This
presents some drawbacks such as some languages still being poorly translated
by current state-of-the-art NMT models (Liu et al., 2020; Costa-jussà et al.,
2022) and some cultural subtleties inherent in each language not being fully
conveyed. Some efforts have been conducted to collect large-scale multilingual
image captioning datasets, such as LAION-5B (Schuhmann et al., 2022), but they
are limited to caption data too, are relatively noisy and more importantly
contain a non-negligible share of “not safe for work” (NSFW) content such as
pædopornographic images (Schuhmann et al., 2022).

This motivated us to collect and release the first large-scale multilingual
and multimodal document dataset derived from Common
Crawl.222https://commoncrawl.org/. The Common Crawl Foundation is a non-profit
organization that crawls the web on a monthly basis.  Our dataset, multimodal
OSCAR (mOSCAR), follows the OSCAR initiative (Ortiz Suárez et al., 2019;
Abadji et al., 2021, 2022) and covers 315M documents in 163 languages, 214B
tokens and 1.2B images. Figure 1 shows an example of a document, more can be
found in Appendix A.3. We carry out extensive filtering to increase its safety
and quality. To prove mOSCAR’s utility, we train a multilingual OpenFlamingo
(Awadalla et al., 2023) from a Gemma-2B language model (Gemma Team et al.,
2024) on a subset of mOSCAR and captioning data from LAION-400M (Schuhmann et
al., 2021), recaptioned with BLIP (Li et al., 2022), filtered with CLIP
(Radford et al., 2021) and translated with NLLB (Costa-jussà et al., 2022). We
compare against a similar model trained on captioning data only and show we
obtain a strong boost in few-shot learning, confirming previous findings for
English (Alayrac et al., 2022; McKinzie et al., 2024; Laurençon et al., 2024).
The dataset and models will be made publicly available.

Figure 1: Example of a French document from mOSCAR.

##  2 Related Work

##### Large-scale web-based datasets

Numerous datasets have been created by filtering web-crawled data. These
include large-scale text-only datasets (Ortiz Suárez et al., 2019; Raffel et
al., 2020; Wenzek et al., 2020; Gao et al., 2020; Abadji et al., 2021; Xue et
al., 2021; Laurençon et al., 2022; Abadji et al., 2022; Penedo et al., 2023)
and multimodal ones (Sharma et al., 2018; Changpinyo et al., 2021; Jia et al.,
2021; Schuhmann et al., 2021, 2022; Byeon et al., 2022; Laurençon et al.,
2023; Zhu et al., 2023; Gadre et al., 2024). Even if these datasets are not as
high quality as smaller and/or hand-crafted ones, they are now the standard to
pretrain foundation models, as it has been shown that training bigger models
on more data leads to better downstream performances (Brown et al., 2020;
Hoffmann et al., 2022; Touvron et al., 2023a, b).

##### English image-text datasets

The first open-source image-text datasets were manually created, small-scale
and English-only (Ordonez et al., 2011; Lin et al., 2014; Plummer et al.,
2015; Krishna et al., 2017). Scaling up these datasets was an appealing
solution to overcome limitations of previous image-text models; a few works
(Sharma et al., 2018; Changpinyo et al., 2021) proposed to collect millions of
image-text pairs from the web before filtering them with well-designed steps.
Relaxing the filtering steps enabled the collection of more data and led to
large-scale datasets to train image-text foundation models (Radford et al.,
2021; Li et al., 2021; Schuhmann et al., 2021, 2022; Byeon et al., 2022).
However, these datasets generally contain caption-like image-text pairs only,
and it is therefore difficult to observe in-context learning abilities
similarly to text-only language models trained on raw documents (Raffel et
al., 2020). Alayrac et al. (2022) overcome this issue by training their model
directly on documents with interleaved image-text data. While their results
are promising, their M3W dataset is English-only and private. Recently, open-
source efforts (Zhu et al., 2023; Laurençon et al., 2023) have been made to
release a similar dataset but they are still monolingual.

##### Multilingual image-text datasets

Only a few image-text datasets are available in multiple languages. One of the
first focused on collecting Google images from short queries based on word
frequencies from Wikipedia pages in 98 languages (Hewitt et al., 2018). Later,
Srinivasan et al. (2021) proposed the WIT dataset, an image-text dataset
composed of Wikipedia pages. Although of high quality, it is only medium-scale
even for high-resource languages and there are fewer than 50k unique images
for most languages. Another approach lies in bootstrapping multilingual and
multimodal data from a model trained with English-only data (Mohammed et al.,
2023). While effective for captioning, it is computationally expensive to
implement in practice. Other multilingual image-text datasets exist but focus
on captions only and are highly domain-specific (Kosar et al., 2022; Leong et
al., 2022).

##  3 Dataset Creation Pipeline

###  3.1 Data collection

We collect mOSCAR from the Web ARchive Content (WARC) files of three 2023
Common Crawl dumps, processing them using the FastWARC library (Bevendorff et
al., 2021). We remove documents smaller than 500 bytes (50% of the documents),
as we find they are usually too small to be considered documents and tend to
contain noisy text. We then navigate through the entire Document Object Model
(DOM) tree with a depth first search algorithm and ChatNoir library
(Bevendorff et al., 2018) to extract nodes of interests corresponding to
specific HTML tags.

Following previous work, we extract text from the tags that usually contain
the main content of web pages (we refer to them as DOM text nodes), i.e. <p>,
<h*>, <title>, <description>, <ul>, <ol>, <aside>, <dl>, <dd>, <dt>. Similarly
to (Laurençon et al., 2023), we choose to remove <table> content as most often
it is irrelevant and difficult to render. We extract all <img> tags (we refer
to them as DOM image nodes). We then remove documents with fewer than 3 text
nodes (as they do not contain enough text) and more than 30 image nodes (as we
found them to be too noisy).

###  3.2 Language identification

We identify the language of each document using the state-of-the-art open-LID
language detector (Burchell et al., 2023), covering 201 languages. We apply
open-LID to each DOM text node and keep the three most probable languages with
their respective probabilities. The language of the document is then
determined by summing over the probabilities of each language detected for
each text segment, weighted by the number of characters in the segment333This
is to avoid mis-assigning the language due to the presence of many short, non-
informative DOM text nodes in the same language (e.g. “Cookies”, “Subscribe”,
“Newsletter” etc.) and because language identification is generally less
reliable for short segments. and taking the language with the highest score.

###  3.3 Text-only filtering

We apply a series of filtering steps to the text content of each document
independently of the images, with the aim of discarding poor quality documents
and cleaning text as best as possible. We first filter at the text-node level
and then at the whole document level, before running near-deduplication to
keep unique text nodes within a document and unique documents in the dataset.

##### Text node filtering

We use a set of heuristics (see Appendix A.2) to extract as much human-
generated content as possible while discarding noisy text related to ads and
website functions (e.g. “Instagram”, “Facebook”). We then keep DOM text nodes
with content over 10 bytes. This step, designed to improve the quality of
extracted text, removes on average 55% of text nodes.

##### Document filtering

We mostly filter “not safe for work” (NSFW) content at the document level. We
use an English regular expression to detect adult content, similar to the one
used by the Université Toulouse 1 Capitole444https://dsi.ut-
capitole.fr/blacklists/index_en.php and remove the entire document if there is
a match with any of the DOM text nodes’ contents, removing on average 0.5% of
documents (mostly English ones). We acknowledge that there is a high
probability that this also discards safe content, e.g. we could remove content
from certain communities who use some explicit words in a non-sexual way (Sap
et al., 2019). However, we explicitly favour recall over precision to minimise
the risk of unsafe content. We additionally remove documents containing fewer
than five DOM text nodes and fewer than 300 characters after the previous
filtering steps, removing 70.6% of documents.

##### Deduplication

We conduct several types of per-language deduplication at different levels, as
this has been shown to improve training efficiency (Abbas et al., 2023).
First, we keep unique documents only by removing exact duplicates at the
document level. We also remove exact duplicates of text nodes within the same
document (4% of text nodes) and near-duplicate text nodes (1% of text nodes)
by computing the Levenshtein ratio (Levenshtein, 1966) between all text nodes
within the same document and applying a threshold of 0.95. If near-duplicates
are found, we keep the first one in the document. Finally, we conduct per
language near-deduplication at the document level with MinHashLSH (Broder,
1997; Gionis et al., 1999) following Smith et al. (2022), removing on average
19% of documents:555With some disparity among languages as we found more
duplicates for low- than high-resource languages. we turn documents into
hashing vectors, compute min hashes from these vectors and perform Locality
Sensitive Hashing to remove duplicates666We performed this using the
datasketch python library. (see Appendix A.5 for more details).

###  3.4 Image-only filtering

We downloaded images from the URLs in DOM image nodes using a modified version
of the img2dataset toolkit (Beaumont, 2021) that includes an antivirus scan
and follows robots.txt instructions to respect the Robots Exclusion Protocol.
We then apply a series of filtering steps, first removing images based on
heuristics, and then applying multiple NSFW detection models to remove
undesirable content. Finally, we conduct a set of deduplication steps.

##### Rule-based filters

Similarly to previous works (Schuhmann et al., 2021) and to avoid extracting
low-resolution images and favicons, we keep images with a minimum height and
width of 150 pixels. We restrict the aspect ratio to be between 3 and 1/3 (to
remove banners), we remove images if their URLs contain the words “logo”,
“banner”, “button”, “widget”, “icon” or “plugin” or if the image name from the
URL matches “twitter”, “facebook” or “rss” (to remove logos). This step
removes 13.6% of the URLs. At this stage, we downloaded 2.5B images with an
average success rate of 55%.

##### NSFW detection

We use multiple NSFW automatic models to remove as much unsafe content as
possible. We first combine two NSFW detectors: nsfw-detector (Laborde, ), a
5-class classifier with a MobileNet (Howard et al., 2017) backbone fine-tuned
on 60GB of annotated data and NudeNet,777https://github.com/vladmandic/nudenet
an object detector trained to detect different types of nudity in images. We
combined the two models as we found the first to be gender-biased while the
second gives a large number of false positives for non-human images.
Concretely, we consider an image an NSFW candidate if the sum of the
probabilities for the classes ‘porn’ and ‘hentai’ is superior to 0.8 using
nsfw-detector. We then tag the image as NSFW if one of the sensitive ‘exposed’
classes of NudeNet gets a probability superior to 0.5. We additionally use
Safer by Thorn888https://safer.io/, a private pornography detector, and tag
the image as NSFW if the probability of the class ‘pornography’ is superior to
0.8. If a document contains an image with an NSFW tag, we remove the entire
document from the dataset, which removes 0.5% of images. We manually
inspecting 1,000 images of the remaining data and found no NSFW content. We
manually inspected 1,000 images of the removed content and found 63.4% of NSFW
images.

##### CSAM content

Child Sexual Abuse Material (CSAM) is widespread on the internet and is
therefore likely to be found in such a large-scale dataset crawled from the
web. Removing CSAM is challenging as there is no training data nor open-source
detection models available as these could be used in a harmful way. We again
rely on Safer, a proprietary 3-class classifier trained to detect CSAM and
pornography content from images. We tag the image as CSAM if the probability
of the class CSAM is superior to 0.4 to favour recall over precision. As
mentioned above, if a document contains an image with a CSAM tag, we remove it
from the dataset. This step removes 0.07% of the images.

##### Deduplication

To avoid memorisation issues often seen in models trained on datasets with
many duplicated images (Somepalli et al., 2023; Carlini et al., 2023; Webster
et al., 2023; Somepalli et al., 2024), we perform deduplication at the image
level. We first remove duplicate images within the same document by URL
matching (removing 8.7% of URLs). We then compute a perceptual hash (pHash)
for each image using the imagehash
library999https://github.com/JohannesBuchner/imagehash and remove images with
the same pHash within the same document, keeping only the first occurrence. We
also limit the number of times an image can appear in the dataset per-language
to 10 using both URL matching and perceptual hashing (this removes 2.5% of
images). We do this per-language and not across languages as having the same
images in documents from different languages could encourage cross-lingual
transfer.

###  3.5 Data decontamination

LLMs and mLLMs are trained on web-crawled data that can contain the benchmarks
they are tested on (Dodge et al., 2021). As they are good at memorizing
training data (Carlini et al., 2023), this data contamination is problematic.
We therefore discard all images with the same perceptual hash as any of the
images from the evaluation benchmarks (and their training sets) we use (see
Section 5.1). This step removes on average 126,016 images for high-resource
languages (up to 300K images for English), 6,862 images for mid-resource
languages and 45 images for low-resource languages.

###  3.6 Text-image joint filtering

Our aim is to obtain truly multimodal documents where all images are related
to at least one of the text nodes in some way101010We do not limit ourselves
to caption-like relation and instead allow all types of text-image relation.
and vice versa. We choose to apply joint text-image filtering to discard
images and/or text nodes that are irrelevant to the rest of the document (e.g.
the case of ads and website functionalities). To do this, we use NLLB-
SIGLIP111111siglip-base-patch16-224 as vision encoder and nllb-distilled-600M
as text encoder. (Visheratin, 2023), a multilingual version of SIGLIP (Zhai et
al., 2023) trained with the encoder of NLLB (Costa-jussà et al., 2022), which
covers all mOSCAR languages.121212We use the open-clip (Ilharco et al., 2021)
model version and the transformers (Wolf et al., 2020) library. We compute
cosine similarity scores between all images and all paragraphs131313We refer
to paragraph as the text content in a DOM text node. within a same document.
To remove irrelevant text nodes or images in a document, we mimic a text-image
retrieval task, which means we avoid using arbitrary cosine similarity
thresholds for each language and can reduce length biases and those in favour
of caption-like paragraphs. For each candidate pair we randomly sample 63
negative images and 63 negative similar-length paragraphs from the same
language but other documents. We tag the text node (resp. image) as valid if
the cosine similarity of the pair is among the top 8 of the text-to-image
(resp. image-to-text) similarity scores computed with the candidate text node
(resp. image) and all the negative images (resp. text nodes). This means that
we tag the text node (resp. image) as valid if it has a significantly higher
score than a score computed with a random image (resp. text) for at least one
of the images (resp. text node) in the document. We then discard text nodes
and images not tagged as valid (on average 35% of the DOM text nodes and 10%
of the images within a document). After this filtering step, we apply
additional text-only filters to keep documents superior to 100 bytes.

##  4 Multimodal Open Super-large Crawled Aggregated coRpus (mOSCAR)

mOSCAR is extracted from three Common Crawl dumps from 2023. Due to
computational constraints and in order to extract a maximum number of
documents for low-resource languages, we extracted all languages from the
first dump only. We removed the 6 most high-resource languages from the second
dump and only extracted the languages with fewer than 1M documents for the
last dump. Table 1 shows a distribution of the total number of languages and
their number of documents.

To avoid data poisoning (Carlini et al., 2024), we release a hash (sha512)
with each mOSCAR image.

#documents | 10M | 5M | 1M | 500K | 200K | 50K | 10K | 5K | 1K
---|---|---|---|---|---|---|---|---|---
#languages | 10 | 15 | 38 | 49 | 58 | 82 | 129 | 142 | 163
Table 1: Number of languages with at least $N$ documents

(a) Number of images

(b) Number of tokens

(c) Number of images and tokens

Figure 2: Distributions of numbers of tokens and images per document

mOSCAR is composed of 315M documents (214B tokens, 1.2B images) from 163
languages. Figure 2 shows the distribution of images and tokens per document
and their joint distribution. As shown in Figure 2(a), the mean and median
number of images per document is 2 and 3.80.

###  4.1 Quality vs Diversity

While improving overall data quality, the filtering steps we applied (see
Section 3) necessarily have a negative impact on diversity. We therefore study
the trade-off between quality and diversity and compare against previously
published, well-used datasets.

####  4.1.1 Text content

##### Diversity

By contruction, mOSCAR is diverse in terms of number of languages, so we focus
on the diversity of mOSCAR’s English documents and compare against mmc4 (Zhu
et al., 2023), OBELICS (Laurençon et al., 2023) and the English subset of WIT
(Srinivasan et al., 2021). We compute the Vendi score (Friedman and Dieng,
2023) on a set of SimCSE embeddings (Gao et al., 2021) with a RoBERTa encoder
(Liu et al., 2019) to evaluate the content diversity. Since embedding-based
diversity metrics target content diversity well but are less relevant for
lexical diversity (Tevet and Berant, 2021), we measure lexical diversity via
the distinct $n$-gram ratio (Li et al., 2016).

##### Comparison with other datasets

Figure 3: Perplexity of 100K random documents from different datasets.

| Vendi score | Dist. $n$-gram ratio
---|---|---
mOSCAR |  69.05 ($\pm$ 0.14) | 0.472 ($\pm$ 0.002)
mmc4 | 67.93 ($\pm$ 0.12) |  0.494 ($\pm$ 0.002)
OBELICS | 58.49 ($\pm$ 0.09) | 0.488 ($\pm$ 0.001)
WIT |  73.30 ($\pm$ 0.09) |  0.530 ($\pm$ 0.001)
Table 2: Average text diversity scores ($\pm$ standard error) of text
documents.

For content diversity, we randomly sample 30M documents for mOSCAR, mmc4 and
OBELICS and 3M documents for WIT and represent the documents by their SimCSE
embedding. We compute the Vendi Score with cosine similarity on a randomly
sampled subset of 65,536 documents. Table 2 shows that mOSCAR English content
is more diverse than mmc4 and OBELICS but less diverse than WIT. For lexical
diversity, we randomly sample 3M documents for mOSCAR, mmc4, OBELICS and WIT
and compute the distinct $n$-gram ratio on a subset of 8,192 documents for $n$
from 1 to 4. Table 2 shows that mOSCAR is slightly less lexically diverse than
OBELICS and mmc4, while WIT is by far the most diverse.

##### Quality

To evaluate document quality, we focus on English documents and compute their
perplexity using Gemma-2B (Gemma Team et al., 2024). Figure 3 shows the kernel
density estimation of the distribution of the perplexity of 100K randomly
sampled documents from different datasets: mOSCAR is comparable to mmc4 and
WIT, while OBELICS appears to be the of the highest quality. mOSCAR is
therefore comparable to other interleaved image-text dataset in terms of
quality and diversity of its English subset. It is however more diverse than
English-only datasets by its multilingual construction and more than 10 times
larger than existing multilingual interleaved image-text datasets such as WIT.

####  4.1.2 Image diversity

##### Comparison with other datasets

mOSCAR | LAION-400M | WIT
---|---|---
55.74 ($\pm$ 0.16) |  67.59 ($\pm$ 0.16) | 36.14 ($\pm$ 0.08)
(a) Comparison of different datasets.

English | All
---|---
52.36 ($\pm$ 0.18) |  54.78 ($\pm$ 2.29)
(b) mOSCAR (English vs. any language).

Table 3: Average Vendi score ($\pm$ standard error) of images.

We compute the Vendi Score on random samples of images for different datasets,
comparing the images from English mOSCAR documents with those from Conceptual
Captions (Changpinyo et al., 2021), LAION-400M (Schuhmann et al., 2021) and
WIT (Srinivasan et al., 2021). We represent each image by its SigLIP141414We
use siglip-base-patch16-224. (Zhai et al., 2023) embedding and compute the
Vendi score on batches of size 65,536 and a total of 1M images for each
dataset. In Table 3(a), we notice that the set of images in mOSCAR documents
are more diverse than images from WIT documents but less diverse than
LAION-400M.

##### Multilingual diversity

We also compare the diversity of images from English documents and of images
sampled from documents of any language (English included). We use multilingual
SigLIP (Chen et al., 2023a) trained on WebLI (Chen et al., 2023b) to compute
image embeddings used to get the Vendi score. We again use a batch of size
65,536 and a total of 3M images, and we do not sample multiple images from a
same document. For the multilingual setting, we randomly sample 50 languages
and an equal number of images for each language to build the batch. As we did
not do any image deduplication across languages, we could expect to have less
diversity in the multilingual setting. However, Table 3(b) shows that the set
of images is on average more diverse when sampled from all documents than from
English-only documents. This means that the distribution of images is not
exactly the same across languages, potentially due to cultural differences.

##  5 Training a multilingual multimodal language model

We train a multilingual Flamingo-like model on mOSCAR. As adding captioning
data to training data has been shown to improve zero-shot performance
(McKinzie et al., 2024), we additionally train on LAION-400M, which we re-
captioned using BLIP (Li et al., 2022), filtered with CLIP score (Radford et
al., 2021) and translated using distilled NLLB-600M (Visheratin, 2023)
following the proportion of languages found in mOSCAR. We use Gemma-2B (Gemma
Team et al., 2024) as the underlying language model and we train the model on
35M mOSCAR documents and 70M randomly sampled image-text pairs. We also train
a model on 300M image-text pairs as a comparison baseline. We additionally
compare with OpenFlamingo-3B-MPT (Awadalla et al., 2023) as the translate-test
baseline. The full list of languages for training and the implementation
details can be found in Appendix A.5.

###  5.1 Evaluation setup

We evaluate the models using a broad set of image-text multilingual tasks and
benchmarks. We use the IGLUE benchmark (Bugliarello et al., 2022) composed of
XVNLI, MaRVL (Liu et al., 2021) to test reasoning, xGQA (Pfeiffer et al.,
2022) to test visual question answering capabilities and xFlickr&CO (Young et
al., 2014; Karpathy and Fei-Fei, 2015; Yoshikawa et al., 2017) for captioning.
We also include Crossmodal-3600 (XM3600) (Thapliyal et al., 2022) and MaXM
(Changpinyo et al., 2022) as they cover a broader range of languages. To test
to what extent models trained on mOSCAR can perform zero-shot multimodal
machine translation (MMT), we also test on Multi30K (Elliott et al., 2016,
2017; Barrault et al., 2018) and CoMMuTE (Futeral et al., 2023). For
captioning we compute the CideR (Vedantam et al., 2015) score and we tokenize
references and model outputs with the Stanford Core NLP tokenizer for English
and Stanza (Qi et al., 2020) tokenizers for other languages. To evaluate
Multi30k, we compute BLEU (Papineni et al., 2002) score from Sacrebleu (Post,
2018) with 13a tokenization and default parameters. We use accuracy for
CoMMuTE. More details can be found in Appendix A.5.3.

###  5.2 Results

Tables 4 and 5 show the average results across all languages. Full results are
available in Appendix A.6. We notice that the multilingual OpenFlamingo
trained additionally on mOSCAR gets better results than the model trained on
captioning data only while having seen fewer image-text pairs during training.
More importantly, when increasing the number of few-shot examples from 0 to
16, it sees gains of on average +8.19 points on VQA benchmarks and +16.07
CideR points on captioning benchmarks. In contrast, the model trained on text-
image pairs only sees gains of +2.82 and +9.08 points respectively. In cross-
modal machine translation, the model additionally trained on interleaved data
is again far better than the one trained on just captioning data, which is not
able to translate the Multi30k benchmark at all.151515Most of the time, the
model is not able to follow the prompt and only outputs the end of sequence
token. Moreover, mOSCAR helps the model to learn to zero-shot disambiguate
translations as shown by the improved average score on CoMMuTE (63.75)
compared to the model trained on captions only (61.36).

Multilingual Open Flamingo trained on mOSCAR & text-image pairs is also better
than OpenFlamingo 3B MPT evaluated on translate test benchmarks. However, we
obtain the best results (except for MaXM) by evaluating our multilingual Open
Flamingo on the translate-test benchmarks since the underlying language model
(Gemma-2B) is far better in English than other languages. We also notice that
all models struggle with reasoning classification tasks (MaRVL, XVNLI) where
they obtain scores close to random guessing.

|  | xFlickR&CO | XM3600 | xGQA | MaXM | MaRVL | XVNLI | Multi30K | CoMMuTE
---|---|---|---|---|---|---|---|---|---
| #shots
Multi. OF | 0 | 19.07 | 8.73 | 25.08 | 19.64 | 49.77 | 33.01 | 22.70 | 63.75
4 | 34.32 | 20.59 | 31.90 | 23.90 | 49.67 | 36.07 | 22.79 | 63.65
full | 8 | 36.77 | 22.15 | 33.9 | 24.41 | 49.72 | 37.16 | 23.21 | 63.00
| 16 | 37.63 | 22.24 | 35.71 | 25.38 | 49.73 | 35.36 | 23.48 | 62.77
Multi. OF | 0 | 9.57 | 4.21 | 8.62 | 4.01 | 49.88 | 33.76 |  00.00 | 61.36
4 | 13.20 | 9.26 | 13.45 | 4.15 | 49.54 | 32.04 |  00.00 | 61.13
cap. only | 8 | 18.00 | 10.35 | 12.82 | 4.88 | 49.65 | 33.71 |  00.01 | 60.90
| 16 | 19.87 | 12.07 | 13.37 | 4.89 | 49.79 | 32.70 |  00.74 | 60.25

Table 4: Results averaged over all languages.

|  | xGQA | MaXM | MaRVL | XVNLI
---|---|---|---|---|---
| #shots
OF-3B MPT | 0 | 18.34 | 7.68 | 49.75 | 32.73
4 | 22.97 | 7.82 | 49.70 | 35.82
8 | 28.57 | 8.32 | 49.71 | 31.29
16 | 31.82 | 9.04 | 49.72 | 33.29
Multi. OF | 0 | 30.16 | 10.06 | 49.93 | 34.66
4 | 35.55 | 9.89 | 48.99 | 36.10
full | 8 | 36.78 | 10.12 | 50.54 | 39.69
| 16 | 37.75 | 11.49 | 49.57 | 37.97

Table 5: Translate-test results averaged over languages.

Figure 4: Score differences averaged over benchmarks and languages between the
model trained on mOSCAR + text-image pairs and the model trained only on text-
image pairs.

We additionally compare results at different training steps, defined by the
number of images seen during training. Figure 4 shows the difference of
averaged scores between the model trained on all data and the model trained
only on text-images pairs. We notice that the gap first decreases until 20M
images seen and keep increasing over training at all training steps after
that. Particularly, the gap is wider for few-shot learning.

##  6 Conclusion, Limitations and Societal Impacts

We introduce mOSCAR, a large-scale multilingual and multimodal dataset
covering 163 languages and composed of 315M documents, 214B tokens and 1.2B
images. We show that mOSCAR is of good quality, diverse and can be used to
train a multilingual and multimodal LLM. We ensure that mOSCAR is as safe as
possible by applying a series of filtering steps to remove NSFW content. We
however did not conduct any toxicity analysis or evaluate its biases as this
is challenging in a multilingual setting. As it is crawled from the internet,
it is possible that mOSCAR reflects biases widespread on it. Nevertheless, by
its multilingual nature, mOSCAR is a step towards the inclusion of more
languages, cultures and people in accessing mLLMs.

## Acknowledgements

This work was granted access to the HPC resources of IDRIS under the
allocation 2024-AD011014232R1, 2023-AD011014232 and 2023-AD011012254 made by
GENCI. It was also partly funded by the last three authors’ chairs in the
PRAIRIE institute funded by the French national agency ANR as part of the
“Investissements d’avenir” programme under the reference ANR-19- P3IA-0001. We
deeply thanks the Jean-Zay support team. We also thanks Filip Šedivý for
insightful discussions regarding the removal of CSAM, Thorn for having
provided access to their CSAM detector, Zeeshan Khan for discussions regarding
the training of the models and Victoria Le Fourner for having manually checked
subsamples of NSFW images.

## References

  * Abadji et al. [2021] Julien Abadji, Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot.  Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus.  In Harald Lüngen, Marc Kupietz, Piotr Bański, Adrien Barbaresi, Simon Clematide, and Ines Pisetta, editors, _Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9)_ , pages 1–9, Limerick, 2021. Leibniz-Institut für Deutsche Sprache.  doi: 10.14618/ids-pub-10468.  URL https://nbn-resolving.org/urn:nbn:de:bsz:mh39-104688.
  * Abadji et al. [2022] Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot.  Towards a cleaner document-oriented multilingual crawled corpus.  In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, and Stelios Piperidis, editors, _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4344–4355, Marseille, France, June 2022. European Language Resources Association.  URL https://aclanthology.org/2022.lrec-1.463.
  * Abbas et al. [2023] Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication.  _arXiv preprint arXiv:2303.09540_ , 2023.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Awadalla et al. [2023] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  _arXiv preprint arXiv:2308.01390_ , 2023.
  * Barrault et al. [2018] Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank.  Findings of the third shared task on multimodal machine translation.  In _Proceedings of the Third Conference on Machine Translation: Shared Task Papers_ , pages 304–323, 2018.
  * Beaumont [2021] Romain Beaumont.  img2dataset: Easily turn large sets of image urls to an image dataset.  https://github.com/rom1504/img2dataset, 2021.
  * Bevendorff et al. [2018] Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast.  Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl.  In Leif Azzopardi, Allan Hanbury, Gabriella Pasi, and Benjamin Piwowarski, editors, _Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)_ , Lecture Notes in Computer Science, Berlin Heidelberg New York, 2018. Springer.
  * Bevendorff et al. [2021] Janek Bevendorff, Martin Potthast, and Benno Stein.  Fastwarc: optimizing large-scale web archive analytics.  _arXiv preprint arXiv:2112.03103_ , 2021.
  * Broder [1997] A.Z. Broder.  On the resemblance and containment of documents.  In _Proceedings of the Compression and Complexity of Sequences 1997_ , pages 21–29. IEEE Computer Society, 1997.  doi: 10.1109/SEQUEN.1997.666900.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Bugliarello et al. [2022] Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić.  Iglue: A benchmark for transfer learning across modalities, tasks, and languages.  In _International Conference on Machine Learning_ , pages 2370–2392. PMLR, 2022.
  * Burchell et al. [2023] Laurie Burchell, Alexandra Birch, Nikolay Bogoychev, and Kenneth Heafield.  An open dataset and model for language identification.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 865–879, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-short.75.  URL https://aclanthology.org/2023.acl-short.75.
  * Byeon et al. [2022] Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim.  Coyo-700m: Image-text pair dataset.  https://github.com/kakaobrain/coyo-dataset, 2022.
  * Carlini et al. [2023] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting training data from diffusion models.  In _Proceedings of the 32nd USENIX Conference on Security Symposium_ , SEC ’23, USA, 2023. USENIX Association.  ISBN 978-1-939133-37-3.
  * Carlini et al. [2024] Nicolas Carlini, Matthew Jagielski, Christopher Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr.  Poisoning web-scale training datasets is practical.  In _Proceedings of the 2024 IEEE Symposium on Security and Privacy (SP)_ , pages 179–179, Los Alamitos, CA, USA, 2024. IEEE Computer Society.  doi: 10.1109/SP54263.2024.00179.  URL https://doi.ieeecomputersociety.org/10.1109/SP54263.2024.00179.
  * Changpinyo et al. [2021] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3558–3568, Nashville, TN, USA, 2021.
  * Changpinyo et al. [2022] Soravit Changpinyo, Linting Xue, Idan Szpektor, Ashish V Thapliyal, Julien Amelot, Michal Yarom, Xi Chen, and Radu Soricut.  Maxm: Towards multilingual visual question answering.  _arXiv preprint arXiv:2209.05401_ , 2022.
  * Chen et al. [2023a] Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al.  Pali-3 vision language models: Smaller, faster, stronger.  _arXiv preprint arXiv:2310.09199_ , 2023a.
  * Chen et al. [2023b] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al.  Pali: A jointly-scaled multilingual language-image model.  In _Proceedings of the International Conference on Learning Representations_ , Kigali, Rwanda, 2023b.
  * Costa-jussà et al. [2022] Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al.  No language left behind: Scaling human-centered machine translation.  _arXiv preprint arXiv:2207.04672_ , 2022.
  * Dodge et al. [2021] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner.  Documenting large webtext corpora: A case study on the colossal clean crawled corpus.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1286–1305, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.98.  URL https://aclanthology.org/2021.emnlp-main.98.
  * Elliott et al. [2016] Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia.  Multi30k: Multilingual english-german image descriptions.  In _Proceedings of the 5th Workshop on Vision and Language_ , pages 70–74. Association for Computational Linguistics, 2016.  doi: 10.18653/v1/W16-3210.  URL http://www.aclweb.org/anthology/W16-3210.
  * Elliott et al. [2017] Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia.  Findings of the second shared task on multimodal machine translation and multilingual image description.  In _Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers_ , pages 215–233, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.  URL http://www.aclweb.org/anthology/W17-4718.
  * Friedman and Dieng [2023] Dan Friedman and Adji Bousso Dieng.  The vendi score: A diversity evaluation metric for machine learning.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=g97OHbQyk1.
  * Futeral et al. [2023] Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, and Rachel Bawden.  Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5394–5413, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.295.  URL https://aclanthology.org/2023.acl-long.295.
  * Gadre et al. [2024] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Gao et al. [2020] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_ , 2020.
  * Gao et al. [2021] Tianyu Gao, Xingcheng Yao, and Danqi Chen.  SimCSE: Simple contrastive learning of sentence embeddings.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.552.  URL https://aclanthology.org/2021.emnlp-main.552.
  * Gionis et al. [1999] Aristides Gionis, Piotr Indyk, Rajeev Motwani, et al.  Similarity search in high dimensions via hashing.  In _Proceedings of the 25th VLDB Conference_ , volume 99, pages 518–529, Edinburgh, Scotland, UK, 1999.
  * Hewitt et al. [2018] John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry Tanti Wijaya, and Chris Callison-Burch.  Learning translations via images with a massively multilingual image dataset.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Melbourne, Australia, July 2018. Association for Computational Linguistics.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Thomas Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karén Simonyan, Erich Elsen, Oriol Vinyals, Jack Rae, and Laurent Sifre.  Training compute-optimal large language models.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 30016–30030. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf.
  * Howard et al. [2017] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ , 2017.
  * Ilharco et al. [2021] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.  OpenCLIP, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Jia et al. [2021] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _Proceedings of the Thirty-Eighth International Conference on Machine Learning_ , pages 4904–4916, online, 2021. PMLR.
  * Karpathy and Fei-Fei [2015] Andrej Karpathy and Li Fei-Fei.  Deep visual-semantic alignments for generating image descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3128–3137, 2015.
  * Kosar et al. [2022] Vaclav Kosar, Antonín Hoskovec, Milan Šulc, and Radek Bartyzal.  GLAMI-1M: A Multilingual Image-Text Fashion Dataset.  In _Proceedings of the 33rd British Machine Vision Conference_ , London, UK, 2022. BMVA Press.  URL https://bmvc2022.mpi-inf.mpg.de/0607.pdf.
  * Krishna et al. [2017] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International journal of computer vision_ , 123:32–73, 2017.
  * [39] Gant Laborde.  Deep nn for nsfw detection.  URL https://github.com/GantMan/nsfw_model.
  * Laurençon et al. [2022] Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gerard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor, Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Sebastian Nagel, Leon Weber, Manuel Muñoz, Jian Zhu, Daniel Van Strien, Zaid Alyafeai, Khalid Almubarak, Minh Chien Vu, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey, Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Margaret Mitchell, Sasha Alexandra Luccioni, and Yacine Jernite.  The bigscience roots corpus: A 1.6tb composite multilingual dataset.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 31809–31826. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/ce9e92e3de2372a4b93353eb7f3dc0bd-Paper-Datasets_and_Benchmarks.pdf.
  * Laurençon et al. [2023] Hugo Laurençon, Lucile Saulnier, Leo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh.  OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 71683–71702. Curran Associates, Inc., 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/e2cfb719f58585f779d0a4f9f07bd618-Paper-Datasets_and_Benchmarks.pdf.
  * Laurençon et al. [2024] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh.  What matters when building vision-language models?  _arXiv preprint arXiv:2405.02246_ , 2024.
  * Leong et al. [2022] Colin Leong, Joshua Nemecek, Jacob Mansdorfer, Anna Filighera, Abraham Owodunni, and Daniel Whitenack.  Bloom library: Multimodal datasets in 300+ languages for a variety of downstream tasks.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8608–8621, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.590.  URL https://aclanthology.org/2022.emnlp-main.590.
  * Levenshtein [1966] Vladimir Iosifovich Levenshtein.  Binary codes capable of correcting deletions, insertions and reversals.  _Soviet Physics Doklady_ , 10(8):707–710, 1966.  Doklady Akademii Nauk SSSR, V163 No4 845-848 1965.
  * Li et al. [2016] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan.  A diversity-promoting objective function for neural conversation models.  In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 110–119, San Diego, California, June 2016. Association for Computational Linguistics.  doi: 10.18653/v1/N16-1014.  URL https://aclanthology.org/N16-1014.
  * Li et al. [2021] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705, 2021.
  * Li et al. [2022] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _Proceedings of the 39 th International Conference on Machine Learning_ , pages 12888–12900, Baltimore, Maryland, USA, 2022. PMLR.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  In _Proceedings of the 40th International Conference on Machine Learning_ , pages 19730––19742, Honolulu Hawaii USA, 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In _Proceedings of the 13th European Conference on Computer Vision_ , pages 740–755, Zurich, Switzerland, 2014. Springer.
  * Liu et al. [2021] Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott.  Visually grounded reasoning across languages and cultures.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10467–10485, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.818.  URL https://aclanthology.org/2021.emnlp-main.818.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Liu et al. [2020] Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742, 2020.  doi: 10.1162/tacl_a_00343.  URL https://aclanthology.org/2020.tacl-1.47.
  * Maaz et al. [2024] Muhammad Maaz, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M Anwer, Tim Baldwin, Michael Felsberg, and Fahad S Khan.  PALO: A Polyglot Large Multimodal Model for 5B People.  _arXiv preprint arXiv:2402.14818_ , 2024.
  * Chameleon Team [2024] Chameleon Team.  Chameleon: Mixed-modal early-fusion foundation models.  _arXiv preprint arXiv:2405.09818_ , 2024.
  * Gemini Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Gemma Team et al. [2024] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al.  Gemma: Open models based on Gemini research and technology.  _arXiv preprint arXiv:2403.08295_ , 2024.
  * McKinzie et al. [2024] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al.  MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training.  _arXiv preprint arXiv:2403.09611_ , 2024.
  * Mohammed et al. [2023] Owais Khan Mohammed, Kriti Aggarwal, Qiang Liu, Saksham Singhal, Johan Bjorck, and Subhojit Som.  Bootstrapping a high quality multilingual multimodal dataset for Bletchley.  In Emtiyaz Khan and Mehmet Gonen, editors, _Proceedings of The 14th Asian Conference on Machine Learning_ , volume 189 of _Proceedings of Machine Learning Research_ , pages 738–753. PMLR, 12–14 Dec 2023.  URL https://proceedings.mlr.press/v189/mohammed23a.html.
  * OpenAI [2023] OpenAI.  GPT-4 Technical Report.  _ArXiv_ , abs/2303.08774, 2023.  URL https://arxiv.org/abs/2303.08774.
  * Ordonez et al. [2011] Vicente Ordonez, Girish Kulkarni, and Tamara Berg.  Im2Text: Describing Images Using 1 Million Captioned Photographs.  In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , volume 24. Curran Associates, Inc., 2011.  URL https://proceedings.neurips.cc/paper_files/paper/2011/file/5dd9db5e033da9c6fb5ba83c7a7ebea9-Paper.pdf.
  * Ortiz Suárez et al. [2019] Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  In Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald Lüngen, and Caroline Iliadi, editors, _Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-7)_ , pages 9 – 16, Cardiff, UK, 2019. Leibniz-Institut für Deutsche Sprache.  doi: 10.14618/ids-pub-9021.  URL http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215.
  * Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  Bleu: a method for automatic evaluation of machine translation.  In Pierre Isabelle, Eugene Charniak, and Dekang Lin, editors, _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics.  doi: 10.3115/1073083.1073135.  URL https://aclanthology.org/P02-1040.
  * Penedo et al. [2023] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 79155–79172. Curran Associates, Inc., 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/fa3ed726cc5073b9c31e3e49a807789c-Paper-Datasets_and_Benchmarks.pdf.
  * Pfeiffer et al. [2022] Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych.  xGQA: Cross-lingual visual question answering.  In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2497–2511, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.findings-acl.196.  URL https://aclanthology.org/2022.findings-acl.196.
  * Plummer et al. [2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649, Santiago, Chile, 2015.
  * Post [2018] Matt Post.  A call for clarity in reporting BLEU scores.  In Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor, editors, _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium, October 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-6319.  URL https://aclanthology.org/W18-6319.
  * Qi et al. [2020] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning.  Stanza: A Python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 2020.  URL https://nlp.stanford.edu/pubs/qi2020stanza.pdf.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38 th International Conference on Machine Learning_ , pages 8748–8763, online, 2021. PMLR.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551, 2020.
  * Sap et al. [2019] Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith.  The risk of racial bias in hate speech detection.  In Anna Korhonen, David Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1668–1678, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1163.  URL https://aclanthology.org/P19-1163.
  * Schuhmann et al. [2021] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs.  In _Proceedings of the Data Centric AI NeurIPS Workshop 2021_ , online, 2021.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev.  Laion-5b: An open large-scale dataset for training next generation image-text models.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 25278–25294. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/a1859debfb3b59d094f3504d5ebb6c25-Paper-Datasets_and_Benchmarks.pdf.
  * Sharma et al. [2018] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Iryna Gurevych and Yusuke Miyao, editors, _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, Melbourne, Australia, July 2018. Association for Computational Linguistics.  doi: 10.18653/v1/P18-1238.  URL https://aclanthology.org/P18-1238.
  * Smith et al. [2022] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al.  Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model.  _arXiv preprint arXiv:2201.11990_ , 2022.
  * Somepalli et al. [2023] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Diffusion art or digital forgery? investigating data replication in diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6048–6058, 2023.
  * Somepalli et al. [2024] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Understanding and mitigating copying in diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Srinivasan et al. [2021] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork.  WIT: Wikipedia-based image text dataset for multimodal multilingual machine learning.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2443–2449, online, 2021.
  * Surís et al. [2022] Dídac Surís, Dave Epstein, and Carl Vondrick.  Globetrotter: Connecting languages by connecting images.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 16474–16484, Canada, 2022.
  * Tevet and Berant [2021] Guy Tevet and Jonathan Berant.  Evaluating the evaluation of diversity in natural language generation.  In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 326–346, Online, April 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.eacl-main.25.  URL https://aclanthology.org/2021.eacl-main.25.
  * Thapliyal et al. [2022] Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut.  Crossmodal-3600: A massively multilingual multimodal evaluation dataset.  In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 715–729, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.emnlp-main.45.  URL https://aclanthology.org/2022.emnlp-main.45.
  * Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  LLaMA: Open and Efficient Foundation Language Models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Vedantam et al. [2015] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4566–4575, 2015.
  * Visheratin [2023] Alexander Visheratin.  NLLB-CLIP – train performant multilingual image retrieval model on a budget.  _arXiv preprint arXiv:2309.01859_ , 2023.
  * Wang et al. [2023] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al.  CogVLM: Visual Expert for Pretrained Language Models.  _arXiv preprint arXiv:2311.03079_ , 2023.
  * Wang et al. [2022] Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  _arXiv preprint arXiv:2208.10442_ , 2022.
  * Webster et al. [2023] Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie.  On the de-duplication of laion-2b.  _arXiv preprint arXiv:2303.12733_ , 2023.
  * Wenzek et al. [2020] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave.  CCNet: Extracting high quality monolingual datasets from web crawl data.  In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 4003–4012, Marseille, France, May 2020. European Language Resources Association.  ISBN 979-10-95546-34-4.  URL https://aclanthology.org/2020.lrec-1.494.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online, October 2020. Association for Computational Linguistics.  URL https://www.aclweb.org/anthology/2020.emnlp-demos.6.
  * Xue et al. [2021] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mT5: A massively multilingual pre-trained text-to-text transformer.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.41.  URL https://aclanthology.org/2021.naacl-main.41.
  * Yoshikawa et al. [2017] Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi.  STAIR captions: Constructing a large-scale Japanese image caption dataset.  In Regina Barzilay and Min-Yen Kan, editors, _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 417–421, Vancouver, Canada, July 2017. Association for Computational Linguistics.  doi: 10.18653/v1/P17-2066.  URL https://aclanthology.org/P17-2066.
  * Young et al. [2014] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78, 2014.  doi: 10.1162/tacl_a_00166.  URL https://aclanthology.org/Q14-1006.
  * Yu et al. [2022] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu.  Coca: Contrastive captioners are image-text foundation models.  _arXiv preprint arXiv:2205.01917_ , 2022.
  * Zhai et al. [2023] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 11975–11986, October 2023.
  * Zhu et al. [2023] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal c4: An open, billion-scale corpus of images interleaved with text.  _arXiv preprint arXiv:2304.06939_ , 2023.

##  Appendix A Appendix

###  A.1 mOSCAR languages & statistics

Languages |  | Statistics
---|---|---
Lang. name | Code | Family | Script |  | #documents | #images | #tokens
Acehnese | ace_Latn | Austronesian | Latin |  | 7,803 | 32,461 | 2,889,134
Mesopotamian Arabic | acm_Arab | Afro-Asiatic | Arabic |  | 2,274 | 10,620 | 1,047,748
Tunisian Arabic | aeb_Arab | Afro-Asiatic | Arabic |  | 7,640 | 41,570 | 2,715,187
Afrikaans | afr_Latn | Indo-European | Latin |  | 54,895 | 247,774 | 39,956,585
South Levantine Arabic | ajp_Arab | Afro-Asiatic | Arabic |  | 12,098 | 87,837 | 5,167,813
Tosk Albanian | als_Latn | Indo-European | Latin |  | 861,678 | 2,569,164 | 452,737,251
Amharic | amh_Ethi | Afro-Asiatic | Ge‘ez |  | 39,588 | 152,646 | 35,089,019
North Levantine Arabic | apc_Arab | Afro-Asiatic | Arabic |  | 19,904 | 128,966 | 9,560,701
Modern Standard Arabic | arb_Arab | Afro-Asiatic | Arabic |  | 3,936,851 | 15,126,931 | 3,401,919,964
Najdi Arabic | ars_Arab | Afro-Asiatic | Arabic |  | 60,229 | 296,741 | 43,610,873
Moroccan Arabic | ary_Arab | Afro-Asiatic | Arabic |  | 142,386 | 698,051 | 204,723,454
Egyptian Arabic | arz_Arab | Afro-Asiatic | Arabic |  | 835,529 | 4,054,632 | 653,626,387
Assamese | asm_Beng | Indo-European | Bengali |  | 3,948 | 9,210 | 640,390
Asturian | ast_Latn | Indo-European | Latin |  | 165,745 | 962,723 | 37,547,944
Awadhi | awa_Deva | Indo-European | Devanagari |  | 29,324 | 107,483 | 4,961,635
Central Aymara | ayr_Latn | Aymaran | Latin |  | 27,384 | 151,889 | 5,148,970
South Azerbaijani | azb_Arab | Turkic | Arabic |  | 8,274 | 38,233 | 5,256,693
North Azerbaijani | azj_Latn | Turkic | Latin |  | 516,021 | 1,808,060 | 257,825,849
Bashkir | bak_Cyrl | Turkic | Cyrillic |  | 4,532 | 17,174 | 3,038,766
Bambara | bam_Latn | Manding | Latin |  | 7,674 | 39,190 | 1,243,332
Balinese | ban_Latn | Austronesian | Latin |  | 1,886 | 11,266 | 542,015
Belarusian | bel_Cyrl | Indo-European | Cyrillic |  | 63,309 | 287,539 | 72,976,520
Bemba | bem_Latn | Atlantic–Congo | Latin |  | 1,096 | 7,479 | 1,340,471
Bengali | ben_Beng | Indo-European | Bengali |  | 270,406 | 947,035 | 35,858,814
Bhojpuri | bho_Deva | Indo-European | Devanagari |  | 6,366 | 28,131 | 875,463
Banjar | bjn_Latn | Austronesian | Latin |  | 5,427 | 27,803 | 1,898,526
Bosnian | bos_Latn | Indo-European | Latin |  | 1,960,599 | 7,633,049 | 1,255,000,505
Buginese | bug_Latn | Austronesian | Latin |  | 3,312 | 18,648 | 588,678
Bulgarian | bul_Cyrl | Indo-European | Cyrillic |  | 2,591,998 | 11,670,028 | 1,760,971,620
Catalan | cat_Latn | Indo-European | Latin |  | 1,153,864 | 4,736,634 | 606,447,390
Cebuano | ceb_Latn | Austronesian | Latin |  | 16,990 | 91,234 | 10,748,818
Czech | ces_Latn | Indo-European | Latin |  | 3,918,837 | 13,291,309 | 2,823,172,996
Central Kurdish | ckb_Arab | Indo-European | Arabic |  | 36,725 | 136,566 | 22,322,689
Crimean Tatar | crh_Latn | Turkic | Latin |  | 6,376 | 24,124 | 1,742,727
Welsh | cym_Latn | Indo-European | Latin |  | 40,408 | 165,897 | 27,748,345
Danish | dan_Latn | Indo-European | Latin |  | 2,076,298 | 9,559,600 | 1,238,277,499
German | deu_Latn | Indo-European | Latin |  | 20,662,696 | 87,976,200 | 8,544,986,218
Southwestern Dinka | dik_Latn | Nilo-Saharan | Latin |  | 1,712 | 6,635 | 1,319,943
Greek | ell_Grek | Indo-European | Greek |  | 4,916,081 | 15,209,058 | 2,923,201,041
English | eng_Latn | Indo-European | Latin |  | 52,215,013 | 207,904,315 | 33,570,108,782
Esperanto | epo_Latn | Artificial | Latin |  | 25,157 | 124,996 | 28,586,195
Estonian | est_Latn | Uralic | Latin |  | 1,040,368 | 5,217,366 | 619,215,048
Basque | eus_Latn | Isolate | Latin |  | 849,043 | 3,445,539 | 277,145,498
Faroese | fao_Latn | Indo-European | Latin |  | 15,411 | 60,340 | 6,691,327
Fijian | fij_Latn | Austronesian | Latin |  | 1,528 | 8,776 | 487,388
Finnish | fin_Latn | Uralic | Latin |  | 2,396,033 | 10,365,333 | 1,781,044,864
French | fra_Latn | Indo-European | Latin |  | 20,305,739 | 78,179,601 | 14,362,579,829
Friulian | fur_Latn | Indo-European | Latin |  | 37,290 | 256,456 | 5,949,600
Nigerian Fulfulde | fuv_Latn | Atlantic-Congo | Latin |  | 1,568 | 7,124 | 401,852
West Central Oromo | gaz_Latn | Afro-Asiatic | Latin |  | 4,058 | 11,763 | 1,786,093
Scottish Gaelic | gla_Latn | Indo-European | Latin |  | 29,710 | 153,249 | 14,605,090
Irish | gle_Latn | Indo-European | Latin |  | 68,858 | 315,132 | 47,438,400
Galician | glg_Latn | Indo-European | Latin |  | 518,973 | 2,381,475 | 217,063,180
Guarani | grn_Latn | Tupian | Latin |  | 490,945 | 2,416,633 | 89,921,114
Gujarati | guj_Gujr | Indo-European | Gujarati |  | 23,062 | 91,320 | 3,324,866
Haitian Creole | hat_Latn | Indo-European | Latin |  | 257,745 | 1,570,699 | 62,847,106
Hausa | hau_Latn | Afro-Asiatic | Latin |  | 25,364 | 104,934 | 13,089,932
Hebrew | heb_Hebr | Afro-Asiatic | Hebrew |  | 1,109,591 | 4,766,483 | 893,327,320
Hindi | hin_Deva | Indo-European | Devanagari |  | 579,430 | 1,830,667 | 122,558,353
Chhattisgarhi | hne_Deva | Indo-European | Devanagari |  | 1,581 | 7,263 | 273,174
Croatian | hrv_Latn | Indo-European | Latin |  | 1,719,617 | 8,425,510 | 1,010,674,096
Hungarian | hun_Latn | Uralic | Latin |  | 3,534,506 | 15,390,083 | 2,831,715,050
Armenian | hye_Armn | Indo-European | Armenian |  | 339,962 | 1,141,885 | 205,635,952
Igbo | ibo_Latn | Atlantic-Congo | Latin |  | 11,529 | 68,049 | 8,701,070
Ilocano | ilo_Latn | Austronesian | Latin |  | 78,872 | 523,195 | 8,116,113
Indonesian | ind_Latn | Austronesian | Latin |  | 7,016,291 | 17,324,777 | 3,981,843,468
Icelandic | isl_Latn | Indo-European | Latin |  | 244,676 | 1,027,465 | 137,015,973
Italian | ita_Latn | Indo-European | Latin |  | 12,937,153 | 47,476,971 | 8,311,790,842
Javanese | jav_Latn | Austronesian | Latin |  | 24,785 | 135,583 | 16,908,805
Japanese | jpn_Jpan | Japonic | Kanji |  | 14,415,292 | 23,893,768 | 8,923,348,944
Kabyle | kab_Latn | Afro-Asiatic | Latin |  | 18,508 | 106,730 | 4,079,553
Kannada | kan_Knda | Dravidian | Kannada |  | 12,978 | 42,621 | 1,442,776
Kashmiri | kas_Arab | Indo-European | Arabic |  | 3,109 | 11,408 | 5,731,910
Georgian | kat_Geor | Kartvelian | Georgian |  | 354,436 | 1,304,281 | 275,223,026
Kazakh | kaz_Cyrl | Turkic | Cyrillic |  | 252,242 | 732,648 | 140,049,214
Halh Mongolian | khk_Cyrl | Mongolic | Cyrillic |  | 124,412 | 508,217 | 84,535,241
Khmer | khm_Khmr | Austroasiatic | Kher |  | 24,495 | 122,243 | 3,043,925
Kinyarwanda | kin_Latn | Atlantic-Congo | Latin |  | 30,401 | 172,201 | 12,049,616
Kyrgyz | kir_Cyrl | Uralic | Cyrillic |  | 53,010 | 199,713 | 34,404,281
Northern Kurdish | kmr_Latn | Indo-European | Latin |  | 39,262 | 164,666 | 23,834,960
Korean | kor_Hang | Koreanic | Hanja |  | 2,614,089 | 13,563,283 | 2,006,080,705
Lao | lao_Laoo | Kra-Dai | Lao |  | 50,611 | 208,768 | 31,029,380
Ligurian | lij_Latn | Indo-European | Latin |  | 8,751 | 56,266 | 2,958,179
Limburgish | lim_Latn | Indo-European | Latin |  | 189,547 | 1,076,047 | 42,534,327
Lingala | lin_Latn | Atlantic-Congo | Latin |  | 24,614 | 152,132 | 4,053,459
Lithuanian | lit_Latn | Indo-European | Latin |  | 1,688,811 | 8,869,443 | 1,161,476,040
Lombard | lmo_Latn | Indo-European | Latin |  | 30,506 | 151,855 | 9,058,614
Latgalian | ltg_Latn | Indo-European | Latin |  | 11,948 | 61,624 | 4,148,492
Luxembourgish | ltz_Latn | Indo-European | Latin |  | 44,987 | 246,346 | 16,676,872
Ganda | lug_Latn | Afro-Asiatic | Latin |  | 1,878 | 7,215 | 789,917
Mizo | lus_Latn | Sino-Tibetan | Latin |  | 7,880 | 26,817 | 4,978,472
Standard Latvian | lvs_Latn | Indo-European | Latin |  | 896,243 | 4,141,648 | 587,653,855
Magahi | mag_Deva | Indo-European | Devanagari |  | 1,097 | 3,847 | 205,763
Malayalam | mal_Mlym | Dravidian | Malayalam |  | 14,140 | 52,679 | 1,689,010
Marathi | mar_Deva | Indo-European | Devanagari |  | 50,391 | 163,868 | 6,689,250
Minangkabau | min_Latn | Austronesian | Latin |  | 9,341 | 35,309 | 1,256,931
Macedonian | mkd_Cyrl | Indo-European | Cyrillic |  | 542,250 | 1,853,070 | 307,232,151
Maltese | mlt_Latn | Afro-Asiatic | Latin |  | 120,888 | 709,242 | 36,097,957
Maori | mri_Latn | Austronesian | Latin |  | 24,322 | 130,137 | 24,957,914
Burmese | mya_Mymr | Sino-Tibetan | Mon |  | 8,144 | 44,188 | 539,527
Dutch | nld_Latn | Indo-European | Latin |  | 17,096,727 | 65,606,013 | 9,670,041,731
Norwegian Nynorsk | nno_Latn | Indo-European | Latin |  | 199,355 | 1,012,313 | 67,799,774
Norwegian Bokmål | nob_Latn | Indo-European | Latin |  | 2,229,702 | 9,698,128 | 1,294,178,095
Nepali | npi_Deva | Indo-European | Devanagari |  | 31,239 | 127,193 | 3,138,539
Nyanja | nya_Latn | Atlantic-Congo | Latin |  | 12,047 | 67,192 | 8,596,769
Occitan | oci_Latn | Indo-European | Latin |  | 164,852 | 671,881 | 59,309,549
Odia | ory_Orya | Indo-European | Odia |  | 4,319 | 15,574 | 378,635
Pangasinan | pag_Latn | Austronesian | Latin |  | 4,214 | 32,287 | 546,071
Eastern Panjabi | pan_Guru | Indo-European | Gurmukhi |  | 11,497 | 46,168 | 1,887,991
Papiamento | pap_Latn | Indo-European | Latin |  | 55,224 | 363,015 | 10,002,655
Southern Pasto | pbt_Arab | Indo-European | Arabic |  | 32,604 | 110,807 | 29,170,322
Western Persian | pes_Arab | Indo-European | Arabic |  | 7,048,946 | 25,200,571 | 6,210,479,015
Plateau Malgasy | plt_Latn | Austronesian | Latin |  | 32,521 | 120,673 | 29,263,848
Polish | pol_Latn | Indo-European | Latin |  | 14,549,605 | 60,639,244 | 11,104,144,109
Portuguese | por_Latn | Indo-European | Latin |  | 8,145,664 | 26,530,423 | 4,760,063,083
Dari | prs_Arab | Indo-European | Arabic |  | 515,041 | 2,589,859 | 517,053,967
Ayacucho Quechua | quy_Latn | Quechuan | Latin |  | 1,578 | 11,817 | 362,690
Romanian | ron_Latn | Indo-European | Latin |  | 5,180,171 | 17,964,048 | 3,548,291,261
Rundi | run_Latn | Atlantic-Congo | Latin |  | 20,001 | 67,096 | 8,686,054
Russian | rus_Cyrl | Indo-European | Cyrillic |  | 15,913,845 | 69,542,828 | 18,909,213,208
Sango | sag_Latn | Atlantic-Congo | Latin |  | 2,124 | 13,556 | 454,455
Sicilian | scn_Latn | Indo-European | Latin |  | 73,199 | 424,362 | 27,110,743
Sinhala | sin_Sinh | Indo-European | Sinhalese |  | 58,767 | 221,183 | 14,270,972
Slovak | slk_Latn | Indo-European | Latin |  | 3,008,599 | 15,067,234 | 1,963,804,563
Slovenian | slv_Latn | Indo-European | Latin |  | 1,472,025 | 7,210,285 | 935,834,754
Samoan | smo_Latn | Austronesian | Latin |  | 12,346 | 71,359 | 14,954,824
Shona | sna_Latn | Atlantic-Congo | Latin |  | 12,698 | 68,782 | 6,112,600
Sindhi | snd_Arab | Indo-European | Arabic |  | 21,095 | 74,289 | 17,647,825
Somali | som_Latn | Afro-Asiatic | Latin |  | 77,343 | 301,429 | 34,554,975
Southern Sotho | sot_Latn | Atlantic-Congo | Latin |  | 7,718 | 43,146 | 6,156,450
Spanish | spa_Latn | Indo-European | Latin |  | 22,713,366 | 78,361,087 | 14,616,773,475
Sardinian | srd_Latn | Indo-European | Latin |  | 675,539 | 4,059,493 | 106,159,957
Serbian | srp_Cyrl | Indo-European | Cyrillic |  | 604,557 | 2,286,171 | 401,223,741
Sundanese | sun_Latn | Austronesian | Latin |  | 44,310 | 236,025 | 13,627,832
Swedish | swe_Latn | Indo-European | Latin |  | 3,302,730 | 10,860,518 | 1,779,284,152
Swahili | swh_Latn | Atlantic-Congo | Latin |  | 137,134 | 593,418 | 59,454,896
Silesian | szl_Latn | Indo-European | Latin |  | 23,535 | 132,459 | 5,996,972
Tamil | tam_Taml | Dravidian | Tamil |  | 36,196 | 167,669 | 4,834,946
Tatar | tat_Cyrl | Turkic | Cyrillic |  | 37,188 | 143,842 | 22,831,350
Telugu | tel_Telu | Dravidian | Telugu |  | 22,974 | 81,033 | 2,273,772
Tajik | tgk_Cyrl | Turkic | Cyrillic |  | 125,236 | 417,591 | 90,503,778
Tagalog | tgl_Latn | Austronesian | Latin |  | 151,437 | 673,814 | 97,708,639
Thai | tha_Thai | Kra-Dai | Thai |  | 2,983,837 | 11,621,786 | 2,839,211,104
Tigrinya | tir_Ethi | Afro-Asiatic | Ge‘ez |  | 2,657 | 8,707 | 1,725,422
Tok Pisin | tpi_Latn | Indo-European | Latin |  | 5,063 | 35,169 | 460,853
Turkmen | tuk_Latn | Turkic | Latin |  | 13,024 | 57,354 | 9,766,999
Turkish | tur_Latn | Turkic | Latin |  | 4,478,700 | 12,401,091 | 2,394,669,068
Twi | twi_Latn | Atlantic-Congo | Latin |  | 3,305 | 13,634 | 495,220
Uyghur | uig_Arab | Turkic | Arabic |  | 10,713 | 41,709 | 6,785,318
Ukrainian | ukr_Cyrl | Indo-European | Cyrillic |  | 2,721,424 | 10,929,796 | 1,928,351,595
Urdu | urd_Arab | Indo-European | Arabic |  | 407,098 | 1,239,125 | 242,007,283
Northern Uzbek | uzn_Latn | Turkic | Latin |  | 156,632 | 798,155 | 89,022,562
Venetian | vec_Latn | Indo-European | Latin |  | 330,611 | 1,830,777 | 71,077,531
Vietnamese | vie_Latn | Viet-Muong | Latin |  | 12,621,521 | 47,411,488 | 11,616,191,199
Wolof | wol_Latn | Atlantic-Congo | Latin |  | 4,658 | 20,380 | 1,596,432
Xhosa | xho_Latn | Atlantic-Congo | Latin |  | 25,950 | 142,387 | 15,809,823
Eastern Yiddish | ydd_Hebr | Indo-European | Hebrew |  | 12,486 | 57,510 | 17,369,727
Yoruba | yor_Latn | Atlantic-Congo | Latin |  | 56,700 | 286,933 | 32,614,558
Yue Chinese | yue_Hant | Sino-Tibetan | Hant |  | 33,671 | 203,513 | 24,172,441
Chinese (Simplified) | zho_Hans | Sino-Tibetan | Hanzi |  | 9,861,262 | 36,152,754 | 8,078,842,701
Chinese (Traditional) | zho_Hant | Sino-Tibetan | Hant |  | 3,967,966 | 16,307,258 | 2,962,854,441
Standard Malay | zsm_Latn | Austronesian | Latin |  | 1,179,744 | 5,488,632 | 432,667,199
Zulu | zul_Latn | Atlantic-Congo | Latin |  | 30,717 | 156,639 | 11,345,288
Table 6: Languages & Statistics

###  A.2 Heuristics to increase the quality of documents

We use a set of heuristics to improve the quality of the documents by
discarding some text nodes. We first consider text nodes to be written in
Latin scripts if more than 50% of the characters are Latin. In detail, we
discard the text node if:

  1. 1.

It is empty.

  2. 2.

It contains fewer than 5 bytes for Latin scripts and fewer than 15 bytes for
non-Latin scripts.

  3. 3.

More than 30% of the characters are digits.

  4. 4.

It contains more than one date.

  5. 5.

It contains the sequence “lorem ipsum”.

  6. 6.

The ratio of non-alphabetic characters is superior to 0.33.

  7. 7.

The symbols ‘{’ or ’‘}’ are in the text.

  8. 8.

The symbols ‘$\geq$’, ‘$\leq$’, ‘>’ or ‘<’ are more than 2 times in the text.

  9. 9.

“Follow us”, “javascript”, “copyright” or “©” are in the text.

  10. 10.

The ratio of capitalized letters is superior to 0.2.

  11. 11.

The text exactly matches with “comment”, “facebook”, “instagram”, “twitter”,
“rss”, “newsletter”, “share” or “follow us”.

  12. 12.

A character is more than 33% of the total number of characters in the string.

We then also apply some filters to clean the text as much as possible:

  1. 1.

Remove URLs from all documents.

  2. 2.

Normalize consecutive special characters (‘\t’, ‘\n’, ‘#’, ‘/’, ‘$’, ‘)’, ‘(’,
‘[’, ‘]’, ‘!’, ‘?’, ‘%’, ‘<’, ‘>’) to keep only one.

Following previous steps, we keep the text node if it is superior to 5 bytes
and we keep the final document if it is superior to 100 bytes.

###  A.3 Examples of documents

Figure 5: Example of a French document.

Figure 6: Example of a Japanese document.

Figure 7: Example of a Russian document. Figure 8: Example of an Italian
document. Figure 9: Example of a Khmer document. Figure 10: Example of an Urdu
document.

###  A.4 Text-Image similarity and DOM Tree

As we rely on the DOM Tree to build the documents and the order of appearance
of the nodes could differ from HTML rendering, we attempt to assess to what
extent it is a relevant way of constructing a multimodal document. To do so,
we rely on the results of the text-image joint filtering step where we compute
the ranks of relevant text nodes (resp images) for each image. We plot the
distribution of the closest most relevant node for each modality in Figures
11(a) and 11(b). We notice that the most relevant node to either a text node
or an image is their closest node in the DOM tree. The cumulative distribution
function of the distribution of the closest node reaches 25% for nodes
positioned between -5 and 5, which confirms the relevance of using the DOM
tree to represent a document.

(a) Relative position in the document of relevant text nodes with respect to
images.

(b) Relative position in the document of relevant images with respect to text
nodes.

Figure 11: Relative positions of most relevant images and text nodes with
respect to the other modality.

###  A.5 Implementation details

####  A.5.1 Text deduplication parameters

Following previous work, we near-deduplicate documents using MinHashLSH. We
first vectorize the documents using HashingVectorizer from scikit-learn with
2,097,152 features computed on 4-grams and 5-grams within word boundaries. We
then compute MinHashes from those vectors with 256 permutations and we finally
run Locality Sensitive Hashing with a threshold Jaccard Similarity of 0.8 for
finding near-duplicates.

####  A.5.2 Training implementation details

We train multilingual OpenFlamingo on mOSCAR and multilingual text-image
pairs. We use a batch of size 64 for mOSCAR and 128 for captioning data,
limiting the number of tokens to 256 for mOSCAR and 32 for captioning data.
Similarly to Flamingo and OpenFlamingo, text tokens can only attend to the
previous image in the sequence. To increase diversity in the training batch,
we randomly reject 2/3 of the documents if they contain only one image. We
limit the maximum number of images in a sequence to 8. We randomly sample 8
languages per batch and upsample low-resource languages. We train multilingual
OpenFlamingo on 43 languages covering all the languages of the benchmarks we
evaluate the models on (see Section A.5.3).

We use Gemma-2B as the underlying language model behind multilingual
OpenFlamingo and CLIP ViT-L-14 as the image encoder. We add a cross-attention
layer after each decoder layer. Following OpenFlamingo, we add the two special
tokens <image> and<|endofchunk|>, whose embeddings were trained. Only the
Perceiver Resampler, cross-attention layers and these two embeddings were
trained; everything else remained frozen. During training, we apply a factor
of 0.2 for the captioning data loss function.

We train the model using the Adam optimizer and a maximum learning rate of
1e-4. We use a constant learning rate scheduler with 1875 warm-up steps. We
use 4 accumulation gradient steps to have an effective batch of size 256 for
mOSCAR and 512 for captioning data. We train the model on 35M documents and
70M image-text pairs on 8 Nvidia A100 for 120h.

####  A.5.3 Evaluation details

| Metric | #examples | Languages
---|---|---|---
xFlickr&CO | CideR | 2,000 | Chinese, English, German, Indonesian, Japanese, Russian, Spanish, Turkish
XM3600 | CideR | 3,600 | Arabic, Czech, Danish, German, Greek, English, Spanish, Farsi,
Finnish, French, Hebrew, Hindi, Croatian, Hungarian, Indonesian, Italian,
Japanese, Korean, Dutch, Norwegian, Poland, Portuguese, Romanian,
Russian, Swedish, Telugu, Thai, Turkish, Ukrainian, Vietnamese, Chinese
xGQA | Accuracy | 9,666 | Bengali, German, English, Indonesian, Korean, Portuguese, Russian, Chinese
MaXM | Accuracy |  $\sim$ 170 | English, French, Hindi, Hebrew, Romanian, Thai, Chinese
MaRVL | Accuracy |  $\sim$ 1,150 | Indonesian, Swahili, Tamil, Turkish, Chinese
XVNLI | Accuracy | 1,164 | English, Arabic, Spanish, French, Russian
Multi30k | BLEU | 1,000 | French, German, Czech
CoMMuTE | Accuracy | 310 | Czech, French, German

Table 7: Overview of the benchmarks used to evaluate our multilingual
OpenFlamingo.

We evaluate on a set of eight benchmarks: xFlickr&CO, XM3600, xGQA, MaXM,
MaRVL, XVNLI, Multi30k (Test2016 subset) and CoMMuTE; covering 5 different
tasks and 43 languages. Details about the languages, the number of examples
and the metric used can be found in Table 7. We used the translate-
test161616Benchmark automatically translated into English. samples provided by
the authors of the benchmarks if available. No translate test samples were
provided for MaXM, so we translated the test set using the NLLB-600M distilled
model. As no training set was available for MaXM, we use the few-shot examples
from xGQA. Since we use Stanza tokenizers, we could not evaluate on all
languages from XM3600 as 3 of them were not available. Filipino was also not
into the list of mOSCAR languages, so we skip this language during evaluation.
The CoMMuTE evaluation set involves choosing between two different
translations of a same source text (one correct and one incorrect depending on
an image provided to disambiguate the text). We use the lowest perplexity
between the two translations as the model’s prediction. We also use Multi30k
training set as few-shot examples.

##### Prompting

Following previous works, the zero-shot setting is composed of two few-shot
examples without providing the images. The prompts we use for the different
tasks are as follows:171717We show the prompts we used with one context
example.

For captioning tasks, we use the prompt:

“<image>Output:[Caption]<|endofchunk|><image>Output:”,

where [Caption] is replaced by the caption.

For visual question answering tasks, we use the prompt:

“<image>Question: [Question] Short Answer: [Answer]
<|endofchunk|><image>Question: [Question] Short Answer:”,

where [Question] and [Answer] are replaced by the question and the answer
respectively.

For multimodal machine translation tasks, we use the prompt:

“<image>Sentence:‘[Caption]’. Translation: [Translation]
<|endofchunk|><image>Output:”,

where [Caption] is replaced by the sentence to translate and [Translation] is
replaced by its translation.

For MaRVL, we use the prompt:

“<image> ‘[Statement]’. True of False? [Answer]<|endofchunk|><image>
‘[Statement]’. True of False?”,

where [Statement] is replaced by the statement and [Answer] by the answer. We
also concatenate the left and right image into a single image.

For XVNLI, we use the prompt:

“<image> ‘[Statement1]’ - ‘[Statement2]’. entailment, neutral or
contradiction? Output: [Answer]<|endofchunk|><image> ‘[Statement1]’ -
‘[Statement2]’. entailment, neutral or contradiction? Output:”,

where [Statement1], [Statement2] and [Answer] are replaced by XVNLI test data.

###  A.6 Detailed results

|  | De | En | Es | Id | Ja | Ru | Tr | Zh
---|---|---|---|---|---|---|---|---|---
| #shots
| 0 | 28.87 | 33.42 | 16.35 | 33.26 |  04.33 | 13.99 |  03.68 | 18.65
Multilingual OF | 4 | 53.99 | 46.53 | 35.87 | 48.85 | 12.44 | 31.88 | 11.35 | 33.64
mOSCAR + caps. | 8 | 56.05 | 52.99 | 37.28 | 52.51 | 15.13 | 32.76 | 11.69 | 35.78
| 16 | 55.44 | 56.78 | 37.72 | 49.06 | 17.10 | 35.43 | 12.72 | 36.82
| 0 | 16.72 | 24.57 |  03.80 | 10.82 |  02.82 |  08.20 |  02.79 |  06.82
Multilingual OF | 4 | 21.10 | 31.05 |  07.52 |  09.63 |  03.84 | 13.21 |  07.01 | 12.20
captions only | 8 | 32.56 | 35.73 | 13.35 | 15.85 |  05.96 | 18.13 |  06.97 | 15.47
| 16 | 29.86 | 40.57 | 13.75 | 23.83 |  06.92 | 20.40 |  07.90 | 15.73
Table 8: Captioning results (CideR scores) on xFlickr&CO. Bold is best result.

|  | Ar | Cs | Da | De | El | En | Es | Fa | Fi | Fr | He
---|---|---|---|---|---|---|---|---|---|---|---|---
| #shots
| 0 |  06.18 |  02.02 | 11.06 |  09.12 |  00.90 | 43.49 | 18.01 | 10.41 |  01.30 | 20.61 |  03.21
Multi. OF | 4 | 21.55 |  05.93 | 28.21 | 21.97 |  03.38 | 74.71 | 36.93 | 26.27 |  06.48 | 39.41 | 10.34
full | 8 | 23.15 |  06.23 | 31.33 | 24.53 |  03.47 | 75.29 | 37.58 | 28.84 |  07.65 | 43.40 | 10.83
| 16 | 23.80 |  06.86 | 31.92 | 24.92 |  03.59 | 76.24 | 38.39 | 26.78 |  07.61 | 43.26 | 11.78
| 0 |  02.24 |  00.97 |  06.42 |  06.46 |  03.68 | 10.02 |  09.32 |  04.95 |  01.14 | 16.15 |  00.78
Multi. OF | 4 |  05.36 |  01.36 | 13.11 | 11.82 |  07.78 | 35.52 | 19.96 |  09.62 |  01.86 | 22.48 |  02.29
Caps only | 8 |  06.76 |  01.40 | 15.29 | 14.39 |  07.21 | 37.28 | 21.90 | 12.19 |  02.08 | 23.27 |  01.71
| 16 |  06.25 |  02.29 | 17.96 | 15.11 |  07.64 | 48.03 | 25.39 |  09.21 |  02.10 | 30.16 |  02.72
|  | Hi | Hr | Hu | Id | It | Ja | Ko | Nl | No | Pl | Pt
| #shots
| 0 |  02.80 |  01.47 |  01.85 |  09.98 | 11.15 |  02.07 |  01.67 | 18.97 |  09.63 |  04.32 | 15.49
Multi. OF | 4 | 10.62 |  07.48 |  05.51 | 22.63 | 27.88 | 16.87 |  09.24 | 42.30 | 22.35 | 14.20 | 29.35
full | 8 | 11.12 |  07.54 |  05.91 | 25.39 | 29.34 | 19.35 |  09.99 | 46.79 | 23.54 | 15.43 | 32.69
| 16 | 12.18 |  07.71 |  06.03 | 23.89 | 29.17 | 18.84 |  09.75 | 46.95 | 23.79 | 15.93 | 31.98
| 0 |  02.29 |  00.97 |  03.51 |  02.98 |  07.96 |  01.85 |  01.05 |  04.88 |  05.78 |  00.92 |  09.79
Multi. OF | 4 |  04.57 |  01.72 |  07.57 |  06.39 | 16.23 |  03.47 |  04.33 | 11.26 | 11.99 |  01.16 | 15.93
Caps only | 8 |  05.94 |  02.17 |  07.83 |  09.93 | 15.40 |  07.93 |  05.34 | 11.87 | 13.79 |  01.38 | 17.50
| 16 |  06.36 |  02.42 |  09.55 | 11.77 | 17.43 | 10.44 |  06.03 | 12.98 | 14.65 |  01.28 | 20.32
|  | Ro | Ru | Sv | Te | Th | Tr | Uk | Vi | Zh |  |
| #shots |  |
| 0 |  02.19 |  06.80 | 11.45 |  00.87 |  08.36 |  03.11 |  03.04 | 21.99 |  07.19 |  |
Multi. OF | 4 |  05.63 | 20.50 | 25.59 |  02.29 | 21.66 | 10.87 | 10.43 | 38.43 | 19.34 |  |
full | 8 |  06.04 | 23.11 | 26.44 |  03.06 | 25.19 | 12.95 | 10.56 | 39.75 | 20.18 |  |
| 16 |  06.43 | 21.15 | 26.51 |  03.42 | 25.31 | 13.57 | 10.45 | 40.85 | 20.46 |  |
| 0 |  02.24 |  01.93 |  04.55 |  00.67 |  02.34 |  02.68 |  00.80 |  08.55 |  02.70 |  |
Multi. OF | 4 |  05.35 |  06.29 | 15.66 |  00.77 |  07.21 |  05.94 |  01.76 | 20.69 |  07.80 |  |
Caps only | 8 |  05.18 |  07.58 | 14.01 |  01.00 |  06.81 |  08.90 |  02.73 | 23.05 |  08.99 |  |
| 16 |  05.06 |  09.06 | 20.60 |  01.18 |  08.35 | 10.25 |  03.47 | 25.16 | 11.05 |  |

Table 9: Captioning results (CideR scores) on XM3600. Bold is best result. |  | Bn | De | En | Id | Ko | Pt | Ru | Zh
---|---|---|---|---|---|---|---|---|---
| #shots
| 0 | 20.99 | 22.57 | 32.60 | 22.45 | 26.44 | 23.54 | 25.31 | 26.77
Multilingual OF | 4 | 25.72 | 32.71 | 38.02 | 30.74 | 31.49 | 31.53 | 31.75 | 33.26
mOSCAR + caps. | 8 | 26.87 | 35.60 | 39.16 | 34.25 | 32.25 | 34.93 | 33.59 | 34.55
| 16 | 29.11 | 37.01 | 40.64 | 36.37 | 34.20 | 37.15 | 34.87 | 36.29
| 0 | 10.54 |  06.51 | 10.43 |  07.74 |  07.50 |  07.79 |  08.62 |  09.84
Multilingual OF | 4 | 12.54 | 11.90 | 15.78 | 13.95 | 13.70 | 12.01 | 12.73 | 15.03
captions only | 8 | 11.62 | 11.70 | 17.29 | 13.86 | 12.85 | 11.60 | 12.65 | 15.35
| 16 |  09.77 | 11.86 | 18.37 | 13.24 | 12.48 | 11.25 | 11.24 | 14.33
Translate Test
OF-3B MPT | 0 | 18.64 | 18.67 | - | 18.36 | 17.54 | 19.21 | 18.88 | 17.11
4 | 23.23 | 23.40 | - | 22.95 | 22.46 | 23.52 | 22.41 | 22.85
8 | 28.22 | 29.44 | - | 28.21 | 27.67 | 29.58 | 28.21 | 28.63
16 | 31.31 | 32.58 | - | 31.82 | 31.42 | 32.74 | 31.62 | 31.22
| 0 | 30.41 | 32.1 | - | 29.35 | 29.99 | 31.39 | 29.06 | 28.81
Multilingual OF | 4 | 34.89 | 36.32 | - | 35.50 | 35.64 | 36.84 | 35.05 | 34.60
mOSCAR + caps. | 8 | 35.95 | 37.65 | - | 36.78 | 37.14 | 37.81 | 36.17 | 35.98
| 16 | 36.78 | 38.78 | - | 37.52 | 37.73 | 38.68 | 37.91 | 36.84
Table 10: VQA results on xGQA. Bold is best result. |  | En | Fr | Hi | He | Ro | Th | Zh
---|---|---|---|---|---|---|---|---
| #shots
| 0 | 34.24 | 15.91 | 18.08 | 16.43 | 13.73 | 25.75 | 13.36
Multi. OF | 4 | 36.19 | 21.21 | 20.77 | 19.29 | 19.01 | 31.72 | 19.13
mOSCAR + caps | 8 | 36.96 | 17.80 | 20.00 | 20.36 | 17.96 | 33.96 | 23.83
| 16 | 36.19 | 18.18 | 21.92 | 20.71 | 16.90 | 41.04 | 22.74
| 0 |  09.73 |  00.38 |  07.69 |  01.43 |  00.00 |  05.22 |  03.61
Multi. OF | 4 |  09.34 |  02.65 |  05.00 |  02.50 |  00.00 |  05.60 |  03.97
captions only | 8 |  09.34 |  01.89 |  08.08 |  05.00 |  01.06 |  03.36 |  05.42
| 16 |  08.56 |  01.14 |  05.00 |  08.21 |  00.35 |  03.36 |  07.58
Translate test
OF-3B MPT | 0 | - | 12.50 | 22.31 |  00.36 | 10.92 |  00.00 |  00.00
4 | - | 10.98 | 25.38 |  00.36 | 10.21 |  00.00 |  00.00
8 | - | 10.98 | 27.31 |  00.36 | 11.27 |  00.00 |  00.00
16 | - | 13.26 | 26.54 |  01.07 | 13.38 |  00.00 |  00.00
| 0 | - | 18.18 | 28.08 |  00.00 | 13.73 |  00.00 |  00.36
Multi. OF | 4 | - | 15.91 | 30.38 |  00.36 | 12.68 |  00.00 |  00.00
mOSCAR + caps | 8 | - | 15.15 | 30.77 |  00.00 | 14.79 |  00.00 |  00.00
| 16 | - | 15.91 | 35.77 |  00.36 | 16.90 |  00.00 |  00.00
Table 11: VQA results on MaXM. Bold is best result. |  | Id | Sw | Ta | Tr | Zh
---|---|---|---|---|---|---
| #shots
Random chance |  | 50.00 | 50.00 | 50.00 | 50.00 | 50.00
| 0 | 50.00 | 49.46 | 49.76 | 49.83 | 49.80
Multilingual OF | 4 | 49.91 | 49.55 | 49.28 | 49.83 | 49.80
mOSCAR + caps | 8 | 50.62 | 48.65 | 49.68 | 49.83 | 49.80
| 16 | 50.18 | 49.01 | 49.76 | 49.83 | 49.90
| 0 | 51.33 | 49.01 | 49.52 | 49.83 | 49.70
Multilingual OF | 4 | 49.73 | 49.64 | 49.19 | 49.41 | 49.70
captions only | 8 | 49.91 | 49.10 | 49.60 | 49.75 | 49.90
| 16 | 50.09 | 49.73 | 49.60 | 49.75 | 49.80
Translate test
OF-3B MPT | 0 | 50.00 | 49.37 | 49.76 | 49.83 | 49.80
4 | 50.00 | 49.64 | 49.52 | 49.75 | 49.60
8 | 49.82 | 49.46 | 49.28 | 50.08 | 49.90
16 | 50.00 | 49.37 | 49.44 | 50.00 | 49.80
| 0 | 49.07 | 49.79 | 49.52 | 50.34 | 49.60
Multilingual OF | 4 | 49.99 | 49.79 | 48.23 | 49.75 | 49.76
mOSCAR + caps | 8 | 50.00 | 48.92 | 50.64 | 50.42 | 48.90
| 16 | 49.84 | 50.00 | 50.24 | 48.90 | 49.75
Table 12: Classification results on MaRVL. Bold is best result. |  | Ar | En | Es | Fr | Ru
---|---|---|---|---|---|---
| #shots
Random chance |  | 33.33 | 33.33 | 33.33 | 33.33 | 33.33
| 0 | 32.90 | 34.02 | 31.44 | 33.85 | 32.82
Multilingual OF. | 4 | 36.94 | 36.17 | 34.19 | 34.54 | 38.49
mOSCAR + caps. | 8 | 37.80 | 39.86 | 34.97 | 35.74 | 37.46
| 16 | 34.97 | 38.92 | 32.30 | 34.97 | 35.65
| 0 | 35.48 | 34.02 | 33.51 | 34.45 | 31.36
Multilingual OF. | 4 | 32.04 | 31.79 | 32.73 | 32.22 | 31.44
captions only | 8 | 34.02 | 33.76 | 32.04 | 35.57 | 33.16
| 16 | 32.04 | 32.99 | 33.76 | 33.17 | 31.53
Translate test
OF-3B MPT | 0 | 32.65 | - | 31.01 | 31.44 | 35.82
4 | 36.25 | - | 35.82 | 35.57 | 35.65
8 | 31.27 | - | 31.10 | 31.10 | 31.70
16 | 33.68 | - | 33.25 | 32.99 | 33.25
| 0 | 34.88 | - | 34.88 | 34.54 | 34.36
Multilingual OF. | 4 | 36.25 | - | 36.17 | 35.91 | 36.08
mOSCAR + caps. | 8 | 39.60 | - | 39.52 | 40.29 | 39.35
| 16 | 37.54 | - | 37.89 | 37.46 | 39.00
Table 13: Classification results on XVNLI. Bold is best result.

|  | Cs | De | Fr
---|---|---|---|---
| #shots
| 0 | 3.22 | 28.86 | 36.01
Multi. OF | 4 | 3.16 | 28.99 | 36.22
full | 8 | 3.44 | 28.76 | 37.41
| 16 | 3.73 | 29.19 | 37.53
| 0 | 0.00 |  00.00 |  00.00
Multi. OF | 4 | 0.00 |  00.00 |  00.00
caps. only | 8 | 0.00 |  00.00 |  00.03
| 16 | 0.00 |  00.40 |  01.82
Table 14: En$\rightarrow$X translation results on Multi30k. Bold is best
result.

|  | Cs | De | Fr
---|---|---|---|---
| #shots
| 0 | 59.09 | 63.67 | 68.51
Multi. OF | 4 | 58.77 | 63.67 | 68.51
full | 8 | 56.82 | 63.67 | 68.51
| 16 | 57.79 | 62.67 | 67.86
| 0 | 58.12 | 61.67 | 64.29
Multi. OF | 4 | 59.09 | 61.00 | 63.31
caps. only | 8 | 59.09 | 59.34 | 64.29
| 16 | 58.12 | 58.67 | 63.96
Table 15: En$\rightarrow$X CoMMuTE results. Bold is best result.