# CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding
and Generation

Wei Chen1,2, Lin Li1∗, Yongqi Yang2∗,
Bin Wen3, Fan Yang3, Tingting Gao3, Yu Wu2†, Long Chen1†
1The Hong Kong University of Science and Technology,
2Wuhan University, 3Kuaishou Technology
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://github.com/HKUST-LongGroup/CoMM
Equal contribution. Work was done when Wei Chen interned in Kuaishou.
†Corresponding authors.

###### Abstract

Interleaved image-text generation has emerged as a vital multimodal task aimed
at creating sequences of interleaved visual and textual content given a query.
Despite notable advancements in recent multimodal large language models
(MLLMs), generating integrated image-text sequences that exhibit _narrative
coherence_ and _entity and style consistency_ remains challenging due to poor
training data quality. To this end, we introduce CoMM, a high-quality Coherent
interleaved image-text MultiModal dataset designed to enhance the coherence,
consistency, and alignment of generated multimodal content. Initially, CoMM
harnesses raw data from diverse sources, focusing on instructional content and
visual storytelling, establishing a foundation for coherent and consistent
content. To further refine the data quality, we devise a multi-perspective
filter strategy that leverages advanced pre-trained models to ensure the
_development of sentences_ , _consistency of inserted images_ , and _semantic
alignment between them_. Various quality evaluation metrics are designed to
prove the high quality of the filtered dataset. Meanwhile, extensive few-shot
experiments on various downstream tasks demonstrate CoMM’s effectiveness in
significantly enhancing the in-context learning capabilities of MLLMs.
Moreover, we propose four new tasks to evaluate MLLMs’ interleaved generation
abilities, supported by a comprehensive evaluation framework. We believe CoMM
opens a new avenue for advanced MLLMs with superior multimodal in-context
learning and understanding ability.

##  1 Introduction

Interleaved image-text generation has burgeoned as an emerging multimodal
task, striving to imitate the human-like capability to alternately create
visual and textual content [1, 2]. Specifically, it aims to generate a
sequence of interleaved text descriptions and illustrative images given a
query [1]. With its interleaved multimodal generation ability, state-of-the-
art models can facilitate a variety of applications, e.g., multimodal
instruction generation, tutorial step generation, and visual storytelling.

Figure 1: Illustration of interleaved image-text content generation results
and dataset quality. (a) Given the query from CoMM, the interleaved image-text
content generation results from the model Emu2 [3] separately trained by MMC4
[2] and CoMM  (Ours). (b) The query is from the MMC4. (c) A training sample is
from the MMC4. (d) A training sample is from the our CoMM.

Recent advancements in multimodal large language models (MLLMs) have exhibited
exceptional capabilities in cross-modal generation, but they still struggle to
generate interleaved image-text sequences coherently [3, 4]. The main reason
is that most of them are trained on single image-text pairs, limited in
generating coherent and contextually integrated multimodal content [4, 5]. To
this end, some efforts integrate multiple models (e.g., GPT-4 [6] and Stable
Diffusion XL [7]) to generate open-domain interleaved image-text contents with
arbitrary formats in a training-free manner [1]. However, their results are
often dominated by the generated text descriptions, limited by the generation
model’s understanding of these textual inputs. In contrast, other research
efforts [8, 9] have attempted to train models with seamless multimodal
understanding and generation capabilities using existing interleaved image-
text datasets (e.g., MMC4 [2]).

Albeit the unprecedented progress, it is becoming increasingly evident that
data-centric methods still suffer several limitations: 1) Narrative Coherence:
The contextual relevance and coherence of generated image-text sentences are
relatively low. In Figure 1(a), the model pre-trained on the MMC4 [2] directly
draws the final cooked dessert, missing the intermediate coherent steps of
generation. These coherent generation steps not only enhance interpretability
but also boost the reasoning capabilities of the model [10]. 2) Entity and
Style Consistency: The entity and subject styles of the generated illustrated
images are inconsistent. As illustrated in Figure 1(b), the style of the blue-
gray sofa image generated by the model pre-trained on MMC4 is inconsistent
with that in the query. This consistency of entities and style has been
proposed as an important metric for evaluating the results of interleaved
image-text content generation [1]. These limitations primarily stem from the
low quality of training data caused by the poor alignment between image and
text content, i.e., _the devil is in the data_. For example in Figure 1(c),
the second image in the training sample has a noticeable gap with the context.
Additionally, the styles of all the images are almost completely different.
Thus, it raises a natural research question: _How to construct a high-quality
interleaved image-text dataset?_

In this paper, we introduce CoMM, a high-quality and coherent interleaved
image-text dataset. By “high-quality”, we mean that the interleaved image-text
content should exhibit _developed and logically structured text steps_ ,
_consistent entities and visual styles in images_ , and _strong semantic and
contextual alignment between them_. To enhance text coherence and image
consistency, we collect raw data from specific websites, e.g., WikiHow [11],
known for their instructional content or visual stories. As illustrated in
Figure 1(d), instructional content, due to its uniform intention (e.g.,
cooking steak) and structured presentation (e.g., step 1), exhibits strong
text coherence and image consistency. Additionally, we devise a multi-
perspective filter strategy consisting of: a text sequence filter, an image
sequence filter, and an image-text alignment filter. The first two leverage
large language models (LLMs), e.g., Llama3 [12], and vision-language models
(VLMs), e.g., CLIP [13], to eliminate incoherent text and images,
respectively, further enhancing overall coherence. The last one employs both
VLMs and MLLMs to comprehensively assess and refine the dataset for better
alignment between visual and textual elements.

To evaluate the data quality and make a comprehensive comparison of existing
interleaved image-text benchmarks [14, 2], we design four metrics to evaluate
the development, completeness, image-text alignment, and consistency of image-
text sequences, separately (cf. Table 1). CoMM significantly outperforms
previous datasets [14, 2] on all metrics, demonstrating its high quality. To
further demonstrate the superiority of CoMM, we conduct few-shot experiments
on various downstream tasks(e.g., VQA) [15, 16, 17, 18, 19], comparing with
MMC4 [2] and OBELICS [14]. The model trained on our CoMM consistently
outperforms others, particularly in long context settings (16 & 32 shots),
showcasing CoMM’s ability to enhance the in-context learning capacities of
MLLMs.

Furthermore, thanks to our dataset, we introduce four new challenging tasks to
evaluate the multimodal understanding and generation abilities of MLLMs:
image-to-text sequence generation, text-to-image sequence generation,
interleaved image-text content continuation, and question-based interleaved
image-text generation. To assess the performance of these newly introduced
tasks, we implement a comprehensive evaluation framework considering metrics
such as METEOR [20], ROUGE [21] for text quality, and FID [22] and IS [23] for
image quality. To accurately evaluate open-ended interleaved image-text
generation content, we develop a series of metrics and leverage powerful MLLM
(e.g., GPT4o [24]) for assessment. These tasks and evaluations pave the way
for novel approaches in evaluating MLLMs by addressing critical gaps in
current benchmarks.

Conclusively, our contributions are as follows:

  * 1)

We introduce CoMM, a high-quality coherent interleaved image-text dataset
designed to address the limitations of existing datasets by ensuring coherent
narrative, consistent entity and style, and strong semantic alignment between
images and text.

  * 2)

CoMM introduces four novel benchmark tasks and associated evaluation metrics
to comprehensively evaluate multimodal understanding and generation
capabilities of MLLMs.

  * 3)

Extensive ablations across various downstream tasks demonstrate the high-
quality of CoMM and its effectiveness in enhancing the understanding and
generation capabilities of MLLMs.

##  2 Related Work

Interleaved Image-Text Web Document Datasets. Training on interleaved image-
text web documents has demonstrated superior performance over image-
description pairs, as evidenced by studies such as Flamingo [25] and KOSMOS-1
[26]. These findings underscore the significant advantages of utilizing the
richer and more meaningful correlations inherent in interleaved image-text
documents[14]. However, the training data in these two studies is not made
publicly available.

The scarcity of interleaved image-text data has been addressed by the
introduction of two datasets: MMC4 [2] and OBELICS [14]. MMC4 builds upon the
text-only C4 corpus [27] by integrating images into text passages using CLIP
features. In contrast, OBELICS emphasizes comprehensive filtering strategies
and preserves the original structure of web pages. These datasets present a
diverse and extensive interleaved image-text corpus for multimodal language
models. Despite their depth and breadth, they exhibit notable deficiencies in
document quality, such as weaker completeness and image-text coherence (cf.,
Table 1). Additionally, as depicted in Figure 2, their documents lack
comprehensiveness in the image modality, with most documents containing only
one or two images. To address this, we propose CoMM that focuses explicitly on
coherence-rich scenarios with more image illustrations. CoMM greatly enhances
the baseline’s [28] performance (in Table 2), highlighting the critical role
of a high-quality coherent interleaved image-text training corpus.

Modeling of Interleaved Image-Text Data. The impressive performance of
language modeling by LLM motivates researchers to delve deeper into the
comprehension and generation capabilities of multimodal data. Flamingo [25]
introduces image tokens into the language modeling process, facilitating
interleaved image-text input and enabling few-shot transfer to tasks (e.g.,
VQA). Emu [29] leverages Stable Diffusion [30] as the image decoder, enabling
the generation of image or text from interleaved image-text input. DreamLLM
[8] pioneers the generation of free-form interleaved image-text outputs by
modeling text and image within a unified multimodal space.

Recent research has increasingly focused on the intrinsic requirement of
interleaved image-text modeling: capturing multimodal coherence while
generating interleaved image-text content. OpenLEAF [1] underscores the
importance of maintaining semantic and stylistic consistency between generated
images. It also proposes a training-free baseline and employs BingChat [31] to
evaluate the two types of consistency. MM-Interleaved [32] introduces the
multimodal feature synchronizer to extract context-sensitive, multi-scale
image features, thereby more effectively capturing multimodal context
coherence and visual consistency. Nonetheless, focusing solely on the model
level to meet the intrinsic multimodal coherence requirement is insufficient.
Therefore, our high-quality data that strictly adhere to this requirement are
essential to fully realize the potential of MLLMs.

##  3 CoMM Dataset

The CoMM dataset is collected from high-quality interleaved image-text content
sourced from various websites, focusing on instructional steps and visual
stories. Besides, we apply multi-perspective quality filter strategies on text
sequences, image sequences, and image-text alignment, leveraging advanced
models, e.g., CLIP [13] and LLMs [12], to enhance the dataset’s coherence and
relevance111Due to space constraints, more details are left in the Appendix..

###  3.1 Dataset Construction

Collection Process. As mentioned above, we gather specific interleaved image-
text data to preliminarily ensure the coherence and consistency of our
dataset. To achieve this, we explore websites that are likely to provide high-
quality interleaved content, such as instructional steps and visual stories.
We collect raw data from these sources1 and download the corresponding text
and images. Initially, we apply basic data cleaning techniques, including de-
duplication and filtering out images with low resolution, to refine the
collected data.

Discarding NSFW images. To address potential ethical implications, we follow
the [2] by utilizing an advanced NSFW binary image classifier [33]. This
classifier is devised as a 4-layer multi-layer perceptron (MLP) that has been
fully trained on the NSFW dataset from LAION-2B [34]. Specifically, it
utilizes features extracted from OpenAI’s CLIP ViT-L/14 model[13] as input,
achieving an impressive accuracy of 97.4% on the NSFW test dataset. For each
image processed, we employ the classifier to predict the NSFW probability.
Images yielding a probability greater than 0.1 are automatically excluded from
further consideration.

Qualities Filter Strategy. Due to the inherent noises in the gathered raw
data, the data still contains inconsistent and irrelevant content. Thus, we
further employ a multi-perspective filter strategy to ensure the quality and
coherence of the dataset. Specifically, it involves three following
components1:

1) Text Sequence Filter.  For text sequences, the most important aspect is the
contextual development and cohesive connection between them. To maintain a
smooth and logical flow of the text, we employ LLMs (e.g., Llama3 [12]) to
evaluate the development and coherence among text steps in a document (prompt
details are in the Appendix). According to evaluation scores, we eliminate
sequences that lack coherence and relevance in their content, producing a
higher-quality text corpus.

2) Image Sequence Filter. As for image sequences, visual consistency and
relevance are key factors in maintaining the quality of the content. To assess
the coherence and development of image sequences, we devise a metric
$\mathcal{F}(\cdot)$ through the CLIP’s visual encoder [13]:

$\mathcal{F}(\\{x_{i}|1\leq i\leq
N\\})=\frac{1}{N-1}\sum_{i=2}^{N}\text{Sim}(x_{i},x_{i-1})-\frac{2}{(N-1)(N-2)}(\sum_{i=2}^{N}\sum_{j=1}^{i-1}\text{Sim}(x_{i},x_{j})),$
(1)

where $\\{x_{i}|1\leq i\leq N\\}$ denotes the sequence of images from 1 to
$N$, $\text{Sim}(\cdot,\cdot)$ denotes the cosine similarity score between two
images calculated by the CLIP’s visual encoder. The first term denotes the
average similarity between consecutive images, promoting smooth transitions
and visual coherence within the sequence, while the second term represents the
average similarity between all pairs of images, ensuring overall diversity and
development. In this way, we can effectively balance coherence and diversity,
ensuring that the image sequence is both visually consistent and contextually
developed. By applying the image sequence filter, we can identify and
eliminate sequences that lack connection or development, resulting in more
polished and visually appealing images.

3) Image-Text Alignment Filter. This filter aims to ensure that the images and
text are not only relevant to each other but also maintain a coherent
narrative throughout the document. We first utilize CLIP [13] to calculate the
CLIP similarity scores between images and text and remove those whose scores
are less than 0.1. Since CLIP is trained on individual image-text pairs, where
the text often serves as the image’s caption, it tends to favor concrete
descriptions (i.e., object descriptions and attributes) [35]. However, in an
interleaved image-text document, the text for each step describes progress
with associated images (e.g., in Figure 1(d), each text focuses on a cooking
action linked to previous steps) which is associated with previous steps.
Therefore, we also employ advanced MLLM (e.g., GPT-4o [24] or Llama3 [12]) to
assess and filter image-text pairs, as these models can better judge current
image-text alignment by considering the previous context.

Figure 2:  Visualization of the image-sentence numbers per document
distribution of three datasets. The $\mu$ and $M$ denote the mean/median
number of images/sentences in documents, respectively.

###  3.2 Dataset Statistic

After filtering, our dataset contains 227K documents with 2.28M images.
Further, Figure 2 illustrates the distribution of image-sentence number per
document across MMC4, OBELICS, and CoMM datasets. Darker colors indicate a
higher number of documents. The $\mu$ and $M$ represent the average and the
most common number of images/sentences per document, respectively. For MMC4
and OBELICS, the distribution of images per document is highly concentrated
around very low values of $M$ (2 and 1 per document, respectively). This
indicates a significant scarcity of the image modality in their multimodal
data, which directly limits the models’ multimodal in-context learning
ability.

Examining Figure 2(a) and Figure 2(b) distributions, it is evident that the
number of images does not significantly change with an increase in the number
of sentences. However, in practical scenarios, longer documents are more
likely to contain a greater number of illustrations, which aids in modeling
longer multimodal contexts. In contrast, CoMM features a larger quantity of
images per document ($M$ = 4, $\mu$ = 10.1), which significantly enhances the
dataset’s image modality while focusing on multimodal coherence. Furthermore,
CoMM exhibits a trend where longer documents tend to include more image
illustrations, thereby facilitating the modeling of longer multimodal
contexts.

##  4 Interleaved Generation Task and Benchmark

###  4.1 Task Formulation

In this section, we propose four benchmark tasks to evaluate MLLMs’
interleaved generation capabilities. For ease of presentation, we utilize
$\\{x_{i}|1\leq i\leq N\\}$ and $\\{t_{i}|1\leq i\leq N\\}$ to represent the
image sequences and text sequences, respectively. Besides, we use
$t_{\text{prompt}}$ as the text prompt, and $\mathcal{F}_{\theta}(\cdot)$ as
the model with parameter $\theta$. The detailed formulations of tasks are
outlined as follows:

Task 1: Image-to-Text Sequence Generation. This task evaluates the model’s
capability to generate coherent and contextually appropriate narratives based
on sequential images. The objective is to predict the corresponding text
sequences from the given image sequences:

$\\{t_{i}|1\leq i\leq
N\\}=\mathcal{F}_{\theta}(t_{\text{prompt}},x_{1},...,x_{N}).$ (2)

Task 2: Text-to-Image Sequence Generation. This task assesses the model’s
proficiency in generating image sequences that accurately represent provided
textual descriptions. The goal is to predict the corresponding image sequences
from the given text sequences:

$\\{x_{i}|1\leq i\leq
N\\}=\mathcal{F}_{\theta}(t_{\text{prompt}},t_{1},...,t_{N}).$ (3)

Task 3: Interleaved Image-Text Content Continuation. It measures the model’s
ability to seamlessly continue content in an interleaved image-text format.
The model needs to predict the outputs of the remaining image-text steps based
on the interleaved image-text content in the previous $k$ steps:

$\\{t_{i},x_{i}|k<i\leq
N\\}=\mathcal{F}_{\theta}(t_{\text{prompt}},t_{1},x_{1},...,t_{k},x_{k}).$ (4)

Task 4: Question-based Interleaved Image-Text Generation. This challenging
task requires the model to generate interleaved content based on a given
question, testing its understanding and generative reasoning abilities across
modalities. The model needs to predict the outcomes of all image-text steps
based on the given question:

$\\{t_{i},x_{i}|1\leq i\leq N\\}=\mathcal{F}_{\theta}(t_{\text{prompt}}).$ (5)

###  4.2 Evaluation Metric

This section lists the evaluation metrics for assessing the quality of
generated results. Our evaluation framework1 measures textual accuracy, visual
fidelity, and the coherence of multi-modal outputs.

Evaluation for Text Sequence Generation. Generating narratives for a series of
images shares similarities with the established task of image captioning.
Consequently, we directly apply traditional image captioning metrics like
ROUGE [21] and METEOR [20] at both the step and document levels. Additionally,
we propose Illustration Relevance Score (IRS) to assess the relevance between
image illustration and text context in the final document. Specifically, we
simulate the human document creation process by first generating detailed
image descriptions from the text context and then assessing the consistency
between these descriptions and corresponding images using GPT-4o [24].

Evaluation for Image Sequence Generation.  We evaluate the quality of image
illustrations on four key assessment aspects including image quality, style
consistency between generated images, reconstruction on referenced images, and
IRS. To evaluate image quality, we use Fréchet Inception Distance (FID) [22]
with ground truth reference and Inception Score (IS) [23]. For assessing style
consistency, we employ the Structural Similarity Index Measure (SSIM), and for
ground truth reconstruction evaluation, we adopt the Peak Signal-to-Noise
Ratio (PSNR).

Evaluation for Interleaved Image-Text Generation. We use MLLM (GPT-4o [24])
for a comprehensive assessment of linguistic and visual coherence. GPT-4o
evaluates four aspects: image sequence coherence, image quality, document
completeness, and IRS. For image sequence coherence, we check style, entity
consistency, and content trend alignment between image and text sequence.
Image quality is evaluated on authenticity, integrity, clarity, and
aesthetics. Document completeness is scored based on how thoroughly the
content covers the topic.

##  5 Experiments

###  5.1 Datasets Quality Comparison

Settings. In this section, we compare the quality of our dataset with previous
interleaved image-text datasets (e.g., MMC4 [2] and OBELICS [14]) focusing on
text sequences, image sequences, and image-text alignment. We utilize powerful
LLMs (e.g., Llama3 [12] and GPT-4o [24]) to assess the text development, text
completeness, and image-text alignment of these datasets. Each metric is
scored on a scale of 0 to 10, with detailed definitions and prompts provided
in the Appendix. For MMC4 and OBELICS, we randomly sample 5,000 documents from
their data. We evaluate our entire dataset with Llama3 and randomly sample
5,000 documents for assessment by GPT-4o. To evaluate image sequence quality,
we employ the metric $\mathcal{F}$ as cited in Eq. (1).

Table 1: Quality comparison of interleaved image-text datasets. “DLP” stands for Development, “CPL” signifies Completeness, “ITA” represents Image-Text Alignment, “ImgS” refers to Image Sequence, “L.3” denotes Llama3 [12], and “G.4o” is GPT-4o [24].  Datasets | DLP | CPL | ITA | ImgS
---|---|---|---|---
L.3 / G.4o | L.3 / G.4o | L.3 / G.4o
MMC4 [2] | 5.56 / 4.75 | 6.28 / 5.12 | 6.53 / 4.66 | 0.21
OBELICS [14] | 5.60 / 5.97 | 6.51 / 5.88 | 5.00 / 3.81 | 1.00
CoMM (Ours) |  6.93 / 7.64 |  7.44 / 7.07 |  7.46 / 8.91 | 4.27

Quantitative Results. As shown in Table 1, our dataset surpasses MMC4 and
OBELICS across all evaluated dimensions. Notably, the score difference in
development, completeness, and image-text alignment is more pronounced when
evaluated by GPT-4o, a more powerful MLLM. Furthermore, the image sequence
metric underscores the superior quality of our data, reflecting higher scores
compared to MMC4 and OBELICS. These results affirm the high quality of our
dataset in interleaved image-text data.

###  5.2 In-Context Multimodal Understanding

Setting.  To further demonstrate the high-quality and validate the efficacy of
our dataset in promoting in-context understanding, we conduct experiments
using few-shot learning settings on several downstream tasks (COCO [18],
Flickr30k [36], VQAv2 [15], OKVQA [19], TextVQA [37], VizWiz [16], and
HatefulMemes [17]). Few-shot learning is critical for assessing a model’s
ability of in-context understanding. In our experiments, few-shot scenarios
are simulated by providing the model with a small number of example queries
and their corresponding answers before it is evaluated on new queries. We
consider various shot settings (0, 4, 8, 16, and 32) to understand the model’s
performance across different levels of supervision.

Quantitative Results. Table 2 presents the results for the baseline model and
its enhancements with MMC4, OBELICS, and our proposed dataset. As observed
from Table 2, our method consistently outperforms the performance of MMC4 and
OBELICS across all shot settings, demonstrating superior generalization and
in-context learning capabilities. Particularly noticeable improvements are
seen in the COCO and TextVQA datasets, where our method exceeds the baseline
by significant margins in nearly all shot settings. This superior performance
underlines the strength of our dataset in enabling models to better learn from
context and apply learned knowledge to unseen instances. Furthermore, our
dataset surpasses the baseline in all tasks with more shot settings,
maintaining its advantage with longer context few-shot data, as shown in the
16-shot and 32-shot scenarios.

Table 2: Performance comparison of different datasets, with the baseline model Open-Flamingo 9B [28]. Evaluations were in an open-ended setting for VQA tasks with random in-context examples. (Task, Metric, Query Split): (COCO [18], CIDEr, test), (Flickr30k [36], CIDEr, test (Karpathy)), (VQAv2 [15], VQA accuracy, testdev), (OKVQA [19], VQA accuracy, val), (TextVQA [37], VQA accuracy, val), (VizWiz [16], VQA accuracy, testdev), (HatefulMemes [17], ROC-AUC, test seen).  | Shot | COCO | Flickr30k | VQAv2 | OKVQA | TextVQA | VizWiz | HatefulMemes
---|---|---|---|---|---|---|---|---
Baseline | 0 | 79.5 | 59.5 | 52.7 | 37.8 | 24.2 | 27.5 | 51.6
\+ MMC4 [2] | 97.1 | 67.0 | 45.8 | 29.8 | 23.4 | 15.0 | 54.6
\+ OBELICS [14] | 83.5 | 51.5 | 50.9 | 38.9 | 25.6 | 21.9 | 53.8
\+ CoMM (Ours) | 100.3 | 69.8 | 52.6 | 41.3 | 33.5 | 23.6 | 54.8
Baseline | 4 | 89.0 | 65.8 | 54.8 | 40.1 | 28.2 | 34.1 | 54.0
\+ MMC4 [2] | 103.3 | 68.8 | 49.5 | 36.7 | 28.7 | 24.4 | 55.0
\+ OBELICS [14] | 103.7 | 65.2 | 53.2 | 43.0 | 31.4 | 29.9 | 55.7
\+ CoMM (Ours) | 107.0 | 73.2 | 53.6 | 42.9 | 35.7 | 30.1 | 55.2
Baseline | 8 | 96.3 | 62.9 | 54.8 | 41.1 | 29.1 | 38.5 | 54.7
\+ MMC4 [2] | 103.6 | 67.9 | 50.5 | 39.0 | 28.8 | 32.7 | 51.3
\+ OBELICS [14] | 101.9 | 63.9 | 53.7 | 44.5 | 32.2 | 37.2 | 51.2
\+ CoMM (Ours) | 108.0 | 74.8 | 54.2 | 44.5 | 35.4 | 37.9 | 52.5
Baseline | 16 | 98.8 | 62.8 | 54.3 | 42.7 | 27.3 | 42.5 | 53.9
\+ MMC4 [2] | 102.9 | 65.5 | 50.0 | 39.1 | 26.4 | 36.9 | 57.2
\+ OBELICS [14] | 100.8 | 58.0 | 53.1 | 45.3 | 30.2 | 42.4 | 57.8
\+ CoMM (Ours) | 109.3 | 71.5 | 54.2 | 46.9 | 35.8 | 43.5 | 59.9
Baseline | 32 | 99.5 | 61.3 | 53.3 | 42.4 | 23.8 | 44.0 | 53.8
\+ MMC4 [2] | 94.9 | 56.4 | 48.3 | 35.6 | 20.0 | 37.7 | 49.2
\+ OBELICS [14] | 96.5 | 53.1 | 52.0 | 43.6 | 26.8 | 44.5 | 56.1
\+ CoMM (Ours) | 111.9 | 70.7 | 54.0 | 46.6 | 34.9 | 45.7 | 57.3
Table 3: Comparison of performance among MiniGPT-5 [4], SEED-Llama [38], and Emu2 [29] on the four generation tasks. “M.S.” and “R.S.” represent METEOR [20] and ROUGE_L [21] for step-by-step evaluation, while “M.W.” and “R.W.” correspond to METEOR and ROUGE_L for whole document evaluation. “CPL.” stands for Completeness, “ImgQ” indicates Image Quality, “IRS” means Illustration Relevance Score, and “$\downarrow$” denotes that lower values are better. Methods | Size | I2T Sequence Generation (Task1) | Continuation Generation (Task3)
---|---|---|---
M.S. | R.S. | M.W. | R.W. | IRS | Style | Entity | Trend | CPL. | ImgQ | IRS
MiniGPT-5 [4] | 7B | 4.60 | 8.64 | 9.10 | 10.67 | 7.39 | 5.58 | 5.21 | 5.24 | 6.33 | 6.36 | 2.56
SEED-Llama [38] | 8B | 2.29 | 6.34 | 4.20 | 7.33 | 6.99 | 6.28 | 5.84 | 5.72 | 6.28 | 6.55 | 2.92
SEED-Llama [38] | 14B | 3.16 | 8.12 | 5.71 | 8.17 | 8.13 | 6.68 | 6.22 | 6.13 | 6.66 | 6.67 | 3.23
Emu2 [3] | 33B | 3.65 | 8.41 | 7.43 | 8.62 | 5.01 | 8.22 | 7.99 | 7.97 | 8.49 | 8.62 | 2.42
Methods | Size | T2I Sequence Generation (Task2) | Question-based Generation (Task4)
FID $\downarrow$ | IS | SSIM | PSNR | IRS | Style | Entity | Trend | CPL. | ImgQ | IRS
MiniGPT-5 [4] | 7B | 56.51 | 7.60 | 20.66 | 7.57 | 5.48 | 5.65 | 5.20 | 5.25 | 5.81 | 6.15 | 2.71
SEED-Llama [38] | 8B | 57.96 | 8.40 | 20.53 | 7.87 | 5.27 | 7.55 | 6.81 | 6.15 | 5.13 | 6.36 | 1.46
SEED-Llama [38] | 14B | 66.23 | 11.06 | 20.83 | 8.12 | 6.24 | 7.51 | 6.61 | 6.30 | 6.13 | 6.66 | 2.50
Emu2 [3] | 33B | 60.45 | 6.34 | 20.96 | 7.91 | 4.17 | 8.41 | 7.56 | 7.63 | 7.54 | 7.59 | 2.02

###  5.3 Interleaved Generation Tasks

Baselines. This section introduces baselines developed from several state-of-
the-art approaches, i.e., MiniGPT-5 [4], SEED-Llama [38], and Emu2 [3], to
assess their effectiveness in generating interleaved image-text content.
Details of these models are thoroughly provided in the Appendix. To construct
the baselines, we fine-tune each of these models using our CoMM dataset. We
employ a rigorous evaluation, assessing them on benchmark tasks involving
interleaved image-text generation and understanding.

Figure 3: Visualization of interleaved image-text content generation from
SEED-Llama [38] (Top) and MiniGPT-5 [4] (Bottom) separately trained by MMC4
[2] and CoMM  (Ours).

Quantative Results. As shown in Table 3, the quantitative results clearly
illustrate the varying performance of MiniGPT-5, SEED-Llama, and Emu2 across
the four interleaved generation tasks. For I2T Sequence Generation (Task1),
MiniGPT-5 achieves the highest scores in METEOR, ROUGE_L, but falls short in
Illustration Relevance Score (IRS) where SEED-Llama (14B) leads with a score
of 8.13. For T2I Sequence Generation (Task2), MiniGPT-5 excels with the lowest
FID score of 56.51, while SEED-Llama (14B) surpasses IS, SSIM, PSNR, and IRS
metrics, suggesting stronger visual fidelity and semantic similarity. Emu2
tops in Continuation Generation (Task3) with the highest scores in Style,
Entity, Trend, Completeness, and Image Quality. In Question-based Generation
(Task4), Emu2 dominates in Style, Entity, Trend, Completeness, and Image
Quality, while MiniGPT-5 manages to secure the lead in IRS. Overall, these
results highlight the strengths and weaknesses of each model in handling
different aspects of interleaved image-text generation tasks, providing
valuable insights for future research and model improvements.

Multimodal Generation Qualitative Analysis. As illustrated in Figure 1 and
Figure 3, the interleaved image-text content generated by models [3, 4, 38]
trained on MMC4 [2] shows a noticeable difference when compared to the content
generated by models trained on our datasets. Our models produce content that
is more consistent with the previous input context (e.g., textual coherence
and entity consistency). By ensuring higher coherence, consistency, and
alignment, CoMM sets a new standard for training datasets in the realm of
multimodal large language models. More visualizations are available in the
Appendix.

##  6 Conclusion

In this paper, we introduced CoMM, a high-quality coherent interleaved image-
text dataset designed to overcome the limitations of existing multimodal large
language models (MLLMs) in generating coherent and contextually integrated
content. By sourcing diverse instructional content and visual storytelling,
and implementing a rigorous filtering strategy, CoMM ensures high-quality
alignment of text and images. Extensive few-shot experiments demonstrate that
CoMM significantly enhances the in-context learning capabilities of MLLMs
across various downstream tasks, surpassing existing datasets in text
completeness, image-text alignment, and entity consistency, thereby setting a
new benchmark in the domain of multimodal datasets.

Additionally, we proposed four novel tasks to evaluate the interleaved
generation abilities of MLLMs, supported by a comprehensive evaluation
framework with detailed and robust metrics. Our contributions underscore the
importance of high-quality interleaved image-text data in advancing MLLM
capabilities. By setting new standards for dataset quality and evaluation,
CoMM paves the way for next-generation AI models that excel in multimodal
understanding and in-context learning.

## References

  * [1] Jie An, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Lijuan Wang, and Jiebo Luo.  Openleaf: Open-domain interleaved image-text generation and evaluation.  arXiv preprint arXiv:2310.07749, 2023.
  * [2] Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi.  Multimodal c4: An open, billion-scale corpus of images interleaved with text.  NeurIPS, 2023.
  * [3] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, et al.  Generative multimodal models are in-context learners.  arXiv preprint arXiv:2312.13286, 2023.
  * [4] Kaizhi Zheng, Xuehai He, and Xin Eric Wang.  Minigpt-5: Interleaved vision-and-language generation via generative vokens.  arXiv preprint arXiv:2310.02239, 2023.
  * [5] Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov.  Generating images with multimodal language models.  NeurIPS, 2024.
  * [6] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [7] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.
  * [8] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al.  Dreamllm: Synergistic multimodal comprehension and creation.  arXiv preprint arXiv:2309.11499, 2023.
  * [9] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan.  Seed-x: Multimodal models with unified multi-granularity comprehension and generation.  arXiv preprint arXiv:2404.14396, 2024.
  * [10] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  NeurIPS, 2022.
  * [11] WikiHow.  https://www.wikihow.com.
  * [12] Meta.  https://llama.meta.com/llama3, 2024.
  * [13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [14] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al.  Obelics: An open web-scale filtered dataset of interleaved image-text documents.  NeurIPS, 2023.
  * [15] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.  Making the v in vqa matter: Elevating the role of image understanding in visual question answering.  In CVPR, 2017.
  * [16] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham.  Vizwiz grand challenge: Answering visual questions from blind people.  In CVPR, 2018.
  * [17] Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine.  The hateful memes challenge: Detecting hate speech in multimodal memes.  NeurIPS, 2020.
  * [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [19] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi.  Ok-vqa: A visual question answering benchmark requiring external knowledge.  In CVPR, 2019.
  * [20] Satanjeev Banerjee and Alon Lavie.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In ACL Workshop, pages 65–72, 2005.
  * [21] Chin-Yew Lin.  Rouge: A package for automatic evaluation of summaries.  In Text summarization branches out, 2004.
  * [22] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  NeurIPS, 2017.
  * [23] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen.  Improved techniques for training gans.  NeurIPS, 2016.
  * [24] OpenAI.  https://openai.com/index/hello-gpt-4o/, 2024.
  * [25] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  NeurIPS, 2022.
  * [26] S Huang, L Dong, W Wang, Y Hao, S Singhal, S Ma, T Lv, L Cui, OK Mohammed, Q Liu, et al.  Language is not all you need: aligning perception with language models.  arXiv preprint arXiv:2302.14045, 2023.
  * [27] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  JMLR, 2020.
  * [28] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al.  Openflamingo: An open-source framework for training large autoregressive vision-language models.  arXiv preprint arXiv:2308.01390, 2023.
  * [29] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang.  Emu: Generative pretraining in multimodality.  In ICLR, 2023.
  * [30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [31] Microsoft.  Bingchat.  https://www.microsoft.com/en-us/edge/features/bing-chat, 2023.  Accessed: 2023-10-09.
  * [32] Changyao Tian, Xizhou Zhu, Yuwen Xiong, Weiyun Wang, Zhe Chen, Wenhai Wang, Yuntao Chen, Lewei Lu, Tong Lu, Jie Zhou, et al.  Mm-interleaved: Interleaved image-text generative modeling via multi-modal feature synchronizer.  arXiv preprint arXiv:2401.10208, 2024.
  * [33] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  NeurIPS, 36, 2023.
  * [34] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  NeurIPS, 35:25278–25294, 2022.
  * [35] Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mahmoud Assran, Andrew Gordon Wildon, Aaron Courville, and Nicolas Ballas.  Modeling caption diversity in contrastive vision-language pretraining.  arXiv preprint arXiv:2405.00740, 2024.
  * [36] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  TACL, 2014.
  * [37] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In CVPR, 2019.
  * [38] Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan.  Making llama see and draw with seed tokenizer.  arXiv preprint arXiv:2310.01218, 2023.
  * [39] Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, and Weidi Xie.  Intelligent grimm–open-ended visual storytelling via latent diffusion models.  arXiv preprint arXiv:2306.00973, 2023.
  * [40] Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang.  Cogvlm: Visual expert for pretrained language models, 2023.
  * [41] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  Lora: Low-rank adaptation of large language models.  arXiv preprint arXiv:2106.09685, 2021.
  * [42] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.

## Appendix

##  Appendix A Details of Our Dataset

Data Source.  As shown in Table 4, we collect our data from five sources. For
StoryGen [39], we use only the original source images and employ Llama3 [12]
to generate a more coherent and developed story text corpus. After filtering,
CoMM contains 227K documents with 2.28M images and 139M text tokens. The ratio
of different data sources is presented in Table 4. We randomly sample 500
documents each for the validation and test splits, with the remaining
documents used for training.

Table 4: Collected data source of CoMM.  Data Source | Document Ratio | Image Ratio | Text Token Ratio
---|---|---|---
www.wikihow.com | 57.78% | 30.02% | 48.71%
www.ehow.com | 3.31% | 4.11% | 2.53%
storybird.com | 1.54% | 1.22% | 0.57%
StoryGen [39] | 2.19% | 1.76% | 1.90%
www.instructables.com | 35.18% | 62.89% | 46.28%

Dataset Visualization.  As displayed in Figure 4, we compared samples from the
MMC4 [2] dataset and our CoMM dataset derived from various data sources. The
comparison reveals: 1) MMC4 exhibits relatively poor style consistency. For
the same “duck” entity, one image is in cartoon style while another is
realistic (cf. Figure 4(a)). 2) Our CoMM dataset maintains high consistency in
both entity representation and style across all data sources. 3) CoMM
demonstrates enhanced narrative and stylistic diversity, covering different
content orientations: Instructables focuses on narrative closure, StoryBird
and StoryGen emphasize cartoon-style storytelling, while WikiHow and eHow
concentrate on instructional content through illustrative methods.

Figure 4: Comparison of samples from different datasets. (a) from the MMC4 [2]
dataset; (b)-(f) from different data sources within CoMM (Ours) dataset.

##  Appendix B Ethical Discussion and License

Ethical Discussion.  Collecting data from online sources comes with the risk
of encountering content that may not be suitable for all audiences.
Fortunately, this risk is minimized in our case because we focus on high-
quality data, such as instructional steps and visual stories. Besides, these
specific websites have their review/ editorial processes, which significantly
improve data quality and reduce potential hazards. For example, WikiHow [11]
claims that "the average WikiHow article has been edited by 23 people and
reviewed by 16 people". To further ensure the integrity of our dataset, we
perform a rigorous screening process to filter out any NSFW content (as
mentioned in Sec 3.1), trying to maintain a clean, and reliable dataset
suitable for all users.

License and Author Statement.  We release this dataset under a CC-BY license
and Terms of Use that require disclosure when used for model training. This
license does not override the original content licenses; all use must comply
with the original licenses and data subjects’ rights. We clarify the user’s
responsibilities and liabilities here. While we’ve tried our best to ensure
data accuracy and legality, we cannot guarantee absolute correctness. We
assume no liability for rights violations, including but not limited to
copyright, privacy issues, or misuse of sensitive information.

By using this dataset, you accept full responsibility for legal or other
consequences. You agree to adhere to all relevant laws, regulations, and
ethical guidelines. Accessing or using this dataset signifies your acceptance
of this statement and the CC-BY license terms. Disagreement with these terms
means you are not authorized to use the dataset.

##  Appendix C Filter Strategy and Evaluation Details

###  C.1 Data Quality Filter Prompt.

Below is the prompt for ensuring data quality in text and image-text
alignment. When using GPT-4o [24], which can see images directly, we input the
original images directly. However, when using Llama3 [12], which cannot see
images, we first employ CogVLM [40] to convert the image into a detailed
caption, then input it in the “<IMAGE>image description</IMAGE>” format.

⬇

You are a master of multi-modal evaluation. Your task is to evaluate the
quality of a docs that contains images and text. Images will be presented in
the format <IMAGE>image description</IMAGE>. The textual content will be
presented as plain text. Evaluate the following criteria:

1. Development: Assess the coherence and logical flow of the data. Only the most logically consistent and well-integrated contexts should receive high scores.

2. Completeness: Check if the content provides a comprehensive and detailed overview of the topic. Full scores should only be given for thorough and exhaustive coverage.

3. Interleaving of Images and Text: Ensure that the images and text are perfectly aligned. Discrepancies or inconsistencies should result in significant deductions.

Scores should range widely to highlight exceptional quality or notable
deficiencies. Each criterion should be evaluated and concluded with a score on
a scale from 0 to 10, where 0-2 indicates major deficiencies and 8-10
indicates exemplary performance. Structure your response as follows:

<Development>

<Problem>Brief description of any issues</Problem>

<Score>Numerical rating</Score>

</Development>

<Completeness>

<Problem>Brief description of any gaps</Problem>

<Score>Numerical rating</Score>

</Completeness>

<Image-Text Interleaving>

<Problem>Brief description of any discrepancies</Problem>

<Score>Numerical rating</Score>

</Image-Text Interleaving>

Emphasize the identification of particularly strong or weak points in the
Problem section. This feedback will guide you to adjust scores to be more
polarized, reflecting a clear distinction between high and low quality.

Data to Review:

<data>

{}

</data>

###  C.2 Evaluation Prompt for Interleaved Generation Content.

We explain the motivation and detailed prompt design of the GPT-4o evaluation
here.

Document Completeness, Image Sequence Coherence and Image Quality. Here are
the prompts for evaluating the document completeness, image sequence
coherence, and image quality.

⬇

We are evaluating the results of a model designed for generating interleaved
image-text documents. The model’s input, starting with "INPUT:", can either be
the beginning of a text-image interleaved document or a specified topic. Its
output, starting with "OUTPUT:", will then be either a continuation of the
document or content generated based on the given topic. The image with the
index i will be enclosed by the symbols "<Img_i>" and "</Img_i>". The images
are numbered sequentially from 0 to N (including the input images).

As an expert in multimodal evaluation, your task is to assess the quality of
the output that includes both images and text. The images are numbered
sequentially from 1 to n (include the input images). Use the guidelines below
to assign a final score.

Scoring Guidelines:

- 0-3: Major deficiencies, misalignment, or inconsistency

- 4-7: Minor gaps, misalignment, or inconsistency

- 8-10: Complete and thorough alignment, strong consistency

Scoring Criteria:

1. Image Coherence:

- Evaluate the consistency of style and entity between the output images. Assess whether the trend shown by the image sequence aligns with the text. Finally, an overall consistency score will be assigned to the image sequence.

2. Completeness:

- Summarize the output document’s topic and evaluate how thorough and comprehensive the output content is.

Evaluate Thoroughness and Comprehensiveness:

Is the text content complete? Is there anything missing?

Is the image content complete? Are any images missing?

Do the images and text fully support each other? Is there any missing image or
text?

3. Image Quality:

- Evaluate the quality of the output images based on the following aspects:

Realism: Determine whether the image resembles a real scene or object and
identify any signs of artificial model synthesis.

Completeness: Check if the objects in the image are fully intact, without any
noticeable missing parts, truncation, or damage.

Clarity: Determine if the details are sufficient and if the image is free of
blurriness or out-of-focus areas.

Composition balance: Evaluate the aesthetic quality and balance of the image
composition, ensuring that the main subjects are well-framed and the
composition is visually pleasing.

Assume the index of the first image in the output is K.

JSON Output Structure:

{

"scores": {

"Image_Coherence": {

"pair_scores": {

"image_K_and_K+1": {

"style_consistency": 0-10,

"entity_consistency": 0-10,

"justification": "Brief explanation of any gap"

},

"image_K+1_and_K+2": {

"style_consistency": 0-10,

"entity_consistency": 0-10,

"justification": "Brief explanation of any gap"

}

// Continue for remaining pairs...

},

"overall_score": {

"style_consistency": 0-10,

"entity_consistency": 0-10,

"trend_consistency": 0-10,

"overall_consistency": 0-10,

"justification": "Brief explanation of overall consistency"

}

},

"Completeness": {

"Summarize": "brief summary",

"Justification": "brief justification of any issue",

"Score": 0-10

},

"Image_Quality":{

"Score": 0-10,

"Justification": "brief justification of any deficiencies in image quality",

}

}

}

Data to Review:

Illustration Relevance Score. In our evaluation tests, we found that GPT-4o
has a good understanding of images, but the text context will seriously
influence this understanding. For example, when only images from a step-by-
step instruction document are input, GPT-4o can accurately describe both the
correctly ordered and reversed-ordered image contents. However, when the
interleaved texts and images are input together, GPT-4o tends to produce
similar descriptions for both the correctly ordered and reversed-ordered
images, which is incorrect.

Thus we design an evaluation process similar to human document creation to
mitigate this limitation. We first generate the required image content based
on the text context and then evaluate the consistency between the image
descriptions and the corresponding images. Specifically, this process involves
two model invocations, the following are the prompts for the first invocation:

⬇

We are evaluating the results of a model designed for generating interleaved
image-text documents. The model’s input, starting with "INPUT:", can either be
the beginning of a text-image interleaved document or a specified topic. Its
output, starting with "OUTPUT:", will then be either a continuation of the
document or content generated based on the given topic. The image with the
index i will be enclosed by the symbols "<Img_i>" and "</Img_i>". The images
are numbered sequentially from 0 to N (include the input images). Now we hide
the output’s images while preserving the "<Img_i></Img_i>". As an expert in
multimodal evaluation, you are responsible for predicting the removed image’s
content based on the input and the output text context.

Tasks:

1. Predict Each Image’s Content:

For each image content prediction, predict the most probable and suitable
image content based on the input and text context in the output. The
description should consider the illustration needs (What should the image
illustrate to complement its surrounding text context?), content description
(Provide a detailed description of what the image should contain.), and
context coherence (Ensure that the final narrative flows well and forms a
complete, coherent document.).

Assume the index of the first removed image in the output is K.

JSON Output Structure:

{

"Tasks": {

"Create an Interleaved Text-Image Document": {

"Content of Image K": "predicted content of image K",

"Content of Image K+1": "predicted content of image K+1",

...

"Content of Image N": "predicted content of image N",

}

}

}

Data to Review:

After the first invocation, we reorganize the output descriptions with
corresponding images and start the second model invocation with the following
prompts:

⬇

As an expert in image description evaluation, your job is to assess the
consistency between two sets of images and their corresponding descriptions.
Use the criteria below to assign a final score.

The input will be formatted as description-image pairs like <Description_i>
image description </Description_i> <Img_i> image </Img_i>. Note that sometimes
one of the descriptions and the image is missing, just score that input data
as 0!

Scoring Guidelines:

0-3: Major deficiencies/misalignment/inconsistency,

4-7: Minor gaps/misalignment/inconsistency,

8-10: Complete and thorough alignment, strong consistency.

Scoring Criteria:

1. Consistency:

- Task: Evaluate the consistency between each image and its corresponding description.

JSON Output Structure:

{

"Consistency": {

"image_1_score": 0-10,

"image_2_score": 0-10,

...

"image_n_score": 0-10,

"overall_score": 0-10,

"Justification": "Brief justification of any issue identified"

}

}

Data to Review:

The output overall score is the IRS.

##  Appendix D Model Training Detail

MiniGPT-5 [4] combines the Stable Diffusion with LLMs through “generative
vokens”. This model adopts a two-stage training strategy tailored for
description-free multimodal generation. Initially, it focuses on extracting
high-quality text-aligned visual features. In the subsequent stage, it ensures
optimal coordination between visual and textual prompts, significantly
enhancing its ability to generate coherent multimodal content.

Training Settings.  We train MiniGPT-5 using 8 A100-80G GPUs, fine-tuning the
parameters of the LoRA [41] layers (the rank is 32) in the LLM backbone and
the Feature Mapper for output visual tokens. The learning rate is set to 5e-5
for the LoRA layers and 5e-4 for the other trainable parameters, with a total
of 5 training epochs. All other settings follow those of MiniGPT-5.

SEED-Llama [38] equips the pre-trained LLM [42] with a VQ-based image
tokenizer (SEED), which processes images into discrete tokens. This tokenizer
utilizes a 1D causal dependency to align visual tokens with the autoregressive
nature of LLMs, enhancing semantic coherence between text and images. Enhanced
by extensive multimodal pretraining and fine-tuning under a next-word-
prediction objective, SEED-Llama excels in handling both comprehension and
generation tasks within a unified multimodal framework.

Training Settings. We train SEED-Llama-8B and SEED-Llama-14B using 8 A100-80G
GPUs. Only the parameters of the LoRA [41] layers (with a rank of 16) in the
LLM backbone are fine-tuned. The learning rate is set to 1e-5, and the
training consists of 10,000 steps.

Emu2 [3] is a generative multimodal model, trained on large-scale multimodal
sequences with a unified autoregressive objective. This model showcases
significant capabilities in multimodal in-context learning, adept at complex
tasks that require on-the-fly reasoning, such as visual prompting and object-
grounded generation.

Training Settings.  Emu2 is trained using 16 A100-80G GPUs. We fine-tune the
parameters in the linear projection layer for input and output visual
embeddings, as well as the LoRA [41] layers (with a rank of 32) within the LLM
backbone. The learning rate is set to 5e-5, and the training lasts for 5
epochs.

##  Appendix E More Generation Visualization

Qualitative Analysis of Interleaved Generation We visualized the results of
three baseline models (Emu2 [3], SEED-Llama [38], and MiniGPT-5 [4]) across
four interleaved generation tasks: image-to-text sequence generation (cf.
Figure 5), text-to-image sequence generation (cf. Figure 6), interleaved
image-text content continuation (cf. Figure 7), and question-based interleaved
image-text Generation (cf. Figure 8). From the results, we can observe that:
1) For the single textual modality generation, the Emu2 model can more
accurately describe entities (e.g., tripod, straw, and sunglasses in Figure 5)
appearing in images, producing more coherent and concise text descriptions. 2)
For the single visual modality generation, SEED-Llama can generate images that
are highly aligned with texts and and exhibit consistent style, as exemplified
by the uniformity in shape and decoration of the pie depicted in Figure 6. 3)
For interleaved image-text generation, SEED-Llama also exhibits more coherent
and stylistically consistent image-text outputs. Conversely, the Emu2 model
generates images with excessive uniformity, which detracts from their
developmental progression. While MiniGPT displays some developmental aspects,
it suffers from inconsistencies in style.

Figure 5: Visualization of image-to-text sequence generation from Emu2 [3],
SEED-Llama [38], and MiniGPT-5 [4], separately. Figure 6: Visualization of
text-to-image sequence generation from Emu2 [3], SEED-Llama [38], and
MiniGPT-5 [4], separately. Figure 7: Visualization of interleaved image-text
content continuation from Emu2 [3], SEED-Llama [38], and MiniGPT-5 [4],
separately. Figure 8: Visualization of question-based interleaved image-Text
generation from Emu2 [3], SEED-Llama [38], and MiniGPT-5 [4], separately.


# TESTING AND CORRECTING SAMPLE SELECTION IN ACADEMIC ACHIEVEMENT COMPARISONS

Onil Boussim  Penn State

###### Abstract.

Country comparisons using standardized test scores may in some cases be
misleading unless we make sure that the potential sample selection bias
created by drop-outs and non-enrollment patterns does not alter the analysis.
In this paper, I propose an answer to this issue which consists of identifying
the counterfactual distribution of achievement (I mean the distribution of
achievement if there was hypothetically no selection) from the observed
distribution of achievements. International comparison measures like means,
quantiles, and inequality measures have to be computed using that
counterfactual distribution which is statistically closer to the observed one
for a low proportion of out-of-school children. I identify the quantiles of
that latent distribution by readjusting the percentile levels of the observed
quantile function of achievement. Because the data on test scores is by nature
truncated, I have to rely on auxiliary data to borrow identification power. I
finally applied my method to compute selection corrected means using PISA 2018
and PASEC 2019 and I found that ranking/comparisons can change.

Keywords: Pisa, Pasec, Sample selection, International student achievement
tests, Learning poverty.

JEL codes: C34, C83, I20

The first version is TBD. This version is of .

##  1\. Introduction

Standardized tests 111PISA( Programme for International Student Assessment),
PIRLS (Progress in International Reading Literacy Study ), TIMSS(Trends in
International Maths and Science Study), PASEC(Programme for the Analysis of
Education Systems), … data on student achievements have been widely used for
economic analysis and countries comparisons (see Nagy (1996), Martin et al.
(2000), McEwan and Marshall (2004), Cromley (2009), Tienken (2008), McGaw
(2008), and Jakubowski and Pokropek (2015). Also, the rankings derived from
those tests are of great interest to policymakers in the area of education. In
fact, they can sometimes be used as a motivation to adjust programs or a
justification to transfer educational reforms from other countries (see,
Feniger and Lefstein (2014), Taylor and Henry (2007), Suleyman (2020)). Also
even within a given country, most measures of inequality in educational
achievement are based on distributions of standardized test scores. However
many critics suggest that the data considered in the comparisons may suffer
from sample selectivity due to countries' differences in enrollment patterns
and can therefore lead to inaccurate comparisons (see Rotberg (1995), Berliner
(1993), Ferreira and Gignoux (2014)). In fact, standardized test samples may
in various cases not be representative of the underlying population of
children who belong to the age group of comparison if a not negligible
proportion of them are out of school. This cannot be ignored given the fact
that enrollment is correlated with some student characteristics that are also
supportive of higher test scores (family socioeconomic background, area of
living, and some unobserved factors …). For example, Rotberg (1990) found that
student selectivity is positively correlated to higher results on the science
and Maths achievement tests.

On the other hand, Hanushek and Woessmann (2011) found a positive correlation
between the enrollment rate and the mean of test scores by combining PISA
(2000, 2003) and TIMSS (1995, 1999, 2003) data. This finding may seem counter-
intuitive since we just explained that students with potentially low
achievement are more likely to leave school, which would suggest that a higher
out-of-school proportion should result in a higher mean (upward bias).
However, the authors explain this discrepancy with the following arguments.
Firstly, almost all developed countries have a 100% enrollment rate, so we
should only be concerned with selection bias when comparing countries at
different levels of development. Secondly, at the country level, any bias due
to low enrollment rates may be outweighed by the fact that such rates are a
sign of a generally underdeveloped or dysfunctional education system.

Because of these two views, it is not clear whether we should worry about
selection bias when comparing countries or simply ignore it. One may think
that if the distribution of counterfactual achievement is statistically the
same as the distribution of observed achievement, then selection bias is not
relevant and has no impact on any economic analysis or comparisons. On top of
that, using standardized test scores on reading, the World Bank developed the
concept of learning poverty (see Azevedo (2020)). It is a simple index whose
aim is to evaluate the effectiveness of the primary school system of a
country. The idea of the index is to measure the proportion of children unable
to read by age 10 as a proxy of effectiveness. It is computed by firstly
multiplying the share of children in school who haven’t achieved minimum
reading proficiency by the probability of being enrolled in school and
secondly, it is adjusted by adding the proportion of children who are out of
school (here all of them are assumed not able to read proficiently). This
assumption may be strong in a context where some kids have dropped before
reaching the considered grade of analysis but still have a good reading
ability or are just home-schooled. Identifying the distribution of the
counterfactual distribution of achievements if all the students were schooled
may help define another measure of primary school effectiveness in the same
logic as learning poverty by a simple adjustment of that index without the
mentioned implicit assumption. Hence, we would be able to compute the
probability of being below the minimum proficiency level based on that
distribution.

Considering the reasons given above, the identification of the counterfactual
distribution of achievement becomes a relevant question. This question is an
interesting econometric problem since we are confronted with a truncated data
problem. In fact, non-enrolled children are not observed at all in the data on
test scores the researcher has access to. In that sense, it differs from the
context of correcting the distribution of wages on the basis of surveys that
contain information on labor force participants and also non-participants
which is a censored data problem (see Heckman (1974), Arellano and Bonhomme
(2017), Chernozhukov et al. (2023)…). The second challenge is that most
methods rely on the existence of a valid instrument while in our case, there
is no justifiable good instrument (at least to my knowledge). Because of these
two challenges, the identification power is limited, and thus auxiliary data
is essential. The econometrics literature offers a vast array of data
combination techniques (refer Ridder and Moffitt (2007) for an exhaustive
survey). I focus on quantiles and I show that selection-corrected quantiles
(quantiles of the counterfactual distribution) can be obtained by suitably
shifting the percentile levels of the observed conditional (on selection)
quantile function. I obtain non-parametric partial identification of the
selection-corrected quantiles (quantiles of the counterfactual distribution of
achievement) with minimal assumptions and I use a parametric assumption to get
point identification. I finally applied to the PISA 2018 and PASEC 2019 data
and I found that the counterfactual distribution of achievement is in many
cases different from the one observed in the data and ranks can change because
of that.

The rest of the paper is as follows. In Section 2, I present the selection
model. In section 3, I discuss the identification and implementation of the
test. In section 4, I present the application and I conclude in section 5. All
the detailed proofs can be found in the appendix.

##  2\. Selection Model

Let $\displaystyle Y^{\ast}$ be the counterfactual achievement (test score in
our context) if there is no selection in school. The distribution of
$\displaystyle Y^{\ast}$ is fully characterized by its quantile function
$\displaystyle q_{Y^{\ast}}(U)$ where $\displaystyle U\sim U[0,1]$ is the
rank. I consider the following sample selection model:

$\displaystyle\displaystyle Y^{\ast}=q_{Y^{\ast}}(U)$
$\displaystyle\displaystyle Y=Y^{\ast}\textit{ if }S=1$

I will refer to $\displaystyle U$ as the rank variable. $\displaystyle S$ is
the indicator variable that takes the value 1 if the individual has been
enrolled in school in the considered year of analysis. We will make more
assumptions about $\displaystyle S$ later.

######  Assumption 1 (Existence of auxiliary data).

$\displaystyle p\equiv\mathbb{P}(S=1)$ can be identified from auxilliary data.

Assumption 1 states that there exists a dataset that would allow us to
retrieve the enrollment/coverage rate for the population of interest. This can
be any nationally representative survey.

##  3\. Identification and Test

In that section, I explain how to recover $\displaystyle q_{Y^{\ast}}$. My
strategy involves applying a suitable shifting of percentile levels of the
already identified quantile function $\displaystyle q_{Y}$. For $\displaystyle
u\in[0,1]$, define $\displaystyle G(u)\equiv\mathbb{P}(U\leq u|S=1)$ as the
selection corrected rank. The first important result of this paper is the
following proposition which reveals the shifting of percentile levels :

######  Lemma 1.

$\displaystyle\forall u\in[0,1],\textit{ we have :}$ $\displaystyle
q_{Y^{\ast}}(u)=q_{Y}(G(u))$

In the absence of more restrictive assumptions, our quantiles are generally
partially identified.

###  3.1. Partial Identification :

In that section, I derive sharp bounds for our quantile function.

######  Lemma 2.

Under Assumptions 1

$\displaystyle\forall u\in[0,1],$

$\displaystyle\displaystyle q_{Y}\left(\frac{\max\\{u+p-1,0\\}}{p}\right)\leq
q_{Y^{\ast}}(u)\leq q_{Y}\left(\frac{\min\\{u,p\\}}{p}\right)$

This lemma follows from an application of the Fréchet–Hoeffding inequality on
the probability of a joint event. The construction of the bounds only
necessitates the knowledge of the propensity score. By assuming that I can get
the propensity score from auxiliary data, I can compute the bounds. However,
one can make the bounds more informative by adding a reasonable structure to
the unobservables.

######  Assumption 2.

[Stochastic Dominance]

$\displaystyle\forall\textit{ u}\in[0,1],\textit{ }\mathbb{P}(U\leq
u|S=1)\leq\mathbb{P}(U\leq u|S=0)$

or equivalently

$\displaystyle\forall\textit{ u}\in[0,1],\textit{ }q_{Y^{\ast}|S=1}(u)\geq
q_{Y^{\ast}|S=0}(u)$

This assumption shows that conditional on being enrolled in school, the
distribution of potential achievement stochastically dominates the
distribution of potential achievement conditional on non-enrollment. In other
words, schooling has a non-negative effect on the potential achievements of
children. Given Assumption 4, I derive new bounds in the following theorem.

######  Theorem 1 (Partial Identification ).

Under Assumptions 1 and 2, the following bounds are valid and sharp

$\displaystyle\forall u\in[0,1]$, $\displaystyle x\in\mathcal{X}$,

$\displaystyle\displaystyle
q_{Y^{\ast}}(u)\in\left[q_{Y}\left(\frac{\max\\{u+p-1,0\\}}{p}\right)\textbf{
, }q_{Y}(u)\right]$

This theorem states that given the model and the assumptions, our latent
quantile function for a given $\displaystyle u\in[0,1]$ lies in the above
interval whose bounds cannot be improved upon without further assumptions.
This interval is smaller than the one derived in Lemma 1 since the upper bound
upper of this one is smaller. One interpretation we have here is that the
observed quantile function will be exactly equal to the latent quantile
function if the selection $\displaystyle S$ is not relevant. This result
clarifies why the observed quantiles are upward biased.

###  3.2. Point Identification :

In the absence of a good instrument, point identification of our quantile
function is almost impossible if we don't rely on parametric assumptions.
Therefore, in order to make the model more tractable, I make the following
assumptions :

######  Assumption 3.

[Structure on S] .

  1. (1)

$\displaystyle S=\mathbb{1}\\{V\leq U\\}$

  2. (2)

$\displaystyle V\sim P_{\theta_{0}}$ with support contained in
$\displaystyle[0,1]$.

  3. (3)

$\displaystyle\theta_{0}$ is identified by
$\displaystyle\mathbb{P}(S=1)=\mathbb{P}(V\leq U)$.

The variables $\displaystyle U$ and $\displaystyle V$ are unobserved,
$\displaystyle U$ can be seen as a return and $\displaystyle V$ as a cost. so
the first assumption says that children are enrolled in school when their
return to schooling is higher than their cost. In the second one, I choose to
parameterize the distribution of $\displaystyle V$ to reach point
identification. Assumption 3 gives us a moment that should be enough to
identify the parameter in the distribution of $\displaystyle V$.

######  Theorem 2 (Identification of $\displaystyle q_{Y^{\ast}}$).

Under Assumptions 1 and 3, $\displaystyle q_{Y^{\ast}}$ is identified and we
have :

$\displaystyle
q_{Y^{\ast}}(u)=q_{Y}\left(\frac{1}{p}\left(uF_{V,\theta_{0}}(u)-\int_{0}^{u}vdF_{V,\theta_{0}}(v)\right)\right)$

where $\displaystyle\theta_{0}$ is such that :

$\displaystyle E_{\theta_{0}}(V)=1-p$

.

This theorem gives us a practical way to correct the selection. We just need
to make sure the parametric distribution assumed for $\displaystyle V$ allows
us to identify the parameter of interest. If for example, we choose
$\displaystyle V\sim\beta(1,\theta_{0})$, we have :

$\displaystyle\frac{1}{1+\theta_{0}}=1-p$

Therefore it leads to :

$\displaystyle\theta_{0}=\frac{1}{1-p}-1$

From the quantiles, we have :

$\displaystyle\displaystyle F_{Y^{\ast}}(y)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\int_{0}^{1}\mathbb{1}\\{q_{Y^{\ast}}(u)\leq
y\\}du$ $\displaystyle\displaystyle\mathbb{E}(Y^{\ast})$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\int_{0}^{1}q_{Y^{\ast}}(u)du$

Let $\displaystyle\bar{y}$ be the minimum proficiency level. One can define a
new measure of learning poverty that we call adjusted learning poverty :

$\displaystyle ALp=F_{Y^{\ast}}(\bar{y})$

###  3.3. Test of selection

We can test for the relevance of selection for economic analysis or
comparisons. The idea is to use a non-parametric test to test whether the
distribution of $\displaystyle Y^{\ast}$ is statistically different from the
distribution of $\displaystyle Y$. Various tests can be used like the
Kolmogorov-Smirnov test, the Anderson-Darling Test, Cramér-von Mises Test,…
which can be used to test whether two underlying one-dimensional probability
distributions are the same almost surely.

##  4\. Applications

In this section, I compute estimates of the selection-corrected means to make
educational achievement comparisons using PISA 2018 and PASEC 2019 data. I
only consider ranking among the countries chosen for the applicatio and for
both applications, $\displaystyle V\sim\beta(1,\theta_{0})$

###  4.1. PISA 2018

The Programme for International Student Assessment (PISA) is an international
assessment conducted by the Organisation for Economic Co-operation and
Development (OECD) that aims to evaluate and compare the educational outcomes
of 15-year-old students from different countries around the world. PISA is
designed to provide insights into how well students are prepared to meet real-
world challenges and to inform education policies and practices.

Here I have selected 10 countries that participated in PISA 2018 and I
illustrate with them how rankings can change after correcting the selection.

Table 1 summarizes the results. From there, we can see that the corrected
means are all below the observed ones, evidence that the selection bias
creates an upward bias of the mean of the distribution of test scores. Because
of that, ranks can change as we observe in the last column of the table. We
can analyze how rankings are affected after the correction of selection. The
pattern is that lower values of $\displaystyle p$ correspond to more decrease
in the counterfactual mean, which is the key to changing the ranking.

Table 1.  PISA 2018

countries | $\displaystyle p$ | subjects | mean($\displaystyle Y$) | mean($\displaystyle Y^{\ast}$) | rank shift
---|---|---|---|---|---
Albania | 0.757 | Maths | 436.70 | 403.24 | 5-6
|  | Reading | 405.35 | 374.51 | 7-7
Australia | 0.894 | Maths | 492.14 | 472.68 | 2-2
|  | Reading | 502.27 | 479.20 | 1-2
Argentina | 0.806 | Maths | 379.69 | 351.95 | 10-9
|  | Reading | 401.19 | 367.97 | 8-8
Czech Republic | 0.954 | Maths | 498.94 | 489.48 | 1-1
|  | Reading | 490.41 | 480.71 | 2-1
Colombia | 0.619 | Maths | 391.13 | 347.10 | 9- 10
|  | Reading | 412.11 | 365.16 | 6- 9
Kazakhstan | 0.920 | Maths | 423.03 | 408.65 | 6-5
|  | Reading | 386.66 | 376.24 | 10-6
Thailand | 0.724 | Maths | 417.58 | 381.03 | 7-8
|  | Reading | 393.27 | 361.46 | 9-10
Turkey | 0.726 | Maths | 453.62 | 415.28 | 3-4
|  | Reading | 465.23 | 427.44 | 4- 4
Ukraine | 0.867 | Maths | 453.16 | 428.51 | 4-3
|  | Reading | 465.42 | 440.43 | 3- 3
Uruguay | 0.780 | Maths | 416.46 | 382.78 | 8-7
|  | Reading | 426.44 | 390.94 | 5-5

###  4.2. PASEC 2019

I also use a dataset from the Programme for the Analysis of Education Systems
(PASEC) conducted in 2019. It is a large data set on International learning
assessment conducted in 14 countries. But in this application, I focus on 6
countries which are: Benin, Burkina Faso, Cote d’Ivoire, Niger, Senegal, and
Togo. The sampling is carried out in such a way as to ensure representativity
for the population of enrolled children. The children are evaluated in reading
and Maths in a multiple choice questions format in the country’s official
language(s) of instruction, (see Pasec (2019)).

I use the harmonized survey on household living conditions (EHCVM) data to
identify $\displaystyle p=\mathbb{P}(S=1)$. It is a harmonized survey carried
out in the period 2018-2019 in WAEMU countries in order to produce nationally
representative household survey data in those countries. Here is a table
summarizing the results. Since I am interested in comparing grade 6 students
(corresponding to late primary school), the corresponding age group is
$\displaystyle[10,14]$ according to the World Bank (see Azevedo (2020)).

in Table 2, one can see that before the correction, for reading Cote d'Ivoire
had a higher mean (499.37) than Togo (492.10). But after the correction, Togo
has a higher mean (473.5) against (463.0) for Cote d'Ivoire. That can be
explained by the fact Togo (almost 90% of enrollment) is much less affected by
the selection than Cote d'Ivoire (around 76% of enrollment) while they have
observed means which are close in value. Also, there is a shifting of rank in
maths for Cote d'Ivoire and Niger

Table 2. Results PASEC

Country | p | subject | mean ($\displaystyle Y$) | mean ($\displaystyle Y^{\ast}$) | rank shift
---|---|---|---|---|---
Benin | 0.703 | Maths | 527.59 | 491.3 | 3-3
|  | Reading | 576.16 | 533.3 | 1-1
Burkina-Faso | 0.713 | Maths | 551.19 | 506.6 | 2-2
|  | Reading | 556.10 | 509.8 | 3-3
Cote d'Ivoire | 0.763 | Maths | 452.88 | 426.8 | 6-5
|  | Reading | 499.37 | 463.0 | 4-5
Niger | 0.561 | Maths | 464.03 | 411.5 | 5-6
|  | Reading | 476.51 | 415.2 | 6-6
Senegal | 0.695 | Maths | 554.64 | 510.6 | 2-2
|  | Reading | 567.63 | 521.9 | 1-1
Togo | 0.896 | Maths | 491.14 | 473.6 | 4-4
|  | Reading | 491.82 | 473.5 | 5-4

##  5\. Conclusion

In this paper, I have introduced a method to account for selection in country
comparisons using test scores data. The idea consists of estimating quantiles
of the counterfactual distribution of achievements if there was no selection
and using that distribution to make comparisons. Under different assumptions,
I explain how we can partially identify and point identify the quantiles of
that distribution. The results of my application on some countries in PISA
2018 and PASEC 2019 countries suggest that the observed quantiles are upward
biased and comparisons can actually be affected.

##  Appendix A Proofs of the results in the main text

###  A.1. Proof of Lemma 1

$\displaystyle\displaystyle G(u)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\mathbb{P}(U\leq u|S=1)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\mathbb{P}(Y^{\ast}\leq q_{Y^{\ast}}(u)|S=1)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\mathbb{P}(Y^{\ast}\leq q_{Y^{\ast}}(u)|S=1)$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
F_{Y^{\ast}|S=1}(q_{Y^{\ast}}(u))$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle F_{Y}(q_{Y^{\ast}}(u))$

From there, we have that :

$\displaystyle q_{Y^{\ast}}(u)=q_{Y}(G(u))$

###  A.2. Proof of Lemma 2

$\displaystyle\displaystyle G(u)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\mathbb{P}(U\leq u|S=1)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\frac{\mathbb{P}(U\leq u,S=1)}{p}$

Now we apply frechet-bounds to the joint probability, and we obtain :

$\displaystyle\displaystyle\frac{\max\\{u+p-1,0\\}}{p}\leq
G(u)\leq\frac{\min\\{u,p\\}}{p}$

Now, using the monononicity of $\displaystyle q_{Y}$, we finally obtain :

$\displaystyle\displaystyle q_{Y}\left(\frac{\max\\{u+p-1,0\\}}{p}\right)\leq
q_{Y^{\ast}}(u)\leq q_{Y}\left(\frac{\min\\{u,p\\}}{p}\right)$

###  A.3. Proof of Theorem 1

:

STEP 1 : Validity

First, we need to prove the validity of the inequalities. By Lemma 2, we
already know that :

$\displaystyle\displaystyle q_{Y^{\ast}}(u)\geq
q_{Y}\left(\frac{\max\\{u+p-1,0\\}}{p}\right)$

We already know that :

$\displaystyle\displaystyle u=\mathbb{P}\left(U\leq
u|S=1\right)p+\mathbb{P}\left(U\leq u|S=0\right)(1-p)$

Using the stochastic dominance assumption, we get :

$\displaystyle\displaystyle\mathbb{P}\left(U\leq
u|S=1\right)p+\mathbb{P}\left(U\leq u|S=1\right)(1-p)$
$\displaystyle\displaystyle\leq$
$\displaystyle\displaystyle\mathbb{P}\left(U\leq
u|S=1\right)p+\mathbb{P}\left(U\leq u|S=0\right)(1-p)$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle u$

which is simply :

$\displaystyle\mathbb{P}\left(U\leq u|S=1\right)\leq u$

Using these probability bounds, and the monotonocity of $\displaystyle q_{Y}$,
we have that

$\displaystyle\displaystyle q_{Y^{\ast}}(u)$ $\displaystyle\displaystyle\leq$
$\displaystyle\displaystyle q_{Y}(u)$

STEP 2 : Sharpness

-The dependence between $\displaystyle U$ and $\displaystyle S$ can be a counter-monotonic relationship and will still be consistent with the data we observe. In that case, we have :

$\displaystyle G(u)=\frac{\max\\{u+p-1,0\\}}{p}$

-Independence between $\displaystyle U$ and $\displaystyle S$ can also produce the observables. In that case:

$\displaystyle G(u)=u$

###  A.4. Proof of Theorem 2

:

We have that :

$\displaystyle\displaystyle G(u)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\mathbb{P}(U\leq u|S=1)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\frac{1}{p}\left(\mathbb{P}(U\leq u,S=1)\right)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\frac{1}{p}\left(\mathbb{P}(U\leq u,U\geq
V)\right)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\frac{1}{p}\left(\int\mathbb{P}(U\leq u,U\geq
v)dF_{V,\theta_{0}}(v)\right)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\frac{1}{p}\left(\int_{0}^{u}(u-v)dF_{V,\theta_{0}}(v)\right)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\frac{1}{p}\left(uF_{V,\theta_{0}}(u)-\int_{0}^{u}vdF_{V,\theta_{0}}(v)\right)$

Now we also have that :

$\displaystyle\displaystyle p$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\mathbb{P}(U\geq V)$ $\displaystyle\displaystyle=$
$\displaystyle\displaystyle\int\mathbb{P}(U\geq v)dF_{V,\theta_{0}}(v)$
$\displaystyle\displaystyle=$
$\displaystyle\displaystyle\int(1-v)dF_{V,\theta_{0}}(v)$
$\displaystyle\displaystyle=$ $\displaystyle\displaystyle
1-\mathbb{E}(V,\theta_{0})$

From there, we have : $\displaystyle E(V,\theta_{0})=1-p$

## References

  * Arellano and Bonhomme (2017) M. Arellano and S. Bonhomme.  Quantile selection models with an application to understanding changes in wage inequality.  _Econometrica_ , 85(1):1–28, 2017.
  * Azevedo (2020) J. P. Azevedo.  Learning poverty.  2020\.
  * Berliner (1993) D. C. Berliner.  International comparisons of student achievement.  In _National Forum_ , volume 73, page 25, 1993.
  * Chernozhukov et al. (2023) V. Chernozhukov, I. Fernández-Val, and S. Luo.  Distribution regression with sample selection and uk wage decomposition.  _arXiv preprint arXiv:1811.11603_ , 2023.
  * Cromley (2009) J. G. Cromley.  Reading achievement and science proficiency: International comparisons from the programme on international student assessment.  _Reading Psychology_ , 30(2):89–118, 2009.
  * Feniger and Lefstein (2014) Y. Feniger and A. Lefstein.  How not to reason with pisa data: An ironic investigation.  _Journal of Education Policy_ , 29(6):845–855, 2014.
  * Ferreira and Gignoux (2014) F. H. Ferreira and J. Gignoux.  The measurement of educational inequality: Achievement and opportunity.  _The World Bank Economic Review_ , 28(2):210–246, 2014.
  * Hanushek and Woessmann (2011) E. A. Hanushek and L. Woessmann.  Sample selectivity and the validity of international student achievement tests in economic research.  _Economics Letters_ , 110(2):79–82, 2011.
  * Heckman (1974) J. Heckman.  Shadow prices, market wages, and labor supply.  _Econometrica: journal of the econometric society_ , pages 679–694, 1974.
  * Jakubowski and Pokropek (2015) M. Jakubowski and A. Pokropek.  Reading achievement progress across countries.  _International Journal of Educational Development_ , 45:77–88, 2015.
  * Martin et al. (2000) M. O. Martin et al.  International comparisons of student achievement.  In _Learning from others_ , pages 29–47. Springer, 2000.
  * McEwan and Marshall (2004) P. J. McEwan and J. H. Marshall.  Why does academic achievement vary across countries? evidence from cuba and mexico.  _Education Economics_ , 12(3):205–217, 2004.
  * McGaw (2008) B. McGaw.  The role of the oecd in international comparative studies of achievement.  _Assessment in Education: Principles, Policy & Practice_, 15(3):223–243, 2008.
  * Nagy (1996) P. Nagy.  International comparisons of student achievement in mathematics and science: A canadian perspective.  _Canadian Journal of Education/Revue canadienne de l'éducation_ , pages 396–413, 1996.
  * Pasec (2019) Pasec.  Education system performance in francophone sub-saharan africa.  _Pasec, Confemen, Dakar._ , 2019.
  * Ridder and Moffitt (2007) G. Ridder and R. Moffitt.  Chapter 75 the econometrics of data combination.  volume 6 of _Handbook of Econometrics_ , pages 5469–5547. Elsevier, 2007.
  * Rotberg (1990) I. C. Rotberg.  I never promised you first place.  _The Phi Delta Kappan_ , 72(4):296–303, 1990\.
  * Rotberg (1995) I. C. Rotberg.  Myths about test score comparisons.  _Science_ , 270(5241):1446–1448, 1995.
  * Suleyman (2020) Y. Suleyman.  International comparison tests and transformation of national education policies: A critical perspective.  _Political Economy and Management of Education_ , 1(1):56–70, 2020.
  * Taylor and Henry (2007) S. Taylor and M. Henry.  Globalization and educational policymaking: a case study.  In _The RoutledgeFalmer reader in education policy and politics_ , pages 111–126. Routledge, 2007.
  * Tienken (2008) C. H. Tienken.  Rankings of international achievement test performance and economic strength: Correlation or conjecture?  _International Journal of Education Policy and Leadership_ , 3(4):1–15, 2008.