# PQPP: A Joint Benchmark for Text-to-Image Prompt and Query Performance
Prediction

Eduard Poesina⋄
University of Bucharest
Adriana Valentina Costache⋄
University of Bucharest
Adrian-Gabriel Chifu
Aix-Marseille Université
Josiane Mothe
Université Toulouse Jean-Jaurès
Radu Tudor Ionescu
University of Bucharest
Corresponding author<EMAIL_ADDRESS>⋄Equal contribution.

###### Abstract

Text-to-image generation has recently emerged as a viable alternative to text-
to-image retrieval, due to the visually impressive results of generative
diffusion models. Although query performance prediction is an active research
topic in information retrieval, to the best of our knowledge, there is no
prior study that analyzes the difficulty of queries (prompts) in text-to-image
generation, based on human judgments. To this end, we introduce the first
dataset of prompts which are manually annotated in terms of image generation
performance. In order to determine the difficulty of the same prompts in image
retrieval, we also collect manual annotations that represent retrieval
performance. We thus propose the first benchmark for joint text-to-image
prompt and query performance prediction, comprising 10K queries. Our benchmark
enables: (i) the comparative assessment of the difficulty of prompts/queries
in image generation and image retrieval, and (ii) the evaluation of
prompt/query performance predictors addressing both generation and retrieval.
We present results with several pre-generation/retrieval and post-
generation/retrieval performance predictors, thus providing competitive
baselines for future research. Our benchmark and code is publicly available
under the CC BY 4.0 license at https://github.com/Eduard6421/PQPP.

##  1 Introduction

Figure 1: We select a set of 10K captions from MS COCO [26] using k-means
clustering. Next, we collect human relevance judgments in two scenarios: image
generation and image retrieval. For each prompt/query, we generate images with
two diffusion models (Stable Diffusion [36] and GLIDE [30]) and retrieve
images from MS COCO with two vision-language models (CLIP [33] and BLIP-2
[24]). Based on the collected relevance judgments, we score each prompt/query
in terms of generation and retrieval performance, respectively. Finally, we
train and evaluate multiple prompt/query performance predictors on the
proposed benchmark. Best viewed in color.

In recent years, more and more people embraced the use of large language
models (LLMs) instead of traditional search engines [1, 5, 53]. The advent of
generative diffusion models [7, 15, 30, 36] capable of generating high-quality
and realistic images triggered a similar trend in text-to-image retrieval.
This paradigm shift in information retrieval (IR) calls for an extensive
exploration of research topics related to generative models. For instance,
query performance prediction (QPP) is an active research topic in IR [10, 13,
18, 27, 32]. QPP refers to the process of estimating the performance level of
an IR system for a given query [6]. Assessing the difficulty of a query before
retrieving the results can help an IR system take informed decisions about
query optimization strategies, indexing, query-specific tuning, and so on. To
the best of our knowledge, there is no prior study that analyzes the
difficulty of queries (prompts) in text-to-image generation and retrieval
using human relevance judgments.

In this study, we introduce the first dataset of prompts which are manually
annotated in terms of image generation performance. We collected over 240K
relevance judgments from 147 human annotators for 40K images generated by
diffusion models in response to 10K prompts. We select these prompts from the
Microsoft Common Objects in Context (MS COCO) dataset [26], which comprises
images with captions. We chose MS COCO as a well-established data collection
that allows cross-task studies and is cost-effective. It provides high-quality
image annotations, as well as a wide range of everyday scenes and objects. To
select a representative and diverse set of prompts from over 590K captions in
MS COCO, we employ a k-means clustering algorithm on the embedding space of a
sentence BERT model [42], where $k=10,000$. We extract the closest caption to
each cluster center to build our set of prompts. For each prompt, we generate
two images with Stable Diffusion [36] and two with GLIDE [30]. Further, we aim
to study the same set of 10K prompts from two perspectives: generation
performance and retrieval performance. To this end, we collect another set of
1.25M annotations from 91 human evaluators to obtain ground-truth relevance
judgments for the retrieval task. We start from preliminary relevance
judgments determined via the pre-trained sentence BERT [42] applied on
captions. If a caption from MS COCO is similar to a query in the BERT
embedding space, the image corresponding to the respective caption is added to
the set of images that are potentially relevant to the respective query. Next,
we collect human relevance judgments for all 10K queries. The retrieval
difficulty score of each query is estimated with respect to two state-of-the-
art vision-language models, CLIP [33] and BLIP-2 [24], which are tasked with
retrieving images for all 10K queries. The entire process used to develop the
Prompt and Query Performance Prediction (PQPP) benchmark is illustrated in
Figure 1.

We conduct preliminary experiments to compare prompt/query difficulty in image
generation versus image retrieval. Our findings show that there is a very low
correlation between the two tasks, which justifies the need to study the novel
task of prompt performance prediction in image generation. We also carry out
experiments with multiple pre- and post-generation/retrieval performance
predictors, providing a set of competitive baselines for future research. We
find that a strong supervised pre-generation/retrieval predictor is a worthy
competitor for post-generation/retrieval predictors in both text-to-image
generation and retrieval.

In summary, our contribution is threefold:

  * •

We propose the first joint benchmark for prompt and query performance
prediction.

  * •

We collect nearly 1.5M relevance judgments from human annotators to score a
total of 10K queries in terms of generation and retrieval performance.

  * •

We experiment with multiple pre-generation/retrieval and post-
generation/retrieval performance predictors to obtain competitive results,
which serve as baselines for future work.

##  2 Related Work

Studies on QPP initially focused on textual ad hoc retrieval, where both pre-
and post-retrieval predictors were considered for sparse retrieval models [8,
22, 29, 48]. Some recent studies investigated dense (neural network) retrieval
models [2, 10, 11, 18, 49], as well as diverse tasks, such as conversational
search [19, 27, 37]. We discuss QPP in textual ad hoc retrieval in Appendix
A.4.

QPP in text-to-image retrieval/generation. QPP in a multimodal (e.g. text-to-
image) context is a relatively new research area [4, 23, 31, 45, 46]. The
exploration of QPP in the context of text-to-image retrieval and generation is
gaining significant interest, particularly with the rapid advancements in
generative methods [7]. This research domain, distinct for its multimodal
nature, aims to enhance the prediction of text query effectiveness for
retrieving relevant images. Initial studies, such as those of Xing et al. [47]
and Tian et al. [43, 44], have laid the groundwork by exploring query
difficulty prediction in image retrieval, utilizing machine learning
algorithms and assessing the utility of various features and information
sources.

Further contributions by Li et al. [25] delve into the challenges of
estimating query difficulty with unigram language models and visual word
verification, highlighting the complexities of aligning text-based queries
with visual data. Meanwhile, the development of a self-supervised framework
for Content-Based Image Retrieval (CBIR) systems marks a significant
advancement [45], addressing the scarcity of labeled data through synthetic
data and rank-based feature training. Recent efforts, such as those of Wu et
al. [46] and Pavlichenko and Ustalov [31], focus on integrating human feedback
into text-to-image models to refine prediction accuracy and enhance the visual
appeal of generated images. Kumari et al. [23] introduced an approach to
concept ablation in text-to-image synthesis, aiming to selectively prevent the
generation of specific concepts.

The closest work to our research is that of Bizzozzero et al. [4], which
introduced the concept of _prompt performance prediction_. The authors
assessed the effectiveness of prompts in generating images. However, in their
study, the ground-truth is automatically generated, which may introduce a
significant bias in the evaluation. In contrast, we are the first to explore
the prompt performance prediction task with respect to human relevance
judgments. In addition, we introduce a benchmark that provides performance
measurements for the same set of prompts/queries in both generation and
retrieval, enabling the comparative study of QPP across both tasks. Another
novel contribution of our work is the study of post-generation prompt
performance predictors.

QPP evaluation. To evaluate QPP, one should determine the degree of dependence
between the predicted performance and the actual performance obtained by an IR
system. A correlation test serves this purpose and, for this reason, it is
widely used. Some studies have highlighted the limitation of correlation
coefficients for evaluating QPP in certain scenarios [17, 21, 28], and
suggested alternative measures, such as the scaled Absolute Ranking Error
(sARE) and its mean (sMARE) to model performance using distributions [17].
Although we agree that correlation may not be the only way to evaluate QPP, we
choose to use it in the evaluation part of our study due to its wide adoption
in QPP literature.

##  3 Proposed Benchmark

Figure 2: An example showing the annotation interface for a random prompt and
the images associated with the respective prompt. For each image, the
annotator can select one of the following options: high relevance, low
relevance, no relevance and unrealistic. The relevance judgments of an
annotator are shown for illustrative purposes. Best viewed in color.

To develop our novel Prompt and Query Performance Prediction (PQPP) benchmark,
we harness the images and captions from the MS COCO [26] training set, which
comprises approximately 118K images. Each of these images is accompanied by
$5$ to $7$ descriptive captions, providing a rich context for our study.

To establish a unified foundation for text-to-image generation and retrieval,
we select a subset of $10,000$ captions to be used across tasks either as
_queries_ for text-to-image retrieval, or as _prompts_ for text-to-image
generation. We aim for a wide variety of captions that can be clearly
associated with images. The selection process entails identifying the most
relevant caption for each image (from the set of captions available in MS COCO
for the respective image), where the (caption, image) similarity is measured
via the cosine similarity in the CLIP embedding space [33]. This reduces the
number of captions to about 118K. The resulting captions are further processed
by a sentence transformer based on BERT [42] to extract sentence embeddings.
Aiming to enhance the distinctiveness of our final set of captions, we next
apply the k-means clustering algorithm based on k-means++ initialization on
the extracted embeddings. For each cluster, we select the nearest caption to
the center of each cluster and include it in our final set of captions. We set
$k=10,000$, thus obtaining a set of 10K prompts/queries. We next describe the
procedures for collecting human relevance judgments for text-to-image
generation and text-to-image retrieval in separate sections.

###  3.1 Prompt Performance Assessment

Generative models. To generate images for the established set of prompts, we
employ the well-established Stable Diffusion [36] and GLIDE [30] generative
models. For the first model, we choose the Stable Diffusion XL (base-1.0)
variant, which produces images with a resolution of $1024\times 1024$ pixels.
In contrast, GLIDE initially produces images at a resolution of $64\times 64$
pixels. The images are subsequently upsampled to a resolution of $256\times
256$ pixels using the upsampling model integrated in GLIDE. For each prompt,
we generate two images with Stable Diffusion and two images with GLIDE.

Control and calibration. Along with the four generated images, we include the
image from MS COCO associated with each prompt into the annotation process.
The ground-truth image is included to calibrate user annotations (assuming
that the ground-truth image should be labeled as highly relevant) or to
exclude annotators that are not seriously engaged in the annotation process.
To further ensure the quality of human relevance judgments, we implement a
verification process based on a control set of 100 prompts. The control
prompts are independently annotated by three authors of this study, following
the same protocol as the other annotators. The Fleiss’ $\kappa$ coefficient
among the control annotators is $0.55$. While running the full-scale
annotation process, a control prompt is randomly inserted among every five
prompts. This allows us to set a minimum threshold for the Cohen’s $\kappa$
coefficient to accept the relevance judgments of an annotator. The relevance
judgments of an annotator are included into the study if the respective
annotator (i) has at least a moderate
agreement111https://en.wikipedia.org/wiki/Cohen%27s_kappa#Interpreting_magnitude
($\kappa>0.4$) with each annotator in the control group, and (ii) has
annotated more than $95\%$ of the ground-truth images as highly relevant.

Annotation interface. We use a custom web interface to collect relevance
judgments. The prompts are presented in a random order to each annotator. Each
prompt is accompanied by four generated images and one ground-truth image,
which are mixed and displayed in a random order, as shown in Figure 2. For
each image, annotators are given a choice among four evaluative categories:
high relevance, low relevance, no relevance, and unrealistic. Participants
receive thorough instructions (summarized below) before starting the
annotation process. The ratings are to be assigned as follows: _high
relevance_ (score $2$) – the image depicts over half of the concepts mentioned
in the prompt; _low relevance_ (score $1$) – the image captures at least one
concept, but fewer than half; _no relevance_ (score $0$) – the image is
unrelated, yet realistic; _unrealistic_ (score $-1$) – the image exhibits
notable artifacts.

Table 1: Statistics about the annotators enrolled in the annotation process for generated images. Statistic | Min | Mean | Max
---|---|---|---
#annotations per person | 30 | 1,664 | 15,045
Fleiss’ $\kappa$ | 0.41 | 0.54 | 1.00

Annotation process. Our aim is to collect at least three annotations per
image. In order to collect the required number of annotations (10,000 prompts
$\times$ 5 images $\times$ 3 annotations = 150,000 annotations in total), we
recruit $173$ annotators. The annotators providing the relevance judgments are
adults having at least a bachelor or college degree. The recruited annotators
willingly agreed to engage in the annotation process, after reading our terms
and conditions. Annotators are allowed to opt out at any time during the
annotation process. To reduce bias or uncertainty, annotators are permitted to
update previously made annotations or skip specific prompts altogether.
Annotators are informed about the inclusion of control prompts within their
tasks, but are not given specifics on the frequency of such prompts. A fair
compensation (proportional to the number of annotated prompts) is given to
each annotator with a Cohen’s $\kappa$ coefficient higher than $0.4$ on the
control prompts. Based on our selection criteria, $26$ annotators are excluded
from the process. This leaves us with $147$ valid annotators. Given the
asynchronous nature of the annotation process, several prompts ended up having
more than three annotations, leading to a total of 244,650 annotations. When a
prompt has more than three relevance judgments, we keep the annotations
provided by the top three annotators with the highest Cohen’s $\kappa$
coefficients (with respect to the control prompts). A brief set of statistics
about the enrolled annotators is presented in Table 1. Notably, the Fleiss’
$\kappa$ coefficient computed across all annotators is consistent with that of
the control annotators. Upon excluding the annotations corresponding to the
ground-truth images, we find that most images are voted as highly relevant
(see Figure 4 in Appendix A.5), confirming that Stable Diffusion and GLIDE
generally produce relevant results.

Measuring prompt performance. To derive the final prompt performance in image
generation, we first map the relevance categories to numerical values, as
follows: _high relevance_ is mapped to 2, _low relevance_ to 1, _no relevance_
to 0, and _unrealistic_ to -1.

To exclude outlier annotations, we group the four relevance categories into
two high-level categories. The first category, combining the _high relevance_
and _low relevance_ annotations, represents images that are at least somewhat
relevant to the prompt. The second category, combining the _unrealistic_ and
_no relevance_ labels, represents images that are not acceptable for the given
prompt, either because they are irrelevant or unrealistic. We hereby
acknowledge that the distinction between _high relevance_ and _low relevance_
is more difficult to determine, involving a fine assessment of how many of the
prompt elements are depicted in the image. This requires evaluators to
consider not just the presence of these elements, but also their significance
and portrayal within the image, making the distinction between high and low
relevance inherently more subjective and challenging. In contrast, the
distinction between the high-level categories (_relevant_ and _irrelevant_)
can be easily assessed.

We employ a majority voting mechanism on the high-level categories to decide
if a generated image is either _relevant_ or _irrelevant_. Since there are
only two high level categories and three annotations per image, there is no
need to break ties (a majority always exists). The majority voting is
performed to rule out outlier annotations. The final relevance of an image is
given by averaging the scores (between $-1$ and $2$) associated with the votes
forming the majority. The performance of a prompt is given by the average
relevance score computed across the four generated images. We further refer to
the resulting score as _human-based prompt performance_ (HBPP).

###  3.2 Query Performance Assessment

Retrieval models. For text-to-image retrieval, we employ two distinct vision-
language models: CLIP [33] and BLIP-2 [24]. For CLIP, we select the base
architecture with patches of $32\times 32$ pixels. For BLIP-2, we choose the
ViT-Large [16] backbone. Both models are equipped with the necessary
components to perform text-to-image retrieval. These models are pre-trained on
natural images, which makes them suitable for image retrieval on MS COCO.

Annotation process. For the retrieval setting, we devise a semi-automatic
labeling process to generate reference (ground-truth) relevance judgments for
the 10,000 image captions chosen as queries. Limited by the annotation budget,
we employ an automatic process to restrict the number of retrieved images to
2,000 per query. We exploit the structure of the MS COCO dataset based on
(image, caption) pairs to generate preliminary relevance judgments using
sentence BERT [42]. More specifically, we compute the cosine similarity in the
embedding space of sentence BERT between each query from the established
subset and each caption in MS COCO. Based on a preliminary exploratory data
analysis, we set the cosine similarity threshold to $0.7$ to determine a
comprehensive set of potentially relevant results. The image corresponding to
each caption that has the cosine similarity with a query higher than $0.7$ is
added to the preliminary set of relevant results for the respective query. For
some short and generic text queries, the preliminary set may contain thousands
of potentially relevant images. For such queries, we refine the results ranked
below 1,000 using a bag-of-words representation. More specifically, a low-rank
image is kept only if the bag-of-words representation of the query is included
in the bag-of-words representation of the aggregated captions of the
respective image. All queries are limited to 2,000 images in the preliminary
set. The preliminary steps described above generate a total of 1,245,186
images, which are further subject to rigorous manual review. There are 100
annotators involved in the manual annotation of the potentially relevant
images. The annotators are asked to label each image as relevant or irrelevant
to the corresponding query. Each image is annotated by two annotators. An
image is kept in the relevant set if it is voted as relevant by one annotator.
Two of the evaluators, who are also the main authors of this paper, annotated
a set of 4,000 queries. These 4,000 queries are used as control queries for
the other enrolled annotators. The remaining queries were randomly divided
into batches of 50 queries. In each batch, there are 5 control queries, which
are used to exclude annotators that provide poor relevance judgments. There
are 98 human evaluators who annotated between 1 and 4 batches. We employed the
$F_{1}$ measure on relevant images to estimate the quality of the relevance
judgments, and set a threshold of $0.4$ to accept annotations. There are 7
annotators who were excluded from the annotation process based on the
considered threshold. For the remaining annotators, we obtain a mean $F_{1}$
score of $0.727$. The minimum $F_{1}$ score is $0.447$, which is significantly
higher than the $F_{1}$ score of $0.150$ of the random chance baseline. The
manual annotation process reduced the total number of relevant images to
431,141. In other words, the annotators removed about a third of the
originally retrieved images.

Measuring query performance. To determine the performance level of a query, we
employ two alternative measures of retrieval effectiveness, namely the
precision for the top $n$ retrieved results (P@$n$) and the reciprocal rank
(RR). The precision@$n$ is the ratio between the number of true positive
images and $n$. Since P@$10$ is often used in text QPP [48], we adopt the same
measure and set $n=10$ for our benchmark. The reciprocal rank of a query is
given by the ratio between 1 and the rank of the first relevant result. For
example, if the first relevant result is at rank 5, then $RR=1/5=0.2$. We
estimate the P@$10$ and RR measures for both CLIP and BLIP-2. Then, the
resulting values are averaged across models, thus obtaining an average P@$10$
and an average RR for each query. However, in Appendix A.7, we also report the
results per reference system.

###  3.3 Evaluation Protocol

We divide the annotated prompts/queries into 6,000 for training, 2,000 for
validation, and 2,000 for testing. To evaluate performance predictors, we
measure the Pearson and Kendall $\tau$ correlation coefficients between the
predicted and the ground-truth performance levels of all test queries,
following conventional evaluation procedures in text [48, 51] and image [32]
QPP. Furthermore, we test the significance of the results with respect to the
random chance baseline using Student’s t-testing [35].

##  4 Predictors

###  4.1 Pre-generation/retrieval Predictors

Basic text predictors. Building on linguistic features to predict query
difficulty [29], we extract a comprehensive suite of pre-retrieval linguistic
indicators: the diversity of concepts (total WordNet synsets per
prompt/query), lexical density (number of words per prompt/query), syntactic
complexity (average word length), and the frequency of specific grammatical
structures (proper nouns, acronyms, numerals, conjunctions and prepositions).
We test a wide variety of basic predictors, but we only report the results of
the top three predictors, namely the number of synsets, the number of words,
and the frequency of conjunctions.

Fine-tuned BERT. We explore the potential of fine-tuning the BERT model [14]
as a pre-retrieval performance predictor. We select the _base_ architecture
based on cased inputs, since the queries contain named entities. The fine-
tuning process involves attaching a custom regression head to the pre-trained
BERT backbone, which learns to predict the performance of prompts/queries in
image generation and retrieval, respectively. The regression head consists of
a dropout layer and two fully connected layers. The dropout rate is set to
$0.3$ to prevent overfitting. The first dense layer is based on ReLU
activation functions, and it takes the [CLS] token returned by BERT and
transforms it into a $512$-dimensional hidden representation. The second layer
contains a single neuron (activated by sigmoid) that predicts prompt/query
performance. Before training, the ground-truth performance values are
normalized to $[0,1]$. We employ grid search on the validation set to
establish the optimal hyperparameter configuration. More specifically, we
consider learning rates between $10^{-3}$ and $10^{-6}$, and weight decays in
the set $\\{0,0.1,0.01\\}$. All versions are trained for 15 epochs with early
stopping, on mini-batches of $256$ samples. We employ AdamW and optimize the
mean squared error (MSE) loss. The fine-tuning is independently carried out
for the two tasks, generation and retrieval.

###  4.2 Post-generation/retrieval Predictors

Fine-tuned CLIP. Our first post-generation/retrieval predictor is based on
fine-tuning a CLIP model on (query, image) pairs. We attach a regression head
for the image generation scenario, and a binary classification head for the
image retrieval scenario. The utilization of CLIP-based embeddings is aimed at
leveraging the model’s capacity to jointly represent text and image modalities
in a single latent space. The fine-tuned CLIP learns to predict the relevance
judgment of each generated or retrieved image for a given prompt/query. The
performance of the prompt/query is then predicted by aggregating the predicted
relevance judgments. For the generative task, the model uses all four images
generated by Stable Diffusion and GLIDE. For the retrieval task, we limit the
training data to the first $25$ images returned by each retrieval model.
Although $10$ images would be enough for the P@$10$ metric, estimating the RR
measure can require more images. A statistical analysis of the training
queries indicates that more than 95% of the queries have the first relevant
image at a rank higher than $25$, which motivates our choice for limiting the
training data to $25$ images per model. The regression/classification head is
composed of a two-layer neural network of 512 and 256 neurons, respectively.
Both layers are based on ReLU activations. A dropout layer with a drop rate of
$0.5$ is added after each dense layer. For the generative task, another layer
comprising a single neuron is added to predict prompt performance. The
objective of the model is to minimize the MSE loss. For the retrieval task,
the last neuron has to determine if an input (query, image) pair is relevant
or irrelevant. This is a binary classification task, so the model is trained
via binary cross-entropy. We perform a grid search to find the best
hyperparameters, considering learning rates between $10^{-3}$ and $10^{-6}$,
and weight decays in the set $\\{0,0.1,0.01\\}$. We employ the AdamW optimizer
for $25$ epochs with early stopping, using a batch size of $256$.

Table 2: Results of the prompt/query performance predictors for the generative and retrieval settings on the test set. On the generative task, we report the correlation of the predicted value with the average HBPP performance of Stable Diffusion and GLIDE. On the retrieval task, the correlation is computed for the average P@${10}$ and the average RR, where the averages are computed over CLIP and BLIP-2. For each task, the highest correlation is highlighted in bold. According to a Student’s t-test, the results marked with $\dagger$ and $\ddagger$ are significantly better than the random chance baseline at p-values ${0.01}$ and ${0.001}$, respectively. Detailed results for each model are in Appendix A.7. Predictor |  | Generative Task | Retrieval Task
---|---|---|---
Type | Predictor Name | HBPP | P@10 | RR
|  Pearson |  Kendall |  Pearson |  Kendall |  Pearson |  Kendall
Pre- | #synsets | $-0.152^{\ddagger}$ | $-0.093^{\ddagger}$ | $-0.141^{\ddagger}$ | $-0.097^{\ddagger}$ | $-0.071^{\dagger}$ | $-0.037^{\dagger}$
#words | $-0.136^{\ddagger}$ | $-0.113^{\ddagger}$ | $-0.199^{\ddagger}$ | $-0.163^{\ddagger}$ | $-0.069^{\dagger}$ | $-0.052^{\dagger}$
#conjunctions / #words | $0.058^{\dagger}$ | $0.045^{\dagger}$ | $-0.117^{\ddagger}$ | $-0.097^{\ddagger}$ | $-0.041$ | $-0.040$
Fine-tuned BERT | $0.568^{\ddagger}$ | $0.416^{\ddagger}$ | $\textbf{0.507}^{\ddagger}$ | $0.318^{\ddagger}$ | ${0.262}^{\ddagger}$ | ${0.182}^{\ddagger}$
Post- | Fine-tuned CLIP | $\textbf{0.600}^{\ddagger}$ | $\textbf{0.448}^{\ddagger}$ | $0.490^{\ddagger}$ | $\textbf{0.330}^{\ddagger}$ | ${0.197}^{\ddagger}$ | ${0.129}^{\ddagger}$
Correlation CNN | $0.471^{\ddagger}$ | $0.347^{\ddagger}$ | $0.327^{\ddagger}$ | $0.196^{\ddagger}$ | $\textbf{0.301}^{\ddagger}$ | $\textbf{0.212}^{\ddagger}$

Correlation-based CNN. Following the work of Sun et al. [41], we employ a
convolutional neural network (CNN) trained on the set of generated or
retrieved images, respectively. The CNN model takes a correlation matrix
between all image pairs as input. For the generative task, the size of the
input correlation matrix is $4\times 4$. For the retrieval task, we apply the
same limit to the number of retrieved images per query as for the fine-tuned
CLIP predictor. Hence, the size of the correlation matrix for one retrieval
model is $25\times 25$. We concatenate the correlation matrices for CLIP and
BLIP-2 models in the channel dimension, which results in a tensor of $25\times
25\times 2$ components, that is given as input to the CNN. The correlation of
an (image, image) pair is given by the cosine similarity between the (pre-
trained) CLIP embeddings of the respective images. The CNN architecture is
composed of four convolutional-pooling blocks, followed by two linear layers.
This is a custom architecture that comprises $3\times 3$ convolutional filters
applied at a stride of $1$, using a padding of $1$. The number of filters in
each of the four convolutional layers is 64, 128, 256 and 512, respectively. A
max-pooling is applied after each convolutional layer. The pooling operation
uses $2\times 2$ filters applied at a stride of $2$. The first fully connected
layer comprises 1024 units. Each hidden neuron is followed by a ReLU
activation. The final layer comprises a single neuron that is trained in a
regression setting via the MSE loss. The hyperparameter tuning is identical to
the one employed for the fine-tuned CLIP model. The correlation-based CNN is
trained for $25$ epochs using AdamW with early stopping, on mini-batches of
$256$ samples.

##  5 Experiments and Results

Figure 3: t-SNE visualization of the test prompts embedded in the latent space
of the fine-tuned BERT on the generation task. The ground-truth HBPP
performance is encoded via a color map from green to red. The visualization
confirms that the fine-tuned BERT predictor learns a meaningful representation
of the prompts. Best viewed in color.

In Table 2, we report the results of the established predictors for both
generative and retrieval tasks, where the reference or ground truth is the
average of the two reference systems. We provide the separate results for each
reference system in Appendix A.7. For the generative task, the highest
correlation between the predicted and the actual performance corresponds to
the fine-tuned CLIP, with a Pearson correlation coefficient of $0.6$. This
predictor shows a lower correlation with P@$10$ in the retrieval task, where
the Pearson score is $0.49$. In general, predictors exhibit higher
correlations with HBPP than with P@$10$. The best predictor among the pre-
generation/retrieval ones is the fine-tuned BERT, for both tasks and all
evaluation measures. Its correlation measures clearly indicate a positive
correlation, e.g. the Pearson correlation with P@$10$ on the retrieval task is
as high as $0.507$. The two post-generation/retrieval predictors exhibit good
performance levels. The fine-tuned CLIP provides better results for the HBPP
and P@$10$ measures, while the correlation CNN is more suitable for the RR
measure. Overall, an interesting finding of our study is that the pre-
retrieval fine-tuned BERT is a worthy competitor for the better post-retrieval
predictor (fine-tuned CLIP). In Table 2, we only report the predictors that
obtain a correlation higher than $0.1$, in at least one evaluation scenario.
The results of the full set of predictors are presented in Appendix A.7. The
very limited effectiveness of the excluded pre-generation/retrieval predictors
can be attributed to their behavior of capturing only a limited portion of the
relevant information (see also Appendix A.6).

In Figure 3, we present a t-SNE visualization of the test prompts embedded in
the latent space of the BERT predictor fine-tuned on the generation task. We
observe that the learned embedding correlates well with the ground-truth HBPP
values, explaining the high accuracy of the fine-tuned BERT predictors on the
generation task. An analogous visualization for the retrieval task is shown in
Figure 6 from Appendix A.8.

Additional results. In Appendix A.6, we discuss empirical evidence regarding
the overlap between generation and retrieval performance. In Appendix A.7, we
present the complete set of quantitative results. In Appendix A.8, we analyze
the generation/retrieval results from a qualitative perspective.

##  6 Conclusion and Future Work

Benchmark collections lay the grounds for the development of effective
approaches to solve new challenges. In this paper, we have developed the first
manually-labeled benchmark for prompt performance prediction in the context of
prompt-to-image generation. Our benchmark is also applicable for query-to-
image retrieval, enabling the direct comparison of the performance prediction
task in generation vs. retrieval scenarios. Our dataset and code are made
publicly available to the research community.

One direction for future work is to develop a model that combines different
pre-retrieval predictors. Combining predictors in a supervised manner has
shown its effectiveness in text [13] and image [32] QPP. Another important
direction is to organize a shared task associated with the proposed PQPP
benchmark, so that the IR community can further explore the novel task of
prompt performance prediction in image generation.

## References

  * Ai et al. [2023] Qingyao Ai, Ting Bai, Zhao Cao, Yi Chang, Jiawei Chen, Zhumin Chen, Zhiyong Cheng, Shoubin Dong, Zhicheng Dou, Fuli Feng, Shen Gao, Jiafeng Guo, Xiangnan He, Yanyan Lan, Chenliang Li, Yiqun Liu, Ziyu Lyu, Weizhi Ma, Jun Ma, Zhaochun Ren, Pengjie Ren, Zhiqiang Wang, Mingwen Wang, Ji-Rong Wen, Le Wu, Xin Xin, Jun Xu, Dawei Yin, Peng Zhang, Fan Zhang, Weinan Zhang, Min Zhang, and Xiaofei Zhu.  Information Retrieval meets Large Language Models: A strategic report from Chinese IR community.  _AI Open_ , 4:80–90, 2023.
  * Arabzadeh et al. [2020] Negar Arabzadeh, Fattane Zarrinkalam, Jelena Jovanovic, and Ebrahim Bagheri.  Neural embedding-based metrics for pre-retrieval query performance prediction.  In _Proceedings of ECIR_ , pages 78–85. Springer, 2020.
  * Arabzadeh et al. [2021] Negar Arabzadeh, Maryam Khodabakhsh, and Ebrahim Bagheri.  Bert-qpp: contextualized pre-trained transformers for query performance prediction.  In _Proceedings of CIKM_ , pages 2857–2861, 2021.
  * Bizzozzero et al. [2023] Nicolas Bizzozzero, Ihab Bendidi, and Olivier Risser-Maroix.  Prompt performance prediction for generative ir.  _arXiv preprint arXiv:2306.08915_ , 2023.
  * Caramancion [2024] Kevin Matthe Caramancion.  Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios.  _arXiv preprint arXiv:2401.05761_ , 2024.
  * Carmel and Yom-Tov [2010] David Carmel and Elad Yom-Tov.  Estimating the query difficulty for information retrieval.  _Synthesis Lectures on Information Concepts, Retrieval, and Services_ , 2(1):1–89, 2010.
  * Croitoru et al. [2023] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion Models in Vision: A Survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(9):10850–10869, 2023.
  * Cronen-Townsend et al. [2002] Steve Cronen-Townsend, Yun Zhou, and W. Bruce Croft.  Predicting query performance.  In _Proceedings of SIGIR_ , pages 299–306, 2002.
  * Cummins et al. [2011] Ronan Cummins, Joemon Jose, and Colm O’Riordan.  Improved query performance prediction using standard deviation.  In _Proceedings of SIGIR_ , pages 1089–1090, 2011.
  * Datta et al. [2022] Suchana Datta, Debasis Ganguly, Derek Greene, and Mandar Mitra.  Deep-QPP: A pairwise interaction-based deep learning model for supervised query performance prediction.  In _Proceedings of WSDM_ , pages 201–209, 2022.
  * Datta et al. [2023] Suchana Datta, Debasis Ganguly, Josiane Mothe, and Md Zia Ullah.  Combining Word Embedding Interactions and LETOR Feature Evidences for Supervised QPP.  In _Proceedings of QPP++_ , volume 3366, pages 7–12, 2023.
  * De Loupy and Bellot [2000] Claude De Loupy and Patrice Bellot.  Evaluation of document retrieval systems and query difficulty.  In _Proceedings of LREC_ , pages 32–39, 2000.
  * Déjean et al. [2020] Sébastien Déjean, Radu Tudor Ionescu, Josiane Mothe, and Md Zia Ullah.  Forward and backward feature selection for query performance prediction.  In _Proceedings of SAC_ , pages 690–697, 2020.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of NAACL_ , pages 4171–4186, 2019.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat GANs on image synthesis.  In _Proceedings of NeurIPS_ , volume 34, pages 8780–8794, 2021.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proceedings of ICLR_ , 2021.
  * Faggioli et al. [2022] Guglielmo Faggioli, Oleg Zendel, J Shane Culpepper, Nicola Ferro, and Falk Scholer.  smare: a new paradigm to evaluate and understand query performance prediction methods.  _Information Retrieval Journal_ , 25(2):94–122, 2022.
  * Faggioli et al. [2023] Guglielmo Faggioli, Thibault Formal, Stefano Marchesin, Stéphane Clinchant, Nicola Ferro, and Benjamin Piwowarski.  Query performance prediction for neural ir: Are we there yet?  In _Proceedings of ECIR_ , pages 232–248, 2023.
  * Hashemi et al. [2019] Helia Hashemi, Hamed Zamani, and W. Bruce Croft.  Performance prediction for non-factoid question answering.  In _Proceedings of SIGIR_ , pages 55–58, 2019.
  * Hauff et al. [2008] Claudia Hauff, Djoerd Hiemstra, and Franciska de Jong.  A survey of pre-retrieval query performance predictors.  In _Proceedings of CIKM_ , pages 1419–1420, 2008.
  * Hauff et al. [2009] Claudia Hauff, Leif Azzopardi, and Djoerd Hiemstra.  The combination and evaluation of query performance prediction methods.  In _Proceedings of ECIR_ , pages 301–312, 2009.
  * He and Ounis [2004] Ben He and Iadh Ounis.  Inferring query performance using pre-retrieval predictors.  In _Proceedings of SPIRE_ , pages 43–54, 2004.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Sheng-Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu.  Ablating concepts in text-to-image diffusion models.  In _Proceedings of ICCV_ , pages 22691–22702, 2023.
  * Li et al. [2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.  In _Proceedings of ICML_ , pages 19730–19742. PMLR, 2023.
  * Li et al. [2012] Yangxi Li, Bo Geng, Linjun Yang, Chao Xu, and Wei Bian.  Query difficulty estimation for image retrieval.  _Neurocomputing_ , 95:48–53, 2012.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence C. Zitnick.  Microsoft COCO: Common Objects in Context.  In _Proceedings of ECCV_ , pages 740–755, 2014.
  * Meng et al. [2023] Chuan Meng, Negar Arabzadeh, Mohammad Aliannejadi, and Maarten de Rijke.  Query performance prediction: From ad-hoc to conversational search.  In _Proceedings of SIGIR_ , pages 2583–2593, 2023.
  * Mothe [2023] Josiane Mothe.  On Correlation to Evaluate QPP.  In _Proceedings of QPP++_ , volume 3366, pages 29–36, 2023.
  * Mothe and Tanguy [2005] Josiane Mothe and Ludovic Tanguy.  Linguistic features to predict query difficulty.  In _Proceedings of SIGIR_ , pages 7–10, 2005.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models.  In _Proceedings of ICML_ , pages 16784–16804, 2021.
  * Pavlichenko and Ustalov [2023] Nikita Pavlichenko and Dmitry Ustalov.  Best prompts for text-to-image models and how to find them.  In _Proceedings of SIGIR_ , pages 2067–2071, 2023.
  * Poesina et al. [2023] Eduard Poesina, Radu Tudor Ionescu, and Josiane Mothe.  iQPP: A Benchmark for Image Query Performance Prediction.  In _Proceedings of SIGIR_ , pages 2953–2963, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In _Proceedings of ICML_ , pages 8748–8763, 2021.
  * Raiber and Kurland [2014] Fiana Raiber and Oren Kurland.  Query-performance prediction: setting the expectations straight.  In _Proceedings of SIGIR_ , pages 13–22, 2014.
  * Roitman [2018] Haggai Roitman.  An extended query performance prediction framework utilizing passage-level information.  In _Proceedings of SIGIR_ , pages 35–42, 2018.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution Image Synthesis with Latent Diffusion Models.  In _Proceedings of CVPR_ , pages 10684–10695, 2022.
  * Samadi and Rafiei [2023] Mohammadreza Samadi and Davood Rafiei.  Performance Prediction for Multi-hop Questions.  _arXiv preprint arXiv:2308.06431_ , 2023.
  * Shtok et al. [2012] Anna Shtok, Oren Kurland, David Carmel, Fiana Raiber, and Gad Markovits.  Predicting query performance by query-drift estimation.  _ACM Transactions on Information Systems_ , 30(2):1–35, 2012.
  * Singh et al. [2023] Ashutosh Singh, Debasis Ganguly, Suchana Datta, and Craig McDonald.  Unsupervised query performance prediction for neural models with pairwise rank preferences.  In _Proceedings of SIGIR_ , pages 2486–2490, 2023.
  * Sparck Jones [1972] Karen Sparck Jones.  A statistical interpretation of term specificity and its application in retrieval.  _Journal of Documentation_ , 28(1):11–21, 1972\.
  * Sun et al. [2018] Shaoyan Sun, Wengang Zhou, Qi Tian, Ming Yang, and Houqiang Li.  Assessing image retrieval quality at the first glance.  _IEEE Transactions on Image Processing_ , 27(12):6124–6134, 2018.
  * Thakur et al. [2021] Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych.  Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks.  In _Proceedings of NAACL_ , pages 296–310, 2021.
  * Tian et al. [2011] Xinmei Tian, Yijuan Lu, and Linjun Yang.  Query difficulty prediction for web image search.  _IEEE Transactions on Multimedia_ , 14(4):951–962, 2011.
  * Tian et al. [2014] Xinmei Tian, Qianghuai Jia, and Tao Mei.  Query difficulty estimation for image search with query reconstruction error.  _IEEE Transactions on Multimedia_ , 17(1):79–91, 2014.
  * Valem et al. [2023] Lucas Pascotti Valem, Vanessa Helena Pereira-Ferrero, and Daniel Carlos Guimarães Pedronette.  Self-supervised regression for query performance prediction on image retrieval.  In _Proceedings of AIKE_ , pages 95–98. IEEE, 2023.
  * Wu et al. [2023] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li.  Human preference score: Better aligning text-to-image models with human preference.  In _Proceedings of ICCV_ , pages 2096–2105, 2023.
  * Xing et al. [2010] Xing Xing, Yi Zhang, and Mei Han.  Query difficulty prediction for contextual image retrieval.  In _Proceedings of ECIR_ , pages 581–585. Springer, 2010.
  * Yom-Tov et al. [2005] Elad Yom-Tov, Shai Fine, David Carmel, and Adam Darlow.  Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval.  In _Proceedings of SIGIR_ , pages 512–519, 2005.
  * Zamani et al. [2018] Hamed Zamani, W. Bruce Croft, and J. Shane Culpepper.  Neural query performance prediction using weak supervision from multiple signals.  In _Proceedings of SIGIR_ , pages 105–114, 2018.
  * Zhang et al. [2018] Zhongmin Zhang, Jiawei Chen, and Shengli Wu.  Query performance prediction and classification for information search systems.  In _Proceedings of APWeb-WAIM_ , pages 277–285, 2018.
  * Zhao et al. [2008] Ying Zhao, Falk Scholer, and Yohannes Tsegay.  Effective pre-retrieval query performance prediction using similarity and variability evidence.  In _Proceedings of ECIR_ , pages 52–64. Springer, 2008.
  * Zhou and Croft [2007] Yun Zhou and W. Bruce Croft.  Query performance prediction in web search environments.  In _Proceedings of SIGIR_ , pages 543–550. ACM, 2007.
  * Zhu et al. [2023] Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and Ji-Rong Wen.  Large language models for information retrieval: A survey.  _arXiv preprint arXiv:2308.07107_ , 2023.

##  Appendix A Appendix

###  A.1 Limitations

We recognize specific limitations in the annotation processes for both
generative and retrieval tasks. In the generative setting, prompts can be
subjectively interpreted by users, which may introduce variability in the
results. To mitigate this, we incorporated a control set, excluded annotations
from annotators who failed on the control set, and computed an average score
from multiple user annotations for each image/prompt pair, enhancing the
robustness of the evaluations.

In the retrieval setting, the initial ground-truth image bank was generated
using a method that combines Sentence-BERT and the bag-of-words model. We
acknowledge that some images may contain content that is not fully or
accurately captured by their paired captions, which could introduce occasional
false negatives or false positives in the ground-truth collection. This
heuristic was chosen to minimize the number of false negatives, while the
false positives were mostly curated by the enrolled annotators.

Another limitation of our study is that we consider prompts that correspond to
naturally occurring images. However, generative models are often used to
generate fictive images. Since retrieval models can only retrieve images from
a database of real images, queries that correspond to fictive images can be
easily labeled as difficult. We therefore choose captions from MS COCO to
serve as prompts/queries for our benchmark, creating an evaluation setting
that is representative for both generation and retrieval.

###  A.2 Potential Negative Societal Impact

The development and deployment of text-to-image generation and retrieval
systems come with several societal implications that warrant careful
consideration. Here, we outline key areas of concern. Our enhanced dataset is
built upon a preexisting large dataset, which may inherit and perpetuate
biases present in the original data. Additionally, user annotations might have
been influenced by their own cultural backgrounds, potentially introducing
subjective biases into the final decisions. The pre-trained generative models
employed in our study could also exhibit inherent biases, affecting the
generated outputs. These combined factors could lead to unfair representations
or reinforce existing stereotypes. We acknowledge the necessity of ongoing
efforts to identify, measure, and mitigate these biases to ensure the fairness
and inclusiveness of our models. In our work, we address these concerns by
promoting transparency in our methodology and unifying multiple user
annotations to mitigate possible individual biases.

The computational resources required for data collection, filtering, and model
training contribute to energy consumption and carbon emissions. We recognize
the environmental impact of our work and emphasize the importance of
optimizing computational processes and exploring sustainable practices to
reduce the ecological footprint of AI research.

Moreover, our benchmark can be used to develop and improve generative models.
Such models can further be used in unethical scenarios, e.g. to generate deep
fakes. In recent years, an increase in deep fake materials flooded the web,
either to spread false information or to steal sensitive information by
impersonating trustworthy individuals. While we strongly believe in the
benefits of very capable generative models, we are aware of the potential
risks. However, we can see that governments are working very closely with
academia and industry on safely developing artificial intelligence, and thus
observe and support the increasing focus on models that detect AI-generated
content to mitigate the aforementioned risks.

###  A.3 Ethical Considerations on Data Annotation

Data annotation by students is a common practice in our three universities
(University of Bucharest, Aix-Marseille Université, Université Toulouse Jean-
Jaurès) and we followed the standard protocols to get approvals from the
corresponding ethics committees. The enrolled students were compensated with
bonus points. We would like to emphasize that the students understood that the
annotation task is optional, and they could also get the extra bonus points by
performing alternative tasks (which did not involve data annotation).
Moreover, all students were given the opportunity to obtain a full grade
without the optional annotation task. Hence, there was no obligation for any
of the students to perform the annotations. The students were also able to opt
out, at any time during the annotation, without any penalties.

The enrolled students come from three universities from two different
countries (France and Romania). They represent different cultural, gender and
age groups, reflecting the varied distribution of students in each of the
three universities.

###  A.4 Related Work on QPP in Text Retrieval

Pre-retrieval features are based on information available prior to the
execution of the query. Some are independent to the document collection, such
as query length, part-of-speech features (including the number of words of
some grammatical categories), query ambiguity [12], and query complexity [29].
Other pre-retrieval features depend on the document collection statistics,
such as the inverse document frequency [40], the query scope [22] (which
measures the coverage of a query within the context of a document collection
and estimates the proportion of documents that are relevant to the query), and
the SQC [51] (which is a similarity score between the query and the
collection). Pre-retrieval predictors have the huge advantage of being
determined before running the search, but they have been found to be less
effective than post-retrieval ones on textual ad hoc retrieval [20, 29, 34,
38]. Unlike pre-retrieval features, post-retrieval features require conducting
document retrieval with the query. Most of these features are calculated based
on the scores of the retrieved documents, quantifying the robustness of the
document list, or considering the distribution of the document scores [6, 8,
9, 34, 50]. On textual IR, the Clarity Score estimates the specificity of a
query considering the language distribution of the document collection and
that of the top-retrieved documents [8]. The Normalized Query Commitment
(NQC), also known as the query drift [38], measures how much the retrieved
documents deviate from the central topic of the query. The Weighted
Information Gain (WIG) calculates the difference in information content
between the documents retrieved for a specific query and a baseline
distribution of information in the collection or corpus, based on the scores
of the top-retrieved documents [52].

The main conclusions from the earlier studies on QPP for textual ad hoc
retrieval are that post-retrieval predictors outperform pre-retrieval ones
[34], and combinations of predictors using supervised approaches are the most
effective [13].

More recent work investigated QPP on neural IR (NIR) systems [2, 10, 18, 27,
39, 49]. Datta et al. [10] employed convolutional neural layers for their
Deep-QPP predictor. This architecture has further been combined with LETOR
post-retrieval predictors with some success [11]. According to Faggioli et al.
[18], QPP models which have been developed for sparse IR methods perform worse
when applied to NIR systems. However, the authors did not consider linguistic-
based predictors in their work. On the other hand, supervised BERT-based QPP
models seem to work better. Arabzadeh et al. [3] used BERT to predict the
performance of search queries in terms of their ability to retrieve relevant
documents from a corpus. Such predictors may better capture the semantic
aspects of the query-document matching.

Other recent studies focused on the transition from ad hoc search to
conversational search [27] or question answering [19, 37]. In conversational
search, the experiments showed that supervised QPP methods outperform
unsupervised ones when a large amount of training data is available, but
unsupervised methods are effective in conversational dense retrieval method
assessment.

###  A.5 Distribution of Human Relevance Judgments for Prompt Performance
Prediction

Figure 4: A histogram showing the number of annotations per category for
images generated by Stable Diffusion XL and GLIDE.

In Figure 4, we show the number of annotations per category label computed for
the four images generated for each prompt. Although the annotations
corresponding to the ground-truth images are excluded, it is clear that most
images are voted as highly relevant, confirming that Stable Diffusion and
GLIDE generally produce relevant results.

###  A.6 Generation vs. Retrieval Performance

Table 3: Pearson and Kendall $\tau$ correlation coefficients between the performance levels measured in image generation vs. image retrieval. According to a Student’s t-test, the results marked with $\dagger$ and $\ddagger$ are significantly better than the random chance baseline at p-values ${0.01}$ and ${0.001}$, respectively. Metric | Pearson | Kendall
---|---|---
HBPP vs. P@10 | 0.118‡ | 0.082‡
HBPP vs. RR | 0.070‡ | 0.046†
P@10 vs. RR | 0.562‡ | 0.504‡

For each query, our benchmark provides performance measurements in both
generation and retrieval settings. Taking advantage of the structure of the
PQPP benchmark, we next analyze the correlation between the studied tasks:
prompt performance prediction (in text-to-image generation) and query
performance prediction (in text-to-image retrieval). We present the
correlation results in Table 3. Although the correlations are statistically
significant, the empirical analysis reveals surprisingly low correlations
between the ground-truth performance measurements for the generative and
retrieval tasks. This observation indicates that the tasks are rather
orthogonal, confirming that image generation requires the development of
dedicated prompt performance predictors.

Figure 5: Representative prompts/queries that exhibit high or low performance
in text-to-image generation (on the horizontal axis) and text-to-image
retrieval (on the vertical axis).

Aside from the quantitative results presented in Table 3, we manually inspect
prompts/queries from the following four categories: (1) high performance in
both generation and retrieval; (2) high performance in generation, low
performance in retrieval; (3) low performance in generation, high performance
in retrieval; and (4) low performance in both generation and retrieval. We
present illustrative captions of the four categories in Figure 5. For the
first category, the queries usually refer to an animate object performing an
action in a location. There are no or few attributes to describe the objects
or the location, which increases the likelihood of finding many matches during
retrieval, and reduces the constraints during generation. The queries from the
second category include inanimate objects with specific descriptions.
Generative models seem to handle many specific attributes rather well, but the
retrieval models have a hard time finding images that match the specific
descriptions. The third category comprises queries about a person performing
an action specific to some sport. The MS COCO database contains many images of
people playing sports, which makes the retrieval task pretty easy. However,
generative models struggle to generate a person in a specific pose, performing
a certain action that typically involves interacting with an object, e.g.
tennis racket or baseball bat. For the fourth category, representative queries
refer to groups of people, e.g. two men, performing specific actions in
specific locations, which are rather uncommon. Queries in this category can
also refer to uncommon objects with specific attributes, which can even be
misspelled, e.g. “gel wrist wrest”. Prompts/queries describing rare
situations/objects are difficult for both generative and retrieval models. On
the one hand, there is some common ground between prompt/query performance in
generation and retrieval, as confirmed by the examples from the first and
fourth categories. On the other hand, the examples from the second and third
categories indicate that there are task-specific characteristics placing
prompt and query performance prediction at opposite poles. Therefore, we
conclude that the task of prompt performance prediction merits further
investigation, motivating the utility of our novel benchmark.

###  A.7 Detailed Quantitative Results

Table 4: Results of the prompt performance predictors for the generative setting on the test set. We report the correlation of the predicted value with the HBPP performance for Stable Diffusion and GLIDE, respectively. For each generative model, the highest correlation is highlighted in bold. According to a Student’s t-test, the results marked with $\dagger$ and $\ddagger$ are significantly better than the random chance baseline at p-values ${0.01}$ and ${0.001}$, respectively. Predictor |  | Generative Task
---|---|---
Type | Predictor Name | Stable Diffusion | GLIDE
|  Pearson |  Kendall |  Pearson |  Kendall
Pre- | #synsets | $-0.115^{\ddagger}$ | $-0.085^{\ddagger}$ | $-0.123^{\ddagger}$ | $-0.082^{\ddagger}$
#words | $-0.118^{\ddagger}$ | $-0.115^{\ddagger}$ | $-0.101^{\ddagger}$ | $-0.092^{\ddagger}$
#conjunctions / #words | $0.037$ | $-0.011$ | $0.051$ | $0.041$
Fine-tuned BERT | $\textbf{0.252}^{\ddagger}$ | $\textbf{0.149}^{\ddagger}$ | $0.558^{\ddagger}$ | $0.405^{\ddagger}$
Post- | Fine-tuned CLIP | $0.247^{\ddagger}$ | $0.141^{\ddagger}$ | $\textbf{0.615}^{\ddagger}$ | $\textbf{0.461}^{\ddagger}$
Correlation CNN | $0.158^{\ddagger}$ | $0.061^{\ddagger}$ | $0.505^{\ddagger}$ | $0.368^{\ddagger}$
Table 5: Results of the query performance predictors for the retrieval setting on the test set. The correlation is computed for the P@${10}$ and RR scores of CLIP. For each measure and retrieval system, the highest correlation is highlighted in bold. According to a Student’s t-test, the results marked with $\dagger$ and $\ddagger$ are significantly better than the random chance baseline at p-values ${0.01}$ and ${0.001}$, respectively. Predictor |  | Retrieval Task (CLIP)
---|---|---
Type | Predictor Name | P@10 | RR
|  Pearson |  Kendall |  Pearson |  Kendall
Pre- | #synsets | $-0.129^{\ddagger}$ | $-0.080^{\ddagger}$ | $-0.052$ | $-0.035$
#words | $-0.161^{\ddagger}$ | $-0.132^{\ddagger}$ | $-0.054$ | $-0.053$
#conjunctions / #words | $-0.086^{\ddagger}$ | $-0.073^{\ddagger}$ | $-0.029$ | $-0.029$
Fine-tuned BERT | ${0.441}^{\ddagger}$ | $0.272^{\ddagger}$ | $0.218^{\ddagger}$ | $0.157^{\ddagger}$
Post- | Fine-tuned CLIP | $\textbf{0.458}^{\ddagger}$ | $\textbf{0.299}^{\ddagger}$ | ${0.173}^{\ddagger}$ | ${0.120}^{\ddagger}$
Correlation CNN | $0.314^{\ddagger}$ | $0.180^{\ddagger}$ | $\textbf{0.279}^{\ddagger}$ | $\textbf{0.210}^{\ddagger}$
Table 6: Results of the query performance predictors for the retrieval setting on the test set. The correlation is computed for the P@${10}$ and RR scores of BLIP-2. For each measure and retrieval system, the highest correlation is highlighted in bold. According to a Student’s t-test, the results marked with $\dagger$ and $\ddagger$ are significantly better than the random chance baseline at p-values ${0.01}$ and ${0.001}$, respectively. Predictor |  | Retrieval Task (BLIP-2)
---|---|---
Type | Predictor Name | P@10 | RR
|  Pearson |  Kendall |  Pearson |  Kendall
Pre- | #synsets | $-0.134{\ddagger}$ | $-0.095{\ddagger}$ | $-0.063^{\dagger}$ | $-0.036$
#words | $-0.206^{\ddagger}$ | $-0.167^{\ddagger}$ | $0.058^{\dagger}$ | $0.042^{\ddagger}$
#conjunctions / #words | $-0.128^{\ddagger}$ | $-0.108^{\ddagger}$ | $-0.038$ | $-0.039$
Fine-tuned BERT | $\textbf{0.499}^{\ddagger}$ | $0.317^{\ddagger}$ | $0.210^{\ddagger}$ | $0.157^{\ddagger}$
Post- | Fine-tuned CLIP | $0.457^{\ddagger}$ | $\textbf{0.339}^{\ddagger}$ | ${0.149}^{\ddagger}$ | ${0.115}^{\ddagger}$
Correlation CNN | $0.298^{\ddagger}$ | $0.190^{\ddagger}$ | $\textbf{0.213}^{\ddagger}$ | $\textbf{0.160}^{\ddagger}$
Table 7: Complete results of the prompt/query performance predictors for the generative and retrieval settings on the test set. On the generative task, we report the correlation of the predicted value with the average HBPP performance of Stable Diffusion and GLIDE. On the retrieval task, the correlation is computed for the average P@${10}$ and the average RR, where the averages are computed over CLIP and BLIP-2. For each task, the highest correlation is highlighted in bold. According to a Student’s t-test, the results marked with $\dagger$ and $\ddagger$ are significantly better than the random chance baseline at p-values ${0.01}$ and ${0.001}$, respectively. Predictor |  | Generative Task | Retrieval Task
---|---|---|---
Type | Predictor Name | HBPP | P@10 | RR
|  Pearson |  Kendall |  Pearson |  Kendall |  Pearson |  Kendall
Pre- | #synsets | $-0.152^{\ddagger}$ | $-0.093^{\ddagger}$ | $-0.141^{\ddagger}$ | $-0.097^{\ddagger}$ | $-0.071^{\dagger}$ | $-0.037^{\dagger}$
#words | $-0.136^{\ddagger}$ | $-0.113^{\ddagger}$ | $-0.199^{\ddagger}$ | $-0.163^{\ddagger}$ | $-0.069^{\dagger}$ | $-0.052^{\dagger}$
avg word length | $0.049$ | $0.039^{\dagger}$ | $-0.116^{\ddagger}$ | $-0.084^{\ddagger}$ | $-0.067^{\dagger}$ | $-0.046^{\dagger}$
avg #acronyms | $0.010$ | $0.004$ | $-0.001$ | $-0.021$ | $0.015$ | $0.012$
avg #numerals | $-0.045$ | $-0.036$ | $-0.065^{\dagger}$ | $-0.066^{\ddagger}$ | $-0.018$ | $-0.001$
#conjunctions / #words | $0.058^{\dagger}$ | $0.045^{\dagger}$ | $-0.117^{\ddagger}$ | $-0.097^{\ddagger}$ | $-0.041$ | $-0.040$
#prepositions / #words | $0.032$ | $0.013$ | $0.060^{\dagger}$ | $0.034$ | $0.035$ | $0.024$
edge count | $0.037$ | $0.062^{\ddagger}$ | $0.027$ | $0.047$ | $0.013$ | $0.010$
edge weight sum | $0.030$ | $0.057^{\ddagger}$ | $0.025$ | $0.046^{\dagger}$ | $0.018$ | $0.004$
inverse weight freq | $0.090^{\ddagger}$ | $0.071^{\ddagger}$ | $0.045$ | $0.039$ | $0.010$ | $0.011$
degree centrality | $0.027$ | $0.005$ | $0.008$ | $0.030$ | $0.028$ | $0.001$
cloneness centrality | $0.085^{\ddagger}$ | $0.065^{\ddagger}$ | $0.048$ | $0.038$ | $0.016$ | $0.011$
between centrality | $0.020$ | $0.004$ | $0.007$ | $0.016$ | $0.020$ | $0.004$
Page Rank | $0.086$ | $0.064$ | $0.049$ | $0.040^{\dagger}$ | $0.014$ | $0.011$
Fine-tuned BERT | $0.568^{\ddagger}$ | $0.416^{\ddagger}$ | $\textbf{0.507}^{\ddagger}$ | $0.318^{\ddagger}$ | $0.262^{\ddagger}$ | $0.182^{\ddagger}$
Post- | Fine-tuned CLIP | $\textbf{0.600}^{\ddagger}$ | $\textbf{0.448}^{\ddagger}$ | $0.490^{\ddagger}$ | $\textbf{0.330}^{\ddagger}$ | ${0.197}^{\ddagger}$ | ${0.129}^{\ddagger}$
Correlation CNN | $0.471^{\ddagger}$ | $0.347^{\ddagger}$ | $0.327^{\ddagger}$ | $0.196^{\ddagger}$ | $\textbf{0.301}^{\ddagger}$ | $\textbf{0.212}^{\ddagger}$
Score Variance @100 | $0.003$ | $0.005$ | $0.016$ | $0.003$ | $0.016$ | $0.004$

In Table 4, we present the results of prompt performance predictors for each
generative model (Stable Diffusion and GLIDE). We observe that predicting
prompt performance for Stable Diffusion is generally more difficult than for
GLIDE. This observation can be attributed to the higher variety (in terms of
relevance to the prompt) of the images generated by GLIDE.

In Table 5 and Table 6, we report the results of query performance predictors
for the CLIP and BLIP-2 retrieval systems, respectively. The results are
consistent with the ones presented in Table 2, which are obtained by averaging
the performance of the two models as a ground truth. Indeed, the pre-retrieval
predictors based on the number of synsets, the number of words, and the ratio
between conjunctions and words are weakly correlated with the individual
system performance. For the RR performance measure, both Pearson and Kendall
correlations indicate that the best predictor for both retrieval models is the
correlation CNN. For P@10, the correlation for the pre-retrieval fine-tuned
BERT and the post-retrieval fine-tuned CLIP are very close to each other, for
both retrieval models and both correlation measures.

In Table 7, we present the results for all the considered predictors, although
as mentioned in Section 5, Table 2 shows the most interesting predictors. It
is nevertheless important to also report failed attempts on specific
predictors. In general, we find that neural predictors clearly outperform
predictors that are based on simple heuristics.

###  A.8 Qualitative Results

Figure 6: t-SNE visualization of the test queries embedded in the latent space
of the fine-tuned BERT on the retrieval task. The ground-truth P@$\mathbf{10}$
performance is encoded via a color map from green to red. The visualization
confirms that the fine-tuned BERT predictor learns a meaningful representation
of the queries. Best viewed in color.

In Figure 6, we present a t-SNE visualization of the test queries embedded in
the latent space of the BERT predictor fine-tuned on the retrieval task. We
observe that the learned embedding correlates well with the ground-truth
P@$10$ values, explaining the high accuracy of the fine-tuned BERT predictor
on the retrieval task. The separation between easy and difficult queries is
evident in the retrieval setting, which is consistent with the quantitative
results reported in Table 2, where the fine-tuned BERT exhibits generally
higher Pearson and Kendall $\tau$ correlation coefficients with P@$10$ than
with HBPP.

Figure 7: Predicted (vertical axis) vs. reference (horizontal axis)
performance for the fine-tuned CLIP predictor, in the text-to-image
_generation_ setting. Figure 8: Predicted (vertical axis) vs. reference
(horizontal axis) performance for the BERT predictor, in the text-to-image
_retrieval_ setting.

In Figure 7, we illustrate the correlation between the HBPP scores predicted
by the post-generation fine-tuned CLIP predictor and the ground-truth HBPP
scores, via box plots. In this image generation setting, the model does not
seem capable of distinguishing between prompts with _unrealistic_ (score $-1$)
and _irrelevant_ (score $0$) images (left-hand side of the figure). In
contrast, there is a more obvious correlation between prompts with
_irrelevant_ (score $0$), _lowly relevant_ (score $1$) and _highly relevant_
(score $2$) images (right-hand side of the figure). An analogous plot for the
retrieval task is displayed in Figure 8, where the comparison is based on the
fine-tuned BERT the P@$10$ measure. In the image retrieval setting, the box
plots indicate that the fine-tuned BERT exhibits low variance for low P@$10$
values. However, queries with high difficulty are not handled well by this
predictor.

Figure 9: Examples of generative (first two columns) and retrieval (last two
columns) results for difficult (first and third columns) and easy (second and
fourth columns) queries. For the retrieval systems, we show only the top two
results. The prompt/query performance values, namely HBPP and P@$\mathbf{10}$,
are predicted by the correlation-based CNN. Best viewed in color.

We showcase examples of easy and difficult prompts/queries for the generation
and retrieval tasks in Figure 9. The generative models exhibit a clear
proficiency with prompts referring to inanimate objects, generating images
with high relevance. However, their capability falls short when faced with
more intricate prompts involving complex actions based on human-object
interactions, leading to inaccuracies in object composition. Such cases
exhibit artifacts, such as duplicate or missing body parts and misplaced
objects, showcasing the lack of deep understanding in both generative models.
The retrieval systems are capable of fetching images for prompts centered
around single, loosely-defined objects. However, they struggle when the
prompts require images containing multiple, specific elements, retrieving
results that only partially match the query. This limitation highlights a gap
in the ability of retrieval systems to interpret and respond to the
multifaceted nature of some queries. This exploration into both generative and
retrieval tasks underscores the nuanced challenges faced by systems in
accurately capturing and responding to the inherent complexity of certain
prompts. It also reinforces the importance of the prompt/query performance
prediction task, setting realistic expectations for the outcomes of both
generative and retrieval models, based on the detailed content and structure
of the prompts/queries.

###  A.9 Computational Resources

We have employed two types of machines to perform our experiments:

  * •

Local Hardware:

    * –

GPU: NVIDIA RTX 3090 with 24GB VRAM

    * –

CPU: Intel i9-10920X @ 3.50GHz

    * –

Memory: 64GB RAM

    * –

Storage: 1TB SSD, 5TB HDD

  * •

Cloud VM:

    * –

GPU: NVIDIA A100 with 40GB VRAM

    * –

CPU: 12 vCPUs

    * –

Memory: 85GB RAM

    * –

Storage: 100GB HDD

Our annotation platform was hosted using Google Cloud Provider, with
authentication developed using Google Firebase Authentication, and image
hosting facilitated by Google Cloud Storage.

By detailing the compute resources utilized, we aim to provide transparency
and reproducibility for our research.

###  A.10 Computational Time Estimation

We present the following estimation of the compute time (in hours) required to
fully replicate the experiments detailed in this paper:

  * •

Pre-processing of the MS COCO dataset: The extraction of Sentence-BERT
embeddings and the subsequent application of the k-means clustering algorithm
across the entire corpus of MS COCO captions requires approximately 48 hours.

  * •

Generative processes: The generative processes employing both the SDXL and
GLIDE methods demand a total time of approximately 120 hours.

  * •

Preliminary relevance judgments: The creation of initial relevance judgments
for the retrieval task takes 72 hours.

  * •

Model fine-tuning: The cumulative time spent on fine-tuning all predictors
involved in our study amounts to 48 compute hours.

These estimates are based on the computational resources and configurations
described in Section A.9.