# “In-Context Learning” or: How I learned to stop worrying and love “Applied
Information Retrieval”

Andrew Parry  1234-5678-9012<EMAIL_ADDRESS>University of
GlasgowGlasgowUK ,  Debasis Ganguly<EMAIL_ADDRESS>1234-5678-9012 University of GlasgowGlasgowUK and  Manish Chandra
<EMAIL_ADDRESS>1234-5678-9012 University of GlasgowGlasgowUK

(2024; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

With the increasing ability of large language models (LLMs), in-context
learning (ICL) has evolved as a new paradigm for natural language processing
(NLP), where instead of fine-tuning the parameters of an LLM specific to a
downstream task with labeled examples, a small number of such examples is
appended to a prompt instruction for controlling the decoder’s generation
process. ICL, thus, is conceptually similar to a non-parametric approach, such
as $k$-NN, where the prediction for each instance essentially depends on the
local topology, i.e., on a localised set of similar instances and their labels
(called few-shot examples). This suggests that a test instance in ICL is
analogous to a query in IR, and similar examples in ICL retrieved from a
training set relate to a set of documents retrieved from a collection in IR.
While standard unsupervised ranking models can be used to retrieve these few-
shot examples from a training set, the effectiveness of the examples can
potentially be improved by re-defining the notion of relevance specific to its
utility for the downstream task, i.e., considering an example to be relevant
if including it in the prompt instruction leads to a correct prediction. With
this task-specific notion of relevance, it is possible to train a supervised
ranking model (e.g., a bi-encoder or cross-encoder), which potentially learns
to optimally select the few-shot examples. We believe that the recent advances
in neural rankers can potentially find a use case for this task of optimally
choosing examples for more effective downstream ICL predictions.

Large Language Models, In-Context Learning, Ranking Models, Query Performance
Prediction

††copyright: acmlicensed††journalyear: 2024††copyright:
acmlicensed††conference: Proceedings of the 47th International ACM SIGIR
Conference on Research and Development in Information Retrieval; July 14–18,
2024; Washington, DC, USA††booktitle: Proceedings of the 47th International
ACM SIGIR Conference on Research and Development in Information Retrieval
(SIGIR ’24), July 14–18, 2024, Washington, DC, USA††doi:
10.1145/3626772.3657842††isbn: 979-8-4007-0431-4/24/07††ccs: Information
systems Information retrieval††ccs: Computing methodologies Machine
learning††ccs: Computing methodologies Natural language processing Figure 1. A
workflow diagram illustrating how three verticals of IR research fit into the
workflow of in-context learning (ICL). Section 3 discusses possible ways of
adjusting unsupervised and supervised QPP approaches for adapting the number
of ICL examples. Section 4 discusses ideas of how to learn the notion of
downstream usefulness of examples. Section 5 discusses methodologies related
to diversifying examples for ICL.

##  1\. Introduction

Research on large language models (OpenAI, 2023) (LLMs) is expanding in scope
and yielding significant scientific advancements rapidly. These language
models are pre-trained on large corpora of documents to capture the inherent
semantics of text in a generic task-independent manner. Common pre-training
methodologies either involve a masked language model (MLM) that predicts
randomly masked tokens from the text (Devlin et al., 2019a; Liu et al., 2019;
Reimers and Gurevych, 2019), or an auto-regressive model or causal language
model (CLM) which predicts a token only from its predecessor tokens (Radford
et al., 2019; Brown et al., 2020; Wang and Komatsuzaki, 2022). While MLM is
employed in BERT (Devlin et al., 2019a) and its successors, such as RoBERTa
(Liu et al., 2019), BART (Lewis et al., 2020a) etc., the latter class of
models, i.e., CLM, is applied to train GPT variants (Radford et al., 2019;
Brown et al., 2020; OpenAI, 2023) and open-source Llama and Mistral variants
(Touvron et al., 2023; Jiang et al., 2023) etc. LLMs, when scaled from
millions to billions of parameters, have demonstrated to be adaptable to a
broad set of tasks due to instruction tuning (Ouyang et al., 2022), in the
sense that they are not only able to produce semantically correct and coherent
text but are also able to adapt themselves surprisingly well with small
changes in contexts supplied as inputs, commonly called prompts (Arora et al.,
2022).

This ability to adapt to unseen data and tasks with only a small number of
examples differs from the standard notion of supervised learning, where the
parameters of a pre-trained model (e.g., BERT (Devlin et al., 2019a)) is then
again learned (commonly referred to as ‘fine-tuning’) from a training set of
labelled examples. Instead, in few-shot learning or in-context learning (ICL),
a small number of labelled examples from a training set are simply appended to
a prompt instruction to control the text generation in a desirable way
beneficial to the downstream task (Mysore et al., 2023; Li et al., 2022; Ni et
al., 2021; Pradeep et al., 2023a). In addition to leveraging ICL for a purely
generative task, e.g., question answering or abstractive summarisation (Brown
et al., 2020; Li et al., 2023; Tang et al., 2023), a more common use is in a
predictive task, such as text classification (Lu et al., 2022; Milios et al.,
2023), where each class is specified by a set of words (commonly called a
verbaliser (Schick and Schütze, 2021)), e.g., for a binary sentiment
classification task the positive class could be defined by the set
$\\{$‘good’, ‘great’, ‘wonderful’…$\\}$. Once each class for a predictive task
is well-defined, the generated text can be mapped to the most likely class(es)
by using the posterior over the vocabulary generated by the decoder.

It is to be realised that ICL is somewhat conceptually similar to a non-
parametric approach, such as $k$-NN, where the prediction for each instance
essentially depends on the local topology, i.e., on a localised set of similar
instances and their labels (called few-shot examples) - the only difference of
ICL with $k$-NN is that the former involves a frozen set of encoder-decoder
parameters of an underlying LLM, where ICL generally works well on any domain
with only a small number of examples because, unlike supervised models, it
does not suffer from overfitting the parameters on a particular set of
labelled examples. Instead, the semantics expressed in the examples
potentially play a key role in controlling the text generation process to
yield the desired output—text generation process to yield the desired output –
either the text itself or a mapping into a class prediction.

The usefulness of localised examples, akin to the nearest neighbor-based
prediction, suggests a strong case of analogy between ICL and ad-hoc IR. More
precisely, a test instance in ICL is analogous to a query in IR, and similar
examples in ICL retrieved from a training set relate to a set of documents
retrieved from a collection in IR. This analogy opens up several interesting
research questions in ICL concerning the effective use of IR to improve ICL
predictions. In this perspective paper, we discuss particular components of
ICL that can be mapped to known and well-researched IR problems. This means
that the solutions to these problems that the IR community has researched over
decades can potentially be applied to improve the effectiveness of ICL.
Moreover, this should also be of interest to an IR researcher to develop new
methodologies for classic problems in IR, such as that of document retrieval
or query performance prediction (QPP) (Datta et al., 2022; Roitman et al.,
2020; Singh et al., 2023), specifically catered to a downstream predictive
task, hence opening up new possibilities of evaluating novel IR methodologies
beyond retrieval tasks.

We now propose the following three main ways of incorporating the ideas from
core IR into ICL. First, during inference in ICL, instead of using a constant
number of examples for each instance, a potentially better approach could be
to make the number of examples variable. A similar problem in IR is to predict
how many documents to retrieve (or equivalently to predict the rank cutoff
threshold (Arampatzis et al., 2009; Bahri et al., 2020)), which is also
closely tied to the problem of query performance prediction (QPP) (Datta et
al., 2022; Roitman et al., 2020; Singh et al., 2023). In the context of ICL,
this means that for some test instances, one can find more ‘useful’ examples
from the training set (an example can be considered useful if including it as
a part of the prompt leads an LLM to the correct prediction). In contrast, it
is difficult for others to find such useful examples. An ICL method that is
aware of the quality of the examples can, hence, potentially adapt itself,
e.g., by using a higher number of examples if the example quality is predicted
to be poor.

Second, we propose to make the underlying metric space – used to compute the
similarities between a test instance and the examples – learnable. The
objective of learning this similarity function would be to rank the ‘useful’
examples ahead of the not useful ones. Unsupervised retrieval models only
consider the similarity between the textual content of a test instance and the
training instances. However, a supervised retrieval model learned via a
standard ranking objective (e.g., noise-contrastive loss (Gutmann and
Hyvärinen, 2010)) using triplets – each triplet constituting a test instance
(query), a useful example (relevant document), and a not useful one (non-
relevant document) – may specifically capture the inherent semantics of the
utility of examples for a particular test instance.

Third, diversity of examples is likely to play a part in the effectiveness of
ICL because an example that is different from the previous one selected should
be more informative to an LLM decoder to generate relevant words which can
then be mapped to the correct class. This can also be traced to facet or
aspect-based IR which attempts to make the top retrieved set of documents
cater to all the latent aspects of an information need (Upadhyay et al., 2020;
Mahdi et al., 2020).

These three core tasks in IR, namely that of QPP, supervised ranking or
learning to rank, and diverse or faceted IR have a long history of thorough
investigation, which have constantly pushed the boundaries of the state-of-
the-art achievable for these tasks. In this perspective paper, we argue that
this knowledge gained by the IR community could be beneficial to further
improve the effectiveness of generative AI for text.

In the next section (Section 2), we provide a brief technical introduction to
the concept of ICL, following which, we structure the remainder of the paper
into three sections detailing on how each of the specific IR tasks can be
applied to an ICL workflow, i.e., QPP for adaptive ICL (Section 3), learning
to rank for learning to order examples in ICL (Section 4), and diversity-based
and faceted IR for obtaining more informative examples in ICL (Section 5).
Although an exhaustive empirical validation of each of these independent ideas
for improving ICL is beyond the scope of a perspective paper, we do, however,
include a preliminary evaluation to support the use-case of QPP in ICL
(Section 6), where we show that adjusting the number of examples in a data-
driven manner does lead to significant improvements. We believe that this
focused investigation, along with the other ideas presented, would motivate
other NLP researchers to apply black-box established IR methodologies or even
IR researchers to adjust the state-of-the-art IR methodologies to specifically
cater to the downstream predictive tasks in ICL.

##  2\. In-Context Learning

We first provide a brief technical introduction to In-Context Learning (ICL)
before describing how the ICL methodology can be improved by incorporating
core ideas from IR.

###  2.1. A Formal Introduction

In-context learning (ICL), unlike supervised learning, does not involve
training a set of parameters $\theta$ on labeled examples. Rather, the
posteriors are now a function of the following: a) text of the input test
instance, b) the decoder parameters of a pre-trained large language model
(LLM), c) a prompt instruction, and d) optionally, a set of $k$ input examples
(commonly called $k$-shot learning). Formally,

(1)
$P(y|\mathbf{x})=f(\mathbf{x},\mathcal{P}_{k}(\mathbf{x});\phi_{\text{LLM}}),$

where, different from a supervised setup, the function $f$ does not have a
parameterized representation that can be learned using a training set with
gradient descent. The function itself depends on the pre-trained parameters
$\phi_{\text{LLM}}$ of an LLM, the current inputs for which a label is to be
predicted, and a prompt comprising a set of $k$ text units denoted by
$\mathcal{P}_{k}(\mathbf{x})$.

Since the decoder of an LLM generates a sequence of words of the form of
$w_{1},\ldots,w_{N}$ ($N$ being the maximum length of a sequence), the class
posterior likelihoods are computed in the following way. A set of classes (say
for a $p$-way classification problem) is mapped to $p$ different equivalent
sets of words, say $V(y)$, where $y\in\mathbb{Z}_{p}$ – these sets commonly
being called verbalisers (Hu et al., 2021). For instance, for a binary
classification problem (e.g., that of a movie review as shown in Figure 2),
$p=2$ (i.e., $y\in\\{0,1\\}$), and a reasonable way to define the verbaliser
sets could be via the following words:
$V(0)=\\{\text{`false'},\text{`negative'}\\}$, and
$V(1)=\\{\text{`true'},\text{`positive'}\\}$.

Note that the word ‘learning’ in ICL is a misnomer because there are no
updates to the decoder parameters of an LLM. For more details on ICL, please
refer to these excellent surveys (Dong et al., 2023; Luo et al., 2024).

Figure 2. Example workflow of In-Context Learning for sentiment
classification. The illustrative example shows a sample test instance for
which a single demonstration (as retrieved from the training set) does not
result in the correct prediction (prediction shown at the top). The example
also shows that increasing the number of demonstrations from one to two
results in the correct prediction (shown at the bottom). Demonstrations
included within the prompt are shown in blue.

###  2.2. The role of IR

One of the most important components of ICL (as shown in Figure 2) is the
search component that outputs a top-$k$ candidate set of similar instances
from the training set, i.e., $\mathcal{P}_{k}(\mathbf{x})$ of Equation 1.
Although, in principle, it is possible to include random examples from the
training set in the prompt, it has been shown that localised examples (i.e.,
examples that are topically similar to the current instance) yield better
performance (Liu et al., 2022; Luo et al., 2024). The reason why this works
can be traced to the fundamental principle of reproducing kernel Hilbert
spaces (RKHS) machine learning – that a predictor function is an aggregation
of parameterised kernel functions pivoted around training data instances
(Paulsen and Raghupathi, 2016).

It is thus crucial to retrieve as many relevant examples as possible from the
training set while imposing a practical constraint on the number of such
examples for efficiency reasons – a classic trade-off of recall and precision
in IR ad-hoc retrieval; the only difference is that relevance for ICL needs to
be defined in terms of the utility or usefulness of an example towards the
correct prediction.

A similar question explored in IR is where to stop reading a ranked list
because there is little utility in retrieving documents due to the low
probability of finding relevant documents beyond a certain rank cutoff
(Arampatzis et al., 2009; Bahri et al., 2020). What is more challenging is
that this rank cut-off depends on the number of relevant documents occurring
in the collection for a specific query, that is to say, while some queries
with well-defined information needs are associated with a small number of
relevant documents satisfying the specific relevance criterion, other queries
with broader information needs usually are associated with a higher number of
relevant documents (Carterette et al., [n. d.]). In core IR research, this
problem is usually addressed by estimating the retrieval qualities of queries
– the assumption being that well-specified queries yield better retrieval
results (in terms of precision and recall), whereas ill-specified ones suffer
from poor retrieval quality due to the apparent ambiguity of information need.
This motivation paves the path to the following section, where we discuss how
query performance prediction (QPP) can also be beneficial to the related
problem of retrieving similar examples in ICL.

##  3\. Adaptive ICL $\mapsto$ QPP?

In this section, we describe an adaptive approach to the selection of examples
for ICL. We outline analogous principles from IR literature that can be
applied in broader tasks.

###  3.1. A Variable Number of Examples

Figure 3.  Motivation behind using a variable sized neighborhood for $k$-NN
classification (Zhang et al., 2017): An instance close to a decision boundary
(black ‘?’) is likely to have a higher heterogeneity in its class
distribution, thus indicating the necessity of a larger neighborhood for an
effective classification.

The observation in IR that different queries exhibit different levels of
retrieval performance can be utilised for ICL, where we can draw an analogy
that some test instances are associated with better candidates for training
examples (i.e., examples which are useful in the sense that including them as
a part of the prompt leads to correct predictions), and hence including a
small number of them should be adequate. On the other hand, the retrieval
quality for some test instances (used as queries for ICL) does not yield good
candidates. As a result, one needs to look down the ranked list further to
collect useful examples.

We call this methodology of using a variable number of demonstrations for ICL
inference by the name ‘Adaptive In-Context Learning’, or AICL for short. The
idea of AICL centres around choosing the context $\mathcal{P}_{k}(\mathbf{x})$
in a data-driven manner, i.e., making $k$ a function of the data (current
instance $\mathbf{x}$) itself. This is somewhat analogous to choosing
different values of $k$ for a $k$-NN based non-parametric modeling (Zhang et
al., 2017), as shown in Figure 3. The motivation is that classifying some
instances would be more difficult than others, in which cases they are
potentially to be benefited from a larger value of $k$ (more context). On the
other hand, for relatively easy data instances using too much context may be
detrimental for effective prediction.

Formally speaking, the difference of AICL with that of ICL (Equation 1) is
that the value $k$, indicating the size of the neighborhood, is no longer a
constant. Instead, we denote it by a parameterised function
$\kappa(\mathbf{x})$ such that

(2)
$P(y|\mathbf{x})=f(\mathbf{x},\mathcal{P}_{\kappa(\mathbf{x})}(\mathbf{x});\phi_{\text{LLM}}),$

where $\kappa:\mathbf{x}\mapsto\\{0,\ldots,M\\}$, $M$ being an upper bound on
the number of example instances. We now suggest how unsupervised or supervised
approaches may be applied to choose the rank cutoff $\kappa$.

###  3.2. Unsupervised Rank Cutoff

Among unsupervised approaches, two main ideas in IR research can be used to
determine the number of examples in ICL.

#### Score Distribution-based Models

The first thread of work is based on the hypothesis that the scores of
relevant and non-relevant documents follow a different statistical
distribution, e.g., Arampatzis et al. propose to use a mixture of Normal-
Exponential distributions – Normal for relevant and Exponential for non-
relevant documents – to model the score distribution of top-ranked documents.
The work in (Arampatzis et al., 2009) uses expectation maximisation (EM) to
estimate the parameters of this mixture distribution and thereby predict the
most likely cutoff rank beyond which the probability of finding a relevant
document is considerably low. Such ideas of utilising the characteristic
differences between the score distributions of relevant and non-relevant
documents have also been used for query performance prediction (QPP) (Cummins,
2014).

While an EM from retrieval scores allows provision for applying a variable
number of examples, the following are some ICL-specific challenges that need
to be researched.

  * •

With the notion of relevance being changed to ‘downstream utility’, the score
distributions of useful and not useful examples may not follow the same
mixture distribution of Normal-Exponential as reported in (Arampatzis et al.,
2009; Cummins, 2014). It will be an interesting future research direction to
investigate the latent relations between the similarity scores and the
downstream utility of the examples in the context of ICL.

  * •

With a threshold on the score-distributions, it is difficult to restrict the
cutoff to a maximum value, which is essential for ICL due to a maximum limit
on the input size to an LLM.

  * •

A score distribution-based approach does not explicitly consider the
information from the queries themselves (equivalently, the test instances in
ICL).

We now describe another thread of work in IR research that may help alleviate
the last two limitations.

#### QPP-based Models

Different from the rank cut-off strategies, query performance prediction (QPP)
models seek to estimate the retrieval quality of a query. As a direct analogy,
such methods can be applied to the top-similar examples retrieved in ICL with
a different objective of predicting the usefulness of the examples.

Most of the classic works in QPP involve unsupervised approaches that make use
of the information from the set of top-retrieved documents to estimate how
topically distinct are the top-retrieved documents from the rest of the
collection – a large difference indicating potentially better retrieval
quality (Cronen-Townsend et al., 2002). Various evidences extracted from the
top-retrieved documents have been shown to be useful for different post-
retrieval QPP estimation methods. This includes i) the KL divergence between
the language model of the top-retrieved documents and the collection model in
Clarity (Cronen-Townsend et al., 2002), ii) the aggregated values of the
information gains of each top-retrieved document with respect to the
collection in WIG (Weighted Information Gain) (Zhou and Croft, 2007), iii) the
skew of the RSVs (Retrieval Status Values) measured with variance in NQC
(Normalized Query Commitment) (Shtok et al., 2012), iv) ideas based on the
clustering hypothesis for a pairwise document similarity matrix (Diaz, 2007),
and, more recently, v) the characteristics of the embedded space of documents
and queries (Roy et al., 2019; Faggioli et al., 2023).

A suitable adaptation of these existing techniques can be applied in a two-
stage pipeline to determine the number of examples in ICL. As a first step,
one can employ a QPP methodology to predict the retrieval quality (in terms of
the usefulness) of a set of ordered examples – a high value likely indicating
that the useful examples can potentially be found at the very top ranks, as a
result of which, a small number of examples should potentially work well. On
the other hand, a low QPP estimate likely indicates that the very top ranked
examples are not likely to be useful for downstream prediction, in which case
it should be better to employ a large number of examples. This approach of
selecting rank cutoffs (with an upper bound) as a function of the QPP scores
has been applied to determine a variable depth of relevance assessments
required for a robust retrieval evaluation (Ganguly and Yilmaz, 2023).

###  3.3. Supervised Rank Cutoff

Instead of devising a heuristic to predict the number of training examples to
use for a test instance $\mathbf{x}$, i.e., $\kappa(\mathbf{x})$, a supervised
approach can be applied to solve this as a classification problem, i.e.,
$\kappa\equiv\text{Softmax}(\mathbf{x}^{\mathrm{T}}\theta)$, where $\theta$ is
a set of layer(s) of parameters. The underlying hypothesis is that if we
provide enough training data constituting the optimal number of examples for a
range of topical content, we should be able to learn to predict the likely
number of examples to use for unseen text during inference time.

To train a classifier that maps a text to a number between 1 to $M$ (the
maximum number of examples), it is necessary to obtain the ground-truth
labels, i.e., the optimal number of examples, for each instance in the
training set. We propose to obtain this by the following methodology: Given a
training set instance $\mathbf{x}$, one can employ a similarity function
(e.g., BM25) to retrieve a candidate set of $M$ examples -
$\\{\mathbf{z}_{1},\ldots,\mathbf{z}_{M}\\}$. Since $\mathbf{x}$ is an
instance from the training set, we can utilise its label to check if the
$k$-shot predictions using an LLM are correct. It may happen that correct
predictions are obtained for several values of $k\in\\{1,\ldots,M\\}$. Several
strategies can be adapted to define the ground-truth number of examples. For
instance, one can stop early and simply select the smallest $k$ that results
in a correct prediction. Alternatively, a potentially more robust procedure
would be to exhaustively check through all possible values of $k=1,\ldots,M$,
and select the one that results in a correct prediction with the least
uncertainty (Rubin et al., 2022; Sorensen et al., 2022).

The workflow of this least uncertainty-based selection of the ground truth for
the number of ICL examples is shown in Algorithm 2. Algorithm 1, which is
invoked during the ground-truth construction, shows a sample prompt template
for text classification.

Input: $\mathbf{x}$ – an instance from the training set

Input: $k(<M)$ – number of examples (max $M$)

Output: $\Delta_{p}$ – Softmax posteriors

begin

$N_{k}(\mathbf{x})\leftarrow\\{\mathbf{z}_{1},\ldots,\mathbf{z}_{k}\\}$

Instruction $\leftarrow$ “Predict the type of $\langle\mathbf{x}\rangle$ as
one of $\\{\langle C_{0}\rangle,\ldots,\langle C_{p-1}\rangle\\}$ given the
following example”.

for  _$i\leftarrow 1$ to $k$_ do

Instruction.append(“Example: $\langle\mathbf{z}_{i}\rangle$ is a
representative of class $\langle y(\mathbf{z}_{i})\rangle$”)

$\Delta_{p}\leftarrow$ LLM(Instruction)

return _$\Delta_{p}$_

Algorithm 1 LLM $k$-shot predictions

Input: $\mathcal{T}$ – a training set of labelled instances

Output: $\mathcal{K}=\cup_{\mathbf{x}\in\mathcal{T}}k^{*}(\mathbf{x})$ –
Number of examples yielding the most confident and correct predictions for
each instance $\mathbf{x}\in\mathcal{T}$

begin

for  _$\mathbf{x}\in\mathcal{T}$_ do

max_confidence $\leftarrow$ 0; $k^{*}\leftarrow 1$

for  _$j\leftarrow 0$ to $M$_ do

$\Delta_{p}\leftarrow$ LLM $k$-shot predictions($\mathbf{x},j$)

// Call Algorithm 1, i.e., try to predict with $j$ examples

$\hat{y}(\mathbf{x})\leftarrow\text{argmax}\Delta_{p}$

// Get the predicted class

confidence$\leftarrow\Delta_{\hat{y}(\mathbf{x})}\mathbb{I}(\hat{y}(\mathbf{x})=y(\mathbf{x}))$

// Check if the predicted class is the correct one and record the prediction
confidence

if  _confidence $>$ max_confidence_ then

max_confidence $\leftarrow$ confidence

// Keep track of the least uncertain correct prediction

$k^{*}\leftarrow j$

$\mathcal{K}\leftarrow\mathcal{K}\cup k^{*}$

return _$\mathcal{K}$_

Algorithm 2 Optimal number of examples

After executing Algorithm 2, we obtain a set of ground-truth labels
$\mathcal{K}$ which could then be used to train a classifier, parameterised by
$\theta$, via optimising:

(3)
$\text{argmin}_{\theta}\sum_{\mathbf{x}\in\mathcal{T},k^{*}\in\mathcal{K}}\mathcal{L}(\mathbf{x}^{\mathrm{T}}\theta,k^{*}),$

where $\mathcal{L}$ is a standard loss function, e.g., the cross-entropy.

During inference, for each $\mathbf{x}\in\mathcal{E}$ ($\mathcal{E}$ denoting
an evaluation set), we propose to apply the classifier
$\kappa:\mathbf{x}\mapsto\\{1,\ldots,M\\}$ – trained via Equation 3 – to
predict the number of examples, and eventually conduct a
$\kappa(\mathbf{x})$-shot prediction on $\mathbf{x}$ (Equation 2).

###  3.4. Open Research Questions and Challenges

Till now in this section, we described how unsupervised and supervised
approaches can be applied to dynamically select the number of examples to be
used for an ICL-based prediction. In this section, we discuss some research
directions that could be explored to adapt ICL in alternative ways to further
improve its effectiveness.

First, we would like to point out to the existing work on generating query
variants, as a part of a data augmentation strategy, to devise alternative
formulations of the same or similar information needs. This has been shown to
improve the effectiveness of rankers (Gao et al., 2023), query performance
prediction (Zendel et al., 2019; Datta et al., 2023) relevance feedback
(Chakraborty et al., 2020), and even act as a tool to measure consistency of
IR models (Sen et al., 2022). Given the recent success of zero-shot query
generation capabilities of LLMs (Alaofi et al., 2023; Wang et al., 2023), we
believe that augmenting a test instance with alternative text representations
can be useful to eventually improve retrieval quality (and hence potentially
improve the downstream ICL effectiveness). The unsupervised and supervised
approaches for predicting the number of examples per query (test instance) may
also lead to better ICL effectiveness, as per the existing findings that
variants do actually help improve QPP (Zendel et al., 2019; Datta et al.,
2023). We thus formulate the following two research questions aligned along
this direction.

  * •

RQ-3.1: Can query variants generated by LLMs (or otherwise) improve the
prediction of the number of examples to use for each instance?

  * •

RQ-3.2: Can relevance feedback based approaches with or without the use of
generated query variants help reorder the top-$k$ initially retrieved
candidate set of examples towards a better prediction of the number of
examples?

The other direction of work involves a dynamic selection of not just the
neighborhood size but also other ICL parameters. For instance, the verbaliser
(Schick and Schütze, 2021) sets can be selected dynamically from a set of
alternatives based on the input instance. Further, a prompt can also be
selected dynamically - again based on the input instance; an unsupervised
approach exploring this idea has already been studied in (Sorensen et al.,
2022). Generally speaking, the research question that can potentially be
explored is the following.

  * •

RQ-3.3: Can other ICL parameters also be chosen in a data-driven manner to
lead to better effectiveness, e.g., the verbaliser, the prompt, or even an LLM
itself (akin to a mixture of experts)?

##  4\. Rank ICL Examples $\mapsto$ Supervised IR?

In this section, we discuss another crucial aspect of ICL that can potentially
be improved by developing ranking models specifically suited to a different
notion of relevance: ICL downstream task-specific usefulness of examples. The
concept of an effective example in core neural IR is well-researched,
particularly the notion of ‘hard’ negatives during fine-tuning (Karpukhin et
al., 2020; Gao et al., 2021a). These negatives have improved downstream
precision on ranking tasks (Xiao et al., 2022) and, more generally,
representation learning (Gao et al., 2021b).

Specific to few-shot learning, Rubin et al. (2022) employed a noise
contrastive estimation (NCE) loss (Gutmann and Hyvärinen, 2010) to train a bi-
encoder-based pairwise ranker using SBERT (Reimers and Gurevych, 2019)
embeddings. For training the ranking model, pairs of instances (relevant and
non-relevant examples) were collected in the following way. For each pivot
instance $\mathbf{x}$ from a training set, the authors employed BM25 to
constitute a top-$k$ candidate set of examples. Each pair
$(\mathbf{x},\mathbf{z}_{i})$ was then tested to check whether a 1-shot
prediction with $\mathbf{z}_{i}$ was correct, in which case, $\mathbf{z}_{i}$
was classified as a relevant example for $\mathbf{x}$, or else it was
considered as a non-relevant one. Batches comprising relevant and non-relevant
pairs were then constituted to train a standard NCE loss. While the work of
Rubin et al. (2022) is a definitive step towards leveraging a task-specific
notion of relevance, the investigation should not be considered complete.
Several potentially promising research directions should be explored to
improve ICL effectiveness further. We now provide a survey of neural ranking
literature introducing core paradigms which may be utilised in example
selection.

#### Bi-Encoder architecture

A bi-encoder architecture encodes text into a latent representation that can
be compared in a vector space; in the context of a retrieval task, these texts
would be queries and documents. While a bi-encoder is implemented either with
a Siamese network of shared parameters (Reimers and Gurevych, 2019) or as a
single encoder (MacAvaney et al., 2019), the latter has become prevalent in
recent years (Karpukhin et al., 2020; Xiao et al., 2022).

The performance of neural models in search was significantly improved with the
release of BERT (Devlin et al., 2019b). Karpukhin et al. (2020) first proposed
the use of ‘hard’ negatives mined from BM25 to improve the precision of BERT-
based rankers. Gao et al. (2021a) then proposed a variant of the NCE
objective, ‘Localised Contrastive Estimation’, in which multiple negatives are
sampled for each query to account for the variance in the notion of non-
relevance. In doing so, they also showed the effectiveness of hard negatives
mined from fine-tuned rankers. To further improve the quality of negative
samples, Xiong et al. (2021) proposed that a model could choose negatives
during training to allow negatives to become continuously ‘harder’ as fine-
tuning progresses.

At a conceptual level, bi-encoders generally represent a text as a single
embedding by using the representation of the BERT [CLS] token as a proxy for
the entire sequence. Other pooling methods are effective, including maximum
sequence similarity (Dai and Callan, 2019) and late interaction in which a max
pooling is performed over the token-level similarity of each query token to
document tokens (Khattab and Zaharia, 2020). More recent works instead use a
BERT-style encoder with a shallow decoder, which places greater emphasis on
the ability of the encoder during pre-training. This architectural development
has yielded not only state-of-the-art recall but new pre-training styles,
including lexical grounding (Shen et al., 2023) and text reconstruction (Xiao
et al., 2022).

The separate encoding of queries and documents allows for the offline encoding
of documents which can vastly improve online latency. This is often coupled
with an approximate nearest neighbour search in a vector space (Khattab and
Zaharia, 2020; Hofstätter et al., 2020). More specifically, after training a
bi-encoder model, the parameters of the trained model act as ‘embeddings’ for
each document in the collection. During inference time, a query is first
embedded into a vector. Then an approximate nearest neighbour search, e.g.,
HNSW (Malkov and Yashunin, 2020), is conducted on an indexed representation of
these dense document vectors. Therefore, exploring the potential benefits
gained from efficient, dense end-to-end retrieval of training examples for
effective ICL can be an interesting research direction.

#### Cross-Encoder architecture

A cross-encoder instead jointly encodes a query and document at inference time
(Nogueira and Cho, 2019), allowing deep interactions between texts that are
impossible in a bi-encoder architecture. Empirically, these models are more
precise than bi-encoders at the expense of latency, as representations cannot
be pre-computed in a standard setting. Both BERT- and T5-based architectures
have been proposed (Nogueira and Cho, 2019; Nogueira et al., 2020); in the
case of a BERT model, a feed-forward classification head is used to output
class probabilities of relevance (Nogueira and Cho, 2019). In the case of a
sequence-to-sequence model, token logits are taken as surrogates for class
probabilities (Nogueira et al., 2020). Recent developments in LLMs have
prompted research in these large decoder-only models as text rankers. A list-
wise approach is commonly taken in which a model receives multiple documents
for a given query and outputs a permutation of the original ranking (Sun et
al., 2023; Pradeep et al., 2023b). The development of these models is still in
its infancy but it offers opportunities to investigate highly precise ranking
models potentially in sample mining beyond simple ad-hoc search.

Therefore, it is reasonable to assume that employing a cross-encoder to learn
ranking examples by their downstream usefulness should yield better results
than a bi-encoder-based approach. An interesting research direction would
hence be to investigate the optimal architecture within an ICL pipeline,
considering the efficiency-effectiveness trade-off.

#### Teacher Distillation

Moreover, a rich literature exists on distilling the more computationally
expensive cross-encoder models into the simpler bi-encoder, the former acting
as a teacher model and the latter as a student (Hofstätter et al., 2020).
Distilling a teacher model into a bi-encoder one allows provision for end-to-
end dense retrieval without requiring any sparse index to retrieve a candidate
top-$k$. Two core paradigms of distillation are homogeneous architecture and
heterogeneous architecture distillation. The former generally will distill one
model into a newly initialised copy via minimisation of a divergence metric
over either the final hidden state (Lin et al., 2021) or internal states such
as attention layers (Izacard et al., 2023). The latter minimises prediction
error between teacher and student models commonly via a mean squared error
criterion over triplet residuals (residual between positive and negative
example scores), allowing for ‘cross-architecture knowledge distillation’
(Hofstätter et al., 2020) as a scalar relevance score is not architecture
dependent. This approach has become a core component of many state-of-the-art
dense retrieval models, frequently beginning with a cross-encoder teacher used
to mine hard negatives and teacher scores before a second stage distillation
is performed using the previous distilled model as a teacher (Xiao et al.,
2022; Shen et al., 2023). A parallel area of work gaining traction is
knowledge sharing between a retrieval system and a generative model (Lewis et
al., 2020b; Izacard and Grave, 2021; Izacard et al., 2023). This paradigm is
directly correlated with our perspective with recent work finding success in
directly optimising a retriever to maximise downstream QA performance (Izacard
et al., 2023). However, these systems are currently brittle with Cuconasu et
al. (2024) finding that the addition of irrelevant content before a gold label
answer as context to a QA system can improve performance against any
intuition, suggesting much work can be done in this area to optimise how we
present a model with ICL examples.

###  4.1. Combined Utility of ICL Examples

There exists a fundamental difference between relevance of documents in IR and
usefulness of examples in ICL. In IR, a document’s relevance is independent of
the relevance of another document, and when combined, the information still
remains relevant. The situation is more complex for ICL. More precisely
speaking, two labeled instances in ICL that are useful examples by themselves
(i.e., when used as a 1-shot demonstration results in a correct prediction)
may not be yielding a correct prediction when combined for a 2-shot inference
(Lu et al., 2022). This is likely to happen because the decoder, on obtaining
too much of a context, can be biased towards a specific topical cluster of
words corresponding to the incorrect class descriptor.

While more investigation is required to analyse the empirical likelihood of
this phenomenon of ‘non-cooperation’ between examples occurring, it is worth
exploring what adjustments may be needed at a methodology level to even define
an ideal ranking of the training data examples for few-shot ICL. The objective
in this case is not just to maximise the number of ‘relevant documents’ (as
per the IR analogy) within the top-$k$, but rather to ensure the combined
usefulness of the examples. A possible direction towards this could be to
adapt a listwise ranking model with this modified notion of combined relevance
(usefulness).

A more computationally efficient approach would be to operate at the level of
pairs, i.e., predict which pairs are concordant and discordant. An element of
each pair takes on a Boolean value (either useful as a 1-shot example or not),
which means that the number of different ways in which a pair can be either
concordant or discordant is the number of possible Boolean functions of $2$
variables, which is $16$ (two such sample functions are Boolean OR, where if
one of the examples is useful - so is the combination, and XNOR where a pair
is discordant if either of the examples are useful as 1-shot). Since, in the
general case, the number of Boolean functions of $n$ variables is $2^{2^{n}}$,
listwise training with $n>3$ will likely be computationally prohibitive.

#### Open research questions

Before concluding this section, we now summarise the importance of the
following IR-specific research questions for ranking examples in ICL.

  * •

RQ-4.1: Is ICL sensitive to the choice of a neural retrieval model, i.e., can
we get an improvement using a basic Siamese model over SBERT as envisaged in
(Rubin et al., 2022)?

  * •

RQ-4.2: How faithful is the assumption that a combination of several 1-shot
useful examples remain useful for ICL prediction?

  * •

RQ-4.3: If the answer to RQ-4.2 is negative, then there is a significant scope
of improving over standard learning to rank approach by explicitly modeling
concordance (or the lack of it) of the usefulness of examples in ICL. How can
we adjust ranking models, and how much improvement can we achieve over a
baseline of the standard few-shot?

##  5\. Informative Examples $\mapsto$ Faceted IR?

In this section, we discuss the last of our proposed verticals towards an
effective ICL workflow as outlined in Figure 2, which is that of seeking to
provide relevant but diverse contexts to an LLM. More precisely speaking,
topical diversity of the examples should play an important role in preventing
a decoder bias towards a single topic. This is more true for text generation
tasks, such as non-factoid question answering, where an LLM decoder needs to
be aware of the different sub-topics to be able to construct a comprehensive
answer. Even for classification tasks, diverse examples are likely to help a
decoder consider a majority of likely topics (the verbalisers of which map to
descriptors of closely related categories) during inference, thus minimising
the risks of misclassification.

Faceted search has been well studied in IR. Explained simply, a faceted search
system extracts the multiple different aspects of the information need from
the top-retrieved set and maps each retrieved document to one of these aspects
(Clarke et al., 2009; Ganguly and Jones, 2018; Ganguly et al., 2013b, a).
Faceted search is particularly useful for queries with broader information
needs, where it can assist a user to reformulate their information need to one
of the more specific aspects of the broader one, e.g., transform a query
‘dehumidifiers’ to ‘price range of dehumidifiers’ where the intention
(information need facet) is to buy one (Carterette et al., [n. d.]).

Faceted search is closely related to the concept of diversified ranking
(Clarke et al., 2008), where search systems seek to improve the retrieval
effectiveness for all possible aspects of a broader information need, e.g.,
for the earlier example query on ‘dehumidifiers’, retrieve documents related
to information on both the aspects of price ranges, technical specifications,
product reviews, and general knowledge on dehumidifiers. Santos et al. (2010)
propose to leverage query variants (what the paper calls ‘sub-queries’) and
their top-retrieved lists for constructing a list of documents potentially
relevant to each facet of the original query. Close to diversity is the
concept of fair search which seeks to mitigate biases towards any particular
aspects of information need, and recently neural approaches have become common
to balance relevance with fairness (Oosterhuis, 2021).

From a search user’s perspective, it has been shown that diversified retrieval
systems play an important role in improving the search experience, by
providing greater coverage of a topic and mitigating potential bias in search
results (Maxwell et al., 2019). Similarly, a greater topical coverage and a
less topical bias can potentially lead an LLM decoder towards contexts more
useful for a downstream task. In fact, Levy et al. (2023) show that
diversifying the few-shot examples on the basis of abstract syntax tree (AST)
structures improves the downstream task of compositional generalisation. This
indeed shows a positive direction of research where the considerable volume of
work conducted on faceted search and diversification by the IR community can
be useful for ICL.

However, similar to relevance, the notion of diversity would also need
suitable adjustments for ICL. A suitable notion of diversity should not just
consider similarities between the input examples but rather also their class
labels and, more importantly, similarities in the ways in which they affect an
LLM decoder’s generation path. Two examples which both output similar output
trees should not be considered diverse. In principle, one can potentially
adapt the classification methodology that we proposed to learn the optimal
number of examples based on minimising the prediction uncertainties for the
purpose of classifying if a given pair of examples is diverse or not.
Furthermore, we posit that neural approaches that take into account both
relevance and fairness or diversity (again both in the context of downstream
ICL) should find a use-case in ICL to help diversify the useful examples.

#### Open research questions

Based on the discussions in this section, we now outline the following
research directions.

  * •

RQ-5.1: How sensitive is ICL towards the topical diversity of the examples?

  * •

RQ-5.2: How can the standard notion of diversity be extended to consider the
latent dependence between the input and the output of an LLM decoder aligning
towards a specific downstream task?

  * •

RQ-5.3: How may existing IR metrics for diversity (e.g., $\alpha$-nDCG (Clarke
et al., 2008)) be adapted to measure how effective is the example retrieval
for downstream ICL?

  * •

RQ-5.4: How can multi-objective neural ranking models be trained to jointly
learn downstream specific usefulness and diversity for ICL?

##  6\. Preliminary Evaluation

In this section, we report the results of our initial investigation, which was
conducted to answer a subset of research questions of the first vertical,
i.e., to develop an effective adaptive version of ICL that can dynamically
select the number of examples.

###  6.1. Research Questions and Dataset

#### Research Questions Investigated

In Section 3.2, we discussed the possibilities of applying QPP-inspired
unsupervised approaches for selecting a cutoff point in the ranked list of
examples. On the other hand, in Section 3.3 we proposed a classifier-based
approach to learn the optimal number of examples. In our experiments, we
compare the supervised approach of Algorithm 2 and an NQC-based unsupervised
approach for adaptive $k$-shot and compare both with static $k$-shot on
standard datasets for text classification. Explicitly stated, we investigate
the following research question.

  * •

CRQ-1: Does adaptively selecting the number of examples in ICL lead to
improved downstream effectiveness?

  * •

CRQ-2: Does an unsupervised approach obtain a reasonable performance as
compared to a supervised one?

Since our experiments answer the above questions, they are not open, unlike
the ones we expounded on in this paper. Therefore, we prefix these questions
with a ‘C’ (closed).

#### Dataset

We conduct experiments on three text classification datasets, namely AGNews
(Del Corso et al., 2005), Jigsaw Toxic
Comment111https://www.kaggle.com/c/jigsaw-toxic-comment-classification-
challenge and SST2 (Socher et al., 2013). Below, we provide more details on
each dataset.

  * •

AGNews: AGNews is a topic classification dataset constituting news articles
from the Web. Each document in the dataset belongs to one of the following 4
classes: World, Sports, Business, and Sci/Tech. The total number of training
instances is $120,000$, while the test set size is $7,600$. Each class
contains $30,000$ samples from the train set and $1,900$ instances from the
test set.

  * •

Jigsaw Toxic Comments: Due to its societal impact, toxicity prediction is a
problem of considerable practical interest. This dataset, released by Jigsaw
and Google as a part of a Kaggle competition, comprises of comments extracted
from Wikipedia’s talk page, each being annotated by human evaluators across
six categories representing toxic behaviors: toxic, ‘severe toxic’, obscene,
threat, insult, and ‘identity hate’.

  * •

SST2: The Stanford Sentiment Treebank (SST) is a corpus with fully labeled
parse trees that allows for a complete analysis of the compositional effects
of sentiment in a language. The corpus consists of $11,855$ sentences
extracted from movie reviews. Being parsed with the Stanford parser, it
constitutes a total of $215,154$ unique phrases from the parse trees, each
annotated by 3 human judges. The SST2 (also called SST-binary) dataset is a
subset of SST, specifically prepared for the task of binary classification.
More precisely, neutral sentences from SST were discarded, and two level, each
for the negative and the positive classes were merged thus yielding two
classes in total.

###  6.2. Methods and Parameters

#### Our proposed methods for Adaptive ICL (AICL)

As novel methods for adaptive ICL, we employ the following:

  * •

The supervised strategy of Algorithm 2, which we call supervised adaptive ICL
(SAICL).

  * •

A QPP-based unsupervised strategy (as per the generic direction outlined in
Section 3.2), where we compute the rank cutoff in a relatively simple way,
stated as follows. First, given a top-$M$ set of candidate examples, we
compute a normalised value of the NQC estimator (Shtok et al., 2012) (we
employ a max normalisation, the normalisation constant being the max NQC value
from the training set). We then quantise the normalised values into $M$ equi-
spaced intervals ranging from $0$ to the max NQC value. As per the hypothesis
that a higher NQC value indicates a better retrieval quality, we employ the
inverse linear relation and end up selecting a value close to $0$ for higher
NQC, and a value close to $M$ for smaller ones. We call this method QPP-AICL.

#### Baselines

As baselines to compare SAICL and QPP-AICL against, we employ the following:

  * •

0-shot: This approach simply inputs an instruction without supplying any
examples.

  * •

Static ICL (SICL): This refers to the standard method of supplying a fixed
number of semantically-similar examples as input, similar to (Liu et al.,
2022). This is different from AICL in that the number of examples in the
prompt is always fixed, however, the examples themselves vary for different
test inputs based on semantic similarity. For a fair comparison with AICL
methods, we report the results obtained with three different values of $k$:
$1$, $\lceil\frac{M}{2}\rceil$ and $M$ representing the most conservative (in
terms of the input size), average, and least conservative situations. In our
case, $M=5$, which means that our standard ICL experiments operate with the
1-shot, 3-shot and 5-shot settings.

#### Model and hyper-parameter settings

Among a relatively large number of available choices for available LLMs –
either open-source models or black-box cloud APIs – we, in particular, conduct
our experiments on GPT-J (Su et al., 2023). GPT-J is an open-source GPT-3-like
model trained on the Pile dataset (Gao et al., 2020). GPT-J-6B yields
performance comparable to the 6.7 billion parameter GPT-3 (Curie) on a variety
of tasks (Wang, 2021). The maximum context length (in terms of number of
tokens) of GPT-J is 2048.

In our experiments, we vary $M$ \- the maximum number of examples, from 1 to 5
(for static ICL this is denoted by $k$). For a fair comparison, we use the
identical prompt template (as shown in Algorithm 1) and greedy decoding with
the same verbalizers across all methods employed in our experiments.

###  6.3. Results

Table 1. Macro-averaged precision, recall and F1-scores for different in-
context learning (ICL) methodologies. The column $k$ denotes the number of
few-shot examples. For AICL approaches, this column denotes the average number
of examples used for the respective method. ‘AIS’ denotes the average input
size measured in terms of the number of tokens rounded off to the nearest
integer.

Evaluation Dataset Method $k$ Precision Recall F-score AIS AGNews 0-shot 0
0.6569 0.5932 0.5849 60 SICL 1 0.9015 0.9017 0.9016 125 SICL 3 0.9008 0.8997
0.8989 252 SICL 5 0.8963 0.8930 0.8917 380 QPP-AICL 3 0.8545 0.8499 0.8486 220
SAICL 1.87 0.9080 0.9096 0.9067 175 Toxicity 0-shot 0 0.5689 0.6238 0.5769 103
SICL 1 0.5760 0.6989 0.5505 195 SICL 3 0.6092 0.7180 0.6254 335 SICL 5 0.6078
0.7248 0.6217 431 QPP-AICL 3 0.5906 0.6942 0.5977 289 SAICL 3.46 0.6194 0.6983
0.6303 359 SST2 0-shot 0 0.7503 0.5022 0.3379 30 SICL 1 0.8703 0.8703 0.8703
61 SICL 3 0.9140 0.9137 0.9137 121 SICL 5 0.9245 0.9230 0.9230 181 QPP-AICL 3
0.8556 0.8479 0.8470 106 SAICL 4.12 0.9302 0.9304 0.9302 154

Table 1 shows the results (in terms of macro-averaged precision, recall and
F1) obtained by the different ICL strategies. It can be seen that SAICL turns
out to be the best among the competing approaches. The reason it outperforms
the best baseline (static ICL) is that SAICL is able to effectively adapt the
number of examples to use, thereby preventing itself from the degradation
effects of non-relevant (not useful) examples. In effect, it learns a latent
relationship between the topical content and the quantity of context required
to guide the decoder’s output in the right direction effectively. Moreover,
SAICL is able to operate more effectively with smaller input sizes (see the
average value of $k$ and also the average size of the input in terms of the
number of tokens), which means that it is computationally faster as compared
to static ICL (SICL). Our observations reveal that CRQ-1 is answered in the
affirmative, i.e., an adaptive selection of the number of examples in ICL does
improve downstream effectiveness and efficiency.

The results with the unsupervised QPP-based approach (QPP-AICL) turned out to
be worse than the baseline of static ICL. From a broader perspective, this
points to an important finding - that off-the-shelf IR approaches without
modifications specifically suited to the underlying characteristics of the
downstream tasks in ICL may not directly yield improvements in the
effectiveness of ICL. For instance, NQC seeks to estimate relevance of
documents, and as we have argued before, that relevance has a different
interpretation for the ICL examples. Although the observations with QPP-AICL
answers CRQ-2 in negative, i.e., an unsupervised approach for an adaptive
selection of ICL examples is substantially worse than a supervised one, they
do suggest that methodologies developed by researchers in the future for
answering any of the open research questions discussed in this paper should be
fundamentally grounded in modeling the notion of relevance (usefulness of
examples) in a robust and effective manner.

##  7\. Conclusion

In this perspective paper, we discuss how some of the recent developments in
generative AI (specifically in-context learning or ICL) can provide a scope to
IR/NLP researchers to revisit some of the well-researched IR topics in a new
light, where the notion of relevance of a document to an information need
changes to that of usefulness of a few-shot example for a downstream AI task,
e.g., text classification, question answering etc. More specifically, we
suggest three main verticals in which this research can be structured - each
offering a set of open questions related to core IR research.

The first vertical aims at adaptively adjusting an ICL workflow, e.g.,
choosing the number of examples to be used in a data-driven manner. Initial
empirical investigations reported in this perspective paper shows that this
direction is promising. The second vertical mainly covers devising novel
ranking models to better distinguish (and thereby retrieve at better ranks) a
useful few-shot context from a noisy one. Finally, the third vertical concerns
an investigation of topical diversity in the few-shot examples for better
downstream prediction.

We believe that the research questions that we have proposed in this paper
will benefit the research community to exploit this synergy between ICL and
IR, and eventually guide the development of new algorithms and techniques.

## References

  * (1)
  * Alaofi et al. (2023) Marwah Alaofi, Luke Gallagher, Mark Sanderson, Falk Scholer, and Paul Thomas. 2023.  Can Generative LLMs Create Query Variants for Test Collections? An Exploratory Study. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Taipei, Taiwan) _(SIGIR ’23)_. Association for Computing Machinery, New York, NY, USA, 1869–1873.  https://doi.org/10.1145/3539618.3591960
  * Arampatzis et al. (2009) Avi Arampatzis, Jaap Kamps, and Stephen Robertson. 2009.  Where to Stop Reading a Ranked List? Threshold Optimization Using Truncated Score Distributions. In _Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Boston, MA, USA) _(SIGIR ’09)_. Association for Computing Machinery, New York, NY, USA, 524–531.  https://doi.org/10.1145/1571941.1572031
  * Arora et al. (2022) Simran Arora, Avanika Narayan, Mayee F. Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022.  Ask Me Anything: A simple strategy for prompting language models.  arXiv:2210.02441 [cs.CL]
  * Bahri et al. (2020) Dara Bahri, Yi Tay, Che Zheng, Donald Metzler, and Andrew Tomkins. 2020.  Choppy: Cut Transformer for Ranked List Truncation. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, China) _(SIGIR ’20)_. Association for Computing Machinery, New York, NY, USA, 1513–1516.  https://doi.org/10.1145/3397271.3401188
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Carterette et al. ([n. d.]) Ben Carterette, Evangelos Kanoulas, Mark M. Hall, and Paul D. Clough. [n. d.].  Overview of the TREC 2014 Session Track. In _Proc. of TREC 2014_.
  * Chakraborty et al. (2020) Anirban Chakraborty, Debasis Ganguly, and Owen Conlan. 2020.  Retrievability based Document Selection for Relevance Feedback with Automatically Generated Query Variants. In _CIKM_. ACM, 125–134.
  * Clarke et al. (2008) Charles L.A. Clarke, Maheedhar Kolla, Gordon V. Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, and Ian MacKinnon. 2008.  Novelty and diversity in information retrieval evaluation. In _Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Singapore, Singapore) _(SIGIR ’08)_. Association for Computing Machinery, New York, NY, USA, 659–666.  https://doi.org/10.1145/1390334.1390446
  * Clarke et al. (2009) Charles L. A. Clarke, Nick Craswell, and Ian Soboroff. 2009.  Overview of the TREC 2009 Web Track. In _Proceedings of The Eighteenth Text REtrieval Conference, TREC 2009, Gaithersburg, Maryland, USA, November 17-20, 2009_ _(NIST Special Publication, Vol. 500-278)_ , Ellen M. Voorhees and Lori P. Buckland (Eds.). National Institute of Standards and Technology (NIST).  http://trec.nist.gov/pubs/trec18/papers/WEB09.OVERVIEW.pdf
  * Cronen-Townsend et al. (2002) Steve Cronen-Townsend, Yun Zhou, and W. Bruce Croft. 2002.  Predicting Query Performance. In _Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’02)_. Association for Computing Machinery, New York, NY, USA, 299–306.
  * Cuconasu et al. (2024) Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024.  The Power of Noise: Redefining Retrieval for RAG Systems.  arXiv:2401.14887 [cs.IR]
  * Cummins (2014) Ronan Cummins. 2014.  Document Score Distribution Models for Query Performance Inference and Prediction.  _ACM Trans. Inf. Syst._ 32, 1, Article 2 (2014), 28 pages.
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper Text Understanding for IR with Contextual Neural Language Modeling. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Paris, France) _(SIGIR’19)_. Association for Computing Machinery, New York, NY, USA, 985–988.  https://doi.org/10.1145/3331184.3331303
  * Datta et al. (2022) Suchana Datta, Debasis Ganguly, Derek Greene, and Mandar Mitra. 2022.  Deep-QPP: A Pairwise Interaction-based Deep Learning Model for Supervised Query Performance Prediction. In _WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ, USA, February 21 - 25, 2022_ , K. Selcuk Candan, Huan Liu, Leman Akoglu, Xin Luna Dong, and Jiliang Tang (Eds.). ACM, 201–209.  https://doi.org/10.1145/3488560.3498491
  * Datta et al. (2023) Suchana Datta, Debasis Ganguly, Mandar Mitra, and Derek Greene. 2023.  A Relative Information Gain-based Query Performance Prediction Framework with Generated Query Variants.  _ACM Trans. Inf. Syst._ 41, 2 (2023), 38:1–38:31.
  * Del Corso et al. (2005) Gianna M. Del Corso, Antonio Gullí, and Francesco Romani. 2005.  Ranking a Stream of News. In _Proceedings of the 14th International Conference on World Wide Web_ (Chiba, Japan) _(WWW ’05)_. Association for Computing Machinery, New York, NY, USA, 97–106.  https://doi.org/10.1145/1060745.1060764
  * Devlin et al. (2019a) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.
  * Devlin et al. (2019b) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Diaz (2007) Fernando Diaz. 2007.  Performance Prediction Using Spatial Autocorrelation. In _Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’07)_. Association for Computing Machinery, New York, NY, USA, 583–590.
  * Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023.  A Survey on In-context Learning.  arXiv:2301.00234 [cs.CL]
  * Faggioli et al. (2023) Guglielmo Faggioli, Nicola Ferro, Cristina Muntean, Raffaele Perego, and Nicola Tonellotto. 2023.  A Geometric Framework for Query Performance Prediction in Conversational Search. In _Proceedings of 46th international ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2023 July 23–27, 2023, Taipei, Taiwan_. ACM.  https://doi.org/10.1145/3539618.3591625
  * Ganguly et al. (2013a) Debasis Ganguly, Manisha Ganguly, Johannes Leveling, and Gareth J. F. Jones. 2013a.  TopicVis: a GUI for topic-based feedback and navigation. In _SIGIR_. ACM, 1103–1104.
  * Ganguly and Jones (2018) Debasis Ganguly and Gareth J. F. Jones. 2018.  A non-parametric topical relevance model.  _Inf. Retr. J._ 21, 5 (2018), 449–479.
  * Ganguly et al. (2013b) Debasis Ganguly, Johannes Leveling, and Gareth J. F. Jones. 2013b.  An LDA-smoothed relevance model for document expansion: a case study for spoken document retrieval. In _SIGIR_. ACM, 1057–1060.
  * Ganguly and Yilmaz (2023) Debasis Ganguly and Emine Yilmaz. 2023.  Query-specific Variable Depth Pooling via Query Performance Prediction. In _SIGIR_. ACM, 2303–2307.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020.  The Pile: An 800GB Dataset of Diverse Text for Language Modeling.  _arXiv preprint arXiv:2101.00027_ (2020).
  * Gao et al. (2021a) Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021a.  Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipeline.  _CoRR_ abs/2101.08751 (2021).  arXiv:2101.08751 https://arxiv.org/abs/2101.08751
  * Gao et al. (2023) Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023.  Precise Zero-Shot Dense Retrieval without Relevance Labels. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 1762–1777.  https://doi.org/10.18653/V1/2023.ACL-LONG.99
  * Gao et al. (2021b) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b.  SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 6894–6910.  https://doi.org/10.18653/V1/2021.EMNLP-MAIN.552
  * Gutmann and Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. 2010.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ _(Proceedings of Machine Learning Research, Vol. 9)_ , Yee Whye Teh and Mike Titterington (Eds.). PMLR, Chia Laguna Resort, Sardinia, Italy, 297–304.  https://proceedings.mlr.press/v9/gutmann10a.html
  * Hofstätter et al. (2020) Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020.  Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation.  _CoRR_ abs/2010.02666 (2020).  arXiv:2010.02666 https://arxiv.org/abs/2010.02666
  * Hu et al. (2021) Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Maosong Sun. 2021.  Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification.  _arXiv preprint arXiv:2108.02035_ (2021).
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021_ , Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, 874–880.  https://doi.org/10.18653/V1/2021.EACL-MAIN.74
  * Izacard et al. (2023) Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2023.  Atlas: Few-shot Learning with Retrieval Augmented Language Models.  _J. Mach. Learn. Res._ 24 (2023), 251:1–251:43.  http://jmlr.org/papers/v24/23-0037.html
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7B.  arXiv:2310.06825 [cs.CL]
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 6769–6781.  https://doi.org/10.18653/V1/2020.EMNLP-MAIN.550
  * Khattab and Zaharia (2020) Omar Khattab and Matei Zaharia. 2020.  ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.). ACM, 39–48.  https://doi.org/10.1145/3397271.3401075
  * Levy et al. (2023) Itay Levy, Ben Bogin, and Jonathan Berant. 2023.  Diverse Demonstrations Improve In-context Compositional Generalization.  arXiv:2212.06800 [cs.CL]
  * Lewis et al. (2020a) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Online, 7871–7880.  https://doi.org/10.18653/v1/2020.acl-main.703
  * Lewis et al. (2020b) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
  * Li et al. (2022) Minghan Li, Xueguang Ma, and Jimmy Lin. 2022.  An Encoder Attribution Analysis for Dense Passage Retriever in Open-Domain Question Answering. In _Proceedings of the 2nd Workshop on Trustworthy Natural Language Processing (TrustNLP 2022)_. Association for Computational Linguistics, Seattle, U.S.A., 1–11.  https://doi.org/10.18653/v1/2022.trustnlp-1.1
  * Li et al. (2023) Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, and Wenhu Chen. 2023.  Few-shot In-context Learning on Knowledge Base Question Answering. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 6966–6980.  https://doi.org/10.18653/v1/2023.acl-long.385
  * Lin et al. (2021) Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021.  In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval. In _Proceedings of the 6th Workshop on Representation Learning for NLP, RepL4NLP@ACL-IJCNLP 2021, Online, August 6, 2021_ , Anna Rogers, Iacer Calixto, Ivan Vulic, Naomi Saphra, Nora Kassner, Oana-Maria Camburu, Trapit Bansal, and Vered Shwartz (Eds.). Association for Computational Linguistics, 163–173.  https://doi.org/10.18653/V1/2021.REPL4NLP-1.17
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022.  What Makes Good In-Context Examples for GPT-3?. In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , Eneko Agirre, Marianna Apidianaki, and Ivan Vulić (Eds.). Association for Computational Linguistics, Dublin, Ireland and Online, 100–114.  https://doi.org/10.18653/v1/2022.deelio-1.10
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  http://arxiv.org/abs/1907.11692
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022.  Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 8086–8098.  https://doi.org/10.18653/v1/2022.acl-long.556
  * Luo et al. (2024) Man Luo, Xin Xu, Yue Liu, Panupong Pasupat, and Mehran Kazemi. 2024.  In-context Learning with Retrieved Demonstrations for Language Models: A Survey.  arXiv:2401.11624 [cs.CL]
  * MacAvaney et al. (2019) Sean MacAvaney, Andrew Yates, Arman Cohan, and Nazli Goharian. 2019.  CEDR: Contextualized Embeddings for Document Ranking. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_ , Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 1101–1104.  https://doi.org/10.1145/3331184.3331317
  * Mahdi et al. (2020) Mohammed Najah Mahdi, Abdul Rahim Ahmad, Roslan Ismail, Hayder Natiq, and Mohammed Abdulameer Mohammed. 2020.  Solution for Information Overload Using Faceted Search–A Review.  _IEEE Access_ 8 (2020), 119554–119585.  https://doi.org/10.1109/ACCESS.2020.3005536
  * Malkov and Yashunin (2020) Yury A. Malkov and Dmitry A. Yashunin. 2020.  Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.  _IEEE Trans. Pattern Anal. Mach. Intell._ 42, 4 (2020), 824–836.  https://doi.org/10.1109/TPAMI.2018.2889473
  * Maxwell et al. (2019) David Maxwell, Leif Azzopardi, and Yashar Moshfeghi. 2019.  The impact of result diversification on search behaviour and performance.  _Inf. Retr. J._ 22, 5 (2019), 422–446.
  * Milios et al. (2023) Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. 2023.  In-Context Learning for Text Classification with Many Labels. In _Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP_ , Dieuwke Hupkes, Verna Dankers, Khuyagbaatar Batsuren, Koustuv Sinha, Amirhossein Kazemnejad, Christos Christodoulopoulos, Ryan Cotterell, and Elia Bruni (Eds.). Association for Computational Linguistics, Singapore, 173–184.  https://doi.org/10.18653/v1/2023.genbench-1.14
  * Mysore et al. (2023) Sheshera Mysore, Andrew McCallum, and Hamed Zamani. 2023.  Large Language Model Augmented Narrative Driven Recommendations.  arXiv:2306.02250 [cs.IR]
  * Ni et al. (2021) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernández Ábrego, Ji Ma, Vincent Y. Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. 2021.  Large Dual Encoders Are Generalizable Retrievers.  arXiv:2112.07899 [cs.IR]
  * Nogueira and Cho (2019) Rodrigo Frassetto Nogueira and Kyunghyun Cho. 2019.  Passage Re-ranking with BERT.  _CoRR_ abs/1901.04085 (2019).  arXiv:1901.04085 http://arxiv.org/abs/1901.04085
  * Nogueira et al. (2020) Rodrigo Frassetto Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020.  Document Ranking with a Pretrained Sequence-to-Sequence Model. In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ _(Findings of ACL, Vol. EMNLP 2020)_ , Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 708–718.  https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.63
  * Oosterhuis (2021) Harrie Oosterhuis. 2021.  Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (, Virtual Event, Canada,) _(SIGIR ’21)_. Association for Computing Machinery, New York, NY, USA, 1023–1032.  https://doi.org/10.1145/3404835.3462830
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems_ , S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 27730–27744.  https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf
  * Paulsen and Raghupathi (2016) Vern I. Paulsen and Mrinal Raghupathi. 2016.  _An Introduction to the Theory of Reproducing Kernel Hilbert Spaces_.  Cambridge University Press.
  * Pradeep et al. (2023a) Ronak Pradeep, Kai Hui, Jai Gupta, Adam D. Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, and Vinh Q. Tran. 2023a.  How Does Generative Retrieval Scale to Millions of Passages?  arXiv:2305.11841 [cs.IR]
  * Pradeep et al. (2023b) Ronak Pradeep, Sahel Sharifymoghaddam, and Jimmy Lin. 2023b.  RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!  _CoRR_ abs/2312.02724 (2023).  https://doi.org/10.48550/ARXIV.2312.02724 arXiv:2312.02724
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI blog_ 1, 8 (2019), 9.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  arXiv:1908.10084 [cs.CL]
  * Roitman et al. (2020) Haggai Roitman, Yosi Mass, Guy Feigenblat, and Roee Shraga. 2020.  Query Performance Prediction for Multifield Document Retrieval. In _Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Information Retrieval_ (Virtual Event, Norway) _(ICTIR ’20)_. Association for Computing Machinery, New York, NY, USA, 49–52.  https://doi.org/10.1145/3409256.3409821
  * Roy et al. (2019) Dwaipayan Roy, Debasis Ganguly, Mandar Mitra, and Gareth J.F. Jones. 2019.  Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction.  _Information Processing and Management_ 56, 3 (2019), 1026 – 1045.
  * Rubin et al. (2022) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022.  Learning To Retrieve Prompts for In-Context Learning. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (Eds.). Association for Computational Linguistics, Seattle, United States, 2655–2671.  https://doi.org/10.18653/v1/2022.naacl-main.191
  * Santos et al. (2010) Rodrygo L.T. Santos, Craig Macdonald, and Iadh Ounis. 2010.  Exploiting query reformulations for web search result diversification. In _Proceedings of the 19th International Conference on World Wide Web_ (Raleigh, North Carolina, USA) _(WWW ’10)_. Association for Computing Machinery, New York, NY, USA, 881–890.  https://doi.org/10.1145/1772690.1772780
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 255–269.  https://doi.org/10.18653/v1/2021.eacl-main.20
  * Sen et al. (2022) Procheta Sen, Sourav Saha, Debasis Ganguly, Manisha Verma, and Dwaipayan Roy. 2022.  Measuring and Comparing the Consistency of IR Models for Query Pairs with Similar and Different Information Needs. In _CIKM_. ACM, 4449–4453.
  * Shen et al. (2023) Tao Shen, Xiubo Geng, Chongyang Tao, Can Xu, Xiaolong Huang, Binxing Jiao, Linjun Yang, and Daxin Jiang. 2023.  LexMAE: Lexicon-Bottlenecked Pretraining for Large-Scale Retrieval. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=PfpEtB3-csK
  * Shtok et al. (2012) Anna Shtok, Oren Kurland, David Carmel, Fiana Raiber, and Gad Markovits. 2012.  Predicting Query Performance by Query-Drift Estimation.  _ACM Trans. Inf. Syst._ 30, 2, Article 11 (2012), 35 pages.
  * Singh et al. (2023) Ashutosh Singh, Debasis Ganguly, Suchana Datta, and Craig MacDonald. 2023.  Unsupervised Query Performance Prediction for Neural Models with Pairwise Rank Preferences. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023_ , Hsin-Hsi Chen, Wei-Jou (Edward) Duh, Hen-Hsen Huang, Makoto P. Kato, Josiane Mothe, and Barbara Poblete (Eds.). ACM, 2486–2490.  https://doi.org/10.1145/3539618.3592082
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Seattle, Washington, USA, 1631–1642.  https://aclanthology.org/D13-1170
  * Sorensen et al. (2022) Taylor Sorensen, Joshua Robinson, Christopher Rytting, Alexander Shaw, Kyle Rogers, Alexia Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022.  An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, 819–862.  https://doi.org/10.18653/v1/2022.acl-long.60
  * Su et al. (2023) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023.  RoFormer: Enhanced Transformer with Rotary Position Embedding.  arXiv:2104.09864 [cs.CL]
  * Sun et al. (2023) Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. 2023.  Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 14918–14937.  https://aclanthology.org/2023.emnlp-main.923
  * Tang et al. (2023) Yuting Tang, Ratish Puduppully, Zhengyuan Liu, and Nancy Chen. 2023.  In-context Learning of Large Language Models for Controlled Dialogue Summarization: A Holistic Benchmark and Empirical Analysis. In _Proceedings of the 4th New Frontiers in Summarization Workshop_ , Yue Dong, Wen Xiao, Lu Wang, Fei Liu, and Giuseppe Carenini (Eds.). Association for Computational Linguistics, Singapore, 56–67.  https://doi.org/10.18653/v1/2023.newsum-1.6
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  arXiv:2307.09288 [cs.CL]
  * Upadhyay et al. (2020) Prajna Upadhyay, Srikanta Bedathur, Tanmoy Chakraborty, and Maya Ramanath. 2020.  Aspect-based academic search using domain-specific KB. In _Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II 42_. Springer, 418–424.
  * Wang (2021) Ben Wang. 2021.  Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang and Komatsuzaki (2022) Ben Wang and Aran Komatsuzaki. 2022.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, 2021.
  * Wang et al. (2023) Liang Wang, Nan Yang, and Furu Wei. 2023.  Query2doc: Query Expansion with Large Language Models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 9414–9423.  https://doi.org/10.18653/v1/2023.emnlp-main.585
  * Xiao et al. (2022) Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022.  RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 538–548.  https://doi.org/10.18653/V1/2022.EMNLP-MAIN.35
  * Xiong et al. (2021) Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021.  Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.  https://openreview.net/forum?id=zeFrfgyZln
  * Zendel et al. (2019) Oleg Zendel, Anna Shtok, Fiana Raiber, Oren Kurland, and J. Shane Culpepper. 2019.  Information Needs, Queries, and Query Performance Prediction. In _Proc. of SIGIR ’19_. Association for Computing Machinery, New York, NY, USA, 395–404.
  * Zhang et al. (2017) Shichao Zhang, Xuelong Li, Ming Zong, Xiaofeng Zhu, and Debo Cheng. 2017.  Learning k for KNN Classification.  8, 3, Article 43 (jan 2017), 19 pages.  https://doi.org/10.1145/2990508
  * Zhou and Croft (2007) Yun Zhou and W. Bruce Croft. 2007.  Query Performance Prediction in Web Search Environments. In _Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’07)_. Association for Computing Machinery, New York, NY, USA, 543–550.