# OLD:Visual In-context LearningNEW:Towards Global Optimal Visual In-Context
Learning Prompt Selection

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh, PA 15213
<EMAIL_ADDRESS>
Use footnote for providing further information about author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

OLD:With the rise of foundation models, Visual In-Context Learning (VICL) as
one way to utilize foundation model becomes prevailing. This paper focuses on
VICL, which leverages contextual information contained in in-context examples
to enhance learning and prediction of query sample. Specifically, we aim to
improve the selection of in-context examples, i.e. choosing the best
performing in-context examples from alternatives for each query sample. In
order to fully explore the alternative set, we propose a novel pipeline
comprising a multi-sample ranking model and a consistency-aware aggregation
strategy. The ranking model, based on a transformer, ranks multiple
alternative samples, providing a more comprehensive understanding of the
relationships between alternatives and the query sample. Based on this model,
the consistent-aware aggregation strategy is further proposed based on Hodge
theory, which aggregates partial ranking predictions via solving a least
square problem and thus generating globally consistent ranking. The
effectiveness of the proposed method is validated through experiments on
foreground segmentation, single object detection and image colorization. The
results demonstrate that the proposed method consistently provides better in-
context examples, leading to new state-of-the-art performance.  NEW: Visual
In-Context Learning (VICL) is a prevailing way to transfer visual foundation
models to new tasks. The fundamental problem in VICL is how to select the best
prompt to activate its power as much as possible. In this paper, we propose a
statistical sample selection method to approximately identify the global
optimal prompt, i.e. choosing the best performing in-context examples from all
alternatives for each query sample. This sample selection problem is
equivalent to the ranking problem to test the in-context behaviour of each
candidates in the alternative set and select the best one. However, given the
huge sample size of alternative set, it is infeasible to directly compare the
candidate prompt with all other alternatives. To solve this issue, we propose
to first train several partial ranking models that learns to rank between a
small subset, and then recover the global ranking from these local
observations. Our method, dubbed Partial2Global, adopts a transformer-based
partial rankers to provide a more comprehensive comparison within several
alternatives, and a global approximator inspired by the statistical Hodge
theory to generate globally consistent ranking. The effectiveness of
Partial2Global is validated through experiments on foreground segmentation,
single object detection and image colorization, demonstrating that
Partial2Global selects consistently better in-context examples compared with
other methods, and thus establish new state-of-the-arts.

##  1 Introduction

[yw: we shall mentioned SD or foundation model, for a good motivation here. ]
Currently, foundation models contribute to the great success of AIGC which has
many powerful applications such as GPT, Gemini and e.t.c. To better make use
of the capcity of foundation models, different finetuning or prompting
paradigms are proposed, Visual In-Context Learning is a prevailing one among
them. Visual In-Context Learning (VICL) has emerged as a pivotal area of study
within the context of computer vision NEW:for adapting Visual Foundation
Models (VFM). This approach leverages contextual information to enhance the
learning process, thereby enabling models to make more accurate predictions
based on visual cues. OLD:The fundamental premise of VICL is the integration
of in-context examples into the inference process of each query sample, which
provides a richer and more comprehensive understanding of both the target task
and data type. NEW:Basically, VICL introduces prompts, termed as in-context
example (including image-label pair), to assist the VM to predict the label of
the testing query image. Such methods mimic the humans ability to understand
and OLD:copyNEW:mimic the guidance. NEW:[highlight what in-context learning
can do? for example fast adaptation to new domains, etc. current version looks
like the benefit of foundation model, instead of ICL.]VICL models can well
serve wide-ranging applications across various domains, for instance, aiding
recognition of traffic signs and road lanes by considering complex scene
information, and assisting medical diagnosis by analyzing history and related
patient cases.

[We need a highlight sentence like " Importantly, samples are of central
important to ICL, by ranking the goodness of samples" ]. [yw: we need a teaser
figure here to show that good V.S. bad examples for ICL means quite
different.] OLD: Selecting suitable samples is the key to VICL. Since the ICL
models highly depend on the information contained in the in-context examples
as in Fig. LABEL:xx, randomly selecting them could lead to undesirable
performance. Visual Prompt Retrieval (VPR) zhang2024vpr first got involved
into such a problem for VICL and built a straightforward method via
contrastive learning based on the simple heuristic about visual similarity. By
adopting contrastive learning, VPR can be actually taken as a special
instantiation of ranking model, which learns a metric regarding the in-context
performance and fetches the global ranking of all alternative samples via
aggregating and sorting the predicted metrics for each alternative, from which
the in-context example is selected by picking the best ranking one.  NEW:
Given the VFM, the core of VICL is how to select the best in-context example
to activate the power of VFM and lead to best performance on the query
example. Empirically, a randomly selected in-context example may cause
undesirable performance, while a carefully-selected example can largely
improve the performance, as in Fig. xx (maybe a comparison between baselines
and our method). In VICL, we usually have a fixed alternative set (for example
the training dataset), and the in-context sample selection problem can be
naturally transformed into a ranking problem. Ideally, for a given query
image, we could run the VICL testing with each candidate prompt, rank the
testing performance, and then identify the best in-context example for this
query image. However, this post-hoc approach is obviously infeasible. The
target of VICL is thus approximating such ranking progress without running
direct testing.

OLD:[However, there are quite many challenges of ranking the data for ICL.
Essentially,] how to utilize the query data more efficiently is among them.
Since such rankings are unavailable for testing data, the ranking observations
from training set have to be utilized so that the model can be generalized to
rank the test set. Unfortunately, while the contrastive objective helps VPR
guarantee the consistency of ranking prediction by simply arranging the order
of similarity between each alternative and the query, it intrinsically limits
the learning capacity due to the design nature of it objective function. To
perform contrastive learning, VPR has to gather the most representative
positive and negative samples for each query, which are implemented the best
and worst performing in-context examples. Consequently, large amount of
ranking observations remain unused during training. Moreover, VPR can only
individually model the alternative-query similarity, which in turns makes it
unaware of the inner relationship between different alternatives. NEW: There
exist several core challenges for selecting the best prompt for VICL. (1) The
ranking metric: The straightforward metric and the eventual target is the
testing performance of the testing performance on the query image given the
condition of in-context example. However, as we want to avoid the directing
testing, we need to find some approximate score to indirectly measure the
testing performance. [I thought I was informed with this conclusion]Previous
works [some cite] demonstrate that visual similarity is positively correlated
with the conditional testing performance. However, we empirically find that
this observation is not always correct (see sec.xxx for detailed discussion).
(2) The comparison set: Given the ranking metric, the optimal comparison is
between the example and all other candidates in the alternative set. However,
empirically the alternative set usually contains millions of examples, making
the global comparison infeasible. Previous approaches tries heuristic rules
such as pre-processing clustering algorithms to reduce the alternative set
size and mini-batch-based comparison to only learn the partial ranking.
However, such methods potentially limit the capacity of VICL due to the lack
of global ranking comparison. The trade-off between feasibility and capacity
is the insurmountable obstacles.

OLD:To solve these problems, we propose a novel pipeline by redesigning the
ranking procedure. Our pipeline includes a multi-sample ranking model and a
consistency-aware aggregation strategy. Concretely, we first train a
transformer-based model to rank multiple alternative samples. This model takes
features of several alternatives together with the query sample from
pretrained foundation models as input. Then by merging information among class
tokens regarding each sample, such a model infers the ranking with more
comprehensive knowledge from multiple alternatives and query sample, thus
being more informative and performing better. On the other hand, since the
proposed ranking model works by ranking partial alternatives at each time,
aggregating these partial predictions into a consistent global prediction is
not trivial. As a conutermeasure, we propose a consistency-aware aggregation
strategy based on Hodge theory. Specifically, partial predictions from the
trained ranking model are collected and divided into several groups, each of
which represents ranking annotations from an individual. After that, we can
reorganize them via solving a least square problem between the global ranking
prediction and all ranking groups. In this way we can not only fully utilize
the comprehensive information contained in the alternative set but also
rectify the informative but inconsistent componential predictions into a
globally consistent one, thus making the best of both worlds and better
boosting the in-context learning performance.

NEW:To solve these problems, we in this paper propose a systematic VICL sample
selection framework, dubbed Partial2Global, towards the global optimal in-
context examples selection. (1) To approximate the ground-truth ranking
metric, we propose a meta-learning based approach to directly train the
ranking model based on the in-context testing performance. Specifically, the
training dataset is processed in the meta-learning manner, sampling a task in
each time to simulate the testing scenarios. In this task, we have access to a
query example, some candidate examples in the alternative set, and the in-
context testing performance which we could utilize as the ranking score
target. This meta-learned ranking model enjoys better transferable ability for
novel query examples. (2) To approximate the global ranking, we propose a
statistical ranking method to strike the balance between feasibility and
ranking capacity. Particularly, we first learn several partial ranking models
that could rank a small subset for a given query example. Given the imprecise,
partial-observed, and noisy ranking models, we recover the global ranking
relationship by the proposed aggregation model. Inspired by the Hodge theory,
our recovered global ranking relationship enjoys better ranking consistency
and thus provides better in-context examples to boost VICL performance. (3) We
propose a transformer-based partial ranking model to rank multiple alternative
samples in a more informative and comprehensive manner.

In order to validate the effectiveness of our method, we conduct experiments
on three different tasks including foreground segmentation, single object
detection and image colorization with datastet including Pascal VOC and
ImageNet, following the previous works. Extensive results show that
Partial2Global not only can provide consistently better in-context examples,
which results in new state-of-the-art performance, but also shed light on the
selection heuristics of in-context examples, facilitating further research on
visual in-context learning.

Contributions of this work are as follows:

  * •

OLD:We propose a novel method for selecting examples in visual in-context
learning, including a multi-sample ranking model and a consistency-aware
aggregation strategy.

  * •

OLD:We import one consistent method to rectify the ranking prediction of in-
context alternatives, which has not been studied before.

  * •

NEW:We emphasize the importance of the global consistent ranking relationships
and correct ranking metric approximation in the selection of in-context
samples for VICL;

  * •

NEW:We approximate the global ranking model by the introduced statistical
ranking model, including several meta-trained transformer-based partial
ranking models and a Hodge-theory inspired global ranking relationship
recovery model.

  * •

Extensive experiment results show that our method consistently works well in
various visual tasks based on pretrained in-context learning model, showing
the practical value of such a method.

##  2 Related Works

##### In-context learning.

Given the current trend of scaling up model sizes, the large-scale models such
as large language models (LLMs) brown2020gpt and their multi-modal
counterparts liu2024llava are shown to gradually learn the ability to perform
in-context learning, i.e. inference with knowledge provided by few-shot
context samples, rather than training the model with extra data. For example,
Pan et. al. pan2023logic designed an in-context learning system to generate
symbolic representation, which can then be used for logical inference. Zhang
et. al. zheng2023can proposed to leverage in-context learning to edit factual
knowledge in LLMs. Handel et. al. hendel2023context showed that in-context
learning works by compressing training set into task representations which can
modulate models during inference for desired output. Such a property enables
nonprofessional users to drive the model simply with several examples, the
same as someone teachers his children. Recently, MAE-VQGAN bar2022maevqgan and
Painter wang2023painter show that by learning to inpaint the image grid
composited by support and query pairs, the vision models can also learn the
in-context learning ability. VPR zhang2024vpr follows MAE-VQGAN and focuses on
the specific problem of selecting good in-context examples. They propose a
simple strategy by learning a contrastive learning based metric network with
performance metrics. Prompt-SelF sun2023promptself analyzes VPR’s strategy and
extend it not pixel-level feature similarity. Moreover, prompt-SelF proposes
an test-time ensemble strategy to gather predictions with different
permutation of in-context image grids. Our work inherits the idea of VPR to
concentrate on in-context example selection. Different from VPR, we stress the
importance of make full use of training sample, and the post-process of model
predictions. To this end, we propose a novel framework which first trains
ranking models with the performance metric data and then process the ranking
predictions with HodgeXX.

##### Ranking in deep learning.

Ranking models are aimed to compare a list of examples with a query example by
considering the similarity of a specific property between them. Burges et al.
burges2006learning proposed an objective function for pairwise ranking, which
was further extended to listwise ranking volkovs2009boltzrank. Recently
several works focus on learning ranking with transformer structures. For
example, Kumar et al. kumar2020deep proposed to adopt attention mechanism to
interact between samples to be ranked. Liu et al. liu2023rankcse proposed to
build the ranking model based on GPT model.

Jiang et. al. jiang2011hodge proposed HodgeRank as a Hodge-theoretic approach
to statistical ranking, which can provide global ranking with incomplete and
inconsistent ranking data. The comprehensive insight of HodgeRank made it
widely applicable to quality of experience assessment. For example, Xu et al.
xu2012hodgerank adopts HodgeRank for subjective video quality assessment based
on random graph models. The similar idea is also applied to crowdsourced
pairwise ranking aggregation xu2018hodgerank. In this paper we propose to
leverage the learning to rank technique and Hodge decomposition to build a
novel framework for visual in-context learning. While such a strategy has
never been studied before, we show that our framework can significantly boost
the visual in-context performance by selecting better examples.

##  3 Preliminary

##### Visual in-context learning.

Visual In-Context Learning (VICL) aims to build the inference procedure of
each testing sample based on knowledge provided by in-context examples. MAE-
VQGAN bar2022maevqgan instantiates such an idea through image inpainting.
Concretely, given a query sample, an example image is selected randomly along
with its annotations (e.g., bounding boxes, masks, etc.) from training set.
Then the example image, example label, query image and a random query label to
be predicted are arranged as an image grid so that the query label is placed
in the lower right corner. Then this composed image is used as input to a
inpainting model. The masked patches are predicted by a transformer
autoencoder and further decoded with the pretrained VQ-VAE van2017vqvae. To
ensure the generalization ability, MAE-VQGAN is pretrained on a large-scale
dataset collected from arxiv papers. While it shows promising results among
different kinds of tasks, the important problem of in-context example
selection is omitted, which leaves this method great potential for
improvement.

##### Visual Prompt Retrieval.

Visual Prompt Retrieval (VPR) zhang2024vpr mainly focuses on the in-context
example selection based on MAE-VQGAN. Specifically, VPR starts with the
heuristic that more similar an image is to the query image, the better it is
as the in-context example. Based on this, the unsupervised and supervised
variants of VPR are designed. In the unsupervised setting, VPR directly
selects the most visually similar training images for each query image. As for
the supervised setting, VPR first creates of a performance metric dataset
collected from the training set, symbolized as
$\\{x_{q}^{i},\mathcal{X}_{R}^{i},y^{i}\\}_{i=1}^{N_{Tr}}$, where $x_{q}$
denotes query sample, $\mathcal{X}_{R}$ denotes a set of $K$ alternatives to
be ranked, and $y$ denotes the performance metric, e.g. IoU for segmentation
or accuracy for classification, for pretrained MAE-VQGAN when testing $x_{q}$
on specific task using $\mathcal{X}_{R}$ as in-context examples respectively.
Then a metric learning model is trained to learn the in-context performance of
training samples, i.e. the best performing samples are labeled as positive
samples and learned against the worst performing ones as negative samples.
Such a method, while outperforming the naive MAE-VQGAN, still cannot fully
explore the knowledge regarding the ranking of in-context examples with the
unprocessed partial observations. To this end, we propose a novel method to
boost VICL with more proper in-context example selection.

##  4 Methodology

###  4.1 Overview

To boost VICL with better in-context example selection, we in this paper
propose a novel pipeline including two main steps. First we train a ranking
model (Sec. 4.2) based on the performance metric dataset collected in the same
way as VPR. Different from VPR, we utilize an extra transformer model to learn
list level ranking instead of instance level scoring so that inner
relationship between alternatives can be better utilized for ranking. Once the
ranking model is trained, the global ranking of alternative set needs to be
gathered from partial ranking predictions. To this end, we propose to adopt
HodgeRank (Sec. 4.3) to process predictions to provide globally consistent
ranking results, thus better facilitating VICL.

###  4.2 Training Rank Model with Contrastive Learning [Ranking ICL Samples?]

Our methodology is rooted in the principles of VPR. Given the query sample
$x_{q}$ and the alternative set $\mathcal{X}_{R}$, we first sample a $k$ size
subset $x$ from $\mathcal{X}_{R}$. Then a ranking model $\phi_{k}$ is built to
provide ranking prediction of $x$ as $\phi_{k}(x,x_{q})$. Concretely, features
$z_{q}\in\mathbb{R}^{(N+1)\times C},z\in\mathbb{R}^{k\times(N+1)\times C}$
corresponding to $x_{q},x$ are extracted with pretrained transformer model
$\phi$ such as CLIP radford2021clip or DINO caron2021dino, where $N+1$
features include class token and patch tokens, $C$ denotes feature channels.
Following the feature extraction, we concatenate all these features to form a
feature sequence $\hat{z}\in\mathbb{R}^{(k+1)(N+1)\times C}$. For the sake of
simplicity and to facilitate further processing, we rearrange this sequence
such that all class tokens are positioned at the beginning. This sequence
$\hat{z}$ is then processed through several newly introduced transformer
layers. These layers are designed to enhance interaction between the global-
level and local-level features contained in different images. This interaction
is crucial as it allows the model to gather the necessary information for
ranking the alternatives. Upon completion of these layers, we collect the
class tokens of alternatives $z_{cls}=\hat{z}_{\left[1:k+1,:\right]}$. These
tokens are then processed through linear layers to generate the ranking
prediction $\hat{y}\in\mathbb{R}^{k}$. This prediction $\hat{y}$ serves as an
indicator of the ranking of the alternatives, providing a quantitative measure
of their relevance to the query sample.

##### Training objectives.

Our ranking model is optimized with a composed objective as follow:

$\mathcal{L}=\underbrace{\sum_{i,j=1,i\neq
j}^{k}\max(0,\mathbf{1}(y_{i}>y_{j})(x_{i}-x_{j}+\delta))}_{\mathcal{L}_{margin}}+\underbrace{\mathrm{NeuralNDCG}(\tau)(\hat{y},y)}_{\mathcal{L}_{sort}}+\underbrace{\mathrm{MSE}(\hat{y}^{\prime},y)}_{\mathcal{L}_{reg}}$
(1)

where $\mathcal{L}_{margin}$ denotes the pair-wise marginal ranking loss,
$\mathcal{L}_{sort}$ denotes list-wise NeuralNDCG pobrotyn2021neuralndcg,
$\delta$ and $\tau$ represents the loss coefficients for two loss terms
respectively. $\mathcal{L}_{margin}$ and $\mathcal{L}_{reg}$ act a similar
role as the supervised contrastive loss in VPR, which encourages the model the
predict higher score for better samples in each pair. $\mathcal{L}_{sort}$, on
the other hand, can drive the model to learn inner relationship among more
alternatives, thus leading to better ranking predictions.

###  4.3 Obtaining Consistent Rank [Statistical Consistency Ranked Samples
with the Insights from combinational Hodge Theory]

[This section needs more work: 1, we may not want to stick to the name of
"Hodgerank", we shall split it into "Statistical ranking and combinatorial
Hodge theory" as its original paper, as there is no one using Hodgerank name.;
2, this part should be further splited into algorithm part and
insight(theoretical) part. ]

For this section, we will talk about how to obtain consistent rank.

####  4.3.1 Motivation

In the process of selecting in-context examples, it is crucial to identify the
most suitable candidates from the available pool. However, the ranking model
we employ only provides partial ranking predictions on a subset instead of the
full alternative set. A naive solution would be splitting the alternative set
into non-overlapping subsets, which are then processed with the trained rank
model respectively. Then the top ranked samples from each subset are gathered
together as a new alternative set, which is then repeatedly split and ranked
again until there is only one sample.

While such a method can provide reasonable results as we will show in the
experiment, it can suffer from three main problems: (1) In total,
$\binom{K}{k}$ subsets with size $k$ can be randomly sampled from an
alternative set with size $K$. However, the naive method only utilizes
$\lceil\frac{K}{k}\rceil$ subsets, which is extremely limited and can result
in selecting poor performing in-context examples. (2) Since this naive method
is iterative, the selection error would accumulate, leading to unbearably
wrong prediction. (3) It is inefficient to fetch other top ranked samples such
as second or third best ones with this method. Therefore a new method, which
can directly provide global ranking prediction by fully utilizing information
contained in the alternative set, can thus better serve the purpose of in-
context example selection. We get insight from the combinational Hodge Theory
and its application in ranking problem yy. This method allows us to derive a
global ranking from the partial information, thereby making it possible to
rank all candidates in relation to each other while guaranteeing global
consistency.

####  4.3.2 Explanation

####  4.3.3 Algorithm

Our method starts with gathering enough useful information from all
alternatives. Given the query sample $x_{q}$, alternative set
$\mathcal{X}_{R}$ and a trained $k$-length ranking model $\phi_{k}$, we can
first build an observation pool
$\mathcal{X}_{k}=\\{\mathcal{X}_{R}^{i}\\}_{i=1}^{N_{p}}$, where
$\mathcal{X}_{R}^{i}$ denotes a randomly shuffled variant of
$\mathcal{X}_{R}$. Then for each $\mathcal{X}_{R}^{i}$ we follow the naive
ranking method to split it into $\lceil\frac{K}{k}\rceil$ non-overlapped
$k$-length sequences and rank these sequences with $\phi_{k}$, resulting in a
predicting set $\mathcal{R}^{i}_{k}$. The predictions are further aggregated
into a preference matrix $S^{i}\in\mathbb{R}^{K\times K}$. If the $m$-th
alternative is favored over the $n$-th alternative, we let $S^{i}_{mn}=1$
otherwise $S^{i}_{mn}=-1$. If not predictions are related with $m$ and $n$-th
alternatives, then $S^{i}_{mn}=0$. $S^{i}$ can then be transformed into an
edge set $E^{i}=\\{(m,n)|S^{i}_{mn}\neq 0\\}$. In this way we can finally get
a preference matrix set $\mathcal{S}=\\{S^{i}\\}$.

Denoting the global ranking of $\mathcal{X}_{R}$ as a score vector
$r\in\mathbb{R}^{K}$, in which higher score denotes higher ranking. If we have
the oracle ranking $r^{*}_{K}$, then if $m$-th candidate is favored than
$n$-th candidate we can get $r^{*}_{m}-r^{*}_{n}>0$. As a result, we can
formulate the ranking problem as follows,

$\min_{r}\sum_{i=1}^{N_{p}}\sum_{(m,n)\in E^{i}}(r_{m}-r_{n}-S_{mn}^{i})^{2}$
(2)

To get the solution we can reformulate Eq. 2 as a Least Square problem by
introducing a transformation matrix
$D^{i}\in\mathbb{R}^{\lceil\frac{K}{k}\rceil\times K}$, each row of which is a
sparse vector with only the two indices from the pairwise set $E^{i}$ as 1 and
-1. Then we have the following formulation,

$\min_{r}\sum_{i=1}^{N_{p}}\frac{1}{2N_{p}}\|D^{i}r-S^{i}\|_{2}^{2}$ (3)

By solving Eq. 3 we can directly get a proper global ranking $r$. Furthermore,
multiple ranking models can be engaged in this process by extending
$\mathcal{S}$ with preference matrices calculated from these different models,
which can in turn enhance the ranking process with more comprehensive
information. Compared with the naive ranking method, this method involves more
partial observations from $\mathcal{X}_{R}$, while avoiding the accumulative
error, thus being both effective and efficient.

Algorithm 1 Our algorithm

0: train set $\mathcal{X}_{train}$, query sample $x_{q}$, trained ranking
models $\\{\phi_{k}\\}$, alternative set size $K$.

1: alternative set
$\mathcal{X}_{R}=topK_{\hat{x}\in\mathcal{X}_{train}}(sim(\hat{x},x_{q}))$

2: initial preference matrix set $\mathcal{S}:=\emptyset$

3: for rank-k model $\phi_{k}$ do

4: build observation pool $\mathcal{X}_{k}$ from $\mathcal{X}_{R}$

5: for randomly shuffled $\mathcal{X}_{k}^{i}$ from $\mathcal{X}_{k}$ do

6: $\mathcal{R}_{k}^{i}=\bigcup_{x\in\mathcal{X}_{k}^{i}}\phi_{k}(x,x_{q})$

7: aggregate $\mathcal{S}^{i}$ from $\mathcal{R}_{k}^{i}$

8: $\mathcal{S}=\mathcal{S}\bigcup\mathcal{S}^{i}$

9: end for

10: end for

11: global ranking $y=Hodge(\mathcal{S})$ (Eq. 3)

12: return top ranked sample.

##  5 Experiments

###  5.1 Dataset and setting

##### Dataset.

We follow MAE-VQGAN bar2022maevqgan and VPR zhang2024vpr to adopt three tasks
including foreground segmentation, single object detection and colorization.
For foreground segmentation, Pascal-5i is utilized which contains 4 data
splits. We conduct experiments and report the mean intesection over union
(mIoU) on all splits together with the averaged mIoU among these splits. For
single object detection, Pascal VOC 2012 is used. Images and predictions are
processed the same as in MAE-VQGAN, in which mIoU is adopted as metric. For
colorization, we first sample 50000 training data from ILSVRC2012 training
set. Then a test set randomly sampled from the validation set of ILSVRC2012 is
used to test the model with mean squared error (MSE) as metric.

##### Implementation Detail.

Considering the training data size for each task, we adopt different sequence
length for ranking. Specifically, we train rank-5 and rank-10 models for
foreground segmentation and single object detection, while rank-3 and rank-5
models are trained for colorization. The training data is built by first
selecting 50 most similar images from the whole training set for each image as
its alternative set. Then we randomly sample 20 sequences with the required
length from this alternative set for training. For all experiments we adopt
DINO v2 oquab2023dinov2 as feature extracter. AdamW optimizer
loshchilov2017adamw is used with learning rate set as 5e-5 and batch size set
as 64.

##### Competitors.

We choose three methods as our competitors, including MAE-VQGAN, VPR which
contains UnsupPR and SupPR and prompt-SelF sun2023promptself, which mainly
facilitates an ensemble strategy. To fairly compare our methods with these
previous ones, we report main results of our models both with and without the
test-time ensemble.

###  5.2 Main Results

The experiment results are shown in Tab. 1. Note that we omit the result of
our model with voting strategy for colorization since prompt-SelF does not
report this term and there is no fair competitors in this setting.

As in Tab. 1, for all three tasks, our model receives best performance on both
variant. Specifically, when not using voting strategy, our model outperforms
the strongest competitors SupPR by 2.84 in terms of average segmentation mIoU,
while the superiority is consistent on other two tasks. When using voting
strategy, our model receives about 4 higher mIoU for segmentation and 2 higher
mIoU for detection, leading to better results than prompt-SelF. One would note
that while SupPR was designed to learn a better metric than the naive visual
similarity used for UnsupPR, its performance on colorization is exactly the
same as UnsupPR. On contrary to that, our model makes a huge leap with 0.04
less MSE, which is not only much better than both UnsupPR and SupPR but also
doubles the improvement these two methods have against the random strategy in
MAE-VQGAN. This can prove the effectiveness of our proposed ranking model for
selecting in-context examples.

To further illustrate the advantage of our proposed method, we visualize
several samples for segmentation and detection in Fig. 1 and Fig.
LABEL:fig:det. We find that in some cases the in-context examples selected by
VPR can let MAE-VQGAN generate totally wrong results, especially when multiple
objects are presented in the query images. For example, in the first sample of
Fig. 1, the query image contains two monitors, while the target label is only
related to the left one. While both VPR and our method select images of
monitors as in-context example, VPR results in 0 IoU while prediction using
our example is much better. This may be resulted from the spatial relation of
the object of interest in the example images. In the example selected by VPR,
the monitor is placed at the right, which is the same as the non-target
monitor in the query image, thus leading to wrong guidance and prediction.
Apart from that, VPR tends to select small objects for detection, as shown in
Fig. LABEL:fig:det. This can lead to failed detection even though the query
objects are large enough for huamns to detect them. In comparison, our method
can generally select more proper in-context examples, thus enjoying better
performance.

Table 1: Comparison of our method with previous in-context learning methods. | Seg. (mIoU) $\uparrow$ | Det. (mIoU) $\uparrow$ | Color. (MSE) $\downarrow$
---|---|---|---
Fold-0 | Fold-1 | Fold-2 | Fold-3 | Avg.
MAE-VQGAN | 28.66 | 30.21 | 27.81 | 23.55 | 27.56 | 25.45 | 0.67
UnsupPR | 34.75 | 35.92 | 32.41 | 31.16 | 33.56 | 26.84 | 0.63
SupPR | 37.08 | 38.43 | 34.40 | 32.32 | 35.56 | 28.22 | 0.63
Ours | 38.81 | 41.54 | 37.25 | 36.01 | 38.40 | 30.66 | 0.58
prompt-SelF | 42.48 | 43.34 | 39.76 | 38.50 | 41.02 | 29.83 | —
Ours+voting | 43.23 | 45.50 | 41.79 | 40.22 | 42.69 | 32.52 | —
Figure 1: Qualitative comparison between our method and VPR, specifically
SupPR, in foreground segmentation. In each item we present the image grid in
the same order as the input of MAE-VQGAN, i.e. in-context example and its
label in the first row, query image and its prediction in the second row. The
IoU is listed below each image grid.

###  5.3 Model Analysis

To fully validate the effectiveness of each module in our method, we conduct a
series of ablation studies. For all ablation studies, we do not use voting
strategy. The experiments contain both foreground segmentation and single
object detection if not specified.

Table 2: Ablation study among different variants of our method. | Seg. (mIoU) $\uparrow$ | Det. (mIoU) $\uparrow$
---|---|---
Fold-0 | Fold-1 | Fold-2 | Fold-3 | Avg.
SupPR | 37.08 | 38.43 | 34.40 | 32.32 | 35.56 | 28.22
Rank-10 Naive | 37.51 | 39.69 | 36.62 | 34.58 | 37.10 | 29.58
Rank-10 Hodge | 38.70 | 41.08 | 37.04 | 35.54 | 38.09 | 29.79
Rank-{5,10} Hodge | 38.81 | 41.54 | 37.25 | 36.01 | 38.40 | 30.66

##### Effectiveness of Hodgerank.

First of all we provide comparison among variants with different designs.
Specifically, we consider three models: (1) Rank-10 Naive: A rank-10 model is
trained, of which the ranking predictions are directly used for example
selection. (2) Rank-10 Hodge: After training the rank-10 model and getting the
ranking predictions, we use the Hodge XX described in Sec. 4.3 to process
these data, then we pick the top-ranked sample in the post-processed global
ranking as in-context example. (3) Rank-{5,10} Hodge: Our full model, in which
the ranking prediction of both ranking models are mixed and processed with
HodgeXX to get the final global ranking. The results are presented in Tab. 2.
We find that the naive ranking model enjoys 1.54 higher average mIoU than
SupPR, which supports our motivation of using ranking model instead of pair-
wise contrastive learning model. Since ranking model can make better use of
the training data, and comparing multiple samples enables the model to
discover the inner relationship between them, rather than simply judging if
one sample is better than another one. Moreover, We find that using HodgeXX to
process the ranking predictions can lead to further improvement, which can be
enlarged by including results from rank-5 model. This is because the
predictions of ranking models can be thought of as from different annotators.
Even for a single model, its predictions will not be consistent when given two
sequences which share partial same samples. Therefore, simply utilizing the
inconsistent predictions can lead to suboptimal results, and HodgeXX helps
regulate the predictions to produce more globally consistent ranking results,
thus having better results.

##### Is quality of in-context examples really aligned with visual similarity?

A basic claim in VPR is that the more similar an image is to the query image,
the better it is as an in-context example. Nonetheless, VPR utilizes
performance metrics instead of visual similarity for contrastive learning and
provides better results, which makes one wonder if such a claim is indeed
solid. To check this out, we take a look at the best performing in-context
examples selected by our method. The results are shown in Fig. 2. We first
visualize the correlation between visual similarity, which is computed with
CLIP features, and mIoU on segmentation for both VPR and our method. It is
clear that visual similarity can be a basic heuristic for choosing visual in-
context examples. Generally well performing examples appear to share high
similarity with query samples. On the other hand, there are also large amount
of failure cases with high similarity, which indicates the ground principle
should be much more complex than visual similarity. Based on this conclusion,
we further visualize several cases in Fig. 2(c) selected by our method, which
better shows that one should consider more than visual similarity, other
factors such as object size, spatial position. On the opposite, we find that
the background similarity, which can contribute significantly to the visual
similarity, is hardly related to the quality of in-context examples.

Figure 2: (a) Scatter plot of visual similarity against IoU for VPR on
segmentation. (b) Scatter plot of visual similarity against IoU for our method
on segmentation. (c) Visualization of several cases with uncorrelated visual
similarity and IoU. The first row presents samples with low similarity but
proper in-context performance. The second row presents samples with high
similarity but poor in-context performance. Captions below each image grid
denote IoU and visual similarity sequentially.

##### Robustness among different backbones.

We compare our model with two variants with other pretrained feature
extractor: CLIP ViT-L/14 and DINO v1. The results are shown in Tab. 3. First,
the naive ranking model performs better than VPR no matter which backbone is
utilized. VPR also conducted such an experiment to try their method with
different backbones such as CLIP, EVA and supervised ViT, but none of those
results can outperform ours, which illustrates the efficacy of our design of
ranking model against the contrastive learning based model. Second, adopting
HodgeXX consistently improves the naive ranking method by nearly 1-2 mIoU for
both segmentation and detection, which is compatible with the results in Tab.
2, showing the robustness of HodgeXX among different backbone choices. Third,
the in-context performance is not totally correlated with the capacity of
backbones. It is commonly known that DINO v2, as an improved version of DINO
v1 with more comprehensive objective functions and more data, should be
equipped with higher capacity. However, we find that DINO v2 performs worse
than DINO v1 on segmentation whether naive ranking or HodgeXX is adopted. We
think such results can be attributed to the different learning difficulty when
using these backbones, and it would be interesting to have further research on
the impact of backbones for visual in-context learning.

Table 3: Ablation study among different backbones of our method. Backbone | Strategy | Seg. (mIoU) $\uparrow$ | Det. (mIoU) $\uparrow$
---|---|---|---
Fold-0 | Fold-1 | Fold-2 | Fold-3 | Avg.
CLIP | Naive | 37.37 | 40.11 | 36.84 | 33.88 | 37.05 | 29.69
Hodge | 38.58 | 41.34 | 37.66 | 35.91 | 38.37 | 30.79
DINOv1 | Naive | 38.78 | 40.02 | 36.92 | 35.12 | 37.71 | 28.03
Hodge | 39.25 | 42.27 | 38.45 | 36.77 | 39.19 | 29.19
DINOv2 | Naive | 37.51 | 39.69 | 36.62 | 34.58 | 37.10 | 29.58
Hodge | 38.81 | 41.54 | 37.25 | 36.01 | 38.40 | 30.66

##### Global consistency of ranking results.

An important property of HodgeRank is it guarantees the global consistency of
ranking predictions. That is to say, apart from the top ranked sample, the
second best and third best samples should also be generally better than other
samples in the alternative set as in-context example. To show that we test two
strategies: (1) directly using samples ranked second and third as in-context
examples and (2) adopting a simple late fusion method to average predictions
generated with 2, 3, 5 top ranked samples. The results are shown in Tab. 4.
One can find that using top-3 best ranked sample have comparable results. For
segmentation, the #2 rank samples only have 0.21 lower mIoU than #1 rank
samples, and #3 rank samples are even slightly better than #2 rank samples,
which evidences the global ranking consistency. The results are consistent for
detection. Moreover, fusing 2 or 3 samples together leads to 0.73 and 1.43
higher average mIoU, which also supports the global consistency brought by
HodgeXX. Such improvement saturates when using 5 samples, indicating that
samples ranked 5-th or later are significantly worse and cannot provide more
useful information for the in-context inference.

Table 4: Ablation study among different example selection strategies of our method. | Seg. (mIoU) $\uparrow$ | Det. (mIoU) $\uparrow$
---|---|---
Fold-0 | Fold-1 | Fold-2 | Fold-3 | Avg.
#1 rank | 38.81 | 41.54 | 37.25 | 36.01 | 38.40 | 30.66
#2 rank | 38.13 | 41.66 | 37.62 | 35.35 | 38.19 | 30.76
#3 rank | 38.66 | 41.08 | 37.36 | 35.91 | 38.25 | 30.61
top2 fusion | 39.08 | 42.61 | 38.17 | 36.67 | 39.13 | 30.16
top3 fusion | 40.07 | 42.48 | 38.77 | 37.61 | 39.73 | 31.85
top5 fusion | 40.12 | 42.59 | 39.09 | 37.28 | 39.77 | 32.08

##  6 Conclusion

This paper proposes a novel pipeline for in-context example selection in
Visual In-Context Learning (VICL). Specifically, we design a transformer-based
ranking model, which can provide desirable ranking predictions by well
utilizing the inner relationship among alternatives. The ranking results are
further aggregated by the proposed consistent-aware aggregation strategy,
which utilizes the classical HodgeRank method. Our method receives state-of-
the-art performance on three different tasks including foreground
segmentation, single object detection and image colorization, showing great
potential of improving VICL researches.