# GeoSpark: Sparking up Point Cloud Segmentation with Geometry Clue

Zhening Huang1 Xiaoyang Wu2 Hengshuang Zhao2 Lei Zhu3 Shujun Wang1
Georgios Hadjidemetriou1 Ioannis Brilakis1
1University of Cambridge 2The Unversity of Hong Kong 3HKUST(GuangZhou)

###### Abstract

Current point cloud segmentation architectures suffer from limited long-range
feature modeling, as they mostly rely on aggregating information with local
neighborhoods. Furthermore, in order to learn point features at multiple
scales, most methods utilize a data-agnostic sampling approach to decrease the
number of points after each stage. Such sampling methods, however, often
discard points for small objects in the early stages, leading to inadequate
feature learning. We believe these issues can be mitigated by introducing
explicit geometry clues as guidance. To this end, we propose GeoSpark, a Plug-
in module that incorporates Geometry clue into the network to Spark up feature
learning and downsampling. GeoSpark can be easily integrated into various
backbones. For feature aggregation, it improves feature modeling by allowing
the network to learn from both local points and neighboring geometry
partitions, resulting in an enlarged data-tailored receptive field.
Additionally, GeoSpark utilizes geometry partition information to guide the
downsampling process, where points with unique features are preserved while
redundant points are fused, resulting in better preservation of key points
throughout the network. We observed consistent improvements after adding
GeoSpark to various backbones including PointNet++, KPConv, and
PointTransformer. Notably, when integrated with Point Transformer, our
GeoSpark module achieves a 74.7% mIoU on the ScanNetv2 dataset (4.1%
improvement) and 71.5% mIoU on the S3DIS Area 5 dataset (1.1% improvement),
ranking top on both benchmarks. Code and models will be made publicly
available.

##  1 Introduction

Figure 1: Illustration of GeoSpark on feature aggregation and downsampling
modules. Current feature learning predominantly depends on local aggregation
(shown in a). Introducing GeoSpark not only expands the receptive field but
also concentrates on the relevant areas (shown in b). For instance, if the
query point is located at the end of the bed, Geometry-Informed Aggregation
will expand the receptive field to cover the entire area of the bed, as well
as the bed mat on the floor. Moreover, GeoSpark also enhances sampling by
using geometry clues as guidance. Compared to FPS (shown in c), Geometric
Downsampling (shown in d) samples coarsely in areas of simple geometry and
densely in areas with complex geometry, such as a dishwasher, a painting on
the wall, and objects on the table. The joint of these two modules yields
impressive segmentation results.

Since PointNet++[31], mainstream point cloud segmentation methods [47, 19, 24,
2, 36] conduct feature aggregation with local neighborhood, because learning
global features for large-scale point clouds is infeasible. However, due to
the redundancy in point clouds, local neighborhoods often contain a high
percentage of similar points, limiting models’ ability to learn from diverse
contexts. For this reason, how to model long-range features in point cloud is
a long-standing challenge in point cloud processing [20, 13, 30].

Moreover, hierarchical paradigms are predominantly utilized in point cloud
understanding architectures, where points undergo multiple downsampling stages
to learn features at various scales. These frameworks typically employ data-
agnostic sampling methods, such as FPS [31], voxelization [11], or random
sampling [13], which tend to dilute points for small objects. This is due to
the severe data imbalance issues present in point clouds. The reduction in the
number of points during later stages restricts the model’s ability to learn
expressive features for small objects, resulting in unsatisfactory
performance.

Figure 2: Illustration of Geometric Partition & Embedding and Geometry-
Informed Aggregation Modules. Geometric Partition & Embedding split point
clouds into simple geometric shapes, such as table surfaces, chair backs,
table edges etc, which are then embedded into superpoints. The query point
$x_{i}$ conducts local aggregation with $k_{local}$ local neighbour points
$x_{j}$, and then performs Geometry Partition aggregation with $k_{global}$
superpoints $\hat{x_{j}}$ to acquire global contexts. The Geometry-Informed
Aggregation module allows the model to learn both local fine details as well
as expressive global contexts.

We argue that these challenges can be mitigated by introducing explicate
geometry clue into the framework and preformed geometry-aware feature learning
and downsampling. We define the geometry information as an initial geometry
partition that can be obtained with conventional point cloud processing
techniques [16, 22, 21, 12, 26] where redundancy point clouds can be clustered
into simple shapes. To this end, we introduce a Plug-in module named GeoSpark
that can be integrated with various point cloud segmentation backbones to
spark up semantic segmentation results by enhancing feature aggregation and
the downsampling process.

Specifically, for feature aggregation, GeoSpark offers a Geometry-Informed
Aggregation (GIA) module which attends to both local neighbor points and
geometry partition entities. Unlike methods that rely solely on local
neighborhoods, our approach offers two compelling advantages. Firstly, it
significantly enlarges the receptive field of the feature aggregation module
without dramatically increasing the number of reference points. This is
achieved by encoding geometry partition into superpoints, which provide a
compact yet information-rich format for the raw input. Secondly, the geometry
partition entities are highly tailored to the input point cloud data.
Consequently, the receptive field for each point is tailored to its specific
geometry, enabling the model to learn from relevant regions rather than data-
agnostic areas. This concept is akin to the methodology employed in the
deformable structure [49, 43, 37, 8], which learns offset values from the
originally prescribed regions. Here, we achieve a similar effect without
requiring additional offset design.

For the downsampling module, GeoSpark introduces Geometric Downsampling (GD).
Unlike conventional methods such as Further Point Sampling, Random Sampling,
and Voxelization which drop points without considering their importance, our
approach uses geometry partition as guidance and fuses points of redundancy
into one while keeping points with unique features. In other words, the
downsampling is the process of reducing point redundancy, ensuring that
important points are retained. This data-dependent downsampling approach has
been shown to be very beneficial, especially for small objects.

These two simple yet powerful module designs can be incorporated into various
backbone models. We tested them on three backbone models, including Pointnet++
[31], KPConv [36], and Point Transformer [10], and consistently observed
improvements. In summary, our contribution is threefold.

  * •

We demonstrate that utilizing geometric partition to enhance feature
aggregation is an effective approach for performance gain. It not only expands
the receptive field in a cost-efficient manner but also concentrates on
relevant regions that are customized for the query point.

  * •

We enhance the downsampling module in the architecture by fusing redundant
points and retaining points with unique features, resulting in better
information retention throughout the network.

  * •

GeoSpark has proven to be universal and transferable to various baseline
models. Notably, when incorporating it with Point Transformer [48], it yielded
a 4.1% improvement in the mIoU score on the ScanNetV2 [7] dataset, and
achieved a score of 71.5% on the S3DIS [1] dataset, ranking high on both
benchmarks.

##  2 Related work

#### Point Cloud Segmentation.

Deep learning methods for point cloud processing can be categorized into three
types: projection-based methods [34, 41, 40, 3, 17], voxel-based methods [6,
11, 29], and point-based methods [4, 31, 47, 14, 19, 25, 2]. Recent work on
point-based methods mostly adopts an encoder-decoder paradigm while applying
various sub-sampling methods to expand the receptive field for point features.
A variety of local aggregation modules were developed, including convolution-
based methods [36, 2], graph-based methods [23, 21], and attention-based
methods [13, 48]. Meanwhile, different downsampling methods [5, 9, 45],
upsampling [32, 38], and post-processing methods [15, 27] have been proposed
to enhance the network’s robustness. The inherent permutation invariance of
attention operations makes them well-suited for point cloud learning, leading
to the application of transformer operations in point clouds [48, 10, 44],
following their success in 2D and NLP. However, global self-attention is
impractical due to massive computational costs, leading to recent work
utilizing local-scale self-attention to learn point features. Point
Transformer [48] proposed a vector self-attention with k nearest neighbor
points for feature aggregation and introduced the concept of learnable
positional embedding, which has proved to be very powerful across various
tasks. Nevertheless, local attention has a limited receptive field and cannot
explicitly model long-range dependencies. Thus, Stratified Transformer [20]
was developed and adopted a shifted window strategy to include long-range
contexts. Similarly, Fast Point Transformer [29] utilized a voxel hashing-
based architecture to enhance computational efficiency. However, these data-
agnostic methods only select receptive fields based on spatial information,
which could fail to focus on relevant areas.

#### Geometric Partition

Geometric partitions are an over-segmentation technique for point clouds that
groups point with similar geometric features. Over the years, several methods
have been developed to learn hand-crafted features of point clouds and utilize
clustering or graph partition methods to generate meaningful partitions [28,
12, 26]. However, the performance of these methods is limited by the
handcrafted feature of point clouds. SPNet [16] introduced a superpoint center
updating scheme that generates superpoints with the supervision of point cloud
labels. Although very efficient, SPNet fails to produce intuitive partitions
beyond grouping points with the same semantic label together. On the other
hand, the supervised superpoint method uses MLP to learn point features and
combines points with graph-structured deep metric learning [23]. Nevertheless,
Geometric partition techniques offer impressive outcomes in finding geometric
shapes in point clouds. However, despite their impressive performance, there
have been few attempts to utilize these techniques in the deep learning
framework [23, 21]. In this regard, we propose GeoSpark, which leverages the
geometry information stored in the partition to enhance feature aggregation
and downsampling.

##  3 GeoSpark

Our goal is to leverage the explicit geometry clue stored in the geometric
partitions to improve the feature aggregation and downsampling process. In
this section, we will elaborate on these are achieved on GeoSpark. We will
begin by explaining how the geometric partition is generated and embedding,
followed by technical details of our Geometry-Informed Aggregation (GIA)
module as well as Geometric Downsampling (GD) module.

Figure 3: The Paradigm of GeoSpark Architecture Initially, the point input is
fed into the ”Geometric Partition & Embedding” module to obtain geometric
partitions and encode them into superpoints. These superpoints are then
processed in the global branch to learn features at multiple scales before
being added to the local branch as geometry guidance. In addition to the
primary prediction loss, a superpoint loss is introduced to guide the training
of the global branch. The aggregation module can be altered to various
backbones.

###  3.1 Geometric Partition & Embedding

The first step of the proposed GeoSpark network is to effectively learn the
low-level features of point clouds and group similar points. This process
significantly reduces the redundancy of point clouds and makes it possible to
encode long-range context inexpensively in future steps. This module will
process the entire input scene and produce an initial geometric partition, so
it needs to be fast and effective. In our experiment, we tested different
geometric partition methods, including VCCS[28], Global Energy Model
(GEM)[12], Graph-Structured Method (GSM) [21] and SPnet [16]. The GEM [12]
offers the best trade-off between speed and performance, so we integrate that
into our Geometric Partition & Embedding module. Specifically, for each point,
a set of geometric features, such as linearity, planarity, scattering, and
verticality $f_{i}\in\mathbb{R}^{c}$ are computed, characterizing local
features and shapes. The geometrically homogeneous partition is defined as the
constantly connected components and therefore as the solution to the following
optimization problem[12]:

$\operatorname*{argmin}_{g\in\mathbb{R}^{c}}\sum_{i\in
C}||g_{i}-f_{i}||^{2}+\lambda\sum_{(i,j)\in E_{nn}}\omega_{i,j}[g_{i}\neq
g_{j}]\vspace{-2mm}$ (1)

where $\lambda$ is the regularization strength that determines the coarseness
of the partition, $\omega_{i,j}$ is a weight matrix that is linearly inversely
proportional to the distance between points and $[\cdot]$ is the Iverson
bracket. We use the $l_{o}$-cut pursuit algorithm[22] to quickly find
approximate solutions for Eq. 1, as it is impractical to find exact solutions
when the number of points is really large. Given point cloud set $X=(P,F)$,
the geometric partition module splits the point cloud into point subsets
$[\hat{X_{1}},\hat{X_{2}},\hat{X_{3}}\dots,\hat{X_{m}}]$, and each set
contains a different number of points.

To encode each partition into a superpoint, we adopted a lightweight MLP
structure, inspired by PointNet [4]. The input point feature firstly go
through MLP layers to gradually project the features of the points to higher
dimension space. Points in each partition are then fused into one superpoint
by applying MaxPooling in feature space $\hat{f_{i}}$ and AvgPooling in
coordinates space $\hat{p_{i}}$. We concatenate global information of each
partition $\hat{f_{i,g}}$, such as partition diameter into the features of the
fused points, before applying other layers of MLP to reduce the feature
dimension to $c$. Formally,

$\displaystyle\hat{F_{i}}$ $\displaystyle=T_{2}\ (\text{MaxPool}\ \\{{T_{1}\
(\hat{f_{i}})\ |\ \hat{f_{i}}\in\hat{X_{i}}}\\}\ \oplus\ \hat{f_{i,g}})$ (2)
$\displaystyle\hat{P_{i}}$ $\displaystyle=\text{AvgPool}\ \\{{\hat{p_{i}}\ |\
\hat{p_{i}}\in X_{i}}\\}$

where $T_{1}:\mathbb{R}^{c}\xrightarrow{}\mathbb{R}^{c_{l}}$ and
$T_{2}:\mathbb{R}^{c_{l}}\xrightarrow{}\mathbb{R}^{c}$ are MLP layers that
enlarge and condense point feature dimensions.

###  3.2 Geometry-Informed Aggregation

The key idea of Geometry-Informed Aggregation (GIA) is to attend point feature
both from local neighbour points and global geometric partition regions,
encoded in superpoints. Therefore, GIA module takes two sets of inputs: local
points $X\ =(p_{i},f_{i})$ and encoded superpoints
$\hat{X}=(\hat{p_{i}},\hat{f_{i}})$. Local point features $f_{i}$ and
superpoint features $\hat{f_{i}}$ are first processed by separated MLP layers
before being fed into the GIA module. The following explain the proposed
method using Point Transformer as backbone. However, similar strategies can be
incorporated with other backbone easily.

#### Local Neighbor Aggregation

The local neighbor aggregation follows the orginal design of backbone
structure. For Point Transformer, it [48] first uses three linear projections
$\phi,\psi,\alpha$ to obtain query, key and value for local points. Following
vector self-attention, a weight encoding function $\omega$ is introduced to
encode the subtraction relations between point queries and keys, before
processing to the softmax operation to form the attention map. A trainable
parameterized position encoding, formed by an MLP encoding function $\theta$,
is added to both the attention map and the value vectors. Formally, for query
point $x_{i}=(p_{i},f_{i})$ and reference points $x_{j}=(p_{j},f_{j})$, the
local attention map is formed as follows:

$\displaystyle\delta_{i,j}$ $\displaystyle=\theta(p_{i}-p_{j})$ (3)
$\displaystyle w_{i,j}$
$\displaystyle=\omega(\phi(f_{j})-\psi(f_{i})+\delta_{i,j})$

#### Geometric Partition Aggregation

To learn point feature from superpoint, a feature map is built between local
points and global superpoints. Specifically, key and value is projected from
superpoint features $\hat{f_{i}}$ with linear operation $\Psi$, and $A$
respectively. Attention is formed similarly to Local Attention. However, an
independent weight encoding function $\Omega$ is used and the positional
embedding also employs a different MLP $\Theta$ to encode the coordinate
difference between the query points and the reference superpoints.

$\displaystyle\Delta_{i,j}$ $\displaystyle=\Theta(p_{i}-\hat{p_{j}})$ (4)
$\displaystyle W_{i,j}$
$\displaystyle=\Omega(\phi(\hat{f_{i}})-\hat{\Psi(f_{j})}+\Delta_{i,j})$

In practice, Local Neighbor Aggregation and Geometric Partition Aggregation
are performed simultaneously and merged to form Geometry-Informed Aggregation,
formally:

$\displaystyle f_{i,local}^{\prime}$ $\displaystyle=\sum_{f_{i}\in
K}\text{softmax}(w_{i,j})\odot(\alpha(f_{j})+\delta_{i,j})$ (5) $\displaystyle
f_{i,global}^{\prime}$
$\displaystyle=\sum_{\hat{f_{i}}\in\hat{K}}\text{softmax}(W_{i,j})\odot(A(\hat{f_{j}})+\Delta_{i,j})$
$\displaystyle f_{i}^{{}^{\prime}}$
$\displaystyle=\xi\left[{f_{i,local}^{\prime}+f_{i,global}^{\prime}}\right]$

where $\odot$ represents Hadamard product, $K$ is defined as reference local
points, taken from $k_{local}$ nearest neighbour, and $\hat{K}$ is the
reference superpoints, sampled with $k_{global}$ nearest neighbour. After
merging the outputs of Local Neighbor Aggregation and Geometric Partition
Aggregation, an MLP layer ($\xi:\mathbb{R}^{c}\xrightarrow{}\mathbb{R}^{c}$)
is employed to further glue global and local features and form the final
outputs. Notice that it is important to maintain a good global and local
balance to ensure that both fine details and long-range dependencies are
included. More details on this can be found in the ablation study.

The features of superpoints $\hat{X}=(\hat{p_{i}},\hat{f_{i}})$ are learned in
a separate global branch. In this branch, the superpoint features are
projected to different dimensions at various stages to match the dimension
space of the local branch. Since the number of superpoints is typically
several orders of magnitude smaller than that of the local branch, the feature
learning in the global branch is very fast.

#### Loss function.

A simple yet important superpoint loss is introduced to assist feature
learning in the global branch. Specifically, given a local point set $X$, and
its label $E=\\{e_{i}\in\mathbb{R}^{l}\ |\ i=1,\dots,n\\}$, where $e_{i}$ is
the one-hot vector. We generate a soft pseudo label by calculating the label
distribution in each geometric partition $W=\\{w_{j}\in\mathbb{R}^{l}\ |\
j=1,\dots,m\\}$. We optimize the global branch by minimising the distance
between superpoint prediction $u$ and soft pseudo label $w$. Overall, our loss
is constructed with per-point prediction loss and the superpoint loss while a
parameter $\beta$ is introduced to determine the weight of the superpoint
loss. Given the pre-point predication as $e^{\prime}$, the loss function is

$L_{total}=\frac{1}{n}\sum_{i=1}^{n}\mathcal{L}_{loss}(e_{i},e_{i}^{\prime})+\beta\frac{1}{m}\sum_{i=1}^{n}\mathcal{L}_{loss}(w_{j},u_{j})$
(6)

where $\mathcal{L}_{loss}$ can be various type of loss function. Cross-entropy
is selected in our experiments.

###  3.3 Geometric Downsampling

Figure 4: Geometric Downsampling. Geometric Downsampling module fuses points
in small partitions into superpoints and further split oversized partitions
with a grid to control the downsampling rate. This method drops redundant
points and retains points with unique features

One notable challenge in the downsampling process is the early loss of the
under-representative points, such as points for small objects, leading to
insufficient feature learning and unsatisfactory prediction results. Data
agnostic sampling methods such as random sampling, and FPS, do not consider
the uniqueness of points and are likely to cause this issue. To mitigate this
problem, we develop the Geometric Downsampling module with the help of
geometric partitioning. The motivation is to preserve points with unique
features and drop redundancy points in the downsampling process. Given a point
set $X=\\{(p_{i},f_{i})\ |\ i=1\dots n\\}$, instead of sampling from the whole
point set, we conduct sampling from each geometric partition. In particular,
we set a target diameter $a$ for new points, and further split every partition
that is larger than the predetermined size $a$ with voxel grids to obtain
subsets, e.g. $[\hat{X_{1}}]$ to
$[\hat{X_{11}},\hat{X_{12}},\hat{X_{13}},\dots]$. Once this process is done,
we fuse the fine partition with pooling operations, where MaxPooling is
applied at features space and AvgPooling is conducted on coordinate space, as
illustrated in Figure 4. Similar to Point Transformer, we apply MLP layers
$D:\mathbb{R}^{c}\xrightarrow{}\mathbb{R}^{c}$ on feature space before
pooling. Mathematically:

$\displaystyle{f_{2,i}}=\text{MaxPool}\\{Df_{1,i}|f_{i}\in X_{1,ii}\\}$ (7)
$\displaystyle{p_{2,i}}=\text{AvgPool}\\{p_{1,i}|p_{i}\in X_{1,ii}\\}$

##  4 Experiments

We evaluated the effectiveness of GeoSpark by integrating it with three
backbones: PointNet++ [31], KPConv [37], and PointTransformer [48]. We
selected two competitive indoor datasets for testing, including the Stanford
3D Indoor dataset (S3DIS) [1] and ScanNetV2 [7]. We compared the results of
GeoSpark with the corresponding baselines and other state-of-the-art methods,
as shown in Table 1 and Table 2.

We also conducted comprehensive ablation studies to evaluate various design
decisions, which involved analyzing individual modules (refer to Table 6),
local-global balance for GIA (refer to Table 6), sampling strategies (refer to
Table 6), and the geometric partition size (refer to Table 6).

###  4.1 Experiments Setting

#### Network architecture.

Method | mAcc (%) | mIoU (%) | ceiling | floor | wall | beam | column | window | door | table | chair | sofa | bookcase | board | clutter
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
PointNet[4] | 49.0 | 41.1 | 88.8 | 97.3 | 69.8 | 0.1 | 3.9 | 46.3 | 10.8 | 59.0 | 52.6 | 5.9 | 40.3 | 26.4 | 33.2
SegCloud[35] | 57.4 | 48.9 | 90.1 | 96.1 | 69.9 | 0.0 | 18.4 | 38.4 | 23.1 | 70.4 | 75.9 | 40.9 | 58.4 | 13.0 | 41.6
TangentConv[34] | 62.2 | 52.6 | 90.5 | 97.7 | 74.0 | 0.0 | 20.7 | 39.0 | 31.3 | 77.5 | 69.4 | 57.3 | 38.5 | 48.8 | 39.8
PointCNN [24] | 63.9 | 57.3 | 92.3 | 98.2 | 79.4 | 0.0 | 17.6 | 22.8 | 62.1 | 74.4 | 80.6 | 31.7 | 66.7 | 62.1 | 56.7
SPGraph [23] | 66.5 | 58.0 | 89.4 | 96.9 | 78.1 | 0.0 | 42.8 | 48.9 | 61.6 | 84.7 | 75.4 | 69.8 | 52.6 | 2.1 | 52.2
PCCN [39] | 67.0 | 58.3 | 92.3 | 96.2 | 75.9 | 0.3 | 6.0 | 69.5 | 63.5 | 66.9 | 65.6 | 47.3 | 68.9 | 59.1 | 46.2
PAT [46] | 70.8 | 60.1 | 93.0 | 98.5 | 72.3 | 1.0 | 41.5 | 85.1 | 38.2 | 57.7 | 83.6 | 48.1 | 67.0 | 61.3 | 33.6
PointWeb[47] | 66.6 | 60.3 | 92.0 | 98.5 | 79.4 | 0.0 | 21.1 | 59.7 | 34.8 | 76.3 | 88.3 | 46.9 | 69.3 | 64.9 | 52.5
HPEIN[18] | 68.3 | 61.9 | 91.5 | 98.2 | 81.4 | 0.0 | 23.3 | 65.3 | 40.0 | 75.5 | 87.7 | 58.5 | 67.8 | 65.6 | 49.4
MinkowskiNet[6] | 71.7 | 65.4 | 91.8 | 98.7 | 86.2 | 0.0 | 34.1 | 48.9 | 62.4 | 81.6 | 89.8 | 47.2 | 74.9 | 74.4 | 58.6
Stratified Transformer[20] | 78.1 | 72.0 | 96.2 | 98.7 | 85.6 | 0.0 | 46.1 | 60.0 | 76.8 | 92.6 | 84.5 | 77.8 | 75.2 | 78.1 | 64.0
Fast Point Transformer [29] | 77.9 | 70.3 | 94.2 | 98.0 | 86.0 | 0.2 | 53.8 | 61.2 | 77.3 | 81.3 | 89.4 | 60.1 | 72.8 | 80.4 | 58.9
CBL [33] | 90.6 | 69.4 | 93.9 | 98.4 | 84.2 | 0.0 | 37.0 | 57.7 | 71.9 | 91.7 | 81.8 | 77.8 | 75.6 | 69.1 | 62.9
PointNet++ [31] | 64.1 | 55.8 | 88.5 | 94.8 | 74.9 | 0.0 | 25.2 | 46.7 | 39.5 | 79.8 | 69.9 | 63.4 | 51.4 | 45.5 | 45.8
\+ GeoSpark | 69.4 (+5.3) | 61.1 (+5.3) | 90.5 | 97.8 | 80.3 | 0.0 | 23.3 | 52.7 | 59.8 | 87.8 | 76.6 | 69.6 | 48.9 | 54.0 | 52.5
KPConv[36] | 70.9 | 65.4 | 92.6 | 97.3 | 81.4 | 0.0 | 16.5 | 54.5 | 69.5 | 90.1 | 80.2 | 74.6 | 66.4 | 63.7 | 58.1
\+ GeoSpark | 72.2 (+1.3) | 65.8 (+0.4) | 92.4 | 98.4 | 83.3 | 0.0 | 37.7 | 60.2 | 53.2 | 89.3 | 80.6 | 69.2 | 65.5 | 73.0 | 52.0
Point Transformer[48] | 76.5 | 70.4 | 94.0 | 98.5 | 86.3 | 0.0 | 38.0 | 63.4 | 74.3 | 89.1 | 82.4 | 74.3 | 80.2 | 76.0 | 59.3
\+ GeoSpark | 77.7 (+1.2) | 71.5 (+1.1) | 93.7 | 98.7 | 86.6 | 0.2 | 48.5 | 60.9 | 66.4 | 83.0 | 92.7 | 82.7 | 76.2 | 81.1 | 59.3

Table 1: Semantic segmentation results on the S3DIS Area 5. Consistency
improvements are observed after adding GeoSpark to three backbone structures.

The network architecture using PointTransformer as the backbone is presented
below. For further information on the settings used for other backbones,
please refer to the Supplementary Materials. The regularization strength
$\lambda$, was set to 3 for S3DIS and 2 for ScanNetV2 to obtain expressive
partitions. Within the global branch, the Superpoints were down-sampled by a
ratio of 1/2 after each stage. While the feature dimensions were chosen as
[32, 64, 128, 256, 512] for stages 1-5. Regarding Geometric Downsampling, a
size cap of [0.10, 0.20, 0.40, 0.80] m was set for S3DIS, and [0.25, 0.50,
0.75, 1.00] m for ScanNetV2, during stages 1-5. In the global branch,
aggregation was only performed once at each stage. Meanwhile, in the local
branch, the depths were set to [1, 2, 2, 6, 2] for stages 1-5, following the
approach in [48]. The depth settings were the same for both S3DIS and
ScanNetV2 datasets.

Method | Input |  mIoU (%)
---|---|---
PointNet++[31] | point | 53.5
PointCov[42] | point | 61.0
JointPointBased[15] | point | 69.2
PointASNL[45] | point | 63.5
KPConv[37] | point | 69.2
SparseConvNet[11] | voxel | 69.3
MinkowskiNet[6] | voxel | 72.2
Fast Point Transformer [29] | point | 72.1
Stratified Transformer[20] | point | 74.3
PointNet++[31] | point | 53.5
\+ GeoSpark | point |  63.5  (+10.0)
KPConv[37] | point | 68.6
\+ GeoSpark | point | 71.1 (+2.5)
Point Transformer[48] | point | 70.6
\+ GeoSpark | point | 74.7 (+4.1)
Table 2: Semantic segmentation results on ScanNetV2 Validation set. Adding
GeoSpark to the three backbone structures resulted in a noticeable, and the
PointTransformer + GeoSpark achieved a higher mIOU than the transformer
variation architectures. ranking top on the benchmarks

#### Implementation details.

All experiments were conducted using two A100 GPUs. The following presents
details of model using Point Transformer as THE backbone. For S3DIS, the model
was trained for 30,000 iterations with a batch size of 16. Following previous
works [48, 20], the input sets were subsampled with a 4mm grid. Area 5 was
used for testing while other areas were used for training. The loss weight
$\beta$ was set to 0.1, and the training was carried out using the AdamW,
optimizer, with a learning rate of 0.004 and a weight decay of 0.02. As for
ScanNetV2, the model was trained for 600 epochs with a batch size of 12. The
input sets were subsampled with a 2mm grid, and the AdamW optimizer was also
used with a learning rate of 0.005 and a weight decay of 0.05. Further
implementation details can be found in the Supplementary Materials.

###  4.2 Comparisons against State-of-the-arts

ID | GD | GIA | $\mathcal{L}_{sup}$ | mIoU | mAcc | OA
---|---|---|---|---|---|---
I |  |  |  | 72.4 | 81.3 | 90.6
II |  | ✓ |  | 72.8 | 81.3 | 90.7
III |  | ✓ | ✓ | 73.8 | 82.1 | 91.0
IV | ✓ |  |  | 73.5 | 81.4 | 90.7
VI | ✓ | ✓ | ✓ | 74.7 | 82.7 | 91.4
Table 3: Module ablation study.. GD: Geometric Downsampling. GIA: Geometry-
Informed Aggregation. $\mathcal{L}_{sup}$: Superpoint loss. Each module
demonstrated its ability to enhance the final results.

Dataset | SP. dia | mIoU | mACC | OA
---|---|---|---|---
S3DIS | 0.5 | 71.1 | 76.8 | 91.2
1 | 71.5 | 77.3 | 91.1
1.5 | 70.8 | 76.5 | 90.7
ScanNetV2 | 2 | 74.2 | 82.4 | 91.2
2.5 | 73.9 | 82.4 | 91.2
3 | 74.7 | 82.7 | 91.4
Table 4: Ablation study of max superpoint diameter setting. Selecting the
appropriate size of superpoint optimize the performance, and the size is
tailored to the dataset.

ID | $k_{local}$ | $k_{global}$ | mIoU | mAcc | OA
---|---|---|---|---|---
I | 8 | 4 | 72.1 | 80.6 | 90.6
II | 8 | 8 | 72.5 | 81.7 | 90.6
III | 10 | 5 | 73.3 | 82.0 | 90.9
IV | 16 | 0 | 73.5 | 82.1 | 91.2
V | 16 | 16 | 74.2 | 82.4 | 91.2
VI | 16 | 4 | 74.4 | 82.7 | 90.7
VII | 16 | 8 | 74.7 | 82.7 | 91.4
Table 5: Ablation study on local-global balance. Maintaining a balance between
global and local features is crucial for optimizing feature aggregation while
minimizing noise introduction.

Method | Sample ratio/size | mIoU | mACC | OA
---|---|---|---|---
FPS | $1/4$ | 73.8 | 82.1 | 91.2
$1/6$ | 72.3 | 81.0 | 90.1
Voxel | $[~{}0.10,0.20,0.40,0.80~{}]$ | 74.2 | 82.1 | 91.2
$[~{}0.25,0.50,0.75,1.00~{}]$ | 74.3 | 81.7 | 90.2
GD | $[~{}0.06,0.15,0.38,0.90~{}]$ | 74.1 | 82.1 | 91.2
$[~{}0.10,0.20,0.40,0.80~{}]$ | 74.3 | 82.1 | 91.2
$[~{}0.25,0.50,0.75,1.00~{}]$ | 74.7 | 82.7 | 91.4
Table 6: Ablation Study on Downsampling Approach Geometric Downsampling (GD)
is generally superior to voxel sampling and FPS, regardless of the partition
size setting. Additionally, fusing-based sampling outperforms FPS.

#### Quantitative comparisons.

We noticed a consistent enhancement in performance across three backbone
structures by integrating the GeoSpark plug-in. For the S3DIS dataset, the
PointNet backbone exhibited a notable 5.3% improvement in mIoU matrix, while
the KPconv backbone (Rigid Version) showed an increase of 0.4% in mIoU and 1.3
% in mACC performance as well. In addition, by adding GeoSpark to the Point
Transformer, we achieved an mIoU of 71.5%, which surpassed the baseline model
by 1.1% and ranked highly on the benchmarks.

On the ScanNetV2 validation set, integrating GeoSpark to PointNet++ backbone
saw a remarkable improvement of 10.0%, outperforming many other complex
designs. This result further demonstrates that modeling long-range features
are crucial for accurate large-scale scene understanding. The KPConv backbone
demonstrated an increase of 2.5% in mIoU performance as well. By incorporating
Geospark into the Point Transformer, we achieved an mIoU score of 74.7%, which
was a 4.1% improvement over the baseline model. This approach also surpassed
other transformer-based networks, like Fast Point Transformer and Stratified
Transformer, indicating the usefulness of geometry guidance in aggregation and
sampling modules. Our method performed exceptionally well for smaller classes
such as chairs, sofas, and boards, as shown in Table 1, highlighting the
significance of geometry features in small class segmentation, as shown in
Table Further information regarding training time and the number of parameters
can be found in the supplementary materials.

#### Visual results

Figure 6 presents the visual prediction outcomes of Point Transformer both
with and without GeoSpark. Furthermore, Figure 6 compared our GD approach to
other sampling techniques, including FPS and voxel-based sampling.

Figure 5: Visual comparison of the Point Transformer with and without
GeoSpark. GeoSpark captures finer details and produces better predictions for
certain objects such as chairs, tables, sofas, and cabinets. The main
difference between the prediction results of the Point Transformer and the
Point Transformer with GeoSpark was highlighted in a red block

Figure 6: Visualization of various sampling methods on the S3DIS dataset. In
comparison to FPS and Voxel-based sampling, Geometric Downsampling preserves
key points better, particularly in geometry complex areas. This provides
significant benefits, especially for small objects.The red block highlights
the preservation of details for small objects.

###  4.3 Ablation Study

A number of controlled experiments were carried out to evaluate the decisions
made during the model design process. The ablations were tested on the
ScanNetV2 dataset and employed Point Transformer as the backbone.

#### Module ablation.

For a fair comparison, Point Transformer is retrained with the same experiment
setting, including feature dimensions, module depths, and data argumentation.
By upgrading the attention module to GIA, a substantial improvement is
observed. With the help of pseudo-label loss, a 1.4% gain in mIoU is achieved.
This significant improvement confirmed our hypothesis that local aggregation
suffers from losing global context and that the GIA module is capable of
providing meaningful long-range contexts for better aggregation. Changing the
sampling strategies also offered a boost in performance, improving the mIoU to
73.5%. The joining of both modules with the pseudo label loss achieves a final
result of 74.7%.

#### Local-global balance.

We then investigate the setting of the number of neighbour points $k_{local}$,
and $k_{global}$, as shown in Table 6. The best results are yielded when the
attention module learns from 16 local points and 8 superpoints for feature
aggregation. Increasing the number of global points does not provide
significant benefits due to the possible introduction of noise. Also, we
notice that it is possible to reduce the number of local points, such as
($k_{local}$ = 10, $k_{global}$ = 5) set, which achieves similar results as
the baseline.

#### Sampling methods.

The ”Voxel” approach partitioned input with a voxel grid and uses pooling
operations to fuse points. In Table 6, we notice that, with the help of
geometric contexts, the GD approach indeed significantly improves the results
compared to the FPS baseline. Interestingly, we also see an improvement in the
voxel-based approaches. A potential reason is that Voxel-based sampling can
better preserve geometry features, than that of FPS (as shown in the
comparison in Figure 4), leading to a boost in performance. With the optimal
sampling size setting, GD yields the best results and outperformed its voxel
counterparts, demonstrating the benefits of sampling with the geometry-aware
approach.

#### Geometric partition size.

We investigate the different sizes of caps for geometric partition (Table 6).
For S3DIS, 1.0 m is the optimal size for geometric partitions, while for
ScanNetV2, 3.0 m yields the best results, since the data are slightly large.
We notice that it is essential to have the partition cap set to be larger than
the maximum receptive field at the end of the encoder, to provide extra global
contexts while reducing the risk of introducing noise.

##  5 Conclusion

Our work, GeoSpark, aims to utilize explicit geometry guidance to enhance
feature learning and downsampling. It leverages easily obtained geometry
partition information to guide the feature learning process, resulting in a
significant boost to the baseline models. Querying global features from
geometric partitions enlarges the receptive fields in a data-dependent way,
which helps to capture more expressive features. The geometric partitions also
guide the downsampling procedure to drop redundant points while keeping unique
points. We hope that this work will draw more attention to integrating
geometric partition clues into point cloud understanding, as redundancy is a
unique problem of point clouds compared to 2D images. In the future, we plan
to investigate integrating the proposed techniques into other 3D scene
understanding tasks, such as object detection and instance segmentation,
Additionally, we aim to test the proposed method on outdoor datasets, where
long-term feature learning and point redundancy pose significant challenges.

## Appendix 1: Runtime & Parameters Analysis

Table 7 presents the training time and a number of parameters on the S3DIS
dataset, based on experiments conducted using the Point Transformer backbone,
which yielded top-ranking results. The inclusion of a global branch results in
an increase in the number of parameters, but it has a manageable effect on
training time, as shown in Table 7. This is because the global branch
processes points that are typically 3-4 orders of magnitude smaller than the
original points (approximately 700-1000 points per S3DIS scan and 900-1200
points on ScanNetV2 scans). The smaller size of global points allows faster
processing by the network. Despite the fact that the Geospark+Point
Transformer slightly underperforms Stratified Transformer on the S3DIS
dataset, we have fewer parameters and shorter training times, demonstrating
the design’s greater efficiency.

The geometry partition module can process 100k points within 2-3 seconds, and
the entire S3DIS dataset can be processed in less than an hour on a modern
eight-core CPU.

Table 7: Run time & No. Parameter Comparison on S3DIS. Tested on two A100 GPUs with batch size 16.  Methods |  Train time(H)$\downarrow$ |  No.Para(M) $\downarrow$
---|---|---
PointTransformer | 26 | 10.2
PointTransformer + GeoSpark |  30 (+15%) |  15.3 (+50%)
Stratified Transformer |  100 (+233%) |  18.8 (+84%)
SparseUNet |  42(+61%) |  38.0 (+272%

## References

  * [1] Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese.  3d semantic parsing of large-scale indoor spaces.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016.
  * [2] Alexandre Boulch.  ConvPoint: Continuous convolutions for point cloud processing.  Computers &amp Graphics, 88:24–34, 2019.
  * [3] Alexandre Boulch, Joris Guerry, Bertrand Le Saux, and Nicolas Audebert.  SnapNet: 3d point cloud semantic labeling with 2d deep segmentation networks.  Computers and Graphics (Pergamon), 71:189–198, 2018.
  * [4] R. Qi Charles, Hao Su, Mo Kaichun, and Leonidas J. Guibas.  PointNet: Deep learning on point sets for 3d classification and segmentation.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
  * [5] Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao.  SASA: Semantics-Augmented Set Abstraction for Point-Based 3D Object Detection.  Proceedings of the AAAI Conference on Artificial Intelligence, 36(1), 2022.
  * [6] Christopher B. Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
  * [7] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Niessner.  ScanNet: Richly-annotated 3d reconstructions of indoor scenes.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
  * [8] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  CoRR, abs/1703.06211, 2017.
  * [9] Oren Dovrat, Itai Lang, and Shai Avidan.  Learning to sample.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2019.
  * [10] Nico Engel, Vasileios Belagiannis, and Klaus Dietmayer.  Point transformer.  IEEE Access, 9:134826–134840, 2021.
  * [11] Ben Graham.  Sparse 3d convolutional neural networks.  In Xianghua Xie, Mark W. Jones, and Gary K. L. Tam, editors, Procedings of the British Machine Vision Conference 2015. British Machine Vision Association, 2015.
  * [12] Stéphane Guinard and Loïc Landrieu.  Weakly supervised segmentation-aided classification of urban scenes from 3D lidar point clouds.  In International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives. International Society for Photogrammetry and Remote Sensing, 2017.
  * [13] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.  Randla-net: Efficient semantic segmentation of large-scale point clouds.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, 2020.
  * [14] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham.  Randla-net: Efficient semantic segmentation of large-scale point clouds.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, 2020.
  * [15] Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew lan Tai.  JSENet: Joint semantic segmentation and edge detection network for 3d point clouds.  In Computer Vision – ECCV 2020, volume 12365 LNCS, pages 222–239. Springer International Publishing, 2020.
  * [16] Le Hui, Jia Yuan, Mingmei Cheng, Jin Xie, Xiaoya Zhang, and Jian Yang.  Superpoint network for point cloud oversegmentation.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021.
  * [17] Maximilian Jaritz, Jiayuan Gu, and Hao Su.  Multi-view PointNet for 3d scene understanding.  In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). IEEE, 2019.
  * [18] Li Jiang, Hengshuang Zhao, Shu Liu, Xiaoyong Shen, Chi-Wing Fu, and Jiaya Jia.  Hierarchical point-edge interaction network for point cloud semantic segmentation.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 2019\.
  * [19] Mingyang Jiang, Yiran Wu, Tianqi Zhao, Zelin Zhao, and Cewu Lu.  PointSIFT: A SIFT-like Network Module for 3D Point Cloud Semantic Segmentation.  arxiv, 2018.
  * [20] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia.  Stratified transformer for 3d point cloud segmentation.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.
  * [21] Loïc Landrieu and Mohamed Boussaha.  Point cloud oversegmentation with graph-structured deep metric learning.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
  * [22] Loïc Landrieu and Guillaume Obozinski.  Cut pursuit: Fast algorithms to learn piecewise constant functions.  In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016, Cadiz, Spain, May 9-11, 2016, volume 51 of JMLR Workshop and Conference Proceedings. JMLR.org, 2016.
  * [23] Loïc Landrieu and Martin Simonovsky.  Large-scale point cloud semantic segmentation with superpoint graphs.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2018.
  * [24] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.  Pointcnn: Convolution on x-transformed points.  In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, 2018.
  * [25] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen.  Pointcnn: Convolution on x-transformed points.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [26] Yangbin Lin, Cheng Wang, Dawei Zhai, Wei Li, and Jonathan Li.  Toward better boundary preserved supervoxel segmentation for 3d point clouds.  ISPRS Journal of Photogrammetry and Remote Sensing, 143:39–47, 2018\.
  * [27] Tao Lu, Limin Wang, and Gangshan Wu.  CGA-net: Category guided aggregation for point cloud semantic segmentation.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2021.
  * [28] Jeremie Papon, Alexey Abramov, Markus Schoeler, and Florentin Worgotter.  Voxel cloud connectivity segmentation - supervoxels for point clouds.  In 2013 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2013.
  * [29] Chunghyun Park, Yoonwoo Jeong, Minsu Cho, and Jaesik Park.  Fast point transformer.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2022.
  * [30] Yifan Peng, Lin Lin, Lexing Ying, and Leonardo Zepeda-Núñez.  Efficient long-range convolutions for point clouds.  Journal of Computational Physics, 473:111692, 2023.
  * [31] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017.
  * [32] Shi Qiu, Saeed Anwar, and Nick Barnes.  Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2021.
  * [33] Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao.  Contrastive boundary learning for point cloud segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8489–8499, June 2022.
  * [34] Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yi Zhou.  Tangent convolutions for dense prediction in 3d.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2018.
  * [35] Lyne Tchapmi, Christopher Choy, Iro Armeni, JunYoung Gwak, and Silvio Savarese.  SEGCloud: Semantic segmentation of 3d point clouds.  In 2017 International Conference on 3D Vision (3DV). IEEE, 2017\.
  * [36] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 2019\.
  * [37] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas.  Kpconv: Flexible and deformable convolution for point clouds.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 2019\.
  * [38] Nina Varney and Vijayan K. Asari.  Pyramid point: A multi-level focusing network for revisiting feature layers.  IEEE Geoscience and Remote Sensing Letters, pages 1–1, 2020.
  * [39] Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun.  Deep parametric continuous convolutional neural networks.  In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. IEEE Computer Society, 2018.
  * [40] Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer.  SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3d LiDAR point cloud.  In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.
  * [41] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer.  SqueezeSegV2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a LiDAR point cloud.  In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2018.
  * [42] Wenxuan Wu, Zhongang Qi, and Fuxin Li.  Pointconv: Deep convolutional networks on 3d point clouds.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
  * [43] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang.  Vision transformer with deformable attention.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4794–4803, June 2022.
  * [44] Saining Xie, Jiatao Gu, Demi Guo, Charles R. Qi, Leonidas Guibas, and Or Litany.  PointContrast: Unsupervised pre-training for 3d point cloud understanding.  In Computer Vision – ECCV 2020, pages 574–591. Springer International Publishing, 2020.
  * [45] Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui.  Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. IEEE, 2020.
  * [46] Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian.  Modeling point clouds with self-attention and gumbel subset sampling.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
  * [47] Hengshuang Zhao, Li Jiang, Chi-Wing Fu, and Jiaya Jia.  Pointweb: Enhancing local neighborhood features for point cloud processing.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundation / IEEE, 2019.
  * [48] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun.  Point transformer.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2021.
  * [49] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  CoRR, abs/2010.04159, 2020.


11institutetext: School of Computer Science and Ningbo Institute, Northwestern
Polytechnical University, Xi’an, China 22institutetext: School of
Cybersecurity, Northwestern Polytechnical University, Xi’an, China
33institutetext: School of Computer Science, Nanyang Technological University,
Singapore 33email<EMAIL_ADDRESS>

# Rethinking and Improving Visual Prompt Selection for In-Context Learning
Segmentation

Wei Suo 1133111These authors contributed equally to this work.111These authors
contributed equally to this work. 0000-0002-8283-8637 Lanqing Lai 11111These
authors contributed equally to this work.111These authors contributed equally
to this work. Mengyang Sun 2233 0000-0003-0638-3295 Hanwang Zhang 33
0000-0001-7374-8739 Peng Wang 11444Corresponding author444Corresponding author
Yanning Zhang 11 0000-0002-2977-8057

###### Abstract

As a fundamental and extensively studied task in computer vision, image
segmentation aims to locate and identify different semantic concepts at the
pixel level. Recently, inspired by In-Context Learning (ICL), several
generalist segmentation frameworks have been proposed, providing a promising
paradigm for segmenting specific objects. However, existing works mostly
ignore the value of visual prompts or simply apply similarity sorting to
select contextual examples. In this paper, we focus on rethinking and
improving the example selection strategy. By comprehensive comparisons, we
first demonstrate that ICL-based segmentation models are sensitive to
different contexts. Furthermore, empirical evidence indicates that the
diversity of contextual prompts plays a crucial role in guiding segmentation.
Based on the above insights, we propose a new stepwise context search method.
Different from previous works, we construct a small yet rich candidate pool
and adaptively search the well-matched contexts. More importantly, this method
effectively reduces the annotation cost by compacting the search space.
Extensive experiments show that our method is an effective strategy for
selecting examples and enhancing segmentation performance.
https://github.com/LanqingL/SCS

###### Keywords:

In-Context Learning Visual Prompt Image Segmentation

##  1 Introduction

Image segmentation is a fundamental task in computer vision, aiming to locate
and identify different semantic concepts at the pixel level. It is essential
in multiple applications such as autonomous driving [43, 54], video
surveillance [41, 20, 36, 39] and complex reasoning [34, 24, 40]. To achieve
precise segmentation, researchers have proposed a variety of segmentation
models [29, 23, 12, 5]. Although these methods have made considerable
progress, they require training the specialist model for each segmentation
task. Recently, inspired by In-Context Learning (ICL) [49, 26, 30], several
generalist segmentation frameworks have been proposed and achieved impressive
performance such as MAE-VQGAN [6] and SegGPT [45]. Different from the
traditional frameworks, ICL-based segmentation methods establish a promising
paradigm for localizing vision concepts by feeding one or a few examples as
references during inference. Meanwhile, the methodology offers a general
solution for various real-world scenarios [44].

ICL learning is a relatively new research line in computer vision, while the
fundamental ideas can be traced back to the Natural Language Processing (NLP)
field [6, 44]. With the development of Large Language Models (LLM), numerous
studies have already proved that selecting a suitable context is crucial for
following human instruction and improving performance [1, 31, 14]. However,
related researches about this topic is still limited in our community. As
shown in Fig.1 (a), existing works typically randomly select examples [44] or
simply apply similarity sorting to construct contextual prompts [56, 37]. The
above insights motivate us to question: 1) Whether different contexts would
significantly impact performance? 2) What is the critical factor of visual
prompt selection for ICL-based segmentation models?

Figure 1: Comparsion of traditional and our method. (a) Existing works rely on
dense annotation to build the exampling space. Then, they use the similarity
sorting manner to select examples for given queries. (b) Instead, our method
significantly alleviates the costs of annotation by searching the typical
examples. Moreover, a novel adaptive search module is designed to further
select well-matched contexts.

In this paper, we first try to answer the above questions. By comprehensive
comparisons, we find that the ICL-based models are sensitive to different
examples. With multiple experiments, the performance gap between using
different contextual examples even exceeds 5 points. Furthermore, we
synthetically investigate the effectiveness of the existing prompt selection
strategy. Surprisingly, although the similarity measure (_i.e.,_ choosing the
most similar examples as contexts) provides a relatively effective approach
for demonstration search, assembling the most dissimilar contexts results in
better performance in $\sim$40% of the test instances. To reveal the potential
reasons, we empirically demonstrate that the diversity of contexts is a
crucial factor in guiding segmentation and enhancing performance (more details
in Sec. 3).

Beyond the above discussions, we rethink the role of contextual examples and
propose a new Stepwise Context Search method, which is called SCS for short.
As shown in Fig. 1 (b), the SCS focuses on maintaining the diversity of the
candidate pool and adaptively searching the well-matched contexts. Meanwhile,
our method significantly reduces the annotation cost by searching the typical
examples. Specifically, given a large unlabeled database, we first apply a
basic cluster algorithm to compact the search space and improve annotation
efficiency. Then, we extract the two most dissimilar members in each cluster
as candidate examples. By collecting these representative members from all
clusters, we construct a small yet rich candidate pool. Furthermore,
considering the significant impact of examples on performance, we further
introduce a novel adaptive search module based on reinforcement learning. It
heuristically utilizes segmentation scores as rewards, guiding the model in
determining final demonstrations. Extensive experiments across various
datasets have shown the effectiveness of our method. To summarize, our
contributions are as follows.

  * •

We take a step forward to systematically analyze the influence of example
selection on the ICL-based segmentation frameworks and empirically demonstrate
that the diversity of contexts is a key factor in guiding segmentation.

  * •

We propose a new SCS method to efficiently select visual prompts by
constructing a small yet rich candidate pool. Meanwhile, an adaptive search
module is designed to further choose the well-matched demonstrations.

  * •

The proposed model significantly improves the segmentation performance of ICL-
based frameworks and achieves state-of-the-art on PASCAL-5i [32], COCO-20i
[21] and iSALD-5i [50].

##  2 Related work

###  2.1 Visual Segmentation

As a fundamental task in computer vision, visual segmentation entails locating
and identifying different semantic concepts at the pixel level. Numerous works
for specific tasks have continued to flourish over the years. For example,
Crossover-Net [52] learns the vertical and horizontal crossover relation for
medical image segmentation. RFNet [35] fuses RGB and depth information which
is designed for real-time road-driving segmentation. [16] proposes DGEN, a
densely connected global entropy network in which noise is suppressed by
context information losses. MSNANet [25] accurately extracts water bodies from
remote sensing scenes by utilizing multiscale attention. Recently, inspired by
ICL learning, several general segmentation methods have been proposed [18, 44,
45]. Different from these specialist models, the paradigm can locate visual
concepts by feeding a few examples, known as the In-Context Learning (ICL)
segmentation frameworks.

###  2.2 In-Context Learning

In-context learning (ICL) is a new learning paradigm that initially emerged
from GPT-3 [7]. It presents a text fill-in-the-blank problem with examples,
eliminating the necessity for fine-tuning model parameters on downstream NLP
tasks. In the vision domain, MAE-VQGAN [6] transforms various visual tasks
into grid inpainting problems and proposes the first ICL-based segmentation
framework. Different from the previous segmentation methods, this new paradigm
provides a promising manner to segment arbitrary objects with a few examples.
Painter [44] adopts masked image modeling on continuous pixels, unifying the
output form across different visual tasks. As a variant of Painter, SegGPT
[45] integrates visual segmentation tasks into in-context coloring problems
and demonstrates powerful capabilities in target segmentation. Beyond the
previous works, we present comprehensive experiments on example selection for
visual ICL models.

###  2.3 Contextual Example Selection

With the development of LLMs, the selection of contextual examples is
important for ICL learning [51, 1, 30]. The emerging studies can be
categorized into rule-based unsupervised methods and supervised methods. KATE
[22] uses the KNN mechanism to enhance a retriever for selecting in-context
examples. EPR [30] adopts a contrastive learning strategy to train a dense
retriever. UDR [19] explores the process of retrieval demonstrations for
various tasks in a unified manner. However, few studies investigate these
effects in the vision domain. Current works typically select examples randomly
[44] or simply apply similarity sorting to construct contextual demonstrations
[56, 37]. Different from the previous rule-based methods, we introduce an
adaptive search method with a small yet rich candidate pool. This strategy
effectively reduces the annotation cost and boosts segmentation performance.

##  3 Preliminary

In this section, we would first present a simple overview of the ICL-based
segmentation frameworks. Then, we construct extensive experiments to
comprehensively analyze the influence of different examples and the
effectiveness of the existing selection strategies. Finally, we experimentally
proved that contextual diversity is an important factor for guiding
segmentation.

###  3.1 Background

The design of segmentation models has achieved tremendous development in
recent years [52, 57, 41, 20]. However, these solutions require expensive
annotation cost and face challenges in adapting to new scenarios. In contrast,
the ICL-based segmentation frameworks [45, 6] can implement various specific
tasks by leveraging several segmented examples. The convenient and versatile
vision models provide a promising perspective for our community.

As shown in Fig. 1, the goal of ICL-based segmentation methods is to predict
segmentation regions based on exhibited one or a few prompts. By reorganizing
the input format, the models can imitate provided contexts to mask specific
object regions. Specifically, given a query image $I_{q}$, these segmentation
frameworks first select $n$ example images ${E_{d}}=\\{I_{i}^{d}\\}_{i=1}^{n}$
with corresponding masks ${G_{d}}=\\{G_{i}^{d}\\}_{i=1}^{n}$ as contextual
demonstrations from a huge database. Here, $I_{i}^{d}$ and $G_{i}^{d}$ denote
the $i$-th selected example and corresponding mask. By reformating the input,
models can perform multiple segmentation tasks across different scenarios. The
above operations can be formulated as:

${\hat{I}_{m}}=\mathbf{S}({I_{q}},{E_{d}},{G_{d}}),$ (1)

where the ${\textbf{S}}(\cdot)$ and $\hat{I}_{m}$ are the generalist
segmentation model and predicted mask results, respectively. Although the
above calculations provide a convenient way to segment objects at arbitrary
granularities, the success of the paradigm greatly relies on the given
contextual examples. In fact, the basic idea of ICL-based vision models comes
from the NLP field, in which it has been proved that contextual prompts play
an important role in executing reasoning and boosting performance. However, in
the vision community, existing approaches mostly simply choose the
demonstrations (_i.e.,_ ${E_{d}}$ and corresponding ${G_{d}}$) based on random
sampling [44] or rely on similarity sorting [56, 37]. To explore the impact of
the visual prompts on performance, we construct a series of quantitative
analyses on the PASCAL dataset [32] with the state-of-the-art ICL-based
segmentation model SegGPT [45]. Note that the SegGPT involves two training
stages (_i.e.,_ pre-training and in-context tuning). In the tuning process,
the SegGPT depends on the whole annotation (including base and new classes) to
improve the performance. Since the main advantage of the ICL-based framework
is to implement few-shot tasks, we only use the pre-trained weights provided
by the SegGPT [3] to construct experiments. Following [6, 44], the mean
intersection over union (mIoU) is adopted as the evaluation metric.

(a)

(b)

Figure 2: The influence of different context selection. (a) Randomly sampling
contextual examples with 5 runs under 1-shot and 5-shot setting on PASCAL. (b)
For each instance, the nearest and farthest examples are retrieved as visual
prompts across different similarity-based sorting. Surprisingly, the most
dissimilar examples achieved better performance on $\sim$40% of the test
samples.

###  3.2 Influence of Different Examples

As mentioned above, research on the influence of different examples in
segmentation remains insufficient. In this paper, we aim to fill this gap by
conducting experiments to quantitatively analyze the impact of different
demonstrations. Specifically, we randomly select examples from the entire
training set with 5 different seeds. As shown in Fig. 2 (a), we report the
best, worst, and average results across the 5 experiments. It reveals that
allocating different contexts would dramatically affect the segmentation
scores under the 1-shot and 5-shot settings. In particular, the best results
can surpass the worst by margins of 5.6 and 4.1 points, respectively. Due to
the huge performance discrepancy between different contextual prompts,
designing an automated method for searching well-matched examples emerges as a
critical challenge.

###  3.3 Example Selection with Similarity

To the best of our knowledge, the related researches to our work are [56, 37],
where they simply rely on cosine similarity to select examples with
qualitative analysis. In this paper, we try to rethink the strategy through a
quantitative discussion across different similarity computation manners.
Specifically, we explore two types of image similarity calculation strategies
including appearance-level and semantic-level. For the appearance-level
comparison, PSNR [47], SSIM [15], FSIM [55] and HaarPSI [28] obtain the
similarity between the test instance and the entire training set. On the other
hand, for semantic-level comparison, we follow [56, 37] to utilize the off-
the-shelf image encoders (_i.e.,_ CLIP [27] and ViT [8]) for feature
extraction. To verify the effectiveness of these similarity-based approaches,
we carefully select the examples based on cosine similarity sorting.

As shown in Fig. 2 (b), each metric is represented by two bars including
$nearest>farthest$ (yellow bar) and $nearest<farthest$ (green bar). We take
yellow as an example, it denotes the proportion of instances where using the
most similar examples (nearest) as contexts results in a better IoU than using
the most dissimilar ones (farthest), and the green bar indicates the opposite.
Surprisingly, we observed that assembling the most dissimilar contexts leads
to superior performance for numerous test instances. There is a substantial
portion of the test samples ($\sim$40%) that achieve better IoU scores by
utilizing the farthest examples. The above results indicate that using
similarity measures is not the only way to search for contextual examples.

(a) Example sampling with SegGPT [45].

(b) Example sampling with MAE-VQGAN [6].

Figure 3: Diversity _vs_ Similarity. Based on similarity sorting, the
performance of the two Nearest examples (NN), the two Farthest examples (FF),
and the Nearest example with the Farthest example (NF) as visual prompts are
shown.

###  3.4 Example Selection with Diversity

Based on the above analyses, it is imperative to explore a more effective way
for searching visual prompts. In Fig. 2 (b), we find that applying the most
dissimilar examples can also yield satisfactory performance for some
instances. The results motivate us to investigate whether combining the
nearest and farthest contexts can obtain a performance boost. Consequently, we
further construct a series of experiments to verify the hypothesis across
different ICL-based models.

More specifically, based on similarity sorting, we use two Nearest examples
(NN), two Farthest examples (FF), and a combination of the Nearest example and
the Farthest example (NF) as contextual demonstrations, respectively. As shown
in Fig. 3, we report related results across different similarity measure
manners. It can be observed that the performance using NF examples
significantly surpasses the other two selection strategies in all measures.
Regarding ViT, the NF achieves better performance that outperforms equipping
the two nearest examples by a margin of 2.4 points.

###  3.5 Discussion

In this section, we would try to summarize the above findings with a deeper
discussion. Based on the insights presented in Sec. 3.2, we find that the ICL-
based segmentation framework is sensitive to contextual examples. Given the
different visual prompts, the performance gap is even over 5 points on the
mIoU. Therefore, building an effective prompt selection method is important
for boosting the segmentation results and achieving stable performance.

To further explore the key factors that influence example selection, we design
experiments in Sec. 3.3 and Sec. 3.4. The results demonstrate that combining
both similar and dissimilar demonstrations can significantly enhance the IoU
scores. We argue that the underlying reason is that this manner provides
richer semantic information about the segmented objects by expanding the
diversity of visual prompts. Based on the above insights, a natural idea is to
select these considerable discrepancy examples as contextual prompts for each
test instance. However, implementing the straightforward strategy would incur
substantial annotation cost and complex computation, since each test sample
needs to compare and rerank the candidate list. To mitigate the above
problems, we introduce a novel stepwise context search (SCS) method that
maintains the diversity of the candidate pool and reduces the cost of
annotation. Moreover, considering the necessity of selecting appropriate
contextual examples for different test queries, we further design an adaptive
search module. It can dynamically select contexts from the candidate pool
based on given test samples.

##  4 Our Method

Based on the discussion in Sec. 3, we design a new SCS method to construct a
small yet rich candidate pool and adaptively search the contextual
demonstrations. Compared to previous works, our method has two potential
advantages: 1) Fewer Annotations. The SCS focuses on building a small yet
diverse candidate pool. It only needs to annotate a portion of samples while
achieving better performance. 2) Adapting Selection. Compared to the previous
rule-based mechanism [56, 37], our method can adaptively search the well-
matched examples for different test instances. Next, we would introduce our
method in detail.

###  4.1 Constructing Candidate Pool

As mentioned in Sec. 3, it is important to provide examples for the ICL-based
segmentation paradigm. However, previous research mostly depend on huge
annotations as contextual candidates, it would inevitably suffers from
expensive annotation cost. Motivated by the of Sec. 3.5, we aim to build a
small yet diverse candidate pool that can be used for subsequent context
searching.

Given $N$ unlabeled samples, follow in [56], we utilize CLIP to extract image
feature into $d$-dimensional vectors, represented by
$F=\\{f_{i}\\}_{i=1}^{N},f_{i}\in\mathbb{R}^{d}$, where $f_{i}$ is the $i$-th
encoded image feature. To construct a candidate pool comprising a set of
representative examples, we employ k-means clustering algorithm [58] to
generate ${M}$ clusters $C=\\{{{c}_{m}}\\}_{m=1}^{M}$, which can be determined
by minimizing the sum of squared intra-cluster norms:

$\mathop{\arg\min_{C}}\sum\limits_{m=1}^{M}\sum\limits_{f_{i}\in{c_{j}}}{\left\|{{{f_{i}}}-{{\bf\mu}}_{m}}\right\|},$
(2)

where ${\mu}_{m}$ is centroid of cluster ${c_{m}}$. Initially, this algorithm
randomly selects $M$ centroids . Each instance is assigned by computing the
distance to all selected centroids. The process is iteratively executed until
the clusters are stable.

To construct the candidate pool, a simple method is to select members closest
to the centroid, while previous studies have shown that samples closest to the
center do not necessarily represent the entire cluster [46, 11]. Moreover,
related works have provided several alternative sampling methods such as
density sorting [46] and entropy sorting [11]. In this paper, we design a
simpler and more efficient sampling strategy without complex computation.
Specifically, for each cluster $c_{m}$, which includes members
$F_{m}=\\{f_{k}\\}_{k=1}^{K}$, where $K$ denotes the number of members in
$c_{m}$. We sort them into a list [$f_{k}^{(1)},f_{k}^{(2)},...,f_{k}^{(K)}$]
in the ascending order of their distance to the cluster’s center. Based on the
findings of Sec. 3.3, aggregating the nearest and farthest examples can cover
more diverse demonstrations. Thus, we consider the $f_{mk}^{(1)}$ and
$f_{mk}^{(K)}$ as typical samples for each cluster. By collecting these
samples from all clusters, we obtain the final candidate pool
$D=\\{f_{1k}^{(1)},f_{1k}^{(K)},f_{2k}^{(1)},...f_{Mk}^{(1)},f_{Mk}^{(K)}\\}$,
which contains a total of $2M$ samples. To simplify, we use $f_{d}^{m}$ and
$I_{d}^{m}$ to represent $m$-th image feature and corresponding image,
respectively.

Figure 4: Overview of our SCS method. Instead of similarity sorting on a large
annotated dataset, we use clustering to select diverse examples from unlabeled
data $D$ and construct the candidate pool. Meanwhile, the search agent is used
to further select contextual demonstrations for various test samples based on
reinforcement learning. During inference, the test sample and candidate pool
examples are fed into the model and adaptively search visual prompts.

###  4.2 Adaptive Search Module

In this module, our goal is to select well-matched examples from the
constructed candidate pool. As shown in Fig. 4, we propose a new adaptive
search module that can further search and assemble the suitable contexts for
given queries.

Specifically, for a given test instance ${I_{q}}$, we first apply the same
image encoder (_i.e_. CLIP) to extract the corresponding feature vector
$f_{q}\in{\mathbb{R}^{d}}$. Then, we concatenate the image feature $f_{q}$ and
each example $f_{d}^{m}$ from the candidate pool. These fused features are fed
into a search agent that consists of a group of light and simple Multi-Layer
Perceptrons (MLP). The search computation can be denoted as follows:

$\displaystyle s_{d}^{m}$ $\displaystyle={\mathrm{MLP}}([f_{d}^{m};f_{q}])),$
(3) $\displaystyle a_{d}^{1},a_{d}^{2},...,a_{d}^{2M}$
$\displaystyle=\mathrm{softmax}(s_{d}^{1},s_{d}^{2},...,s_{d}^{2M}),$ (4)

where $a_{d}^{m}$ represents the probability of $m$-th example in the
candidate pool. To guide the search agent in learning select examples, we
apply the IoU scores as rewards for selecting well-matched demonstrations from
the candidate pool. Specifically, following the format of the ICL-based
segmentation model, we reorganize query $I_{q}$, the original image of context
examples $I_{d}$ and corresponding mask $G_{d}$ into a joint image, as shown
in Fig. 4. These images are then fed into the ICL-based segmentation model to
obtain the IoU scores $U=\\{u_{d}\\}_{d=1}^{2M}$. Inspired by [4, 38], we use
the average IoU score $u_{avg}$ as the base score. By applying reinforcement
learning, the gradient is calculated by:

$\displaystyle\nabla_{\theta}L(\theta)=-\frac{1}{2M}\sum\limits_{m=1}^{2M}(u_{d}^{m}-u_{avg})\nabla_{\theta}log(a_{d}^{m}),$
(5)

where $a_{d}^{m}$ and $\theta$ are the probability of $m$-th example and the
parameters of the search agent. Based on the above computation, this gradient
would tend to increase the probability of the $m$-th context when its IoU
scores $u_{d}^{m}$ are higher than the average IoU score $u_{avg}$.

During inference, we directly input the test samples and candidate pool
examples into the search agent. The contextual examples are then determined by
sorting them based on probability scores $a_{d}$ as detailed in Eq. 4.

##  5 Experiment

We use different prompt selection strategies and various generalist
segmentation frameworks to demonstrate the superiority of our method.
Furthermore, we compare our approach with traditional methods and conduct
extensive qualitative and quantitative studies on multiple datasets.

Table 1: The results on COCO-20i and PASCAL-5i based on example-based semantic segmentation. To ensure a fair comparison and follow the few-shot setting, $\dagger$ indicates that only using the pre-trained weights provided by the SegGPT [3], and we report related results based on randomly sampling (_i.e.,_ w/R) and similarity sorting [56]. |  COCO-20i [21] |  PASCAL-5i [32]
---|---|---
$\mathbf{Method}$ | 1-shot | 5-shot | 1-shot | 5-shot
| mIoU | FBIoU | mIoU | FBIoU | mIoU | FBIoU | mIoU | FBIoU
Painter [44] | 32.8 | 50.2 | 32.6 | 50.1 | 64.5 | 74.2 | 64.6 | 74.4
MAE-VQGAN [6] | 12.1 | 45.7 | - | - | 27.5 | 55.6 | - | -
MAE-VQGAN w/ [56] | 18.0 | 46.5 | - | - | 33.6 | 57.8 | - | -
SegGPT$\dagger$ w/R [45] | 48.6 | 71.3 | 52.7 | 73.1 | 72.1 | 83.1 | 76.2 | 80.3
SegGPT$\dagger$ w/ [56] | 53.3 | 72.2 | 56.1 | 75.5 | 74.4 | 84.4 | 76.9 | 85.9
MAE-VQGAN w/Ours | 19.1 | 47.0 | - | - | 35.0 | 61.2 | - | -
SegGPT$\dagger$ w/Ours | 57.2 | 75.1 | 59.5 | 76.8 | 76.6 | 85.4 | 78.2 | 86.5

###  5.1 Experimental Setting

Dataset. We follow the previous works [6, 56, 45] and evaluate our approach on
PASCAL-5i [32], COCO-20i [21] and iSALD-5i [50]. PASCAL-5i includes 20
semantic categories and is created from PASCAL VOC 2012 [9] with external
annotations from SDS [10]. COCO-20i is built upon MSCOCO [21], which is a more
challenging dataset and contains 80 categories. For all categories, both
COCO-20i and PASCAL-5i are evenly divided into 4 folds for cross-validation,
where each fold is selected as the test set and the remaining three folds are
used for training. The iSALD-5i contains 15 geospatial categories which
derived from iSALD [48] in the remote sensing image semantic segmentation
task. Similarly, the iSALD-5$i$ is also partitioned into 3 folds for cross-
validation.

Evaluation Metric. Following [13], mean intersection over union (mIoU) and
foreground-background IoU (FBIoU) are adopted as evaluation metrics. The
former measures the average overall IoU values for all object classes, while
the latter reports the average IoU scores for the foreground and background.

Implementation details. For both datasets, the Adam [17] is used as our
optimizer and the maximum number of training epochs is 10. We use the CLIP-ViT
[27] image encoder to extract image features. During training, all images are
resized to 224×224 and fed into the image encoder. The dimension of extracted
feature vectors is 1024. We train on one 4080 GPU with an initial learning
rate of 1e-3, which is dropped by half every 5 epochs. The number of clusters
$M$ and batch size is set to 10 and 8.

###  5.2 Quantitative Evaluation

In Table 1, we show a performance comparison of two general ICL-based
segmentation models with and without proposed SCS. Following [56, 37], all
results are reported by repeating the random experiment five times. It can be
observed that our method can significantly boost performance in all settings
including 1-shot and 5-shot settings. In particular, for the 1-shot setting,
the SCS leads to MAE-VQGAN with the mIoU gain of 7.0 and 6.5 on the PASCAL-5i
and COCO-20i dataset, respectively. Besides, compared to previous rule-base
models [56] in 1-shot and 5-shot semantic segmentation tasks, our method
surpasses it by a considerable margin. Moreover, following the few-shot
setting and ensuring a fair comparison, we report related results based on the
pre-trained weights for SegGPT [3]. It can be observed that our method
significantly outperforms these rule-based visual prompt selection methods for
SegGPT by 9.7 and 6.8 on the 1-shot and 5-shot settings for COCO-20i. More
importantly, the above results demonstrate that the proposed SCS can be
considered as a “plug-and-play” module to enhance the segmentation ability of
ICL-based frameworks.

###  5.3 Ablation Study

To clarify the effectiveness of our method, we conduct further ablation
studies on the COCO-20i dataset with the 5-shot setting. The SegGPT is used as
the basic model due to its outstanding segmentation performance.

Table 2: Ablation study. Similarity: Cosine similarity ranking; Cluster: Cluster compacting; Sorting: The nearest-farthest sampling; ASM: Adaptive search module. | Similarity | Cluster | Sorting | ASM | mIoU | FBIoU
---|---|---|---|---|---|---
1 |  |  |  |  | 52.7 | 73.1
2 | ✓ |  |  |  | 56.1 | 74.5
3 | ✓ | ✓ |  |  | 55.7 | 74.3
4 | ✓ | ✓ | ✓ |  | 57.6 | 75.7
5 |  | ✓ | ✓ | ✓ | 59.5 | 76.8

Effects of Cluster. In Table 2, we first build a baseline by randomly
selecting examples from the entire training set. The average result is
reported across 5 different seeds. In the second row of Table 2, we follow
[56] and utilize cosine similarity ranking to select the top-5 examples as
contexts. In the third row, we investigate the impact of the cluster on in-
context example selection, using the closest examples to each cluster centroid
to construct the candidate pool. Compared to the baseline, the results show
that compacting the search space not only reduces annotation cost but also
improves model performance.

Table 3:  The effects of different settings for our method.  | Model | 1-shot | 5-shot
---|---|---|---
| mIoU | FBIoU | mIoU | FBIoU
1 | Entropy | 53.1 | 72.3 | 56.1 | 75.0
2 | Foreground | 53.5 | 72.5 | 56.7 | 79.3
3 | ViT | 56.9 | 75.0 | 58.7 | 76.0
4 | CLIP-resnet | 56.3 | 74.7 | 58.5 | 76.4
5 | Density-sort | 57.6 | 75.1 | 59.1 | 76.3
6 | Entropy-sort | 56.9 | 74.5 | 58.4 | 75.9
7 | Ours | 57.2 | 75.1 | 59.5 | 76.8

Effects of Sort. As shown in row 4 of Table 2, we experimentally demonstrate
that integrating our sorting mechanism results in 1.9 mIoU improvement
compared to directly using the closest examples to each cluster centroid. The
inherent reason is that centroid examples describe a limited feature space
[46, 33], making it challenging to support a variety of different contexts for
test instances. In contrast, our sorting mechanism introduces the nearest and
farthest examples , which constructs more diverse demonstrations for ICL
learning.

Effects of Adaptive Search Module. Finally, we further explore the
effectiveness of our adaptive search module in the last row of Table 2.
Compared to using a simple similarity sorting method, our approach shows an
improvement of 3.4 points. Meanwhile, benefiting from the diverse candidate
pool, our SCS achieves state-of-the-art performance and outperforms the
baseline by a considerable margin.

Table 4: The results on PASCAL-5i [32] based on instance segmentation. Model | 1-shot | 5-shot
---|---|---
SegGPT$\dagger$ w/R [45] | 46.5 | 46.9
SegGPT$\dagger$ w/ [56] | 47.1 | 48.5
SegGPT w/Ours | 50.5 | 51.4

Table 5: The results on iSALD-5i [48] based on semantic segmentation. Model | 1-shot | 5-shot
---|---|---
SegGPT$\dagger$ w/R [45] | 29.8 | 30.6
SegGPT$\dagger$ w/ [56] | 34.7 | 35.4
SegGPT w/Ours | 37.9 | 39.5

###  5.4 Different Model Settings

In this section, we explore several alternative model settings to further
discuss the proposed method. All experiments are conducted on the COCO-20i
with the SegGPT model. In the first two rows, we first compare the performance
using different metrics. Specifically, we use the rule-based methods to select
the contextual examples from the complete training set. which including
Entropy sorting [11] and Foreground sorting [53]. In particular, “Entropy” is
executed by calculating the maximum entropy values. The “Foreground” refers to
a two-stage approach [53]. It uses an object detector [42] to extract the
foreground boxes and the foreground similarity is used as the metric. From the
1-shot and 5-shot results, it can be observed that our SCS can dramatically
outperform these methods by a large margin. In rows 3-4, we construct the
experiments with different feature encoders. We replace the CLIP-ViT with ViT
[8] and CLIP-resnet [27] to obtain image feature vectors, These extracted
features would be utilized to construct the candidate pool and perform the
adaptive searching. The results show that our SCS is robust for different
image encoders. Finally, in rows 5-6 of Table 3, we replace our selection
mechanism with several alternative manners. However, these approaches mostly
involve complex computation. For example, the Density-sort [2] requires
additional computations on the order of $\boldsymbol{O(MN)}$ in each iteration
to update density lists, where $M$ is the number of clusters and $N$ is the
size of the unlabeled data. Instead, our method uses the basic clustering
algorithm and obtains a comparable performance.

###  5.5 Further Discussion

More Segmentation tasks. In this section, we evaluate the effect of our method
on more segmentation tasks and datasets. Because MAE-VQGAN cannot distinguish
between individual instances of objects within the same class, we construct
more experiments with the SegGPT model. Specifically, we report instance
segmentation results on PASCAL-5i in Table 5 and aerial image semantic
segmentation results on iSALD-5i in Table 5. The above results indicate that
the SCS can enhance the segmentation ability for various real-world scenarios.

Figure 5:  Qualitative results under 1-shot setting on COCO-$20^{i}$. The
similarity-based selection method is viewed as our baseline. The green
regions, red regions and purple regions are example masks, predicted masks and
ground-truth masks.

Qualitative Results. As shown in Fig. 5, to further analyze and verify the
proposed method, we visualized the qualitative segmentation results on
COCO-20i with the SegGPT. The first and third columns represent the visual
prompts (highlighted in green) based on similarity sorting and our SCS method,
respectively. The second and fourth columns show the corresponding
segmentation results (highlighted in red). The last column represents the
ground truth mask (highlighted in purple). As illustrated in the first row,
the baseline model selects the most similar context to the query (i.e.,
chopping board, knife and carrot), while the segment result is not
satisfactory, as the model can not recognize the knife in the image. Instead,
Our method selects an image from the candidate pool that accurately segments
out the mask. The above results demonstrate that similarity sorting is not the
only golden standard and improving the diversity of context prompts can boost
the segmentation results.

##  6 Conlusion

In this paper, we focus on the example selection method for the ICL-based
segmentation frameworks. By building comprehensive comparisons, we empirically
demonstrate that diversity of contexts is a key factor in guiding segmentation
and boosting performance. Beyond the above insights, we propose a new stepwise
context search method that improves the diversity of the candidate pool and
significantly alleviates the annotation cost. We hope that this work can
promote further research to better understand in-context learning in computer
vision.

Acknowledgement. This work was supported by National Natural Science
Foundation of China (No.62102323), the Innovation Capability Support Program
of Shaanxi(Program No. 2023KJXX-142) and National Natural Science Foundation
of China (U23B2013).

## References

  * [1] Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., Ghazvininejad, M.: In-context examples selection for machine translation. arXiv preprint arXiv:2212.02437 (2022)
  * [2] et al., W.: "unsupervised selective labeling for more effective semi-supervised learning". In: ECCV 2022. pp. 427–445. Springer (2022)
  * [3] et al, W.: "seggpt: Segmenting everything in context". In ICCV 2023 (2023)
  * [4] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)
  * [5] Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence 39(12), 2481–2495 (2017)
  * [6] Bar, A., Gandelsman, Y., Darrell, T., Globerson, A., Efros, A.: Visual prompting via image inpainting. Advances in Neural Information Processing Systems 35, 25005–25017 (2022)
  * [7] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)
  * [8] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [9] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. International journal of computer vision 88, 303–338 (2010)
  * [10] Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse detectors. In: 2011 international conference on computer vision. pp. 991–998. IEEE (2011)
  * [11] Haussmann, E., Fenzi, M., Chitta, K., Ivanecky, J., Xu, H., Roy, D., Mittel, A., Koumchatzky, N., Farabet, C., Alvarez, J.M.: Scalable active learning for object detection. In: 2020 IEEE intelligent vehicles symposium (iv). pp. 1430–1435. IEEE (2020)
  * [12] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
  * [13] Hong, S., Cho, S., Nam, J., Lin, S., Kim, S.: Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In: European Conference on Computer Vision. pp. 108–126. Springer (2022)
  * [14] Hongjin, S., Kasai, J., Wu, C.H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N.A., et al.: Selective annotation makes language models better few-shot learners. In: The Eleventh International Conference on Learning Representations (2022)
  * [15] Hore, A., Ziou, D.: Image quality metrics: Psnr vs. ssim. In: 2010 20th international conference on pattern recognition. pp. 2366–2369. IEEE (2010)
  * [16] Hu, H., Li, Z., Li, L., Yang, H., Zhu, H.: Classification of very high-resolution remote sensing imagery using a fully convolutional network with global and local context information enhancements. IEEE Access 8, 14606–14619 (2020)
  * [17] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [18] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [19] Li, X., Lv, K., Yan, H., Lin, T., Zhu, W., Ni, Y., Xie, G., Wang, X., Qiu, X.: Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320 (2023)
  * [20] Li, Y., Padmanabhan, A., Zhao, P., Wang, Y., Xu, G.H., Netravali, R.: Reducto: On-camera filtering for resource-efficient real-time video analytics. In: Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication. pp. 359–376 (2020)
  * [21] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [22] Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., Chen, W.: What makes good in-context examples for gpt-$3$? arXiv preprint arXiv:2101.06804 (2021)
  * [23] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3431–3440 (2015)
  * [24] Luo, G., Zhou, Y., Sun, X., Cao, L., Wu, C., Deng, C., Ji, R.: Multi-task collaborative network for joint referring expression comprehension and segmentation. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 10034–10043 (2020)
  * [25] Lyu, X., Fang, Y., Tong, B., Li, X., Zeng, T.: Multiscale normalization attention network for water body extraction from remote sensing imagery. Remote Sensing 14(19) (2022). https://doi.org/10.3390/rs14194983, https://www.mdpi.com/2072-4292/14/19/4983
  * [26] Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., Zettlemoyer, L.: Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202.12837 (2022)
  * [27] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [28] Reisenhofer, R., Bosse, S., Kutyniok, G., Wiegand, T.: A haar wavelet-based perceptual similarity index for image quality assessment. Signal Processing: Image Communication 61, 33–43 (2018)
  * [29] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. pp. 234–241. Springer (2015)
  * [30] Rubin, O., Herzig, J., Berant, J.: Learning to retrieve prompts for in-context learning. arXiv preprint arXiv:2112.08633 (2021)
  * [31] Rubin, O., Herzig, J., Berant, J.: Learning to retrieve prompts for in-context learning. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 2655–2671 (2022)
  * [32] Shaban, A., Bansal, S., Liu, Z., Essa, I., Boots, B.: One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410 (2017)
  * [33] Shen, F., Zhao, L., Du, W., Zhong, W., Qian, F.: Large-scale industrial energy systems optimization under uncertainty: A data-driven robust optimization approach. Applied Energy 259, 114199 (2020)
  * [34] Shi, H., Li, H., Meng, F., Wu, Q.: Key-word-aware network for referring expression image segmentation. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 38–54 (2018)
  * [35] Sun, L., Yang, K., Hu, X., Hu, W., Wang, K.: Real-time fusion network for rgb-d semantic segmentation incorporating unexpected obstacle detection for road-driving images. IEEE robotics and automation letters 5(4), 5558–5565 (2020)
  * [36] Sun, M., Suo, W., Wang, P., Niu, K., Liu, L., Lin, G., Zhang, Y., Wu, Q.: An adaptive correlation filtering method for text-based person search. International Journal of Computer Vision pp. 1–16 (2024)
  * [37] Sun, Y., Chen, Q., Wang, J., Wang, J., Li, Z.: Exploring effective factors for improving visual in-context learning. arXiv preprint arXiv:2304.04748 (2023)
  * [38] Suo, W., Sun, M., Liu, W., Gao, Y., Wang, P., Zhang, Y., Wu, Q.: S3c: Semi-supervised vqa natural language explanation via self-critical learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2646–2656 (2023)
  * [39] Suo, W., Sun, M., Niu, K., Gao, Y., Wang, P., Zhang, Y., Wu, Q.: A simple and robust correlation filtering method for text-based person search. In: European conference on computer vision. pp. 726–742. Springer (2022)
  * [40] Suo, W., Sun, M., Wang, P., Zhang, Y., Wu, Q.: Rethinking and improving feature pyramids for one-stage referring expression comprehension. IEEE Transactions on Image Processing 32, 854–864 (2022)
  * [41] Vasudevan, A.B., Dai, D., Van Gool, L.: Semantic object prediction and spatial sound super-resolution with binaural sounds. In: European conference on computer vision. pp. 638–655. Springer (2020)
  * [42] Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al.: Internimage: Exploring large-scale vision foundation models with deformable convolutions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14408–14419 (2023)
  * [43] Wang, X., Ma, H., You, S.: Deep clustering for weakly-supervised semantic segmentation in autonomous driving scenes. Neurocomputing 381, 20–28 (2020)
  * [44] Wang, X., Wang, W., Cao, Y., Shen, C., Huang, T.: Images speak in images: A generalist painter for in-context visual learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 6830–6839 (2023)
  * [45] Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 (2023)
  * [46] Wang, X., Lian, L., Yu, S.X.: Unsupervised selective labeling for more effective semi-supervised learning. In: European Conference on Computer Vision. pp. 427–445. Springer (2022)
  * [47] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)
  * [48] Waqas Zamir, S., Arora, A., Gupta, A., Khan, S., Sun, G., Shahbaz Khan, F., Zhu, F., Shao, L., Xia, G.S., Bai, X.: isaid: A large-scale dataset for instance segmentation in aerial images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 28–37 (2019)
  * [49] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837 (2022)
  * [50] Yao, X., Cao, Q., Feng, X., Cheng, G., Han, J.: Scale-aware detailed matching for few-shot aerial image semantic segmentation. IEEE Transactions on Geoscience and Remote Sensing 60, 1–11 (2021)
  * [51] Ye, J., Wu, Z., Feng, J., Yu, T., Kong, L.: Compositional exemplars for in-context learning. arXiv preprint arXiv:2302.05698 (2023)
  * [52] Yu, Q., Gao, Y., Zheng, Y., Zhu, J., Dai, Y., Shi, Y.: Crossover-net: Leveraging vertical-horizontal crossover relation for robust medical image segmentation. Pattern Recognition 113, 107756 (2021)
  * [53] Yu, Y., Yang, X., Li, Q., Zhou, Y., Da, F., Yan, J.: H2rbox-v2: Incorporating symmetry for boosting horizontal box supervised oriented object detection. In: Thirty-seventh Conference on Neural Information Processing Systems (2023)
  * [54] Zendel, O., Schörghuber, M., Rainer, B., Murschitz, M., Beleznai, C.: Unifying panoptic segmentation for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21351–21360 (2022)
  * [55] Zhang, L., Zhang, L., Mou, X., Zhang, D.: Fsim: A feature similarity index for image quality assessment. IEEE transactions on Image Processing 20(8), 2378–2386 (2011)
  * [56] Zhang, Y., Zhou, K., Liu, Z.: What makes good examples for visual in-context learning? arXiv preprint arXiv:2301.13670 (2023)
  * [57] Zhang, Y., Liu, H., Hu, Q.: Transfuse: Fusing transformers and cnns for medical image segmentation. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24. pp. 14–24. Springer (2021)
  * [58] Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022)