.tifpng.pngconvert #1 .tif

[1,3]Zhenguo Gao

These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Department of Computer Science and Technology, Huaqiao University, No.668
Jimei Avenue, Xiamen, 361021, Fujian, China 2]Department of Mechanical
Engineering and Automation, Huaqiao University, No.668 Jimei Avenue, Xiamen,
361021, Fujian, China 3]Key Laboratory of Computer Vision and Machine Learning
of Fujian Provincial Universities, Xiamen, 361021, Fujian, China

# Support-Query Prototype Fusion Network for Few-shot Medical Image
Segmentation

Xiaoxiao Wu<EMAIL_ADDRESS><EMAIL_ADDRESS>Xiaowei Chen<EMAIL_ADDRESS>Yakai Wang<EMAIL_ADDRESS>Shulei Qu<EMAIL_ADDRESS>Na Li
<EMAIL_ADDRESS>*  [  [

###### Abstract

In recent years, deep learning based on Convolutional Neural Networks (CNNs)
has achieved remarkable success in many applications. However, their heavy
reliance on extensive labeled data and limited generalization ability to
unseen classes pose challenges to their suitability for medical image
processing tasks. Few-shot learning, which utilizes a small amount of labeled
data to generalize to unseen classes, has emerged as a critical research area,
attracting substantial attention. Currently, most studies employ a prototype-
based approach, in which prototypical networks are used to construct
prototypes from the support set, guiding the processing of the query set to
obtain the final results. While effective, this approach heavily relies on the
support set while neglecting the query set, resulting in notable disparities
within the model classes. To mitigate this drawback, we propose a novel
Support-Query Prototype Fusion Network (SQPFNet). SQPFNet initially generates
several support prototypes for the foreground areas of the support images,
thus producing a coarse segmentation mask. Subsequently, a query prototype is
constructed based on the coarse segmentation mask, additionally exploiting
pattern information in the query set. Thus, SQPFNet constructs high-quality
support-query fused prototypes, upon which the query image is segmented to
obtain the final refined query mask. Evaluation results on two public
datasets, SABS and CMR, show that SQPFNet achieves state-of-the-art
performance.

###### keywords:

Deep Learning, Few-shot Segmentation, Prototypical Network

##  1 Introduction

Semantic segmentation is a cornerstone task in computer vision, aimed at
pixel-level classification of images by assigning each pixel to a specific
class, such that the image is segmented into several regions. Convolutional
Neural Networks (CNNs) have achieved remarkable success in natural image
analysis, as well as a widespread application in the medical domain. Semantic
segmentation of medical images plays a pivotal role in advancing medical image
analysis, enabling the automatic identification and characterization of
anatomical structures and pathological regions, including tumors, organs,
blood vessels, and lesions. This in turn facilitates accurate diagnosis,
treatment planning, and disease monitoring.

However, semantic segmentation of medical images presents unique challenges
compared to natural image segmentation, as medical images often exhibit low
contrast, high variability, heterogeneity, and noise, thereby intensifying the
complexity of segmentation tasks. Furthermore, the success of CNNs rely on
large amount of well-annotated training data, otherwise their performance will
be seriously degraded. However, being of more paramount importance for medical
image analysis, precise annotations are seriously limited as requiring
expensive labor efforts of high specialized technicians. Hence, the scarcity
of annotated medical image datasets also poses a significant challenge to
medical image analysis.

To overcome the dataset scarcity challenge, few-shot learning is increasingly
attracting research efforts, which aims to quickly learn the underlying
patterns from limited annotated images (in support set) and well adapt to new
images (in query set). Few-Shot Segmentation (FSS) is a typical paradigm of
few-shot learning. FSS consider images from some classes. An image with its
corresponding ground truth segmentation mask is called an annotated image, and
the mask is called the annotation of the image. A pair of an image and its
mask is called an image-annotation pair. As an annotated image is always
accompanied with its mask, the phrase annotated image is usually referred to
the image-annotation pair by default for description clarity. The actual
meaning can be distinguished from the context.

Most existing FSS models in this domain rely on prototypical networks, which
extract class prototypes from the support set for each class and then can be
well applied to segment the query set. However, this approach often results in
an over-reliance on the support set, potentially overlooking valuable
information in the query set. Recent research have explored various methods to
address this issue. For example, Wang et al.[1] focused on adapting to the
differences between the support and query images by including alignment loss.
Ding et al.[2] leveraged pixel-level relationships between the query and
support images using periodic similarity attention module. Additionally,
method proposed by Sun et al.[3] exploited connections between the support and
query sets, while that of Shen et al.[4] allowed continuous prototype
adaptation even in the testing stage, aiming to refine segmentation
predictions. Despite of these efforts, existing approaches often under-exploit
the information in the query set during training. To fill this gap, we
incorporate query features into the prototype construction, enabling to obtain
a support-query fused prototype by utilizing information from both the support
set and the query set. Our approach involves segmenting the foreground area of
the support image into parts and constructing support prototypes for them.
These support prototypes serve as the basis for building a comprehensive class
prototype, not only for the support set but also for the query set. Finally,
we aggregate the heterogeneous prototypes to obtain the final support-query
fused prototype, which is used to obtain a refined segmentation result of the
query image. We call our model as Support-Query Prototype Fusion Network
(SQPFNet). Extensive simulation results on public datasets confirms that, by
leveraging both support and query set information effectively, our SQPFNet
approximately obtain the state-of-the-art performance.

Figure 1: Network structure of SQPFNet.

The contributions of this paper are summarized as follows:

  * •

We proposed an new support-query prototype fusion approach to jointly learns
information from both the support images and the query image. It achieves this
goal by constructing a refined query prototype via exploiting the query image
to refine the comprehensive prototype which combines the heterogeneous
prototypes generated from the segmentation parts in foregrounds of the support
images.

  * •

We proposed SQPFNet following our support-query prototype fusion approach for
few-shot segmentation of medical images.

  * •

Extensive experiments were carried out on two different medical datasets, and
the simulation results verified the superiority of our SQPFNet.

##  2 Related Work

###  2.1 Medical Image Segmentation

UNet proposed by Ronneberger et al.[5] presents a symmetrical encoder-decoder
architecture with skip connections. In the encoder, features are progressively
extracted using convolutional operations, while the decoder combines low-level
encoder features with high-level ones via skip connections, facilitating the
fusion of features across different levels. UNet is regarded as a significant
milestone in medical image segmentation, serving as a foundational framework
for subsequent research endeavors.

To leverage 3D medical image data more effectively, Cicek et al.[6] replaced
UNet’s convolutions with 3D convolutions. Milletari et al.[7] brought residual
connections into UNet to address the gradient vanishing issue, while VoxResNet
proposed in et al.[8] augments UNet with residual structures, enhancing its
capacity to harness deep features. Additionally, Xiao et al.[9] proposed
weighted-ResUNet, which incorporates weighted attention mechanisms and skip
connections to deal with thin blood vessel segmentation and optic disc
recognition in retinal vascular segmentation tasks. Although these 3D networks
achieve improved segmentation performance, their increased model sizes lead to
high computation costs.

For complex segmentation tasks, model cascading offers a solution. In this
approach, multiple segmentation models are integrated using a cascaded
structure, where each model is dedicated to solving a specific sub-task and
generating outputs to feed subsequent models. For instance, Christ et al.[10]
proposed a cascade FCN[11] approach for liver and lesion segmentation, while
Kaluva et al.[12] devised DenseNet, which is a two-stage cascade segmentation
method for liver and tumor segmentation. Moreover, Yan et al.[13] introduced a
multi-scale encoder-decoder cascade model to leverage automatic context
integration of multi-scale information, in order to meet the demands of
higher-resolution segmentation tasks. Although these networks have achieved
promising results, they often require substantial amounts of annotated data,
posing challenges in clinical medical image analysis.

###  2.2 Few-shot Learning

Deep learning has achieved remarkable success across various domains,
including image recognition, natural language processing, and video analysis.
However, the effectiveness of deep learning models heavily depends on large
labeled datasets, which are often scarce in practical applications.
Consequently, research efforts have shifted towards few-shot learning, aiming
to achieve satisfactory results with limited labeled data. Currently, few-shot
learning has three approaches: model-based learning, optimization-based
learning, and metric-based learning.

In model-based approach, techniques like Neural Turing Machines (NTM)[14] are
employed to incorporate short-term memory via external storage and long-term
memory through gradual weight updates. The Meta Network[15] utilizes a fast-
weight mechanism generated from training gradients to adaptively update
parameters during inference.

In optimization-based approach, Ravi et al.[16] recognized that gradient-based
optimization algorithms, especially of non-convex problems with numerous
hyper-parameters, leads to slow convergence. To deal with this issue, they
proposed a model which learns to initialize and update the parameters of
classifier networks on new tasks by learning parameter updating rules.

Metric-based approach addresses over-fitting in low-data scenarios by
employing non-parametric methods, such as nearest neighbors or K-Means, to
construct end-to-end classifiers. Siamese networks[17] leverages cosine
similarity to assess feature similarity, where similarity is computed during
training either between an anchor point and the positive/negative samples or
between pairs of vectors. The prototypical network proposed in [18] projects
samples into a shared space and compute class prototypes by averaging samples
within each class, where distances between samples and class prototypes are
computed during testing using Euclidean distance, and probabilities are
calculated using the softmax function. The class with the highest score is
then regarded as the sample’s category.

###  2.3 Few-shot Semantic Segmentation

In recent years, significant progress has been made in the field of few-shot
semantic segmentation. Shaban et al.[19] pioneered this area by introducing
the concept of few-shot segmentation. Their approach involves a two-arm
structure comprising a conditioning arm and a segmentation arm. The
conditioning arm processes annotated support images to generate feature
representations, whereas the segmentation arm executes the segmentation task.
To further leverage support set information, Wang et al.[1] introduced an
alignment strategy, where an alignment loss is employed to swap the positions
of a support image and a query image during training, enabling the use of the
query image to predict the support image. This approach helps narrow the gap
between the support set and the query set. Additionally, Ouyang et al.[20] and
Hansen et al.[21] respectively introduced the concepts of superpixel and
supervoxel for facilitating segmentation tasks. Shen et al.[4] proposed a
prototype refinement module aimed at continuously updating acquired class
prototypes during testing, leading to more precise predictions. Feng et
al.[22] introduced a spatial branch to incorporate spatial information into
segmentation tasks, integrating this with prototype similarity for improved
predictions.

Most existing methods focus on facilitating interaction between support sets
and query sets. Ding et al.[2] enhanced the non-local block by selectively
boosting query and support features through extensive similarity comparison
between query and support pixels. Sun et al. [3] proposed a global correlation
module to encourage feature clustering, enhancing similarity between support
and query images. Wei et al.[23] introduced a network for embedding anatomical
knowledge, where anatomical prior knowledge from support images is initially
embedded into the model, and then a similarity-guided strategy is utilized to
align query and support sets.

##  3 SQPFNet for Few-Shot Segmentation

###  3.1 Problem Definition

Given a dataset of annotated images from a set of $n_{train}$ classes in
$C_{train}{=}\\{c_{1},c_{2},\ldots,c_{n_{train}}\\}$ classes, FSS aims to
quick adaption to new classes
$C_{test}{=}\\{c_{1},c_{2},\ldots,c_{n_{test}}\\}$, such that it can well
segment images from classes $C_{test}$ when exposed to only a few annotated
sample images from $C_{test}$. Here $C_{train}{\cap}C_{test}{=}\emptyset$ is
required. An FSS model is trained on the training set and evaluated on the
test set.

In FSS, the training and testing are performed in an episodic manner [27]. In
each episode of the training stage, a set of $N$ classes are sampled from
$C_{train}$, and $K$ annotated images from each of the $N$ classes are sampled
from the dataset to create a support set $\mathcal{S}$. Let $C_{S}$ be the set
of $N$ selected classes in this episode. Meanwhile, a query set $\mathcal{Q}$
is also created by selecting an annotated image from a random class
$c{\in}C_{S}$. The testing stage is identical to the training stage, except
that the $N$ classes are selected from $C_{test}$. We refer an image in the
support set $\mathcal{S}$ as a support image, and that in the query set
$\mathcal{Q}$ as a query image. This setting is named as $N$-way-$K$-shot
setting. To be specific, in an $N$-way-$K$-shot setting, we can denote the set
$\mathcal{S}$ as
$\mathcal{S}{=}\\{(x^{s}_{i,k},m^{s}_{i,k}|i{\in}\\{1,2,\ldots,N\\},k{\in}\\{1,2,\ldots,K\\}\\}$
where $x^{s}_{i,k}$ represents the $k$-th image in the support set
$\mathcal{S}$ from the $i$-th class in $c{\in}C_{S}$, and $m^{s}_{i,k}$
represents the corresponding ground truth segmentation mask. Similarly, we can
denote the query set $\mathcal{Q}$ as
$\mathcal{Q}=\\{(x^{q}_{j},m^{q}_{j})\\}$, where $x^{q}_{j}$ and $m^{q}_{j}$,
$j{\in}C_{S}$, represent the query image and its corresponding ground truth
mask, respectively. The model’s task is to learn from the support set
$\mathcal{S}$ and generalize its knowledge to accurately segment the image in
the query set $\mathcal{Q}$.

###  3.2 SQPFNet Structure Overview

Our SQPFNet adopts the prototypical network model approach. For facilitating
the readers, the structure of SQPFNet is re-illustrated in Figure 2.

Figure 2: Network structure of SQPFNet.

The process and the key idea of SQPFNet is as follow. Initially, it extracts
features from both the support and query images using a feature extractor.
Then, it constructs a class prototype for the support set (called as support
prototype) via the prototype generation module. A class prototype for the
query set (called as query prototype) is then generated by exploiting guiding
hues from a coarse query mask predicted using the support prototype. The
support prototype and query prototype are fused to yield the final query
prototype. Ultimately, the query image is segmented by leveraging the final
query prototype, leading to a refined query mask as the segmentation result of
the query image.

###  3.3 Support-Query Prototype Fusion

Many existing medical few-shot segmentation models rely on prototypical
networks. The essence of a prototypical network lies in acquiring the class
information from the support sets and representing them as prototypes. When
both the query sets and prototypes are projected into a vector space, an image
in the query set is categorized based on its proximity to the prototypes. This
metric-based learning approach is both simple and effective, making it
particularly suitable for medical image segmentation due to the significant
disparity between the foreground and the background pixels.

Firstly, the model utilizes a feature extractor ${f_{\theta}}$ to extract
support features ${F_{s}}$ and query features ${F_{q}}$. The dimensions of the
feature map are denoted as ${H^{\prime}}$ for height, ${W^{\prime}}$ for
width, and ${C}$ for the number of channels, respectively. Following the
state-of-the-art approach, we employ the ResNet101 model pre-trained on the
COCO dataset as the feature extractor. The extracted feature is then reduced
to 1/4 of the input size. In practical implementation, we utilized the
torchvision library’s deeplabv3_resnet101 model, yet remove the ASPP (Atrous
Spatial Pyramid Pooling) module.

We employ a prototype-based approach for few-shot segmentation tasks within
this domain. Due to the divergence between the images in the support set and
the query set, constructing a single prototype for the entire support set may
not adequately capture the class information of the support images. Hence, we
opt to partition a support image into different regions and establish a
prototype for each region.

Initially, we identify the foreground region ${R_{n}}$ based on the support
image ${x_{s}}$ and its corresponding mask ${m_{s}}$. We then partition both
the foreground region ${R_{n}}$ and its mask ${M_{n}}$ into ${N_{r}}$
segments, denoted as $\mathcal{R}_{n}{=}\\{r_{i}\\}_{i{=}1}^{{N_{r}}}$ and
$\mathcal{M}_{n}{=}\\{u_{i}\\}_{i{=}1}^{N_{r}}$, respectively. For each
segment, we derive a regional prototype by conducting pixel-level Mask Average
Pooling (MAP) on the support features (features in support images) and masks
within that specific segment. The amalgamation of all regional prototypes
yields the ultimate support prototype ${p_{s}}$, which is calculated using Eq.
(1) where ${\odot}$ denotes Hadamard product.

${p_{s}}=\frac{1}{N_{r}}\sum_{i{=}1}^{N_{r}}\text{MAP}({F_{s}},{u_{i}})=\frac{1}{N_{r}}\sum_{i{=}1}^{N_{r}}{\cdot}\frac{\sum\nolimits_{x,y}{\left(F_{s}(x,y){\odot}u_{i}(x,y)\right)}}{\sum\nolimits_{x,y}u_{i}(x,y)}$
(1)

When calculating the average mask, we sample the support mask of the region
${u_{i}}$ to the size of the support feature ${F_{s}}$, which is
${(C,H^{\prime},W^{\prime})}$.

After obtaining the support prototype ${p_{s}}$, the initial query mask
${\tilde{m}^{q}}$ is derived by computing the cosine similarity between the
support prototype ${p_{s}}$ and the query features ${F_{q}}$ as Eq. (2), where
$\cos$ denotes cosine similarity and ${F_{q}}$ denotes query features.

${\tilde{m}^{q}}=\text{softmax}(\cos({F_{q}},{p_{s}})).$ (2)

For few-shot segmentation tasks, a common challenge arises from intra-class
gaps. Even though the support set and the query set belong to the same
category, variations in imaging sensors can lead to differences, even among
organs of the same category. Direct utilization of the support set’s extracted
prototypes may bring excessive impacts from the support set, resulting in
inaccurate segmentation of query image. To address this issue, we propose
building query prototypes akin to how supporting prototypes are constructed.
The key difference lies in the absence of a genuine query mask for the query
prototype. To compensate for it, we suggest utilize the query mask
${\tilde{m}^{q}}$ derived from the support prototype and selecting a portion
of it as the ground-truth query mask. Despite potential biases in the query
mask obtained from the support prototype, it can be viewed as a coarse query
mask, containing partially relevant information. The query prediction
generated based on the support prototype is ${\tilde{m}^{q}}$, it is
partitioned into two components: the foreground prediction mask
${\tilde{m}_{f}^{q}}$ and the background prediction mask
${\tilde{m}_{b}^{q}}$, where

$\tilde{m}_{b}^{q}=1-\tilde{m}_{f}^{q}.$ (3)

As previously discussed, we employ a partial filtering mechanism for the query
prediction mask utilizing support prototypes. In essence, we adopt a hard
selection method to establish thresholds for both the foreground prediction
mask and the background prediction mask. Typically, the threshold for the
foreground ${T_{f}}$ is set relatively high, whereas the threshold for the
background ${T_{b}}$ is set relatively low. Empirically, we set the background
threshold to 0.5, while the formula for computing the foreground threshold
${T_{f}}$ is as follows

${T_{f}}=\sum\nolimits_{x,y}{\frac{\max\left(\tilde{m}_{f}^{q}(x,y)\right)+\text{mean}\left(\tilde{m}_{f}^{q}(x,y)\right)}{2}}.$
(4)

After foreground threshold ${T_{f}}$ and background threshold are obtained,
the query prediction mask after hard selection ${m^{\prime}}$ can be expressed
as Eq. (5) and Eq. (6), where the subscripts ${f}$ and ${b}$ are used to
indicate foreground and background, respectively.

$m_{f}^{\prime}=\left\\{\begin{array}[]{lr}1,&\hskip 14.22636pt\text{if
}\tilde{m}_{f}^{q}{\geq}{T_{f}};\\\ 0,&\hskip
14.22636pt\text{otherwise}.\end{array}\right.$ (5)

$m_{b}^{\prime}=\left\\{\begin{array}[]{lr}1,&\hskip 14.22636pt\text{If
}\tilde{m}_{b}^{q}{\geq}T_{b};\\\ 0,&\hskip
14.22636pt\text{otherwise}.\end{array}\right.$ (6)

We consider the filtered query prediction mask ${m^{\prime}}$ as the ground-
truth and proceed to obtain the query prototype ${p_{q}}$ in a similar manner
to acquiring the support prototype. Specifically, we perform average pooling
on the combination of the query feature and the query mask after hard
selection ${m^{\prime}}$ using Eq. (7).

${p_{q}}=\frac{{\sum\nolimits_{x,y}{{F_{q}}(x,y)\odot
m^{\prime}(x,y)}}}{\sum\nolimits_{x,y}{m^{\prime}(x,y)}}.$ (7)

We fuse the support prototype ${p_{s}}$ with the query prototype ${p_{q}}$ to
obtain the final prototype ${p_{\text{final}}}$ using Eq. (8), where $\alpha$
and $\beta$ are two parameters for controlling the relative importantce of the
two prototypes.

${p_{\text{final}}}{=}\alpha{\cdot}{p_{s}}{+}\beta{\cdot}{p_{q}}.$ (8)

Utilizing the final prototype ${p_{\text{final}}}$, we obtain the query
foreground prediction mask ${\hat{m}_{f}^{q}}$ using Eq. (9).

$\hat{m}_{f}^{q}=\text{softmax}(\cos(p_{final},F_{q}))$ (9)

##  4 Experiment

###  4.1 Dataset

To validate the effectiveness of our method, we conducted experiments on two
representative publicly available datasets: SABS [24] and CMR[25]. The SABS
dataset originates from the Multi-Atlas Abdominal Label Challenge released by
MICCAI2015. It comprises CT scans of the abdomen with multi-organ
segmentation, featuring organs such as the pancreas, kidneys (left and right),
gallbladder, adrenal glands (left and right), liver, and inferior vena cava,
among others. This dataset consists of 30 3D CT scans. The CMR dataset, on the
other hand, is a cardiac segmentation dataset published by MICCAI in 2019. It
encompasses 35 scans, each with approximately 13 slices per scan.

To ensure fair comparison with other methods, we employ a consistent setting
by adjusting all images to either 2D axial or 2D short-axis. Specifically, for
the SABS dataset, we convert it to a 2D axial representation, while for the
CMR dataset, we transform it to a 2D short-axial format. Subsequently, all
adjusted images are resize to 256×256 size. Focusing on the SABS dataset, akin
to prior approaches, we concentrate our analysis on the model’s performance
concerning the kidneys, liver, and spleen. Furthermore, all experiments follow
a five-fold cross-validation methodology.

Table 1: Comparison results of SQPFNet with other methods on the SABS dataset. Settings | Methods | LK | RK | Spleen | Liver | Mean Dice Score$\uparrow$
---|---|---|---|---|---|---
1 | ALPNet[20] | 72.36 | 71.81 | 70.96 | 78.29 | 73.35
ADNet[21] | 68.76 | 61.18 | 68.10 | 79.49 | 69.38
QNet[4] | 74.14 | 74.04 | 78.32 | 80.70 | 76.80
AAS-DCL[26] | 74.58 | 73.19 | 72.30 | 78.04 | 74.52
CRAPNet[2] | 74.69 | 74.18 | 70.37 | 75.41 | 73.66
SQPFNet(Ours) | 74.88 | 75.10 | 73.20 | 84.79 | 77.00
2 | ALPNet[20] | 63.34 | 54.82 | 60.25 | 73.65 | 63.02
ADNet[21] | 36.52 | 37.60 | 46.10 | 76.25 | 49.12
QNet[4] | 67.73 | 62.11 | 62.35 | 81.45 | 68.41
AAS-DCL[26] | 64.71 | 69.95 | 66.36 | 71.61 | 68.16
CRAPNet[2] | 53.31 | 63.97 | 60.25 | 78.22 | 63.94
SQPFNet(Ours) | 68.93 | 66.99 | 62.08 | 81.47 | 69.87
Table 2: Comparison results of SQPFNet with other methods on the CMR dataset. Settings | Methods | LV-BP | LV-MYO | RV | Mean Dice Score$\uparrow$
---|---|---|---|---|---
1 | ALPNet[20] | 83.99 | 66.74 | 79.96 | 76.90
ADNet[21] | 86.62 | 62.41 | 74.93 | 74.65
Q-Net[4] | 89.23 | 64.42 | 75.70 | 76.45
SQPFNet(Ours) | 89.97 | 64.96 | 77.29 | 77.41

###  4.2 Evaluate Metric

In accordance with common medical segmentation practices, we employ the
average Dice coefficient as an evaluation metric. The Dice coefficient,
ranging between 0 and 100, indicates the degree of overlap between the
segmented region and the ground truth. A score of 0 implies complete
disjointedness between the segmentation result and the actual mask, while a
score of 100 signifies perfect alignment between the segmentation result and
the ground truth mask. The Dice score is calculated using the following
formula:

${\rm{D}}(A,B)=2\frac{{|A\cap B|}}{{|A|+|B|}}\cdot 100\%$ (10)

###  4.3 Comparison of SQPFNet with State-of-the-Art Methods

ALPNet[20]

ADNet[21]

Q-Net[4]

CRAPNet[2]

SQPFNet(Ours)

GT

Figure 3: Segmentation results using Setting1 on the SABS dataset. From left
to right: ALPNet, ADNet, Q-Net, CRAPNet, SQPFNet (Ours), and Ground Truth
(GT). From top to bottom: Left Kidney (LK), Right Kidney (RK), Spleen, and
Liver.

ALPNet[20]

ADNet[21]

Q-Net[4]

CRAPNet[2]

SQPFNet(Ours)

GT

Figure 4: Segmentation results using Setting2 on the SABS dataset. From left
to right: ALPNet, ADNet, Q-Net, CRAPNet, SQPFNet (Ours), and GT(Ground Truth).
From top to bottom: Left Kidney (LK), Right Kidney (RK), Spleen, and Liver.

ADNet[21]

Q-Net[4]

CRAPNet[2]

SQPFNet(Ours)

GT

Figure 5: Comparison of segmentation results from Setting1 on CMR dataset.
From left to right: ADNet, Q-Net, CRAPNet, SQPFNet(Ours) and GT(Ground Truth).
From top to bottom: LY-BP, LV-MYO and RV.

We conducted comparisons of our methods with ALPNet[20], ADNet[21], QNet[4],
AAS-DCL[26], and CRAPNet[2] on both the SABS[24] and CMR[25] datasets. To
ensure a fair comparison, we evaluated them under two settings: Setting 1,
where the test class may appear during the training phase, and Setting 2,
where the test class is not visible to the model at all, following the
approach of SSL-APLNet. For the CMR datasets, since organs consistently appear
together, it’s not feasible to exclude the test class from the model’s
training stage. Therefore, we only consider Setting 1 for the CMR datasets.

Table 1 presents the model and other methods’ performance on the SABS dataset,
while Table 2 displays their performance on the CMR dataset. Our model
exhibits superior performance across both settings, as illustrated in Table 1.
Specifically, under Setting 1, our model achieves an average Dice score of
77.00%. Notably, in liver organ segmentation, our model outperforms ALPNet and
QNet by 6.50% and 4.09%, respectively.

Figure 3 illustrates the specific segmentation results on the SABS dataset
under Setting 1. Our model demonstrates higher accuracy in segmentation and
effectively reduces unnecessary segmentation. Even under Setting 2, where the
test class is entirely invisible to the model, our model performs well, as
depicted in Figure 4. Given that SABS is a multi-organ dataset with many
organs of small size, precise segmentation proves challenging. However, our
model excels, particularly in the scenario of completely invisible classes,
which closely resembles real-world conditions. Figure 4 highlights our model’s
ability to accurately segment very small target organs, such as the right
kidney. In contrast, the ADNet method fails to produce effective segmentation
predictions, and other methods exhibit varying degrees of over-segmentation.
On the CMR dataset, as shown in Table 2, our model generally outperforms
others.

Figure 5 provides a visual representation of our model’s segmentation,
indicating its superior accuracy, particularly in the right ventricle (RV)
organ. While other methods display complete segmentation, our segmentation
aligns more closely with the real mask.

###  4.4 Ablation Study

In order to verify the effectiveness of our proposed method, we conducted
ablation experiments on different modules, and the relevant experimental
results are shown in Table 3. All ablation experiments were conducted on the
SABS dataset at setting 2, and we also listed the results of 5 cross-
validation sets in Table 3.

We first consider the impact of supporting prototypes on model performance.
The traditional way is to build a class prototype for each class. The way we
use is to divide the foreground area, build a class prototype for each part,
and then average the prototypes of different parts to get the final class
prototype. We call our approach multiple supporting prototypes. As you can see
from the table, building multiple support prototypes is a significant
improvement over building a single support prototype, with an increase of
2.68%. In addition, we built a query prototype following the way we built a
support prototype. As you can see from the table, building query prototypes on
top of multiple supported prototypes improved the performance of the model by
0.9%. Finally, through the combination of multiple support prototypes and
query prototypes, the performance of the model has been greatly improved
compared with the single support prototype, from 66.41% to 69.87%.

The predictions produced by the different prototypes are shown in Figure 6.
Among them, in the segmentation of liver organs, it can be seen that the use
of a single support prototype cannot produce a comprehensive segmentation, and
the use of multiple support prototypes has significantly improved the effect,
and the combination of multiple support prototypes and query prototypes can
produce a more comprehensive segmentation. On the left kidney, the use of a
single supporting prototype will produce a relatively large amount of over-
segmentation, which will be reduced after the introduction of the query
prototype, but some pixels are still over-segmented. However, the combination
of multiple support prototypes and query prototypes can avoid over-
segmentation.

Table 3: Ablation study results of different modules on the SABS dataset under
setting 2.

SSP | MSP | QP | Flod1 | Fold2 | Fold3 | Fold4 | Fold5 | Mean Dice Score$\uparrow$
---|---|---|---|---|---|---|---|---
✓ |  |  | 68.95 | 55.24 | 65.79 | 75.39 | 66.67 | 66.41
| ✓ |  | 69.65 | 59.95 | 68.76 | 73.57 | 73.54 | 69.09
✓ |  | ✓ | 68.63 | 56.74 | 67.69 | 73.56 | 65.83 | 66.49
| ✓ | ✓ | 68.57 | 61.94 | 72.04 | 72.99 | 73.80 | 69.87

SupportImg

QueryImg

SSP

MSP

SSP+QP

MSP+QP

GT

Figure 6: Predictions generated by various prototypes. From left to right:
support image, query image, predictions of SSP, MSP, SSP+QP, MSP+QP, and GT
(Ground Truth), where SSP denotes single support prototype, MSP denotes
multiple support prototypes, QP denotes query prototype, and GT denotes Ground
Truth.

Furthermore, we conducted an ablation experiment on the combination
coefficient ${\alpha}$, which represents the balance between the support
prototype and the query prototype. The results are depicted in Figure 7. It’s
evident from the figure that as ${\alpha}$ decreases, indicating a gradual
introduction of the query prototype, the model’s performance improves. The
optimal performance of 69.87% is achieved when ${\alpha}$ is set to 0.5.
However, further decreasing ${\alpha}$ beyond this point leads to a decline in
performance, reaching a sub-optimal level of 68.56% when ${\alpha}$ is reduced
to 0.1.

Figure 7: Ablation study results of ${\alpha}$.

Moreover, we compared the number of model parameters and floating-point
calculation times across different methods, with a unified input size of
256×256. The experimental results are summarized in Table 4. Given the
similarity in the model backbone across methods, the number of parameters is
comparable. Notably, our model stands out as a relatively lightweight model.
Despite the introduction of the query prototype, our model achieves improved
performance without increasing the number of model parameters.

Table 4: Model size and computational complexity of various models. Methods |  Parameters$\downarrow$ |  ${\approx}$ GFLOPs$\downarrow$
---|---|---
ALPNet[20] |  43.02 M |  90.17442
ADNet[21] |  43.02 M |  90.17439
QNet[4] |  46.12 M |  266.86460
CRAPNet[2] |  44.01 M |  91.51657
SQPFNet(Ours) |  43.02 M |  90.17439

##  5 Conclusion

In this paper, we propose a new Support-Query Prototype Fusion network
(SQPFNet) to alleviate the problems of intra-class and inter-class gaps in
few-shot segmentation. Our model first divides the foreground area of the
support picture into multiple parts, constructs prototypes for each part, and
then averages the multiple part prototypes to get the support prototypes and
the coarse mask predictions generated by the support prototypes. Then, the
query prototype is constructed according to the pattern of the prototypical
network by using the query features and the segmented coarse mask. Ultimately,
we implement weighted combination support and query prototypes to build high
quality prototypes and complete the segmentation task.

##  6 Declarations

###  6.1 Conflict of interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

###  6.2 Ethical Approval

Not applicable.

###  6.3 Funding

Natural Science Foundation of China under Grant 62372190 and Industry
University Cooperation Project of Fujian Province under Grant 2021H6030.

###  6.4 Availability of data and materials

The datasets we use are all public open-source data, they can be obtained from
[24, 25]

## References

  * [1] Wang K, Liew J. H, Zou Y, et al. PaNet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9197–9206 (2019).
  * [2] Ding H, Sun C, Tang H, et al. Few-shot medical image segmentation with cycle-resemblance attention. in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2488-2497 (2023).
  * [3] Sun, L., Li, C., Ding, X., Huang, Y., Chen, Z., Wang, G., and Paisley, J. Few-shot medical image segmentation using a global correlation network with discriminative embedding. Computers in Biology and Medicine, 2022, 140, 105067.
  * [4] Shen Q, Li Y, Jin J, et al. Q-Net: Query-informed few-shot medical image segmentation. ArXiv:2208.11451. (2022).
  * [5] Ronneberger, O.; Fischer, P.; and Brox, T. U-Net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 234–241 (2015).
  * [6] Çiçek Ö, Abdulkadir A, Lienkamp S, et al. 3d U-Net: learning dense volumetric segmentation from sparse annotation. In: Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 424–432 (2016)
  * [7] Milletari F, Navab N, Ahmadi S.-A. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In Proceedings of the Fourth International Conference on 3D Vision (3DV), pp. 565–571 (2016).
  * [8] Chen H, Dou Q, Yu L,et al. VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images. NeuroImage, 2018, 170: 446–455.
  * [9] Xiao X, Lian S, Luo Z, et al. 2018\. Weighted Res-UNet for high-quality retina vessel segmentation. In Proceedings of the 9th International Conference on Information Technology in Medicine and Education (ITME), pp.327-331 (2018).
  * [10] Christ P, Elshaer M, Ettlinger F, et al. Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 415–423 (2016).
  * [11] Long. J, Shelhamer. E, and Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 3431–3440 (2015).
  * [12] Kaluva K C, Khened M, Kori A, et al. 2D-densely connected convolution neural networks for automatic liver and tumor segmentation. Arxiv:1802.02182, 2018.
  * [13] Yan Y, Conze. P, Decencière. E, et al. Cascaded multi-scale convolutional encoder-decoders for breast mass segmentation in high-resolution mammograms. In Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). pp. 6738–6741 (2019)
  * [14] Santoro A, Bartunov S, Botvinick M, et al. Meta-learning with memory-augmented neural networks. In Proceedings of Machine Learning Research(PMLR), pp. 1842-1850 (2016).
  * [15] Munkhdalai T, Yu H. Meta networks. In Proceedings of Machine Learning Research(PMLR), pp. 2554-2563 (2017).
  * [16] Ravi S, Larochelle H. Optimization as a model for few-shot learning. In Proceedings of the International Conference on Learning Representations, (2016).
  * [17] Koch G, Zemel R, Salakhutdinov R. Siamese neural networks for one-shot image recognition. In Proceedings of the ICML Deep Learning Workshop. pp. 1-8 (2015).
  * [18] Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems, 2017, 4077-4087.
  * [19] Shaban A, Bansal S, Liu Z, et al. One-shot learning for semantic segmentation. ArXiv:1709.03410. (2017)
  * [20] Ouyang C, Biffi C, Chen C, et al. Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. In Proceedings of the European Conference on Computer Vision(ECCV), pp. 762–780 (2020).
  * [21] Hansen S, Gautam S, Jenssen R, et al. Anomaly detection-inspired few-shot medical image segmentation through self-supervision with supervoxels. Medical Image Analysis, 2022, 78: 102385.
  * [22] Feng Y, Wang Y, Li H, et al. Learning what and where to segment: A new perspective on medical image few-shot segmentation. Medical Image Analysis, 2023, 87: 102834.
  * [23] Wei Y, Tian J, Zhong C, et al. AKFNET: An anatomical knowledge embedded few-shot network for medical image segmentation. In Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP), PP. 11-15 (2021).
  * [24] Landman B, Xu Z, Igelsias J, et al. Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge, In Proceedings of MICCAI Workshop. 5: 12 (2015).
  * [25] Zhuang X. Multivariate mixture model for cardiac segmentation from multi-sequence MRI. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 581–588 (2016).
  * [26] Wu H, Xiao F, and Liang C. Dual contrastive learning with anatomical auxiliary supervision for few-shot medical image segmentation. In Proceedings of the European Conference on Computer Vision(ECCV), pp. 417–434 (2022).
  * [27] Vinyals O, Blundell C, Lillicrap T, Wierstra D, et al. Matching networks for one shot learning, In Proceedings of Advances in neural information processing systems, pp. 3630–3638 (2016).