# One-shot and Partially-Supervised Cell Image Segmentation
Using Small Visual Prompt

Sota Kato
Meijo University
<EMAIL_ADDRESS>Kazuhiro Hotta
Meijo University
<EMAIL_ADDRESS>
###### Abstract

Semantic segmentation of microscopic cell images using deep learning is an
important technique, however, it requires a large number of images and ground
truth labels for training. To address the above problem, we consider an
efficient learning framework with as little data as possible, and we propose
two types of learning strategies: One-shot segmentation which can learn with
only one training sample, and Partially-supervised segmentation which assigns
annotations to only a part of images. Furthermore, we introduce novel
segmentation methods using the small prompt images inspired by prompt learning
in recent studies. Our proposed methods use a pre-trained model based on only
cell images and teach the information of the prompt pairs to the target image
to be segmented by the attention mechanism, which allows for efficient
learning while reducing the burden of annotation costs. Through experiments
conducted on three types of microscopic cell image datasets, we confirmed that
the proposed method improved the Dice score coefficient (DSC) in comparison
with the conventional methods. Our code is available at
https://github.com/usagisukisuki/Oneshot-Part-CellSegmentation.

##  1 Introduction

Semantic segmentation, which assigns a class label to each pixel in an image,
is a crucial technique for image analysis in the fields of medicine [27, 9,
26, 14] and biology [12, 16]. It has become possible to obtain objective
results automatically by using deep learning and various methods have been
proposed [30, 4, 31, 13, 29, 10]. However, it is necessary to require a large
number of images and ground truth labels when we design a deep learning model.
Particularly, generating ground truth requires the knowledge of human experts
and takes a lot of time and costs.

In recent studies, to tackle the above problem, few-shot segmentation [21, 18,
5, 6] for less training data, zero-shot segmentation [8, 3] for only inference
without training, and semi-supervised segmentation [17, 35], which learns with
a small number of supervised training samples, have been proposed.
Additionally, the novel idea of prompt learning has been gaining popularity in
the field of natural language processing [2]. This idea is an inference method
using large-scale pre-trained models, and it has been reported that it can
achieve higher accuracy than conventional few-shot and zero-shot learning
methods in the field of image recognition [22, 34, 37, 15].

Figure 1: Overview of our proposed strategies. Firstly, we build a pre-trained
model using only cell images. Secondly, we use the pre-trained model and the
small visual prompt to learn one-shot segmentation and partially-supervised
segmentation.

However, these approaches have been trained by natural images. As shown in
Figure 2, it cannot be adapted well to images from other fields such as
biology due to differences in domains between data. Additionally, although
conventional prompt learning methods often use both image and textual
information [20, 34, 37, 22], it is possible that general language models used
by prompt learning cannot deal with specialized terms not included in the
training data. In order to perform prompt learning to the data from different
fields, we consider that it is necessary to use a pre-trained model, which is
more specialized to the field.

Therefore, we propose three strategies for cell image segmentation as shown in
Figure 1. Firstly, we build a pre-trained model using only cell image
datasets. By using the pre-trained model, we can learn more effectively even
when training on another cell image dataset. Secondly, we present novel
strategies for one-shot segmentation and partially-supervised segmentation
employing the above pre-trained model and visual prompts with small regions.
In one-shot segmentation, we assume learning with a single image and propose a
novel learning method that utilizes a prompt image and label of a small
region. In partially-supervised segmentation, we assume that a part of an
image has annotation, and pseudo-labels are predicted using the same framework
as one-shot segmentation, and segmentation network is trained with the pseudo
labels. Since cell images often have a fractal structure in which similar
structures spread over the entire image, we consider that it is possible to
segment the entire cell image by successfully using the information from small
prompt regions. Additionally, since labeling the prompt image with the small
region is also easier than the entire image, it can reduce the annotation cost
of human experts.

We evaluated our methods on three cell image datasets with different shapes.
From the experimental results, we confirmed the effectiveness of the proposed
method in one-shot segmentation as well as the effectiveness of the pre-
training model. Furthermore, the results by the proposed partially-supervised
method and training on generated pseudo-labels demonstrated that the
difference in accuracy between the proposed method and the method using the
original dataset was within about 1.00% for all cell image datasets. We have
also confirmed that our proposed partially-supervised strategy can produce
sufficient supervised labels with only part annotations.

This paper is organized as follows. Section 2 describes related works. Section
3 describes the details of the proposed methods. Section 4 shows the
experimental results. Finally, we describe our summary and future works in
Section 5.

The main contributions of this paper are as follows:

  * •

We build a pre-trained model using only cell image datasets. By using this
pre-trained model for one-shot segmentation and partially-supervised
segmentation, we can create a model with more accuracy even with less data.

  * •

We present a novel method for one-shot segmentation employing the pre-trained
model and visual prompts with small regions. By using this method, we can
achieve efficient learning even if only one training sample is used.

  * •

Furthermore, we present a novel strategy for partially-supervised
segmentation. We can generate pseudo labels from part labels using the same
method as in one-shot segmentation, and we train the model again using these
pseudo labels. By using this approach, we confirmed that the difference in
accuracy between the proposed method and the method using all training samples
in the original dataset was within about 1.00% for all cell image datasets.

(a)

(b)

Figure 2: Visualization results of CLIPSeg [22].

##  2 Related works

###  2.1 One-shot segmentation

Few-shot segmentation [21, 18, 19, 5, 6, 7] is a method that uses a small
amount of training data. A query image of an unknown category, a support image
of the same category, and a corresponding mask image are given as a support
set for both training and inference. The goal is to learn efficiently with a
small number of training data by using the information from the support set.
In the field of biology, [5, 6, 7] have been proposed. Especially, when only
one training image is used, it is called one-shot segmentation [28, 24], which
is a more hard problem than few-shot segmentation. Shaban et al. [28] propose
a two-branched approach to one-shot semantic image segmentation taking
inspiration from few-shot learning, and show significant improvements over the
baselines on this benchmark. Raza et al. [24] present a simple yet effective
approach, whereby exploiting information from base training classes in the
conventional one-shot segmentation set-up allows for weak supervision to be
easily used.

Furthermore, as an even more challenging problem, zero-shot segmentation has
been proposed [8, 3]. It is a method for segmenting unknown categories by
using embedded features of words and similarity between features of the pre-
trained model. Recently, the novel idea of prompt learning for zero-shot
segmentation has been proposed [22].

Although numerous approaches have been proposed, there are a few one-shot
segmentation methods for microscopic biology images. We believe that one-shot
segmentation is more necessary than few-shot segmentation because it reduces
the burden of annotation cost furthermore. Additionally, as shown in Figure 2,
conventional zero-shot segmentation cannot work well due to the differences
between data domains. Therefore, we propose a novel method for one-shot cell
image segmentation that can be improved segmentation accuracy in comparison
with conventional approaches.

###  2.2 Partially-supervised segmentation

Semantic segmentation using deep learning requires a large number of
annotations, and the performance decreases significantly when the number of
annotations is small. However, generating a large number of annotations is a
hardship for human experts. Therefore, semi-supervised learning, which
maintains performance with a small number of annotations, has been attracting
attention [17, 33, 35, 32, 38]. In the field of medical and biological
imaging, [32, 38] have been proposed, and furthermore, a similar problem
setup, which is called partially-supervised segmentation, has also been
proposed in a recent study. In semi-supervised segmentation, the setting is to
use a few pieces of data annotated throughout an image, whereas, in partially-
supervised segmentation, only a part of the image is annotated. Xu et al. [36]
propose an active learning framework. This strategy is used to select the most
uncertain partially-label image based on the probability predicted by the
segmentation model until finishing selecting the maximum number of annotated
images, and the MixUp augmentation is used to learn a proper visual
representation of both the annotated and unannotated images.

We believe that annotating a part of the image is less burden than annotating
the full image in the case of the cell image because it is very hard to
annotate each individual cell. Additionally, cell images often have a fractal
structure and we consider partially-supervised segmentation is effective.
However, the conventional technique for partially-supervised segmentation [36]
is difficult to learn because of too many active learning steps. Our proposed
partially-supervised learning is simple, and it is possible to achieve an even
level of accuracy with the original data with only two training stages.
Further, since our methods for one-shot and partially-supervised segmentations
are nearly identical, our training strategy can be used for multiple tasks.

##  3 Methodology

Figure 3: Overview of the proposed method for one-shot segmentation. Figure 4:
Overview of methods for acquiring final output using the prompt label. Prompt
label is converted to a one-hot label and the class with the highest value is
output as a prediction from the inner product result with the attention map.

In Section 3, we present our novel approach for one-shot and partially-
supervised segmentation for cell images. By using these approaches, it is
possible to learn with fewer annotations by simply preparing a visual prompt
with a small region.

In Section 3.1, we present a detail of the pre-trained model. We present a
novel architecture for one-shot learning in Section 3.2. In Section 3.3, we
present a novel learning strategy using pseudo labels under partially-
supervised segmentation.

###  3.1 Building the pre-trained model

Firstly, we train a model using only cell image datasets. These datasets used
in this study are ISBI2012 [1], ssTEM [11], and iRPE [23] datasets. The
network is U-Net [25] and the details of each dataset and training conditions
are the same in Section 4.1. We build pre-trained models using two datasets
other than the target dataset because of the fairness of the experiments. For
instance, for an evaluation experiment on ISBI2012, the pre-trained model
using only ssTEM and iRPE datasets is adopted.

Consequently, three types of pre-trained models are built in this study. These
models are used in our proposed strategies for one-shot segmentation in
Section 3.2 and partially-supervised segmentation in Section 3.3.

###  3.2 One-shot segmentation

Figure 5: Overview of our proposed strategies for partially-supervised
segmentation.

Figure 3 shows the overview of the proposed method for one-shot segmentation.
In one-shot segmentation, we use three types of input data: the target
microscopic cell image to be segmented, the prompt pairs of the small cell
image, and the corresponding ground truth. The target microscopic cell image
and the prompt cell image are fed into the segmentation network. The weights
of two networks are shared, and the number of channels in output feature maps
is the same as the number of classes in ground truth. Final output features
$\boldsymbol{x}\in\mathbb{R}^{C\times N}$ of the input image to be segmented
and the final output $\boldsymbol{p}\in\mathbb{R}^{C\times M}$ of the prompt
image are an inner product to generate a normalized attention map using a
softmax function as

$\displaystyle\beta_{i,j}=\frac{exp(s_{i,j})/\tau}{\sum_{j}^{M}exp(s_{i,j})/\tau},\>\>\>where\>\>s_{i,j}=\boldsymbol{x_{c,i}}^{\top}\boldsymbol{p_{c,j}}$
(1)

where $\beta_{i,j}$ is the attention map and represents the degree to which
the $j$-th region is associated with the $i$-th region, $C$ is the number of
classes, and $N=H_{t}\times W_{t},M=H_{s}\times W_{s}$ are the number of
feature locations from the final output feature maps. $\tau$ is a temperature
parameter that is to align probabilities for attention maps. By using one-hot
teacher label $\boldsymbol{q}\in\mathbb{R}^{C\times M}$ corresponding to the
prompt cell image, the final output is
$\boldsymbol{o=(o_{1},o_{2},...,o_{i},...,o_{N})}\in\mathbb{R}^{C\times N}$
and can be calculated in Equation (2).

$\displaystyle\boldsymbol{o_{i}}=\sum_{j}^{M}\beta_{i,j}\boldsymbol{q_{j}}^{\top}$
(2)

where $\beta_{i,j}$ is the attention map calculated by the inner product of
$\boldsymbol{x}$ and $\boldsymbol{p}$. We can obtain the similarity between
the output features from the input image and the prompt image in the attention
map. Subsequently, as shown in Figure 4, a final prediction can be performed
by referring to the class labels of output features of the prompt image that
have high similarity to the output feature of the image to be segmented.

We employ the cross entropy loss in equation (3).

$\displaystyle Loss=-\frac{1}{C}\sum_{c=1}^{C}\sum_{i=1}^{H_{t}\times
W_{t}}y_{i}^{c}\log\sigma(o_{i}^{c})$ (3)

where $C$ is the number of classes, $y_{i}^{c}$ is the teacher labels
associated with the input image, and $\sigma(o_{i}^{c})$ is the probability
value after a softmax function as
$\sigma(o_{i}^{c})=\frac{exp(o_{i}^{c})}{\sum_{j}exp(o_{i}^{j})}$. Further,
$o_{i}^{c}$ is the $c$-th element of $\boldsymbol{o_{i}}$, which is a final
output vector of the deep neural network.

Since the proposed method can utilize the information of the prompt image and
their corresponding teacher labels for both training and inference, we
consider that it can create a highly accurate model even with one-shot
segmentation.

###  3.3 Partially-supervised segmentation

Figure 5 shows the overview of our proposed strategy for partially-supervised
segmentation. In partially-supervised segmentation, it is assumed that only a
part of the input cell image is given an annotation, and cannot be trained in
the basic training method for segmentation using deep learning. Therefore, the
proposed strategy consists of two stages. First, the network is trained to
generate pseudo-labels using only part of the label information. Second,
another segmentation network is trained using generated pseudo-labels in a
full-scratch learning.

In the stage of learning to generate pseudo-labels, we crop the part region of
a cell image which is assigned annotation, and feed it with the partial label
into the network as the prompt pairs. Consequently, we use three types of
input to the segmentation network in the same way as in Section 3.2. As in the
proposed one-shot segmentation, the final output is obtained by referring to
the ground truth corresponding to the prompt image using the attention
architecture. Then, the final output is translated a mask label with one-hot
type by the argmax function, and the network trains using the mask label as a
self-supervised learning. Additionally, it trains bringing the output from the
prompt image closer to the prompt label.

We design loss functions in Equation (4)-(6). We use the cross-entropy loss
between the pseudo mask labels and the final predictions in Equation (5), and
the cross-entropy loss between the prediction regions for the prompt image and
the prompt label in Equation (6). As the final loss function, we employ in
combination in Equation (4).

$\displaystyle Loss$ $\displaystyle=$ $\displaystyle
Loss_{pseudo}+Loss_{prompt}$ (4) $\displaystyle Loss_{pseudo}$
$\displaystyle=$
$\displaystyle-\frac{1}{C}\sum_{c=1}^{C}\sum_{i=1}^{H_{t}\times
W_{t}}m_{i}^{c}\log\sigma(o_{i}^{c})$ (5) $\displaystyle Loss_{prompt}$
$\displaystyle=$
$\displaystyle-\frac{1}{C}\sum_{c=1}^{C}\sum_{i=1}^{H_{s}\times
W_{s}}q_{i}^{c}\log\sigma(p_{i}^{c})$ (6)

where $C$ is the number of classes, $m_{i}$ is the pseudo mask label, $q_{i}$
is the prompt label, $\sigma(p_{i}^{c})$ is the probability value from the
input image, and $\sigma(p_{i}^{c})$ is the probability value from the prompt
image after a softmax function as
$\sigma(p_{i}^{c})=\frac{exp(p_{i}^{c})}{\sum_{j}exp(p_{i}^{j})}$, further,
$p_{i}^{c}$ is the $c$-th element of $\boldsymbol{p_{i}}$, which is a final
output vector of the deep neural network. In one-shot segmentation, we do not
use the loss for the prompt image since we use only one prompt pair and there
is a possibility of over-fitting. However, in partially-supervised
segmentation, we add $Loss_{prompt}$ to improve the quality of the pseudo
label because we can use different prompt pairs during learning.

In the stage of training using pseudo-labels, we crop the part region of the
cell image, which is assigned annotation, and feed it with the partial label
into the network as the prompt pairs.

By using the proposed strategy, even when only some of the teachers are given,
the proposed method can extend the information of the sample images to the
whole image by using the attention structure, which enables segmentation of
the whole image.

##  4 Experiments

Table 1: Comparison results for one-shot segmentation.

---
| ISBI2012 [1] | ssTEM [11] | iRPE [23]
DSC | Average | Background | Membrane | Average | Background | Membrane | Average | Background | Membrane
$Original\>\>learning$ |  |  |  |  |  |  |  |  |
U-Net (Full-scratch) | 86.66±0.50 | 94.04±0.15 | 79.28±0.86 | 91.60±0.14 | 96.54±0.05 | 86.65±0.23 | 74.39±0.19 | 84.60±0.34 | 64.17±0.67
U-Net (Pre-trained) | 86.92±0.40 | 94.15±0.20 | 79.70±0.61 | 90.99±0.19 | 96.28±0.12 | 85.70±0.27 | 73.70±0.11 | 84.00±0.21 | 63.41±0.38
$One\mathchar 45\relax shot\>\>learning$ |  |  |  |  |  |  |  |  |
U-Net (Full-scratch) | 78.80±1.73 | 89.83±0.86 | 67.77±2.67 | 78.74±2.46 | 92.54±0.26 | 64.94±4.66 | 59.25±0.05 | 78.96±0.44 | 39.53±0.40
U-Net (Pre-trained) | 81.01±1.86 | 91.51±1.06 | 70.51±2.69 | 82.42±0.71 | 92.71±0.36 | 72.13±1.07 | 53.85±1.44 | 79.77±0.61 | 27.93±2.28
Co-FCN [24] | 78.19±2.34 | 89.87±1.62 | 66.51±3.48 | 83.61±0.24 | 93.14±0.07 | 74.09±0.46 | 54.81±3.57 | 81.88±0.32 | 27.75±7.38
OSLSM [28] | 78.67±1.43 | 90.03±0.58 | 67.32±2.29 | 84.00±0.83 | 93.09±0.50 | 74.91±1.17 | 62.94±0.76 | 79.52±0.72 | 46.36±2.25
FSMICS (one-shot) [7] | 79.85±0.93 | 90.67±0.43 | 69.03±1.42 | 83.42±0.10 | 92.82±0.17 | 74.02±0.30 | 61.14±2.18 | 79.48±0.30 | 42.79±4.65
Ours (Full-scratch) | 80.18±1.26 | 91.11±0.66 | 69.25±1.87 | 83.13±0.26 | 92.30±0.25 | 73.95±0.29 | 64.00±1.33 | 78.85±0.72 | 49.16±2.03
Ours (Pre-trained) | 81.25±1.55 | 91.68±1.04 | 70.81±2.18 | 85.47±0.08 | 94.41±0.01 | 76.53±0.15 | 63.85±1.53 | 78.56±1.69 | 49.15±1.61

Table 2: Comparison results for partially-supervised segmentation.

---
| ISBI2012 [1] | ssTEM [11] | iRPE [23]
DSC | Average | Background | Membrane | Average | Background | Membrane | Average | Background | Membrane
$Original\>\>learning$ |  |  |  |  |  |  |  |  |
U-Net (Full-scratch) | 86.66±0.50 | 94.04±0.15 | 79.28±0.86 | 91.60±0.14 | 96.54±0.05 | 86.65±0.23 | 74.39±0.19 | 84.60±0.34 | 64.17±0.67
U-Net (Pre-trained) | 86.92±0.40 | 94.15±0.20 | 79.70±0.61 | 90.99±0.19 | 96.28±0.12 | 85.70±0.27 | 73.70±0.11 | 84.00±0.21 | 63.41±0.38
$Pseudo\>\>label\>\>learning$ |  |  |  |  |  |  |  |  |
Ours (Full-scratch) | 85.94±0.28 | 94.03±0.13 | 77.85±0.43 | 89.78±0.02 | 95.80±0.05 | 83.76±0.05 | 74.35±0.12 | 85.21±0.10 | 63.49±0.25
Ours (Pre-trained) | 86.27±0.32 | 94.10±0.18 | 78.43±0.46 | 90.53±0.12 | 96.12±0.06 | 84.94±0.18 | 73.30±0.25 | 84.18±0.20 | 62.41±0.71

###  4.1 Datasets

We used 2D electron microscopy images of the ISBI2012 challenge (ISBI2012)
[1], serial sectioning transmission electron microscopy (ssTEM) [11], and
absorbance microscopy images of human iRPE cells (iRPE) [23] as datasets. All
datasets are for binary segmentation of tubular structures spread over an
image, i.e., cell membrane and background. The number of iRPE images is
$1,032$ and the pixel count is $256\times 256$. Since the resolution of ssTEM
image is $1,024\times 1,024$ and the resolution of ISBI2012 image is
$512\times 512$, and we cropped a region of $256\times 256$ pixels from the
original images due to the limitation of GPU memory. There is no overlap for
cropping areas, and consequently, the total number of crops is 320 in ssTEM
and 120 in ISBI2012. Figure 6 shows the examples in the datasets.

We randomly rearranged the images. Afterward, we divided each dataset into 2
to 1 in index order and prepared them as training or inference data, and used
three-fold cross validation while switching the training and inference data.

In the experiment of one-shot segmentation, we used only one fixed training
image from the training data, and in inference, we used the original inference
data. The prompt image was selected randomly from the training data with a
different image other than the training image, and the fixed coordinates were
cropped. Consequently, different prompt pairs were used for each 3-fold cross
validation.

In the experiment of partially-supervised segmentation, all labels in the
training data were masked by filling in zeros except for the fixed
coordinates, and we used the region where the label exists as prompt images.
The size of the prompt was set to $64\times 64$ pixels in all experiments.
Ablations of different prompt sizes are shown in Section 4.3.

(a) (b) (c)
---
Figure 6: Examples of datasets. (a) ISBI2012 dataset [1], (b) ssTEM dataset
[11], and (c) iRPE dataset [23]. All datasets are labeled as: cell membrane
(white) and background (black).

###  4.2 Training conditions

The batch size for training was set to 4, and we used Adam ($betas=0.9,0.999$)
for optimization. The initial learning rate was 0.001, and we used a learning
rate schedule that decays the learning rate by $0.1$ at 180 epoch and again at
190 epoch. For data pre-processing, training samples were flipped
horizontally, rotated with an angle randomly selected within $\theta$ =
$-90^{\circ}$ to $90^{\circ}$, and normalized from zero to one. For inference,
images were normalized from zero to one. All experiments were conducted using
a three-fold cross validation and we applied the Dice score coefficient (DSC)
as evaluation metric. The average DSC from three validations was used for
evaluation. We used a single Nvidia RTX Quadro 8000 GPU as a calculator.

###  4.3 Experimental results

---

---

(a) (b) (c) (d) (e)
---

Figure 7: Qualitative results on one-shot segmentation. The top row is for the
ISBI2012, the middle row is for the ssTEM, and the bottom row is for the iRPE
dataset. (a) Input image, (b) Label image, (c) Original U-Net, (d) Prompt
images ($64\times 64$ pixels), and (e) Ours.

####  4.3.1 One-shot segmentation

Table 1 shows comparison results for one-shot segmentation on the three
datasets. As the comparison method, we employed conventional methods for one-
shot segmentation [24, 28, 7]. Original learning in Table 1 indicates the case
when the model is trained by full training images, and one-shot learning
indicates the case where it is only one training image. Full-scratch in Table
1 indicates the case where the pre-trained model is not used, and pre-trained
indicates the case where it is used. The bold letters show the best DSC. By
using our proposed method, the highest average DSC was achieved for all
datasets. Particularly, when we used the pre-trained model, the average DSC
was improved by over 2.45% for ISBI2012, 6.73% for ssTEM, and 4.75% for iRPE
in comparison with a baseline using the original U-Net. Furthermore, our
method was more improved DSC than conventional approaches for one-hot
segmentation. When the iRPE was used, the model with full-scratch was more
accurate than the model with pre-trained. We consider that this may be because
the iRPE cell image had a different structure compared to the two other cell
images.

Figure 7 shows qualitative comparisons of one-shot segmentation through
visualizations of three types of cell image datasets when we used a full-
scratch learning. As shown in Figure 7, the proposed method could segment the
cell membrane that U-Net could not segment well. These results demonstrated
the effectiveness of our method using the small visual prompt.

Table 3 shows an ablation study of one-shot segmentation when we evaluated
various $\tau$ parameters, which are used in the softmax function of the
attention mechanism, and smaller prompt sizes. We evaluated
$\tau=0.01,0.1,1.0,2.0$ as a temperature parameter for attention maps and
further evaluated $32\times 32$ pixels of the prompt image, which is half size
of $64\times 64$. Comparison results with various temperature parameters
demonstrated that adequate parameters depended on the data, and we consider
that this is influenced by the thickness and complexity of the cell membrane
to be segmented. Our proposed method can be adapted to a variety of cell
images by setting appropriate $\tau$. Furthermore, the accuracy did not
decrease much with smaller prompt sizes. By using smaller prompt images, we
can further reduce the burden on the annotating cost.

####  4.3.2 Partially-supervised segmentation

---

---

(a) (b) (c)
---

Figure 8: Segmentation results on partially-supervised segmentation. The top
row is for the ISBI2012, the middle row is for the ssTEM, and the bottom row
is for the iRPE dataset. (a) Input image, (b) Label image, and (c) Ours.

---

---

(a) (b) (c) (d)
---

Figure 9: Qualitative results on partially-supervised segmentation. The top
row is for the ISBI2012, the middle row is for the ssTEM, and the bottom row
is for the iRPE dataset. (a) Training image, (b) Original label image, (c)
Partial label image ($64\times 64$ pixels), and d) Generated pseudo label
image in Stage 1 of Figure 5. Table 3: Ablation study of one-shot
segmentation.

---
Methods | Prompt size | ISBI2012 [1] | ssTEM [11] | iRPE [23]
$\tau$=0.01 (Full-scratch) | $64\times 64$ | 69.39±12.16 | 81.81±0.82 | 59.01±7.15
$\tau$=0.1 (Full-scratch) | 78.61±1.80 | 83.12±0.39 | 64.00±1.33
$\tau$=1.0 (Full-scratch) | 79.94±1.21 | 83.13±0.26 | 62.73±1.31
$\tau$=2.0 (Full-scratch) | 80.18±1.26 | 83.12±0.41 | 63.31±1.21
$\tau$=0.01 (Pre-trained) | $64\times 64$ | 76.86±4.24 | 85.47±0.08 | 58.88±6.61
$\tau$=0.1 (Pre-trained) | 81.13±2.05 | 85.40±0.09 | 58.13±5.25
$\tau$=1.0 (Pre-trained) | 81.25±1.55 | 81.73±1.09 | 62.24±1.09
$\tau$=2.0 (Pre-trained) | 81.15±1.52 | 84.18±0.37 | 63.85±1.53
$\tau$=0.01 (Full-scratch) | $32\times 32$ | 77.95±1.70 | 82.25±0.19 | 60.19±7.53
$\tau$=0.1 (Full-scratch) | 77.41±2.21 | 82.21±0.05 | 63.26±1.83
$\tau$=1.0 (Full-scratch) | 78.73±1.04 | 82.75±0.38 | 62.60±1.08
$\tau$=2.0 (Full-scratch) | 79.37±1.11 | 82.28±0.27 | 62.00±0.97
$\tau$=0.01 (Pre-trained) | $32\times 32$ | 78.06±3.26 | 82.91±0.28 | 49.66±6.15
$\tau$=0.1 (Pre-trained) | 80.45±0.84 | 82.93±0.29 | 60.17±4.45
$\tau$=1.0 (Pre-trained) | 75.89±4.77 | 81.46±0.67 | 61.75±0.81
$\tau$=2.0 (Pre-trained) | 80.35±0.44 | 75.35±2.73 | 62.84±2.66

Table 2 shows the results of pseudo-label learning for partially-supervised
segmentation on the three datasets. Original learning in Table 2 indicates the
case when the model is trained by a full annotation dataset, and pseudo label
learning indicates the case where it is trained by pseudo labels generated by
our proposed strategy. In addition, full-scratch in Table 2 indicates the case
where the pre-trained model is not used when we generate pseudo-labels, and
pre-trained indicates the case where it is used. All subsequent training with
pseudo labels is done in full scratch. We fixed 2.0 to the value of $\tau$
because we consider it to be a suitable parameter for all three types of
datasets from the results of Table 3. By using our pseudo label, there was
almost no difference in the average DSC compared to results trained using the
original overall label images even though the annotation is only in one part
of the image. Particularly, when we used the pre-trained model, the difference
of average DSC was within $0.65\%$ for ISBI2012, $0.46\%$ for ssTEM, and
$0.40\%$ for iRPE in comparison with a baseline using the original U-Net with
the pre-trained model.

Figure 8 shows segmentation results when the model was trained by our pseudo
labels in Stage 2 of Figure 5. As shown in Figure 8, we can confirm that the
predicted results are almost identical to the correct label images. Figure 9
shows the comparison results between the generated pseudo labels by our method
in Stage 1 of Figure 5 and the correct labels. Even though the hard setting,
where only part of the annotations is attached in the image, the generated
pseudo labels were of largely incorrect quality compared to the original label
images. These results demonstrated that, in the case of cell images, there is
no need to attach an annotation to the entire image, and our proposed method
can be used to reduce the annotation cost.

##  5 Conclusion

In this study, we proposed a novel one-shot segmentation method and a
partially-supervised segmentation method for microscopic cell images.
Experiments on three different cell image datasets demonstrated that our
proposed methods, which is used the pre-trained model and small visual prompt
images, can produce highly accurate models even with a small number of
training data. Furthermore, no need to attach an annotation to the entire
image by using the proposed strategy and our proposed method can be used to
reduce the annotation cost. However, we were only evaluating simple binary
cell image datasets. Therefore, evaluating whether it is equally effective for
multi-class segmentation is our future work. Additionally, we would like to
evaluate zero-shot segmentation using the proposed method.

## Acknowledgements

This work was supported by JSPS KAKENHI Grant Number 22H04735, and Tateishi
Science and Technology Promotion Foundation.

## References

  * [1] Web page of the em segmentation challenge.  http://brainiac2.mit.edu/isbi_challenge/.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, and Patrick Pérez.  Zero-shot semantic segmentation.  Advances in Neural Information Processing Systems, 32, 2019.
  * [4] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III, pages 205–218. Springer, 2023.
  * [5] Sixian Chan, Cheng Huang, Cong Bai, Weilong Ding, and Shengyong Chen.  Res2-unext: a novel deep learning framework for few-shot cell image segmentation.  Multimedia Tools and Applications, 81(10):13275–13288, 2022.
  * [6] Youssef Dawoud, Arij Bouazizi, Katharina Ernst, Gustavo Carneiro, and Vasileios Belagiannis.  Knowing what to label for few shot microscopy image cell segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3568–3577, 2023.
  * [7] Youssef Dawoud, Julia Hornauer, Gustavo Carneiro, and Vasileios Belagiannis.  Few-shot microscopy image cell segmentation.  In Machine Learning and Knowledge Discovery in Databases. Applied Data Science and Demo Track: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part V, pages 139–154. Springer, 2021.
  * [8] Jian Ding, Nan Xue, Gui-Song Xia, and Dengxin Dai.  Decoupling zero-shot semantic segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11583–11592, 2022.
  * [9] Michael Ebner, Guotai Wang, Wenqi Li, Michael Aertsen, Premal A Patel, Rosalind Aughwane, Andrew Melbourne, Tom Doel, Anna L David, Jan Deprest, et al.  An automated localization, segmentation and reconstruction framework for fetal brain mri.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 313–320. Springer, 2018.
  * [10] Haruki Fujii, Hayato Tanaka, Momoko Ikeuchi, and Kazuhiro Hotta.  X-net with different loss functions for cell image segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3793–3800, 2021.
  * [11] Stephan Gerhard, Jan Funke, Julien Martel, Albert Cardona, and Richard Fetter.  Segmented anisotropic sstem dataset of neural tissue.  figshare, pages 0–0, 2013.
  * [12] Simon Graham and Nasir M Rajpoot.  Sams-net: Stain-aware multi-scale network for instance-based nuclei segmentation in histology images.  In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 590–594. IEEE, 2018.
  * [13] Yuki Hiramatsu, Kazuhiro Hotta, Ayako Imanishi, Michiyuki Matsuda, and Kenta Terai.  Cell image segmentation by integrating multiple cnns.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2205–2211, 2018.
  * [14] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen.  Kvasir-seg: A segmented polyp dataset.  In MultiMedia Modeling: 26th International Conference, MMM 2020, Daejeon, South Korea, January 5–8, 2020, Proceedings, Part II 26, pages 451–462. Springer, 2020.
  * [15] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, pages 709–727. Springer, 2022.
  * [16] David Joon Ho, Chichen Fu, Paul Salama, Kenneth W Dunn, and Edward J Delp.  Nuclei segmentation of fluorescence microscopy images using three dimensional convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 82–90, 2017.
  * [17] Donghyeon Kwon and Suha Kwak.  Semi-supervised semantic segmentation with error localization network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9957–9967, 2022.
  * [18] Chunbo Lang, Gong Cheng, Binfei Tu, and Junwei Han.  Learning what not to segment: A new perspective on few-shot segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8057–8067, 2022.
  * [19] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim.  Adaptive prototype learning and allocation for few-shot segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8334–8343, 2021.
  * [20] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang.  Open-world semantic segmentation via contrasting and clustering vision-language embedding.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XX, pages 275–292. Springer, 2022.
  * [21] Weide Liu, Chi Zhang, Guosheng Lin, and Fayao Liu.  Crnet: Cross-reference networks for few-shot segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4165–4173, 2020.
  * [22] Timo Lüddecke and Alexander Ecker.  Image segmentation using text and image prompts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7086–7096, 2022.
  * [23] Michael Majurski, Petru Manescu, Sarala Padi, Nicholas Schaub, Nathan Hotaling, Carl Simon Jr, and Peter Bajcsy.  Cell image segmentation using generative adversarial networks, transfer learning, and augmentations.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [24] Hasnain Raza, Mahdyar Ravanbakhsh, Tassilo Klein, and Moin Nabi.  Weakly supervised one shot segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [25] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [26] Abhijit Guha Roy, Sailesh Conjeti, Nassir Navab, and Christian Wachinger.  Inherent brain segmentation quality control from fully convnet monte carlo sampling.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 664–672. Springer, 2018.
  * [27] Jo Schlemper, Ozan Oktay, Wenjia Bai, Daniel C Castro, Jinming Duan, Chen Qin, Jo V Hajnal, and Daniel Rueckert.  Cardiac mr segmentation from undersampled k-space using deep latent representation learning.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 259–267. Springer, 2018.
  * [28] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots.  One-shot learning for semantic segmentation.  2017\.
  * [29] Eisuke Shibuya and Kazuhiro Hotta.  Cell image segmentation by using feedback and convolutional lstm.  The Visual Computer, 38(11):3791–3801, 2022.
  * [30] Suprosanna Shit, Johannes C Paetzold, Anjany Sekuboyina, Ivan Ezhov, Alexander Unger, Andrey Zhylka, Josien PW Pluim, Ulrich Bauer, and Bjoern H Menze.  cldice-a novel topology-preserving loss function for tubular structure segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16560–16569, 2021.
  * [31] Athanasios Tragakis, Chaitanya Kaul, Roderick Murray-Smith, and Dirk Husmeier.  The fully convolutional transformer for medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3660–3669, 2023.
  * [32] Kuo-Kun Tseng, Ran Zhang, Chien-Ming Chen, and Mohammad Mehedi Hassan.  Dnetunet: a semi-supervised cnn of medical image segmentation for super-computing ai service.  The Journal of Supercomputing, 77:3594–3615, 2021.
  * [33] Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le.  Semi-supervised semantic segmentation using unreliable pseudo-labels.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4248–4257, 2022.
  * [34] Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister.  Learning to prompt for continual learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139–149, 2022.
  * [35] Huisi Wu, Zhaoze Wang, Youyi Song, Lin Yang, and Jing Qin.  Cross-patch dense contrastive learning for semi-supervised segmentation of cellular nuclei in histopathologic images.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11666–11675, 2022.
  * [36] Yanyu Xu, Xinxing Xu, Lei Jin, Shenghua Gao, Rick Siow Mong Goh, Daniel SW Ting, and Yong Liu.  Partially-supervised learning for vessel segmentation in ocular images.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24, pages 271–281. Springer, 2021.
  * [37] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816–16825, 2022.
  * [38] Yanning Zhou, Hao Chen, Huangjing Lin, and Pheng-Ann Heng.  Deep semi-supervised knowledge distillation for overlapping cervical cell instance segmentation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I 23, pages 521–531. Springer, 2020.