11institutetext: Faculty of Engineering, Tel Aviv University22institutetext:
Icahn school of Medicine, Mount Sinai, NY

# ProtoSAM: One Shot Medical Image Segmentation With Foundational Models

Lev Ayzenberg 11 Raja Giryes 11 Hayit Greenspan 1122

###### Abstract

This work introduces a new framework, ProtoSAM, for one-shot medical image
segmentation. It combines the use of prototypical networks, known for few-shot
segmentation, with SAM – a natural image foundation model. The method proposed
creates an initial coarse segmentation mask using the ALPnet prototypical
network, augmented with a DINOv2 encoder. Following the extraction of an
initial mask, prompts are extracted, such as points and bounding boxes, which
are then input into the Segment Anything Model (SAM). State-of-the-art results
are shown on several medical image datasets and demonstrate automated
segmentation capabilities using a single image example (one shot) with no need
for fine-tuning of the foundation model. Our code is available at:
https://github.com/levayz/ProtoSAM/

###### Keywords:

One Shot Segmentation Medical Images Foundational Models. SAM Prototypical
Networks

##  1 Introduction

Deep learning models are now the go-to method for segmenting medical images.
However, using these models usually requires a large set of manually labeled
data, which is expensive and time-consuming. Additionally, these models
struggle with new categories they have not seen before, requiring more
training and adjustments. Few-shot segmentation (FSS) has been introduced as a
solution to these problems. FSS trains the model to learn from just a few
labeled examples, reducing the need for large, manually labeled datasets.
Prototypical Networks (PN) [23] are widely used for FSS. These networks use
prototypes, which represent the key features of different classes, to make
predictions based on similarity. One notable method is ALPNet [17, 18], which
is the leading FSS approach for medical data. Its key innovations are the
Adaptive Local Prototypes Pooling module (ALP), which helps to better capture
fine details in medical images, and utilizing superpixels as labels for
training.

Recently, a segmentation foundational model, Segment Anything Model (SAM)
[12], was introduced. This model was trained on a curated dataset of 1B masks
and 11M natural images. In our work, we aim to implement SAM for medical image
segmentation. We hypothesize that with appropriately designed prompts, SAM
could serve as an effective one-shot segmentation model for medical images, as
we show in Section 5. We demonstrate the performance of ProtoSAM on the CHAOS
[11] and MICCAI 2015 Multi-Atlas Abdomen Labeling challenge [14] datasets and
show SOTA results. We also demonstrate the advantages of our strategy on
several polyp datasets.

Figure 1: ProtoSAM Framework: A DINOv2 encoder derives features from query and
support images. Foreground and background prototypes are crafted from support
features and masked through the ALP module. Initial segmentation is achieved
by comparing these prototypes with query features using cosine similarity. The
system extracts prompts from an initial prediction to guide the SAM model for
enhanced segmentation.

##  2 Related Work

Few Shot Segmentation. In the supervised setting of medical image
segmentation, deep learning models are trained to predict labels for each
pixel in the input images. This process typically starts from scratch for each
new task, requiring extensive annotated datasets. Few-shot segmentation offers
a more practical solution in this context, enabling models to perform well
with limited annotated data, typically referred to as the support set [7, 13].
Prototypical Networks (PN) [23] are a popular choice for addressing few-shot
learning tasks. They focus on exploiting representation prototypes of semantic
classes extracted from the support. These prototypes are utilized to make
similarity-based predictions.
ALPNet [17, 18] is a novel PN approach that introduced the Adaptive Local
Prototypes (ALP) module, which is capable of generating prototypes at both
local and class-wide levels. Additionally, they proposed a self-supervised
learning strategy that employs superpixels as labels for training in a
supervised manner.
Foundational Models are large-scale, pre-trained models that serve as a base
for various downstream tasks, typically offering strong generalization
capabilities. Recently, several vision foundational models have been
introduced, demonstrating significant advancements in the field [19]. DINOv2
[16] is a recent Self Supervised Learning framework based on vision
transformers (ViT) [5]. It was trained on a large curated dataset of natural
images and exhibits strong feature extraction capabilities. Throughout this
paper, the term "DINOv2" refers to the ViT that has been pretrained using the
DINOv2 framework. In [2], the advantage of substituting the encoder in the
ALPNet framework with a DINOv2 encoder was demonstrated. In this work, we take
an alternative strategy and use ALPNet with DINOv2 for box and point proposals
for the Segment Anything Model (SAM) [12], which is a foundational model for
segmentation. SAM was trained on a large curated set consisting of 1B masks
and 11M natural images. SAM supports fully automatic and prompt-based (points
and bounding boxes) segmentation. Various adaptation schemes of SAM for
medical data have been proposed [21, 27, 26]. Recently, MedSAM was introduced
[15], which fine-tuned the SAM model for medical images on over 1 million
image-mask pairs across 15 different modalities. Yet, unlike our work, MedSAM
requires getting bounding box prompts.

In [1], we have employed DINOv2 with ALPNet, showing that using the features
generated by DINOv2 led to better segmentation performance on CT and MRI
datasets. In this work, we extend our earlier work to support also the use of
SAM in an automatic manner, requiring only a single support image and mask.
Furthermore, we expand our evaluation to include polyp datasets.

##  3 Method

Problem Formulation. In one-shot segmentation, the goal is to train a
function, $f(\mathbf{x}^{q},S)$, capable of generating a binary mask for an
unseen class $c$ when provided with a support set, $S$, and a query image,
$\mathbf{x}^{q}$. The support set $S$, comprises pairs of images and their
annotated masks ${(\mathbf{x}^{s}_{i}(c),\mathbf{y}^{s}_{i}(c))_{i=1}^{k},c\in
C_{test}}$. Since we are dealing with one-shot learning, we set $k=1$. Here,
$\mathbf{x}^{s}$ represents the image in the support image set, and
$\mathbf{y}^{s}(c)$ represents the annotated mask of the support image
corresponding to the unseen class $c$. For training, we split the data into a
training dataset, $D_{train}$, and a testing dataset, $D_{test}$. These
datasets are comprised of image-binary mask pairs, with $D_{train}$
annotations belonging to $C_{train}$ and $D_{test}$ annotations belonging to
$C_{test}$. It is important to note that $C_{train}\cap C_{test}=\emptyset$.

The training process of FSS networks is comprised of episodes. An episode
consists of a pair of a support set and a query image, $\langle
S,\mathbf{x}^{q}\rangle$, where $S$ is a subset of the training dataset
$D_{train}$ ($S\subset D_{train}$), and $\mathbf{x}^{q}$ is a query image. It
is crucial to note that the query mask $\mathbf{y}^{q}(c)$ is utilized solely
for training purposes as a ground-truth label. Each episode is randomly
selected from $D_{train}$, with the support set $S$ containing an image-mask
pair for a specific class $c$. Given that the problem involves $n$ classes
from $C_{test}$, each episode is an "n-way k-shot segmentation sub-problem."
In this work, we focus on 1-way 1-shot learning.

Figure 1 provides an overview of ProtoSAM. We turn to detail each stage:

Initial Segmentation Stage. For the initial segmentation stage we employ the
ALPNet [17] framework, but we replace the encoder with a DINOv2 [16] encoder.
The purpose of this stage is to extract a coarse segmentation mask and logits
from the query image, based on the support set. We encode the support and
query image using the encoder, $f_{\theta}(.)$ and extract the support feature
map, $f_{\theta}(\mathbf{x}^{s})\in\mathbb{R}^{D\times H\times W}$, and the
query feature map $f_{\theta}(\mathbf{x}^{q})\in\mathbb{R}^{D\times H\times
W}$, where $D$ is the number of channels, and $H,W$ are the feature map
spatial dimensions. The ALP module extracts local prototypes from
$f_{\theta}(\mathbf{x}^{s})$ by average pooling features for the support
feature map using a sliding window. We define a local prototype at location
$(m,n)$ for class $c$ as:

$P(m,n;c)=\frac{1}{L_{H}L_{W}}\sum_{h}\sum_{w}f_{\theta}(\mathbf{x}^{s}(c)(h,w),$
(1)

where $(L_{H},L_{W})$ are the sliding window dimensions and $mL_{H}\leq
h<(m+1)L_{H},\ nL_{W}\leq w<(n+1)L_{W}$.

Additionally, we extract a global prototype defined as:

$P^{g}(c)=\frac{\sum\limits_{h,w}\mathbf{y}_{i}^{s}(c)(h,w)f_{\theta}(\mathbf{x}^{s}(c))(h,w)}{\sum\limits_{h,w}\mathbf{y}^{s}(c)(h,w)}$
(2)

For cases where the foreground class may be much smaller than the pooling
window. For each class (foreground or background), the prototypes are
aggregated into a set $\mathbb{P}_{c}=\\{P_{l}(c)\\}$. A similarity map is
generated between each prototype of class $c$ and the query feature map,
$f_{\theta}(\mathbf{x}^{q})$, using:

$S_{l}(c^{j})(h,w)=\alpha\cdot P_{l}(c)\odot f_{\theta}(\mathbf{x}^{q})(h,w)$
(3)

Where $\odot$ denotes cosine similarity, and $l$, the index for the prototype.
We set $\alpha=20$ as in [17]. These similarity maps are merged into a class-
wise similarity map using:

$S^{\prime}(c)(h,w)=\underset{l}{\sum}S_{l}(c)(h,w)\cdot\underset{l}{\text{softmax}}[S_{l}(c)(h,w)]$
(4)

We normalize the similarity maps for the foreground and background class into
probabilities using:

$\mathbf{\hat{y}}^{q}(h,w)=\underset{c}{\text{softmax}}[S^{\prime}(c)(h,w)]$
(5)

It is important to note that we do not perform any fine-tuning unless
explicitly stated otherwise, and rely solely on the feature extraction
capabilities of the DINOv2 encoder.

Prompt Extraction. This stage receives as input the initial prediction
probabilities. From these, we extract prompts in the form of points and
bounding boxes for SAM [12]. Using connected component analysis, we first
extract the most confident connected component from the initial prediction
using the following formula to rank the prediction confidence of each
connected component:

$\text{Confidence}=\frac{\sum_{i}p_{i}\cdot\mathbf{\hat{y}}_{i}}{\sum_{i}\mathbf{\hat{y}}_{i}}$
(6)

where $p_{i}$ is the probability that pixel $i$ belongs to the foreground
class, $\mathbf{\hat{y}}_{i}$ is the $i$-th pixel’s predicted label. From this
point on, we only consider the probabilities and mask corresponding to that
connected component. We extract the following prompts from the predicted
logits and mask: bounding box (Bbox), centroid points (Cent) and confidence-
based points (Conf). We set the bounding box to be the bounding box of the
connected component, the centroid point is selected to be the centroid point
of the connected component, and for the confidence-based point, we choose the
point corresponding to the highest probability of the foreground class inside
the connected component.

Final Segmentation. The prompts of the previous stage are fed into the Segment
Anything Model (SAM) which produces the final finer segmentation map.

Optional: Encoder fine-tuning (EFT). To enhance performance, it is possible to
fine-tune the DINOv2 encoder on the target data. For that, we employ the
ALPNet framework [17]. We emulate real-life scenarios by constructing
episodes, comprised of a support set and a query set. A preprocessing step is
taken, where we generate superpixels using Felzenszwalb [6] for all the
available slices. At each episode an image $\mathbf{x}$ is chosen from the
train-set, together with a random superpixel $\mathbf{y}^{r}(c^{p})$, which is
used as the pseudo-label. This forms the support set
$S={(\mathbf{x},\mathbf{y}^{r}(c^{p}))}$, where $c^{p}$ denotes the superpixel
class and $r$ is the index of the random superpixel. The query set is formed
by augmenting the chosen image $\mathbf{x}$, i.e,
$Q_{i}={(\mathcal{T}_{g}(\mathcal{T}_{I}(\mathbf{x}}))$, where
$\mathcal{T}_{g}$ and $\mathcal{T}_{I}$ are geometric and intensity transforms
respectively. The loss is

$\displaystyle\mathcal{L}^{i}_{\text{seg}}(\theta;S,Q)=-\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}\sum_{c\in\\{c^{0},c^{p}\\}}\mathcal{T}_{g}(\mathbf{y}^{r}(c))(h,w)\log(\hat{\mathbf{y}}^{r}(c)(h,w)),$
(7)

where $\hat{\mathbf{y}}^{r}(c^{p})$ is the prediction of the pseudolabel,
$c^{0}$ is the background class and $\theta$ \- the model parameters. Also, as
in [17], we incorporate the prototype alignment regularization [25], where the
roles of the support label and the prediction are reversed. The prediction
assumes the role of the support label, and our aim is to segment the original
superpixel accordingly. The regularization loss is

$\displaystyle\mathcal{L}_{\text{reg}}(\theta;\mathcal{S}^{\prime},\mathcal{S})=-\frac{1}{HW}\sum_{h=1}^{H}\sum_{w=1}^{W}\sum_{c\in\\{c^{0},c^{p}\\}}\mathbf{y}^{r}(c)(h,w)\log(\bar{\mathbf{y}}^{r}(c)(h,w)),$
(8)

where $S^{\prime}=(\mathbf{x},\hat{\mathbf{y}}^{r}(c^{p}))$, and
$\bar{\mathbf{y}}^{r}(c^{p})$ is the prediction of the superpixel label
$\mathbf{y}^{r}(c^{p})$.

##  4 Experiments

Datasets. We employ two datasets for abdominal organ segmentation, each
associated with a different modality (CT and MRI), and four polyp datasets.
The first two datasets include (i) Abd-CT: Derived from the MICCAI 2015 Multi-
Atlas Abdomen Labeling challenge [14], containing 30 3D abdominal CT scans and
(ii) Abd-MRI: Sourced from the ISBI 2019 Combined Healthy Abdominal Organ
Segmentation Challenge (Task 5) [11], comprising 20 3D T2-SPIR MRI scans. The
poylp datasets are: Kvasir-SEG [10], ClinicDB [3], ColonDB [24] and ETIS [22].
The test splits of each dataset are as follows: 100 images from ClinicDB, 64
from Kvasir-SEG, 380 from ColonDB, 196 from ETIS, following [21].

Evaluation. To evaluate 2D segmentation on 3D volumetric images, we follow the
evaluation protocol established by [20]. In a 3D image, when dealing with each
specific class denoted as $c^{j}$, we divide the images that fall between the
top and bottom slices containing this class into equal sections. These
sections, in our experiments, are set to be $C=3$ in number. For each of these
sections, we choose the middle slice from the corresponding section of the
support scan as a reference point. Then, this reference slice is used to guide
the segmentation of all the slices within the current section in the query
scan. It is important to note that the support and query scans are obtained
from different scans. We employ Dice score and Intersection over Union (IoU)
as evaluation metrics. The Dice score, used for the CT and MRI datasets, is
defined as follows:

$\text{Dice}=\frac{2\cdot|A\cap B|}{|A|+|B|}$ (9)

where $A$ and $B$ are the sets of predicted and ground truth binary
segmentation masks, respectively.

The IoU score, used alongside the Dice score for the polyp datasets, is
defined as follows:

$\text{IoU}=\frac{|A\cap B|}{|A\cup B|}$ (10)

For the CT and MRI datasets, we use the Dice score since IoU was not available
for comparison with other models. For the polyp datasets, we use both the Dice
score and IoU. For the CT and MRI evaluation, we perform 5-fold cross-
validation for each task. The results, including the mean and standard
deviation, are presented in Table 1. Additionally, we perform a statistical
analysis using the Wilcoxon rank test (Table 2) using compiled results for all
organs per modality (4 organs × 5 folds = 20 data points per model). Some
models were excluded due to missing fold results.

Implementation Details. Our code111https://github.com/levayz/ProtoSAM/ is
freely available. For inference images are resized to $672\times 672$
resolution. For (optionally) fine-tuning the encoder, we employ the ALPNet
[17] framework as described in Section 3, together with Low Rank Adaption
(LoRA) [8]. We train for 100k steps with a learning rate of 1e-4 and resize
the images to $256\times 256$. For the CT and MRI datasets, we use a setting
introduced in [17], where the testing class may not appear during training,
meaning we discard any slice that contains the testing class. This setting is
referred to as “Setting 2” in [17].

For the MRI and CT datasets, we divide the organs into two groups: (Spleen,
Liver), (RK, LK), where RK and LK represent Right-Kidney and Left-Kidney
respectively. For the DINOv2 encoder we use the ’large’ architecture. We use
the SAM ’huge’ architecture for ProtoSAM, AutoSAM, and SAM models, and the
’base’ architecture for ProtoSAM-base and ProtoMedSAM.

##  5 Results

Comparison to SOTA methods. Our method was benchmarked against leading methods
across different datasets. For the polyp dataset, we compared our model to the
SOTA supervised segmentation model, AutoSAM [21]. For fair comparison, we
trained AutoSAM with support sets of 1, 5, and 100 images and their masks,
before evaluating it. We also assessed SAM, using its automatic segmentation
mode, and manually selecting its best-performing mask. Additionally, we
compared ProtoSAM with both the default DINOv2 encoder and a fine-tuned
version as described in Section 3. Finally, we evaluated the performance of
ProtoMedSAM, where we replace SAM with MedSAM in ProtoSAM and use only
bounding boxes as prompts, as MedSAM was fine-tuned this way.

Table 1: MRI/CT 1-Shot Results (in Dice score) on abdominal images

Method | LK | RK | Spleen | Liver | Mean
---|---|---|---|---|---
MRI / CT | MRI / CT | MRI / CT | MRI / CT | MRI / CT
SSL-ALPNet [17] | 73.63$\pm$4.60 / 63.34$\pm$9.33 | 78.39$\pm$5.51 / 54.82$\pm$11.7 | 67.02$\pm$7.98 / 60.25$\pm$6.67 | 73.05$\pm$3.49 / 73.65$\pm$3.61 | 73.02$\pm$6.01 / 63.02$\pm$7.97
SSL-ALPNet+BP [18] | 78.77 / 66.04 | 83.44 / 62.14 | 70.02 / 68.39 | 75.01 / 73.90 | 76.81 / 67.62
CRAP-Net [4] | 74.66 / 70.91 | 82.77 / 67.33 | 73.82 / 70.17 | 70.82 / 70.45 | 73.82 / 69.72
CRTPNet [9] | 76.74 / 66.37 | 80.15 / 61.05 | 70.07 / 67.92 | 73.36 / 73.88 | 75.08 / 67.31
AutoSAM [21] | 61.07 / 43.20 | 64.46 / 38.77 | 69.03 / 54.50 | 68.10 / 70.68 | 65.66 / 51.79
SAM (best mask) [12] | 77.32$\pm$4.54 / 85.21$\pm$3.29 | 80.75$\pm$3.85 / 85.36$\pm$2.63 | 66.37$\pm$4.25 / 76.56$\pm$4.44 | 27.61$\pm$6.61 / 69.58$\pm$1.39 | 63.01$\pm$1.39 / 79.18$\pm$3.29
SSL-DINOv2 [2] | 75.06$\pm$11.61 / 69.96$\pm$11.61 | 80.21$\pm$2.14 / 66.40$\pm$12.99 | 71.86$\pm$10.30 / 73.00$\pm$6.49 | 73.50$\pm$5.95 / 76.40$\pm$3.89 | 75.16$\pm$2.14 / 71.44$\pm$6.49
SSL-DINOv2+CCA[2] | 81.43$\pm$13.84 / 66.40$\pm$13.84 | 84.40$\pm$4.07 / 69.96$\pm$13.65 | 73.30$\pm$10.27 / 74.60$\pm$6.58 | 74.20$\pm$5.46 / 81.67$\pm$4.41 | 78.43$\pm$4.07 / 73.16$\pm$6.58
ProtoMedSAM | 69.97$\pm$4.97 / 66.08$\pm$9.45 | 77.16$\pm$3.92 / 67.15$\pm$6.01 | 69.68$\pm$8.38 / 60.53$\pm$2.94 | 71.99$\pm$6.12 / 78.64$\pm$3.03 | 72.20$\pm$4.44 / 68.10$\pm$4.44
ProtoSAM-base | 70.47$\pm$3.93 / 67.54$\pm$8.33 | 79.03$\pm$3.06 / 64.52$\pm$7.85 | 69.56$\pm$7.87 / 57.99$\pm$3.40 | 69.88$\pm$4.89 / 77.56$\pm$2.50 | 72.23$\pm$3.93 / 66.90$\pm$8.33
ProtoSAM | 71.46$\pm$6.85 / 69.44$\pm$10.21 | 81.43$\pm$5.16 / 71.04$\pm$10.52 | 76.51$\pm$7.97 / 65.50$\pm$7.72 |  81.94$\pm$6.83  / 87.84$\pm$4.71 | 77.83$\pm$6.85 / 73.45$\pm$10.21
ProtoSAM +EFT |  85.61$\pm$3.90 / 73.88$\pm$16.50 |  87.94$\pm$1.79 / 70.05$\pm$14.27 |  80.92$\pm$7.12 / 81.14$\pm$3.83 | 80.60$\pm$3.33 / 86.34$\pm$ 3.20 |  83.69$\pm$3.33 / 77.85$\pm$14.27

Comparison | CT (p-value) | MRI (p-value)
---|---|---
ProtoSAM vs ALPNet | 0.0027 | 0.001
ProtoSAM+EFT vs SSL-DINOv2+CCA | 0.0017 | 3.62E-05
Table 2: Wilcoxon tests results

Figure 2: (Left) MRI segmentation. Top to bottom: SSL-Dinov2 + CCA, SAM (best
mask), ProtoSAM, Ground Truth, Query Image. Left to right: RK, LK, Spleen,
Liver. (Right) Polyp segmentation. Top to bottom: SAM (best mask),
ProtoMedSAM, ProtoSAM, Ground Truth, Query Image. Table 3: Polyp segmentation
benchmarks results

Method | Kvasir33 [10] | Clinic [3] | Colon [24] | ETIS [22] | All
---|---|---|---|---|---
Dice | IoU | Dice | IoU | Dice | IoU | Dice | IoU | Dice | IoU
AutoSAM [21] 1-shot | 35.64 | 25.18 | 20.31 | 14.12 | 10.34 | 6.29 | 13.97 | 9.00 | 15.57 | 10.23
AutoSAM [21] 5-shot | 59.56 | 48.14 | 38.90 | 29.34 | 35.60 | 25.75 | 26.57 | 19.53 | 36.72 | 33.33
AutoSAM [21] 100-shot | 73.44 | 62.51 | 62.64 | 52.26 | 53.27 | 43.24 | 38.52 | 29.94 | 52.87 | 43.08
AutoSAM [21] full-set | 91.00 | 87.00 | 92.80 | 89.33 | 83.00 | 76.70 | 79.70 | 74.00 | 86.25 | 81.75
SAM (best mask) [12] | 77.91 | 69.28 | 57.23 | 50.66 | 46.08 | 39.73 | 52.32 | 47.42 | 59.94 | 53.98
ProtoMedSAM (1-shot) | 81.11 | 72.75 | 66.54 | 57.31 | 64.34 | 55.61 | 59.899 | 51.66 | 65.62 | 57.02
ProtoSAM-base (1-shot) | 79.98 | 71.12 | 65.79 | 56.12 | 65.48 | 57.11 | 63.07 | 55.50 | 66.83 | 58.50
ProtoSAM (1-shot) | 81.55 | 73.66 | 69.01 | 60.80 | 67.31 | 59.60 | 62.33 | 55.47 | 68.06 | 60.49

Discussion. As shown in Table 1, for CT images, ProtoSAM achieves a mean Dice
score of 73.45. It surpasses other methods in the liver region with a score of
87.84, outperforming other trained methods and the best mask produced by the
SAM model. However, in the spleen region, its score of 65.50 is lower than the
scores of some other methods. Additionally, in certain cases, SAM provides
better masks than ProtoSAM, indicating that there may be room for improvement
in prompt generation. This suggests that while ProtoSAM is promising, further
training and refinement could enhance its performance in specific areas. Note
though that in SAM, we select the best mask w.r.t GT, which is not done in
ProtoSAM.

Table 2 presents results of the paired Wilcoxon tests. These results indicate
significant differences between the models. For ProtoSAM vs ALPNet, the
p-values were 0.0027 for CT and 0.001 for MRI. For ProtoSAM EFT vs SSL-
DINOv2+CCA, the p-values were 0.0017 for CT and 3.62E-05 for MRI,
demonstrating statistically significant improvements in both cases.

For MRI, ProtoSAM outperforms previous trained methods in the Liver, Spleen
organs, as well as in the overall mean, with scores of 81.94, 76.51 and 77.83,
respectively. It is also better than SAM on average in this case. By applying
the EFT described in Section 3, we observe a boost in performance, enabling
ProtoSAM to surpass all previous methods on all benchmarks, excluding SAM
(that does best mask selection w.r.t GT) on LK and RK in CT.

Table 3 presents the results of the polyp segmentation benchmarks to compare
the performance of ProtoSAM (1-shot) with SAM and AutoSAM (under different
shot settings, including training on the full training dataset). The
evaluation metrics used are the Dice coefficient and Intersection over Union
(IoU). ProtoSAM (1-shot) achieves a mean Dice score of 68.06 and a mean IoU of
60.49 across all datasets. This performance surpasses that of SAM when
choosing its best mask w.r.t GT. This indicates that our prompt generation
process is effective.

Observe that ProtoSAM consistently outperforms ProtoMedSAM across all our test
cases. Note that the ProtoSAM with SAM-base model provided comparable
performance to ProtoMedSAM for MRI and CT but better results for the polyps.
We believe the gap stems from the fact that MedSAM was fine-tuned using
bounding boxes derived from ground-truth masks, ensuring high accuracy in the
targeted regions. Since the ProtoSAM framework relies on an initial coarse
segmentation, the bounding boxes are not as accurate. In contrast, SAM in
ProtoSAM employs both bounding boxes and points. Since it was not fine-tuned
with precise bounding boxes, it is more resilient to Bbox noise.

When comparing ProtoSAM (1-shot) to AutoSAM under different shot settings, it
is evident that our untrained method achieves better performance using fewer
images. Specifically, ProtoSAM (1-shot) outperforms AutoSAM (1-shot), AutoSAM
(5-shot), and even AutoSAM (100-shot) in terms of mean Dice and IoU scores,
despite using only one image. This highlights the efficiency of ProtoSAM in
leveraging pre-training to achieve superior segmentation results compared to
the best supervised framework on these datasets, namely AutoSAM. However,
there remains a performance gap between the fully-supervised model and the
1-shot performance of ProtoSAM. Using EFT with Felzenszwalb generated
superpixels for poylps, lead to degregation of results compared to using the
pretrained encoder. This is a topic for future research.

Ablation Study. We wish to examine the effects of using different prompt
extraction methods. We tested the effects of the prompts on all the polyp
segmentation datasets and the first evaluation fold from the MRI dataset.
Table 4 present the results of this study on polyp segmentation datasets and
the first evaluation fold of the MRI dataset, respectively. In these tables,
we compare various combinations of prompts, including centroid (Cent),
confidence (Conf), negative (Neg) point, and bounding box (Bbox). Negative
points are generated in the same way as Conf points, only for the background
class instead of the foreground class. Our findings indicate that the
combination of bounding box, confidence, and centroid prompts (Bbox+Conf+Cent)
yielded the best overall performance across both datasets.

Table 4: Ablation study of different prompts

Cent | Conf | Neg | Bbox | MRI LK | MRI RK | MRI Spleen | MRI Liver | Polyps | Mean
---|---|---|---|---|---|---|---|---|---
Dice | IoU | Dice | IoU | Dice | IoU | Dice | IoU | Dice | IoU | Dice | IoU
✓ | ✗ | ✗ | ✗ | 56.31 | 51.51 | 77.11 | 72.22 | 72.67 | 67.29 | 47.86 | 36.29 | 55.38 | 48.33 | 61.87 | 55.13
✗ | ✓ | ✗ | ✗ | 65.60 | 60.92 | 78.82 | 73.98 | 75.55 | 69.60 | 35.67 | 27.81 | 58.73 | 51.32 | 62.87 | 56.73
✓ | ✓ | ✗ | ✗ | 60.76 | 55.42 | 80.11 | 75.04 | 73.46 | 68.00 | 53.59 | 43.00 | 62.32 | 54.89 | 66.05 | 59.27
✗ | ✗ | ✗ | ✓ | 62.42 | 57.38 | 82.40 | 76.58 | 74.61 | 68.43 | 77.37 | 67.56 | 67.43 | 59.80 | 72.85 | 65.95
✓ | ✓ | ✗ | ✓ | 61.71 | 56.83 | 82.81 | 77.27 | 74.00 | 68.08 | 78.87 | 69.18 | 68.06 | 60.49 | 73.14 | 66.42
✓ | ✓ | ✓ | ✓ | 61.69 | 56.79 | 82.59 | 76.89 | 73.87 | 67.88 | 78.47 | 68.48 | 67.99 | 60.24 | 72.11 | 65.18

##  6 Limitations

The current method, selects prompts based on the most confident component of
the initial segmentation phase, limiting it a single object segmentation,
making it unsuitable for tasks requiring multiple object segmentation.
Additionally, while ProtoSAM outperforms many existing methods, there are
still cases where SAM provides better masks, indicating room for improvement
in the prompt generation process. Furthermore, there is a performance gap
between fully-supervised models and ProtoSAM’s one-shot performance, evident
in the polyp segmentation results, suggesting that further refinement of the
method is needed to match SOTA supervised approaches. Lastly, the encoder
fine-tuning (EFT) technique, which improved results for CT and MRI datasets,
led to degradation of results for polyp segmentation when using Felzenszwalb
generated superpixels, highlighting the need for a more robust fine-tuning
strategy across diverse datasets.

##  7 Conclusion

In this work we introduce ProtoSAM, a novel automatic framework for one-shot
image segmentation. The experimental results demonstrate that the untrained
ProtoSAM achieves competitive performance across various organs and
modalities, often surpassing trained methods. Encoder fine-tuning further
boosts its performance, enabling it to outperform all previous methods,
including the best masks generated by the original SAM model, in specific
areas. Future work could explore the refinement of prompt generation,
including prompt generation for MedSAM and extension of the framework to other
segmentation tasks. To conclude, ProtoSAM presents a promising approach to
one-shot image segmentation in medical imaging, with potential applications in
scenarios where labeled data is scarce or where rapid adaptation to new
classes is required.

###  7.0.1 Acknowledgements

The research in this publication was supported in part by the Israel Science
Foundation (ISF) grant number 20/2629, the Israel Ministry of Science and
Technology, KLA research fund, and TAD.

## References

  * [1] Ayzenberg, L., Giryes, R., Greenspan, H.: Dinov2 based self supervised learning for few shot medical image segmentation. In: 2024 IEEE 21st International Symposium on Biomedical Imaging (ISBI). IEEE (2024)
  * [2] Ayzenberg, L., Giryes, R., Greenspan, H.: Dinov2 based self supervised learning for few shot medical image segmentation. arXiv preprint arXiv:2403.03274 (2024)
  * [3] Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodríguez, C., Vilariño, F.: Wm-dova maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized medical imaging and graphics 43, 99–111 (2015)
  * [4] Ding, H., Sun, C., Tang, H., Cai, D., Yan, Y.: Few-shot medical image segmentation with cycle-resemblance attention. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2488–2497 (2023)
  * [5] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [6] Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. International journal of computer vision 59, 167–181 (2004)
  * [7] Feyjie, A.R., Azad, R., Pedersoli, M., Kauffman, C., Ayed, I.B., Dolz, J.: Semi-supervised few-shot learning for medical image segmentation (2020)
  * [8] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * [9] Huang, Y., Liu, J.: Cross-reference transformer for few-shot medical image segmentation. arXiv preprint arXiv:2304.09630 (2023)
  * [10] Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., de Lange, T., Johansen, D., Johansen, H.D.: Kvasir-seg: A segmented polyp dataset (2019)
  * [11] Kavur, A.E., Gezer, N.S., Barış, M., Aslan, S., Conze, P.H., Groza, V., Pham, D.D., Chatterjee, S., Ernst, P., Özkan, S., Baydar, B., Lachinov, D., Han, S., Pauli, J., Isensee, F., Perkonigg, M., Sathish, R., Rajan, R., Sheet, D., Dovletov, G., Speck, O., Nürnberger, A., Maier-Hein, K.H., Akar, G.B., Ünal, G., Dicle, O., Selver, M.A.: CHAOS challenge - combined (CT-MR) healthy abdominal organ segmentation. Medical Image Analysis 69, 101950 (apr 2021)
  * [12] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. arXiv preprint arXiv:2304.02643 (2023)
  * [13] Kotia, J., Kotwal, A., Bharti, R., Mangrulkar, R.: Few Shot Learning for Medical Imaging, pp. 107–132. Springer International Publishing (2021)
  * [14] Landman, B., Xu, Z., Igelsias, J., Styner, M., Langerak, T., Klein, A.: Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge. In: Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge. vol. 5, p. 12 (2015)
  * [15] Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15(1), 654 (2024)
  * [16] Oquab, M., Darcet, T., Moutakanni, T., Vo, H.V., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P.Y., Xu, H., Sharma, V., Li, S.W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., Bojanowski, P.: Dinov2: Learning robust visual features without supervision. arXiv:2304.07193 (2023)
  * [17] Ouyang, C., Biffi, C., Chen, C., Kart, T., Qiu, H., Rueckert, D.: Self-supervision with superpixels: Training few-shot medical image segmentation without annotation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16\. pp. 762–780. Springer (2020)
  * [18] Ouyang, C., Biffi, C., Chen, C., Kart, T., Qiu, H., Rueckert, D.: Self-supervised learning for few-shot medical image segmentation. IEEE Transactions on Medical Imaging 41(7), 1837–1848 (2022). https://doi.org/10.1109/TMI.2022.3150682
  * [19] Rani, V., Nabi, S.T., Kumar, M., Mittal, A., Kumar, K.: Self-supervised learning: A succinct review. Archives of Computational Methods in Engineering 30(4), 2761–2775 (2023)
  * [20] Roy, A.G., Siddiqui, S., Pölsterl, S., Navab, N., Wachinger, C.: ‘squeeze & excite’guided few-shot segmentation of volumetric images. Medical image analysis 59, 101587 (2020)
  * [21] Shaharabany, T., Dahan, A., Giryes, R., Wolf, L.: Autosam: Adapting sam to medical images by overloading the prompt encoder. arXiv preprint arXiv:2306.06370 (2023)
  * [22] Silva, J., Histace, A., Romain, O., Dray, X., Granado, B.: Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer. International journal of computer assisted radiology and surgery 9, 283–293 (2014)
  * [23] Snell, J., Swersky, K., Zemel, R.: Prototypical networks for few-shot learning. Advances in neural information processing systems 30 (2017)
  * [24] Tajbakhsh, N., Gurudu, S.R., Liang, J.: Automated polyp detection in colonoscopy videos using shape and context information. IEEE transactions on medical imaging 35(2), 630–644 (2015)
  * [25] Wang, K., Liew, J.H., Zou, Y., Zhou, D., Feng, J.: Panet: Few-shot image semantic segmentation with prototype alignment. In: proceedings of the IEEE/CVF international conference on computer vision. pp. 9197–9206 (2019)
  * [26] Wu, J., Ji, W., Liu, Y., Fu, H., Xu, M., Xu, Y., Jin, Y.: Medical sam adapter: Adapting segment anything model for medical image segmentation (2023)
  * [27] Zhang, Y., Shen, Z., Jiao, R.: Segment anything model for medical image segmentation: Current applications and future directions. Computers in Biology and Medicine 171, 108238 (2024)


affect line ratios including atomic carbon by decreasing the $R^{\rm
CO10}_{\rm CI10}$ ratio and increasing the $R^{\rm CI10}_{\rm CO21}$ ratio.
Finally, both $X_{\rm CO}$ and $X_{\rm CI}$ conversion factors remain
relatively constant as a function of the FUV intensity.

###  9.3 Effect of metallicity

Low metallicities decrease the CO and H2 shielding against FUV, as well as the
H2 formation rate on dust grains, therefore impacting the carbon cycle in
relation to the Hi-to-H2 transition and increasing the overall gas
temperature. We find that for $Z=0.1\,{\rm Z}_{\odot}$, the dense cloud
remains molecular and becomes equally rich in Cii and Ci abundances as it is
in CO abundance. The diffuse cloud becomes Hi-dominated with the carbon
abundance to be almost entirely in Cii form. For super-solar metallcities,
both clouds are entirely molecular CO-dominated. Consequently, the emission of
all cooling lines modelled, change accordingly. For $Z=0.1\,{\rm Z}_{\odot}$
it is particularly interesting to note that the emission of [Cii] originates
from higher column densities corresponding to H2-rich gas, making this line a
promising tracer for molecular gas in low-metallicity environments (see also
Madden et al., 2020). As a result of the CO photodissociation in metal-poor
ISM, low $Z$ affect the mid- and low-$J$ transitions of the CO line and hence
the corresponding SLEDs. In regards to the bridge-effect signature, we find
that its shape depends only weakly on metallicity for $Z\geq 0.5\,{\rm
Z}_{\odot}$, but it diminishes for ${\sim}0.1\,{\rm Z}_{\odot}$ as these fine
structure lines originate from higher column densities that may not carry this
collision information any longer. We find that as metallicity increases, the
line ratios of $R^{\rm CI10}_{\rm CI21}$ and $R^{\rm CO10}_{\rm CI10}$
increase, while $R^{\rm CI10}_{\rm CO21}$ decrease. Finally, we find that both
$X_{\rm CO}$ and $X_{\rm CI}$ factors are depending strongly on metallicity in
agreement with several observations, as well as on the density distribution of
the ISM.

As a final conclusion, future studies using ALMA, SOFIA and the forthcoming
CCAT-prime telescope of ISM environments that differ from the local one, such
as the Galactic Centre, low-metallicity galaxies and galaxies with high star-
formation rates, can provide data to help understand the ISM environment in
the high-redshift Universe, particularly at $z\sim 2-3$ marking the ‘cosmic
noon’ of galaxy assembly. The models we have presented will be helpful in the
interpretation of such data.

## Data availability

The data underlying this article will be shared on reasonable request to the
corresponding author.

## Acknowledgements

The authors thank the anonymous referee for their comments and the overall
detailed and timely review which improved the clarity of the work. The authors
also thank Zhi-Yu Zhang, Daniel Seifried, Stefanie Walch, Brandt Gaches,
Loretta Dunne, Francesco Valentino and Munan Gong for the useful discussions.
TGB acknowledges support from Deutsche Forschungsgemeinschaft (DFG) grant No.
424563772. JCT acknowledges support from the Chalmers Foundation, NSF grant
AST-2009674 and ERC Advanced Grant 788829 (MSTAR). KEIT acknowledges support
from NAOJ ALMA Scientific Research grant No. 2017-05A, and JSPS KAKENHI grant
Nos. JP19H05080, JP19K14760.

TGB dedicates this work to the memory of Professor John-Hugh Seiradakis.

## References

  * Accurso et al. (2017a) Accurso G., Saintonge A., Bisbas T. G., Viti S., 2017a, MNRAS, 464, 3315
  * Accurso et al. (2017b) Accurso G., et al., 2017b, MNRAS, 470, 4750
  * Amorín et al. (2016) Amorín R., Muñoz-Tuñón C., Aguerri J. A. L., Planesas P., 2016, A&A, 588, A23
  * Andree-Labsch et al. (2017) Andree-Labsch S., Ossenkopf-Okada V., Röllig M., 2017, A&A, 598, A2
  * Aravena et al. (2014) Aravena M., et al., 2014, MNRAS, 442, 558
  * Arzoumanian et al. (2011) Arzoumanian D., et al., 2011, A&A, 529, L6
  * Bate (2014) Bate M. R., 2014, MNRAS, 442, 285. doi:10.1093/mnras/stu795
  * Bate (2019) Bate M. R., 2019, MNRAS, 484, 2341. doi:10.1093/mnras/stz103
  * Bell et al. (2006) Bell T. A., Roueff E., Viti S., Williams D. A., 2006, MNRAS, 371, 1865
  * Bell et al. (2007) Bell T. A., Viti S., Williams D. A., 2007, MNRAS, 378, 983
  * Beuther et al. (2020) Beuther H., et al., 2020, A&A, 638, A44
  * Beuther et al. (2014) Beuther H., et al., 2014, A&A, 571, A53
  * Bialy & Sternberg (2015) Bialy S., Sternberg A., 2015, MNRAS, 450, 4424
  * Bialy & Sternberg (2016) Bialy S., Sternberg A., 2016, ApJ, 822, 83
  * Bialy et al. (2017) Bialy S., Burkhart B., Sternberg A., 2017, ApJ, 843, 92
  * Bisbas et al. (2012) Bisbas T. G., Bell T. A., Viti S., Yates J., Barlow M. J., 2012, MNRAS, 427, 2100
  * Bisbas et al. (2014) Bisbas T. G., Bell T. A., Viti S., Barlow M. J., Yates J., Vasta M., 2014, MNRAS, 443, 111
  * Bisbas et al. (2015a) Bisbas T. G., Papadopoulos P. P., Viti S., 2015a, ApJ, 803, 37
  * Bisbas et al. (2015b) Bisbas T. G., Haworth T. J., Barlow M. J., Viti S., Harries T. J., Bell T., Yates J. A., 2015b, MNRAS, 454, 2828
  * Bisbas et al. (2017a) Bisbas T. G., van Dishoeck E. F., Papadopoulos P. P., Szűcs L., Bialy S., Zhang Z.-Y., 2017a, ApJ, 839, 90
  * Bisbas et al. (2017b) Bisbas T. G., Tanaka K. E. I., Tan J. C., Wu B., Nakamura F., 2017b, ApJ, 850, 23
  * Bisbas et al. (2018) Bisbas T. G., et al., 2018, MNRAS, 478, L54
  * Bisbas et al. (2019) Bisbas T. G., Schruba A., van Dishoeck E. F., 2019, MNRAS, 485, 3097
  * Bolatto et al. (2013) Bolatto A. D., Wolfire M., Leroy A. K., 2013, ARA&A, 51, 207
  * Bothwell et al. (2017) Bothwell M. S., et al., 2017, MNRAS, 466, 2825
  * Bryan et al. (2014) Bryan G. L., et al., 2014, ApJS, 211, 19
  * Burton et al. (1990) Burton M. G., Hollenbach D. J., Tielens A. G. G. M., 1990, ApJ, 365, 620
  * Cardelli et al. (1996) Cardelli J. A., Meyer D. M., Jura M., Savage B. D., 1996, ApJ, 467, 334
  * Carniani et al. (2018) Carniani S., Maiolino R., Smit R., Amorín R., 2018, ApJL, 854, L7
  * Cartledge et al. (2004) Cartledge S. I. B., Lauroesch J. T., Meyer D. M., Sofia U. J., 2004, ApJ, 613, 1037
  * Chen et al. (2015) Chen B.-Q., Liu X.-W., Yuan H.-B., Huang Y., Xiang M.-S., 2015, MNRAS, 448, 2187
  * Clark et al. (2019) Clark P. C., Glover S. C. O., Ragan S. E., Duarte-Cabral A., 2019, MNRAS, 486, 4622
  * Clark et al. (2013) Clark P. C., Glover S. C. O., Ragan S. E., Shetty R., Klessen R. S., 2013, ApJ, 768, L34
  * Cooke et al. (2015) Cooke R. J., Pettini M., Jorgenson R. A., 2015, ApJ, 800, 12
  * Crocker et al. (2019) Crocker A. F., Pellegrini E., Smith J.-D. T., Draine B. T., Wilson C. D., Wolfire M., Armus L., et al., 2019, ApJ, 887, 105
  * Cummings et al. (2016) Cummings A. C., et al., 2016, ApJ, 831, 18
  * Cummings et al. (2015) Cummings A., et al., 2015, ICRC, 34, 318
  * Dalgarno (2006) Dalgarno A., 2006, PNAS, 103, 12269
  * Dame et al. (2001) Dame T. M., Hartmann D., Thaddeus P., 2001, ApJ, 547, 792
  * Draine (1978) Draine B. T., 1978, ApJS, 36, 595
  * Federman et al. (1996) Federman S. R., Rawlings J. M. C., Taylor S. D., Williams D. A., 1996, MNRAS, 279, L41
  * Feldmann et al. (2012) Feldmann R., Gnedin N. Y., Kravtsov A. V., 2012, ApJ, 747, 124
  * Franeck et al. (2018) Franeck A., et al., 2018, MNRAS, 481, 4277
  * Gaches & Offner (2018) Gaches B. A. L., Offner S. S. R., 2018, ApJ, 854, 156.
  * Gaches et al. (2019a) Gaches B. A. L., Offner S. S. R., Bisbas T. G., 2019a, ApJ, 878, 105
  * Gaches et al. (2019b) Gaches B. A. L., Offner S. S. R., Bisbas T. G., 2019b, ApJ, 883, 190
  * Galliano et al. (2008) Galliano F., Dwek E., Chanial P., 2008, ApJ, 672, 214
  * Genovali et al. (2014) Genovali K., Lemasle B., Bono G., Romaniello M., Fabrizio M., Ferraro I., Iannicola G., et al., 2014, A&A, 566, A37
  * Genzel et al. (2012) Genzel R., et al., 2012, ApJ, 746, 69
  * Girichidis et al. (2016) Girichidis P., Naab T., Walch S., Hanasz M., Mac Low M.-M., Ostriker J. P., Gatto A., et al., 2016, ApJL, 816, L19
  * Giveon et al. (2002) Giveon U., Sternberg A., Lutz D., Feuchtgruber H., Pauldrach A. W. A., 2002, ApJ, 566, 880
  * Glover et al. (2010) Glover S. C. O., Federrath C., Mac Low M.-M., Klessen R. S., 2010, MNRAS, 404, 2
  * Glover & Mac Low (2011) Glover S. C. O., Mac Low M.-M., 2011, MNRAS, 412, 337
  * Glover et al. (2015) Glover S. C. O., Clark P. C., Micic M., Molina F., 2015, MNRAS, 448, 1607
  * Glover & Clark (2016) Glover S. C. O., Clark P. C., 2016, MNRAS, 456, 3596
  * Goldsmith (2019) Goldsmith P. F., 2019, ApJ, 887, 54
  * Gong et al. (2020) Gong M., Ostriker E. C., Kim C.-G., Kim J.-G., 2020, ApJ, 903, 142. doi:10.3847/1538-4357/abbdab
  * Gong et al. (2018) Gong M., Ostriker E. C., Kim C.-G., 2018, ApJ, 858, 16
  * Goto et al. (2008) Goto M., et al., 2008, ApJ, 688, 306
  * Guevara et al. (2020) Guevara C., Stutzki J., Ossenkopf-Okada V., Simon R., Pérez-Beaupuits J. P., Beuther H., Bihr S., et al., 2020, A&A, 636, A16. doi:10.1051/0004-6361/201834380
  * Grassi et al. (2014) Grassi T., Bovino S., Schleicher D. R. G., Prieto J., Seifried D., Simoncini E., Gianturco F. A., 2014, MNRAS, 439, 2386
  * Grenier et al. (2015) Grenier I. A., Black J. H., Strong A. W., 2015, ARA&A, 53, 199
  * Gullberg et al. (2016) Gullberg B., et al., 2016, A&A, 591, A73
  * Habing (1968) Habing H. J., 1968, BAN, 19, 421
  * Haworth et al. (2015a) Haworth T. J., et al., 2015a, MNRAS, 450, 10
  * Haworth et al. (2015b) Haworth T. J., et al., 2015b, MNRAS, 454, 1634
  * Haworth et al. (2018) Haworth T. J., Glover S. C. O., Koepferl C. M., Bisbas T. G., Dale J. E., 2018, NewAR, 82, 1
  * Heintz & Watson (2020) Heintz K. E., Watson D., 2020, ApJL, 889, L7
  * Hodge et al. (2012) Hodge J. A., Carilli C. L., Walter F., de Blok W. J. G., Riechers D., Daddi E., Lentati L., 2012, ApJ, 760, 11
  * Hollenbach & Tielens (1999) Hollenbach D. J., Tielens A. G. G. M., 1999, RvMP, 71, 173
  * Hitschfeld et al. (2008) Hitschfeld M., et al., 2008, A&A, 479, 75
  * Indriolo & McCall (2012) Indriolo N., McCall B. J., 2012, ApJ, 745, 91
  * Indriolo et al. (2015) Indriolo N., et al., 2015, ApJ, 800, 40
  * Inno (2020) Inno L., 2020, IAUGA
  * Inoue et al. (2020) Inoue S., Yoshida N., Yajima H., 2020, MNRAS, 498, 5960. doi:10.1093/mnras/staa2744
  * Israel (1997) Israel F. P., 1997, A&A, 328, 471
  * Israel & Baas (2002) Israel F. P., Baas F., 2002, A&A, 383, 82
  * Jiao et al. (2017) Jiao Q., Zhao Y., Zhu M., Lu N., Gao Y., Zhang Z.-Y., 2017, ApJL, 840, L18
  * Joblin et al. (2018) Joblin C., et al., 2018, A&A, 615, A129
  * Kaufman et al. (1999) Kaufman M. J., Wolfire M. G., Hollenbach D. J., Luhman M. L., 1999, ApJ, 527, 795
  * Kaufman et al. (2006) Kaufman M. J., Wolfire M. G., Hollenbach D. J., 2006, ApJ, 644, 283. doi:10.1086/503596
  * Kirsanova et al. (2020) Kirsanova M. S., Ossenkopf-Okada V., Anderson L. D., Boley P. A., Bieging J. H., Pavlyuchenkov Y. N., Luisi M., et al., 2020, MNRAS, 497, 2651. doi:10.1093/mnras/staa2142
  * Kraemer et al. (1998) Kraemer K. E., Jackson J. M., Lane A. P., 1998, ApJ, 503, 785. doi:10.1086/306002
  * Kramer et al. (2004) Kramer C., Jakob H., Mookerjea B., Schneider N., Brüll M., Stutzki J., 2004, A&A, 424, 887
  * Kramer et al. (2020) Kramer C., Nikola T., Anderl S., Bertoldi F., Boquien M., Braine J., Buchbender C., et al., 2020, A&A, 639, A61. doi:10.1051/0004-6361/201936754
  * Krips et al. (2016) Krips M., et al., 2016, A&A, 592, L3
  * Lamperti et al. (2020) Lamperti I., et al., 2020, ApJ, 889, 103
  * Langer & Penzias (1990) Langer W. D., Penzias A. A., 1990, ApJ, 357, 477. doi:10.1086/168935
  * Langer & Penzias (1993) Langer W. D., Penzias A. A., 1993, ApJ, 408, 539. doi:10.1086/172611
  * Langer et al. (2010) Langer W. D., Velusamy T., Pineda J. L., Goldsmith P. F., Li D., Yorke H. W., 2010, A&A, 521, L17
  * Langer et al. (2014) Langer W. D., Velusamy T., Pineda J. L., Willacy K., Goldsmith P. F., 2014, A&A, 561, A122
  * Le Petit et al. (2006) Le Petit F., Nehmé C., Le Bourlot J., Roueff E., 2006, ApJS, 164, 506
  * Leroy et al. (2011) Leroy A. K., Bolatto A., Gordon K., Sandstrom K., Gratier P., Rosolowsky E., Engelbracht C. W., et al., 2011, ApJ, 737, 12
  * Li et al. (2018) Li Q., Tan J. C., Christie D., Bisbas T. G., Wu B., 2018, PASJ, 70, S56
  * Lim et al. (2020) Lim W., et al., 2020, PASJ..tmp, doi:10.1093/pasj/psaa035
  * Lim et al. (2016) Lim W., Tan J. C., Kainulainen J., Ma B., Butler M. J., 2016, ApJL, 829, L19. doi:10.3847/2041-8205/829/1/L19
  * Lo et al. (2014) Lo N., et al., 2014, ApJL, 797, L17
  * Luo et al. (2020) Luo G., et al., 2020, ApJL, 889, L4
  * McCall et al. (2003) McCall B. J., et al., 2003, Natur, 422, 500
  * McElroy et al. (2013) McElroy D., Walsh C., Markwick A. J., Cordiner M. A., Smith K., Millar T. J., 2013, A&A, 550, A36
  * Mackey et al. (2019) Mackey J., Walch S., Seifried D., Glover S. C. O., Wünsch R., Aharonian F., 2019, MNRAS, 486, 1094
  * Madden et al. (2020) Madden S. C., Cormier D., Hony S., Lebouteiller V., Abel N., Galametz M., De Looze I., et al., 2020, A&A, 643, A141. doi:10.1051/0004-6361/202038860
  * Maiolino & Mannucci (2019) Maiolino R., Mannucci F., 2019, A&ARv, 27, 3
  * Maloney et al. (1996) Maloney P. R., Hollenbach D. J., Tielens A. G. G. M., 1996, ApJ, 466, 561
  * Mashian et al. (2015) Mashian N., et al., 2015, ApJ, 802, 81
  * Meijerink et al. (2006) Meijerink R., Spaans M., Israel F. P., 2006, ApJL, 650, L103
  * Meijerink et al. (2011) Meijerink R., Spaans M., Loenen A. F., van der Werf P. P., 2011, A&A, 525, A119
  * Narayanan & Krumholz (2014) Narayanan D., Krumholz M. R., 2014, MNRAS, 442, 1411
  * Narayanan et al. (2012) Narayanan D., Krumholz M. R., Ostriker E. C., Hernquist L., 2012, MNRAS, 421, 3127
  * Nelson & Langer (1997) Nelson R. P., Langer W. D., 1997, ApJ, 482, 796
  * Neufeld et al. (2010) Neufeld D. A., et al., 2010, A&A, 521, L5
  * Offner et al. (2013) Offner S. S. R., Bisbas T. G., Viti S., Bell T. A., 2013, ApJ, 770, 49
  * Offner et al. (2014) Offner S. S. R., Bisbas T. G., Bell T. A., Viti S., 2014, MNRAS, 440, L81
  * Oka et al. (2005a) Oka T., et al., 2005a, ApJ, 623, 889
  * Oka et al. (2005b) Oka T., Geballe T. R., Goto M., Usuda T., McCall B. J., 2005b, ApJ, 632, 882
  * Okada et al. (2019a) Okada Y., Güsten R., Requena-Torres M. A., Röllig M., Stutzki J., Graf U. U., Hughes A., 2019, A&A, 621, A62
  * Okada et al. (2019b) Okada Y., Higgins R., Ossenkopf-Okada V., Guevara C., Stutzki J., Mertens M., 2019, A&A, 631, L12. doi:10.1051/0004-6361/201936685
  * Ossenkopf et al. (2013) Ossenkopf V., Röllig M., Neufeld D. A., Pilleri P., Lis D. C., Fuente A., van der Tak F. F. S., et al., 2013, A&A, 550, A57. doi:10.1051/0004-6361/201219837
  * Pak et al. (1998) Pak S., Jaffe D. T., van Dishoeck E. F., Johansson L. E. B., Booth R. S., 1998, ApJ, 498, 735
  * Papadopoulos et al. (2004) Papadopoulos P. P., Thi W.-F., Viti S., 2004, MNRAS, 351, 147
  * Papadopoulos et al. (2010a) Papadopoulos P. P., van der Werf P., Isaak K., Xilouris E. M., 2010a, ApJ, 715, 775
  * Papadopoulos (2010b) Papadopoulos P. P., 2010b, ApJ, 720, 226
  * Papadopoulos et al. (2011) Papadopoulos P. P., Thi W.-F., Miniati F., Viti S., 2011, MNRAS, 414, 1705
  * Papadopoulos et al. (2012) Papadopoulos P. P., van der Werf P., Xilouris E., Isaak K. G., Gao Y., 2012, ApJ, 751, 10
  * Papadopoulos et al. (2012b) Papadopoulos P. P., van der Werf P. P., Xilouris E. M., Isaak K. G., Gao Y., Mühle S., 2012b, MNRAS, 426, 2601
  * Papadopoulos et al. (2018) Papadopoulos P. P., Bisbas T. G., Zhang Z.-Y., 2018, MNRAS, 478, 1716
  * Peñaloza et al. (2018) Peñaloza C. H., Clark P. C., Glover S. C. O., Klessen R. S., 2018, MNRAS, 475, 1508
  * Peretto et al. (2010) Peretto N., et al., 2010, A&A, 518, L98
  * Pineda et al. (2013) Pineda J. L., Langer W. D., Velusamy T., Goldsmith P. F., 2013, A&A, 554, A103
  * Pon et al. (2016) Pon A., et al., 2016, ApJ, 827, 107
  * Pineda et al. (2008) Pineda J. E., Caselli P., Goodman A. A., 2008, ApJ, 679, 481
  * Pineda et al. (2017) Pineda J. L., et al., 2017, ApJ, 839, 107
  * Rémy-Ruyer et al. (2014) Rémy-Ruyer A., Madden S. C., Galliano F., Galametz M., Takeuchi T. T., Asano R. S., Zhukovska S., et al., 2014, A&A, 563, A31. doi:10.1051/0004-6361/201322803
  * Requena-Torres et al. (2016) Requena-Torres M. A., et al., 2016, A&A, 589, A28
  * Röllig et al. (2006) Röllig M., Ossenkopf V., Jeyakumar S., Stutzki J., Sternberg A., 2006, A&A, 451, 917
  * Röllig et al. (2007) Röllig M., et al., 2007, A&A, 467, 187
  * Rosenberg et al. (2015) Rosenberg M. J. F., et al., 2015, ApJ, 801, 72
  * Safranek-Shrader et al. (2017) Safranek-Shrader C., Krumholz M. R., Kim C.-G., Ostriker E. C., Klein R. I., Li S., McKee C. F., Stone J. M., 2017, MNRAS, 465, 885
  * Sandstrom et al. (2013) Sandstrom K. M., Leroy A. K., Walter F., Bolatto A. D., Croxall K. V., Draine B. T., Wilson C. D., et al., 2013, ApJ, 777, 5
  * Schruba et al. (2012) Schruba A., Leroy A. K., Walter F., Bigiel F., Brinks E., de Blok W. J. G., Kramer C., et al., 2012, AJ, 143, 138
  * Schruba et al. (2018) Schruba A., Bialy S., Sternberg A., 2018, ApJ, 862, 110
  * Schneider et al. (2018) Schneider N., Röllig M., Simon R., Wiesemeyer H., Gusdorf A., Stutzki J., Güsten R., et al., 2018, A&A, 617, A45. doi:10.1051/0004-6361/201732508
  * Seifried & Walch (2016) Seifried D., Walch S., 2016, MNRAS, 459, L11
  * Seifried et al. (2017) Seifried D., et al., 2017, MNRAS, 472, 4797
  * Seifried et al. (2020) Seifried D., Haid S., Walch S., Borchert E. M. A., Bisbas T. G., 2020, MNRAS, 492, 1465
  * Shi et al. (2016) Shi Y., Wang J., Zhang Z.-Y., Gao Y., Hao C.-N., Xia X.-Y., Gu Q., 2016, NatCo, 7, 13789
  * Shi et al. (2015) Shi Y., Wang J., Zhang Z.-Y., Gao Y., Armus L., Helou G., Gu Q., et al., 2015, ApJL, 804, L11
  * Smith et al. (2014) Smith R. J., Glover S. C. O., Clark P. C., Klessen R. S., Springel V., 2014, MNRAS, 441, 1628
  * Sternberg & Dalgarno (1995) Sternberg A., Dalgarno A., 1995, ApJS, 99, 565
  * Strom et al. (2018) Strom A. L., Steidel C. C., Rudie G. C., Trainor R. F., Pettini M., 2018, ApJ, 868, 117
  * Strong & Mattox (1996) Strong A. W., Mattox J. R., 1996, A&A, 308, L21
  * Strong et al. (2007) Strong A. W., Moskalenko I. V., Ptuskin V. S., 2007, ARNPS, 57, 285
  * Stutzki et al. (1988) Stutzki J., Stacey G. J., Genzel R., Harris A. I., Jaffe D. T., Lugten J. B., 1988, ApJ, 332, 379
  * Szűcs, Glover & Klessen (2016) Szűcs L., Glover S. C. O., Klessen R. S., 2016, MNRAS, 460, 82
  * Tacconi et al. (2008) Tacconi L. J., et al., 2008, ApJ, 680, 246
  * Tafelmeyer et al. (2010) Tafelmeyer M., et al., 2010, A&A, 524, A58
  * Tanaka & Omukai (2014) Tanaka K. E. I., Omukai K., 2014, MNRAS, 439, 1884. doi:10.1093/mnras/stu069
  * Tanaka et al. (2018) Tanaka K. E. I., Tan J. C., Zhang Y., Hosokawa T., 2018, ApJ, 861, 68. doi:10.3847/1538-4357/aac892
  * Valentino et al. (2018) Valentino F., et al., 2018, ApJ, 869, 27
  * Valentino et al. (2020) Valentino F., et al., 2020, ApJ, 890, 24
  * Vallini et al. (2018) Vallini L., Pallottini A., Ferrara A., Gallerani S., Sobacchi E., Behrens C., 2018, MNRAS, 473, 271
  * van Dishoeck & Black (1986) van Dishoeck E. F., Black J. H., 1986, ApJS, 62, 109
  * van Dishoeck & Black (1988) van Dishoeck E. F., Black J. H., 1988, ApJ, 334, 771
  * van Dishoeck (1992) van Dishoeck E. F., 1992, IAUS, 150, 143, IAUS..150
  * Van Loo et al. (2013) Van Loo S., Butler M. J., Tan J. C., 2013, ApJ, 764, 36
  * Visser et al. (2009) Visser R., van Dishoeck E. F., Black J. H., 2009, A&A, 503, 323
  * Vlahakis et al. (2013) Vlahakis C., van der Werf P., Israel F. P., Tilanus R. P. J., 2013, MNRAS, 433, 1837
  * Wakelam & Herbst (2008) Wakelam V., Herbst E., 2008, ApJ, 680, 371. doi:10.1086/587734
  * Wakelam et al. (2017) Wakelam V., Bron E., Cazaux S., Dulieu F., Gry C., Guillard P., Habart E., et al., 2017, MolAs, 9, 1. doi:10.1016/j.molap.2017.11.001
  * Walch et al. (2015) Walch S., et al., 2015, MNRAS, 454, 238
  * Wang et al. (2017) Wang X., et al., 2017, ApJ, 837, 89
  * Weingartner & Draine (2001) Weingartner J. C., Draine B. T., 2001, ApJ, 548, 296
  * Wilson (1999) Wilson T. L., 1999, RPPh, 62, 143. doi:10.1088/0034-4885/62/2/002
  * Wolfire et al. (2003) Wolfire M. G., McKee C. F., Hollenbach D., Tielens A. G. G. M., 2003, ApJ, 587, 278
  * Wolfire et al. (2010) Wolfire M. G., Hollenbach D., McKee C. F., 2010, ApJ, 716, 1191
  * Wright et al. (1991) Wright E. L., et al., 1991, ApJ, 381, 200
  * Wu et al. (2015) Wu B., Van Loo S., Tan J. C., Bruderer S., 2015, ApJ, 811, 56
  * Wu et al. (2017) Wu B., Tan J. C., Nakamura F., Van Loo S., Christie D., Collins D., 2017, ApJ, 835, 137
  * Xie et al. (1995) Xie T., Allen M., Langer W. D., 1995, ApJ, 440, 674
  * Yusef-Zadeh et al. (2013) Yusef-Zadeh F., et al., 2013, ApJ, 762, 33
  * Yusef-Zadeh et al. (2007) Yusef-Zadeh F., Muno M., Wardle M., Lis D. C., 2007, ApJ, 656, 847
  * Zhang et al. (2014) Zhang Z.-Y., et al., 2014, A&A, 568, A122
  * Zhu & Huang (2014) Zhu J., Huang M., 2014, A&A, 564, A111

##  Appendix A Additional emission maps and optical depth diagrams

Figures 19 and 20 present emission maps for the lines of [Ci] (2-1), [Oi]
$146\mu$m and CO with $J=2-1$ up to $J=10-9$ for the FUV intensity and the
metallicity as the free ISM parameters, respectively.

Figures 21 – 24 show the average optical depth ($\tau$) weighted with the
brightness temperature for the lines of [Cii] $158\mu$m, [Ci] (1-0), CO
$J=1-0$ and [Oi] $63\mu$m versus the column density of H2, respectively. The
aforementioned optical depth is given by the expression:

$\displaystyle\tau=\frac{\int_{\varv_{\rm min}}^{\varv_{\rm
max}}\tau_{\varv}T_{\rm A}d\varv}{\int_{\varv_{\rm min}}^{\varv_{\rm
max}}T_{\rm A}d\varv},$ (9)

where $\tau_{\varv}$ is the optical depth of a given velocity channel and the
integration is over velocities as discussed in §2.3. In each panel we show a
scatter plot of $\tau$, coloured according to the probability density. High
values of the probability density gives the dominant value of $\tau$ under all
ISM conditions explored.

Note that the probability density of the $\tau_{\rm CII}$ in the dense cloud
appears to contain four different ‘groups’. This occurs for the cases of the
fiducial model, of $\zeta_{\rm CR}=10^{-17}\,{\rm s}^{-1}$ and of $Z=2\,{\rm
Z}_{\odot}$, and it is a result of edge effects in the boundaries of the map
i.e. where the isotropic radiation impinges from. Furthermore, in the
particular case of $\tau_{\rm CO10}$ in the diffuse cloud and at $Z=0.1\,{\rm
Z}_{\odot}$, there are only a few scatter points as a result of the CO
depletion and photodissociation.

Figure 19: The rest of emission lines following Fig. 8 for the FUV intensity
as the free parameter. From top-to-bottom, the lines of [Ci] (2-1), [Oi]
$146\mu$m and then CO $J=2-1$ to $J=10-9$ are presented. Figure 20: The rest
of emission lines following Fig. 9 for the metallicity as the free parameter.
From top-to-bottom, the lines of [Ci] (2-1), [Oi] $146\mu$m and then CO
$J=2-1$ to $J=10-9$ are presented. Figure 21: Optical depth of the [Cii]
$158\mu$m line versus N(H2) for all conditions explored. Figure 22: Optical
depth of the [Ci] (1-0) line versus N(H2) for all conditions explored. Figure
23: Optical depth of the CO $J=1-0$ line versus N(H2) for all conditions
explored. Figure 24: Optical depth of the [Oi] $63\mu$m line versus N(H2) for
all conditions explored.