[table]capposition=top (eccv) Package eccv Warning: Package ‘hyperref’ is
loaded with option ‘pagebackref’, which is *not* recommended for camera-ready
version

11institutetext: DAMO Academy, Alibaba Group 22institutetext: Hupan Lab,
310023, Hangzhou, China
22email<EMAIL_ADDRESS>
# Towards a Comprehensive, Efficient and Promptable Anatomic Structure
Segmentation Model using 3D Whole-body CT Scans

Heng Guo 1122 Jianfeng Zhang 1122 Jiaxing Huang 11 Tony C. W. Mok 1122
Dazhou Guo 11 Ke Yan 1122 Le Lu 11 Dakai Jin 11 Minfeng Xu 1122

###### Abstract

Segment anything model (SAM) demonstrates strong generalization ability on
natural image segmentation. However, its direct adaption in medical image
segmentation tasks shows significant performance drops with inferior accuracy
and unstable results. It may also requires an excessive number of prompt
points to obtain a reasonable accuracy. For segmenting 3D radiological CT or
MRI scans, a 2D SAM model has to separately handle hundreds of 2D slices.
Although quite a few studies explore adapting SAM into medical image volumes,
the efficiency of 2D adaption methods is unsatisfactory and 3D adaptation
methods only capable of segmenting specific organs/tumors. In this work, we
propose a comprehensive and scalable 3D SAM model for whole-body CT
segmentation, named CT-SAM3D. Instead of adapting SAM, we propose a 3D
promptable segmentation model using a (nearly) fully labeled CT dataset. To
train CT-SAM3D effectively, ensuring the model’s accurate responses to higher-
dimensional spatial prompts is crucial, and 3D patch-wise training is required
due to GPU memory constraints. For this purpose, we propose two key technical
developments: 1) a progressively and spatially aligned prompt encoding method
to effectively encode click prompts in local 3D space; and 2) a cross-patch
prompt learning scheme to capture more 3D spatial context, which is beneficial
for reducing the editing workloads when interactively prompting on large
organs. CT-SAM3D is trained and validated using a curated dataset of 1204 CT
scans containing $107$ whole-body anatomies, reporting significantly better
quantitative performance against all previous SAM-derived models by a large
margin with much fewer click prompts. Our model can handle segmenting unseen
organ as well. Code, data, and our 3D interactive segmentation tool with
quasi-real-time responses will be made publicly available.

###### Keywords:

CT-SAM3D Comprehensive and Efficient Interactive Segmentation 3D Click Based
Prompt

##  1 Introduction

Figure 1: Illustration of the enhanced TotalSeg++ dataset and CT-SAM3D.
TotalSeg++ complements TotalSeg dataset with added skeletal muscle, visceral
and subcutaneous fat voxel-level lable maps for a more comprehensive body-
parsing purpose. The proposed CT-SAM3D is a pure 3D point-promptable
segmentation model trained to segment 107 anatomies and even unseen organs in
whole-body CT scans.

Image segmentation is a fundamental and ubiquitous task in medical image
analysis and various clinical applications, such as for disease quantification
[24, 14], computer-aided diagnosis [53, 8, 43, 41], and radiotherapy planning
[27, 69, 26] and so on. Automatic segmentation achieves great improvement for
the past decade with rapidly evolving advanced deep learning techniques [34,
52, 42, 16, 23, 6, 59]. On the other hand, fully-automated segmentation still
have limited accuracy and robustness for daily clinical use caused by large
variations in medical images including different imaging protocols, imaging
noises/artifacts, and abnormalities or pathological changes among patients
[20, 1]. Interactive segmentation or intelligent image editing with human-in-
the-loop [40, 75, 54, 62, 64, 25, 72, 29] techniques are still needed to
further improve the segmentation performance with reduced manual workload by
incorporating users’ inputs often in the forms of point, scribble or mask
edits. Recently, segment anything model (SAM) [28] shows great success for
general-purpose promptable object segmentation in natural images with strong
generalization ability and efficient human interaction. Direct deployment of
SAM to medical imaging domain exhibits significant performance drops [61, 38,
19, 22, 10], but its core design principles of class-agnostic segmentation,
prompt encoding, and progressive training scheme can be exploited and applied
for medical image segmentation problems, with huge potentials to serve as
strong pretraining models and significantly boost the interactive segmentation
efficiency and accuracy.

A few recent studies attempt to apply fine-tuning to SAM by incorporating
lightweight plug-and-play adapters [7, 15, 4, 73, 70, 67]. Simple 2D
adaptation methods are directly used in [73, 7], which completely ignore the
intrinsic 3D information from CT scans and require hundreds of clicks when
segmenting hundreds of 2D CT slices. Thus these models are inapplicable or
infeasible in real clinical practice. Instead, other efforts focus on 3D
adaptation with careful modifications of SAM architecture by incorporating a
set of 3D adapters into each transformer block of the encoder to extract 3D
information [15, 4], or introducing space and depth adapters specifically
designed to process 3D information while retaining all the weights in the
original SAM model [67]. Nevertheless, these methods only report segmenting a
limited number of organs/tumors and their transferability to segment a large
number of 3D anatomical categories has not been validated. Note that a more
recent work, SAM-Med3D [63], simply replaces all 2D operations in the SAM
network with 3D operations and trains a 3D SAM network from scratch using
$21K$ medical images with $131K$ 3D masks. Another recent work, SegVol [13]
incorporates a zoom-out-zoom-in mechanism into 3D SAM development based on
$6K$ CT scans and $150K$ segmentation masks. As demonstrated later in our
experiments, merely extending SAM’s network architecture to 3D and scaling up
the dataset can lead to significant inefficiency and unsatisfactory results.

In this work, we aim to develop a 3D point-promptable segmentation model that
can interactively segment nearly all anatomic structures within whole-body CT
scans with high accuracy and efficiency (e.g., using 1 $\sim$ 5 clicks to
accurately segment a complete 3D organ). To achieve this, we present a
comprehensive, scalable and efficient 3D SAM network, named CT-SAM3D, working
on whole-body CT scans. Instead of directly adapting the original SAM [28], we
propose a 3D promptable segmentation model trained on a moderate number of
1204 fully labeled CT scans. We identify several key technical issues for
developing an effective and efficient 3D SAM model from scratch. 1) SAM’s
densely annotated dataset (SA-1B) guarantees that each pixel position in its
input space has the opportunity to be positively prompted. It is ideal to have
an analogous fully labeled whole-body CT dataset, i.e., each voxel in the
valid body region has an anatomic label. Otherwise, some anatomic regions
would always be background, thus not learned by the developed network,
limiting the underlying model’s capacity in zero-shot or interactive
segmentation scenarios. 2) SAM encodes 2D spatial prompts (point/box) into
one-dimensional embeddings using random Fourier features [50, 58]. However,
when scaling up the model to 3D, this one-dimensional prompt encoding (PE)
loses spatial alignment thereby encounters optimization difficulty. 3) Because
of the additional third dimension, SAM’s model complexity and input data scale
can increase dramatically.

To solve these issues, we first enrich our whole-body CT scan dataset based on
TotalSeg [66] by curating and adding the segmentation masks of three important
yet under-explored anatomic structures of skeletal muscle, visceral fat, and
subcutaneous fat, as illustrated in Fig. 1. This results in a more
comprehensive whole-body CT dataset, namely TotalSeg++, where overall
$\sim$83% of voxels within the body region are semantically labeled (i.e.,
with an anatomical label assigned per-pixel), substantially increased from the
previous 37% in TotalSeg [66] dataset. Then, we propose a progressively and
spatially aligned prompt encoding method to ensure the model responds to the
3D spatial prompts accurately. Lastly, 3D patch-wise training is necessary to
effectively train a 3D SAM model. Yet, if simply training on 3D local image
patches, the inference efficiency would be reduced and drastically more clicks
are needed to capture the whole spatial context when segmenting organs larger
than the local patch dimensions. Therefore, we propose a cross-patch prompt
scheme to alleviate this problem.

We outline our main contributions as follows:

  * •

We present a comprehensive, scalable and efficient 3D promptable segmentation
model on whole-body CT scans, named CT-SAM3D. It has the versatility of
segmenting hundreds of anatomies and achieves new state-of-the-art interactive
segmentation results on various quantitative benchmarks.

  * •

To effectively train CT-SAM3D, we propose two technical novelties: 1) a
progressively and spatially aligned prompt encoding method to effectively
encode click prompt in local 3D space; 2) a cross-patch prompt scheme to make
the local click take effect in a broader spatial context, beneficial for
reducing workloads when prompting on large organs.

  * •

We develop an interactive 3D visualization and segmentation tool with the
direct GPU access for model inference. This reports efficient quasi-real-time
3D interactive segmentation performance for the first time.

  * •

We enhance the TotalSeg dataset by adding annotations of three important
anatomical structures: skeletal muscle, visceral fat, and subcutaneous fat.
This results in a more comprehensively labeled whole-body CT dataset, and
facilitates the training of our CT-SAM3D model and future research on whole-
body organ segmentation.

##  2 Related Work

Segmentation foundation models. Foundation models have become a pivotal topic
in natural language processing [11, 2] that leads to the development of Vision
Foundation Models (VFM) [3, 49, 44, 51] as well. Based on transformer
architectures and massive multi-modality dataset training, VFM offers the
potential of substantial improvement for various downstream tasks. Segment
Anything Model [28] is the first foundation model for generalized image
segmentation, validating its zero-shot capability for segmenting objects in
the wild. Using input prompts, SAM can perform a wide range of tasks with high
accuracy. The concurrent work SegGPT [65] introduces a general interface that
can be compatible with various segmentation tasks. SEEM [77] offers a unified
method using varied prompts to segment and identify objects in images all at
once.

SAM adaption in medical imaging. Substantial disparities exist between natural
images and medical imaging, and object characteristics in natural images
differ drastically from those in medical images [56]. Although SAM
demonstrates strong zero-shot segmentation capabilities, its performance when
applied directly to the medical imaging domain varies significantly among
different objects, anatomies and modalities [22]. Considerable efforts have
been invested to harness the full potential of SAM in medical imaging [36, 67,
73, 7, 63, 4, 15, 13]. MedSAM [36] curates a medical image cohort of $200K$
masks across 11 modalities and adapts SAM to general medical image
segmentation. As the low-rank adaptation [21] greatly reduces the number of
trainable parameters for fine-tuning tasks, these adapters serve as
alternatives to expand the generality of SAM in medical imaging. SAM-Med2D [7]
and SAMed [73] use 2D adapters for medical image segmentation. MA-SAM [4] and
3DSAM-adapter [15] incorporate a set of 3D adapters into each transformer
block of the encoder to extract 3D information in medical scans. MSA [67]
introduces space and depth adapters specifically designed to process 3D
information while retaining all weights in the original SAM model. However,
adaptation methods often restrict to a limited number of organs/tumors or show
inferior results on unseen anatomies. Alternatively, training 3D models from
scratch could straightforwardly capture important 3D contexts in medical
images. SAM-Med3D [63] simply reforms the 2D SAM into its 3D counterpart to
train a 3D SAM model from scratch using $21K$ medical images and $131K$ masks.
SegVol [13] incorporates a zoom-out-zoom-in mechanism into 3D SAM development
based on $6K$ CT scans and $150K$ masks. Despite the great expansion of images
and masks by aggregating vast partially labeled datasets and generating pseudo
labels, the issues associated with developing a 3D SAM model, as
aforementioned, remain unsolved.

In this work, we develop a 3D point-promptable model (CT-SAM3D) from scratch
by enhancing TotalSeg [66] with 1204 CT scans, only $\sim$5% of data used in
SAM-Med3D and $\sim$20% of that used in SegVol, but with a higher annotated
ratio ($\sim$83%) within valid body region. We also propose a progressively
and spatially aligned prompt encoding technique to ensure the model accurately
responds to user clicks in a higher-dimensional space. Furthermore, we propose
an innovative cross-patch prompt learning approach to capture additional
spatial context, which is particularly beneficial for segmenting organs that
extend beyond the local patch dimensions in 3D patch-wise training. As
validated from our extensive experiments, CT-SAM3D significantly outperforms
previous leading SAM-derived models by a large margin while requiring only a
few interactive clicks as prompts to accurately segment 107 anatomies even
untrained organs.

##  3 Method

Overview of CT-SAM3D. The original SAM [28] is designed to segment 2D natural
images, and adopting it to handle inputs of 3D volumetric images is non-
trivial. In this work, we aim to develop an efficient promptable 3D SAM
network for segmenting 107 classes of anatomies and unseen organs on whole-
body CT scans. The encoder and decoder design in CT-SAM3D is explained for
incorporating the 3D volumetric information in Sec. 3.1. Our two main
technical contributions, progressively and spatially aligned prompt, and
cross-patch prompt for segmenting large organs, are described to ensure the
effective and efficient 3D SAM implementation in Sec. 3.2 and Sec. 3.3,
respectively. The overall framework of CT-SAM3D is illustrated in Fig. 2.

###  3.1 CT-SAM3D Architecture

Recall that SAM’s ViT [12] backbone employs a convolution kernel $(16\times
16)$ to patchify the input image $(1024\times 1024)$, producing a sequence of
tokens with a length of 4096. When encountering 3D space, we need to avoid the
exponential increase of tokens. On the other hand, it has been extensively
demonstrated that hierarchical multi-scale features play a crucial role in
semantic segmentation [52, 9, 23, 59]. A recent SAM-derived study reports that
simply dividing the ViT into four stages and establishing connections between
the feature maps of each stage and the corresponding decoder layers does not
confer any advantages [4]. Taking these into consideration, we incorporate a
hierarchical and memory-efficient Transformer network ResT [74] and construct
skip connections [52] to form another U-shape architecture, as illustrated in
Fig. 2(A). The four-stage ResT-based image encoder has a stem block consisting
of two consecutive 3D convolution layers with a stride of 2 as the initial
feature extractor. For a given 3D input patch $I\in\mathbb{R}^{W\times H\times
D}$, it will be down-scaled by factors of [4, 8, 16, 32] in the hierarchical
encoding path. The decoding feature integration is achieved by channel-wise
concatenation. The 3D transposed convolutions with the kernel size of 2 and
the stride of 2 are used for feature upscaling. The prompt signals are applied
for each decoder stage of the network.

Figure 2: Overall framework of the proposed CT-SAM3D, which is composed of a
ResT-based image encoder and a mask decoder with skip connections (A),
progressively and spatially aligned prompt (B), and cross-patch prompt
training (C). The segmentation prediction is able to leverage all 26 neighbors
for the final result (D).

###  3.2 Progressively and Spatially Aligned Prompt

The vanilla SAM’s spatial prompt has showcased a robust capability of encoding
2D positions through the utilization of random Fourier features (RFF) [50,
58]. In addition to learning the relationship between the designed one-
dimensional prompt embeddings and sampled masks, the SAM model also needs to
learn an additional embedding for distinguishing positive/negative points. For
a prompted position, it will firstly be normalized to
$\mathbf{v}\in[-1,1]^{d}$ w.r.t. input size, $d=2$ in SAM’s space, then a set
of sinusoids is obtained as:

$\textit{RFF}(\mathbf{v})=\left[\cos\left(2\pi\mathbf{b}_{1}^{\mathrm{T}}\mathbf{v}\right),\sin\left(2\pi\mathbf{b}_{1}^{\mathrm{T}}\mathbf{v}\right),\ldots,\cos\left(2\pi\mathbf{b}_{m}^{\mathrm{T}}\mathbf{v}\right),\sin\left(2\pi\mathbf{b}_{m}^{\mathrm{T}}\mathbf{v}\right)\right]^{\mathrm{T}},$
(1)

where $m$ is a configurable feature length, and $\mathbf{b}_{j}(j\in[1,m])$ is
sampled from an isotropic distribution. The final prompting encoding of
$\mathbf{v}$ is $\textit{PE}(\mathbf{v})=\textit{RFF}(\mathbf{v})+\mathbf{E}$,
where $\mathbf{E}$ is the learned positive/negative embedding. We
experimentally find that simply adapting this technique to 3D can make the
optimization process be more complex since it intrinsically lacks spatial
alignment, leading to unexpected interactive behavior.

Partially inspired by the spatially-adaptive normalization [46] that
effectively preserves the location geometry of the semantic label map, we
propose a progressively and spatially aligned prompt (PSAP) for 3D prompt
encoding. In contrast to SAM’s prompt encoding, our proposed method encodes
positive and negative points into two separate click maps, eliminating the
need to learn the positive/negative attribute embedding for prompts. This
principle is also adopted by some previous 2D interactive segmentation methods
[68, 57, 5, 33]. Our PSAP differs in that it omits the need for forwarding
through a computationally intensive image encoder, ensuring faster interactive
responses. Concretely, for a given click point $\mathbf{v}=(x,y,z)$, a
Gaussian heatmap is generated centered around this point. This heatmap is
regarded as feature map $\mathbf{P}$ if it is a positive click; otherwise, it
is feature map $\mathbf{N}$. Subsequently, the mask prediction $\mathbf{Y}$
from the previous iteration (filled with zeros in the initial iteration), is
concatenated with $\mathbf{P}$ and $\mathbf{N}$ to form the composite feature
map $\mathbf{M}$. Then, let $\mathbf{X}_{i-1}$ denote the feature map output
of the $(i-1)^{th}$ decoder layer, $\mathbf{M}$ will be downsampled to
$\mathbf{M}_{i-1}$ that has the same spatial dimensions as $\mathbf{X}_{i-1}$.
As illustrated in Fig. 2(B), we compute
$\gamma(\mathbf{M}_{i-1})=Conv_{\gamma}(Conv(\mathbf{M}_{i-1}))$ and
$\beta(\mathbf{M}_{i-1})=Conv_{\beta}(Conv(\mathbf{M}_{i-1}))$, then the
output of the proposed PSAP is formulated as follows:

$\displaystyle\mathbf{X}_{i}^{{}^{\prime}}=\gamma(\mathbf{M}_{i-1})\text{IN}(\mathbf{X}_{i-1})+\beta(\mathbf{M}_{i-1}),$
(2)

where $\gamma(\mathbf{M}_{i-1})$ and $\beta(\mathbf{M}_{i-1})$ are the learned
modulation parameters of the instance normalization (IN) layer [60], and they
both have the same spatial dimensions as $\mathbf{X}_{i-1}$. Finally,
$\mathbf{X}_{i}^{{}^{\prime}}$ will be upscaled and concatenated with the
corresponding encoder features to form $\mathbf{X}_{i}$. This technique
enables the model to encode 3D clicks along the decoding path progressively
and spatially aligned, resulting in improved efficiency and accuracy for 3D
interactive segmentation.

Algorithm 1 Training Algorithm for Cross-Patch Prompt

1:

2:$U,V$: Patch samples with overlapping;

3:$U_{gt},V_{gt}$: Ground truth masks;

4:$U_{C},V_{C}$: Sampling clicks;

5:$\epsilon$: Learning rate;

6:$N$: Number of iterations per sample;

7:$\mathcal{S},\mathcal{P}$: Segmentation and CPP prediction modules.

8:Optimal network parameters $\theta$

9:while $N>0$ do

10: $U_{p}=\mathcal{S}(U,U_{C})$, $V_{p}=\mathcal{S}(V,V_{C})$

11:
$\mathcal{L}_{local}=\mathcal{L}_{seg}(U_{p},U_{gt})+\mathcal{L}_{seg}(V_{p},V_{gt})$

12:
$V_{C}^{{}^{\prime}}=\mathcal{P}(U,U_{p},V),U_{C}^{{}^{\prime}}=\mathcal{P}(V,V_{p},U)$

13:
$\mathcal{L}_{cross}=\mathcal{L}_{cpp}(U_{C}^{{}^{\prime}},U_{gt}^{{}^{\prime}})+\mathcal{L}_{cpp}(V_{C}^{{}^{\prime}},V_{gt}^{{}^{\prime}})$

14: $\mathcal{L}=\mathcal{L}_{local}+\mathcal{L}_{cross}$

15: $\theta\leftarrow{\theta-\epsilon\nabla_{\theta}{\mathcal{L}}}$

16: Update $U_{C},V_{C}$ according to the error regions

17: $N\leftarrow N-1$

18:end while

###  3.3 Cross-Patch Prompt for Large Anatomy Segmentation

Due to limitations in GPU memory and computation efficiency, it is not
feasible to take as input an entire 3D volume (e.g., $512\times 512\times 192$
for a typical Thoracic CT scan) for network training. However, cropping 3D
medical images into smaller sub-volumes can result in truncation of larger or
tubular-shaped anatomies, e.g., liver or aorta. Training on only 3D local
image patches reduces inference efficiency and requires more manual clicks. To
alleviate this problem, we introduce an innovative cross-patch prompt (CPP)
learning approach to capture a broader spatial context. Building upon the
promptable segmentation network, condensed as the blue module in Fig. 2(C), we
add a light-weight encoder-decoder subnetwork dedicated to predicting CPP.

During CPP’s training iterations, we sample two patches with overlapping
regions in each anatomy. As illustrated in Algorithm 1, the training pipeline
is executed in $N$ iterations to simulate a real-world interactive scenario.
Let $U$ and $V$ denote two patches sharing overlapping regions, $U_{C}$ and
$V_{C}$ denote the sampling clicks from the simulated user. The outputs of the
segmentation network for the two input patches are $U_{p}$ and $V_{p}$, and
the associated segmentation loss for two local patches $\mathcal{L}_{local}$
is calculated (line 3), where $\mathcal{L}_{seg}(\cdot)$ denotes the
combination of Focal loss [32] and Dice loss [42]. Then, the CPP prediction
module takes as inputs the crossed and concatenated tuples $(U,U_{p},V)$ and
$(V,V_{p},U)$ to build a mutual anatomical region identification task (line
4). In this task, we aim to predict a heatmap that can be used as click
prompts as aforementioned. The ground truth heatmaps $V_{gt}^{{}^{\prime}}$
and $U_{gt}^{{}^{\prime}}$ are constructed based on the centroids of the
foregrounds in $V_{gt}$ and $U_{gt}$, respectively. After obtaining the CPP
predictions $U_{C}^{{}^{\prime}}$ and $V_{C}^{{}^{\prime}}$, we compute
$\mathcal{L}_{cpp}(\cdot)$ using a mean squared error loss, then the cross-
patch loss $\mathcal{L}_{cross}$ is calculated (line 5). The CT-SAM3D is then
updated (line 7), considering both the local segmentation loss and cross-patch
prompts prediction loss. We omit the mask input and its updating in the
pseudo-code for simplicity. This training process is also illustrated in Fig.
2(C).

When segmenting a large anatomical structure such as the liver or aorta via
prompting, a simple approach is to start with a small training patch size and
gradually increase the number of clicks until the complete results are
obtained. Yet, this is not user-friendly. As demonstrated in Fig. 2(D), CPP
can decrease workloads by utilizing the nearest 26 spatial neighboring patches
surrounding the selected patch, thereby attaining precise segmentation
outcomes.

##  4 Experiments

We train CT-SAM3D using the training set of the curated TotalSeg++ dataset.
Then, the testing set of TotalSeg++ is used as the internal testing set and
two public datasets (FLARE22 and BTCV) and one in-house Chest CT dataset are
treated as external testing sets. SAM and several SAM-derived methods in
medical imaging are extensively compared. We first introduce the datasets.

TotalSeg dataset consists of 1204 CT scans with 104 anatomical structures
annotated. CT scans were randomly sampled from routine clinical studies
presenting different ages, pathologies, scanners, body parts, sequences, and
sites [66]. Using an in-house developed muscle/fat segmentation model with
manual examination and curation, we enhance the TotalSeg dataset by
introducing annotations of three important yet under-explored anatomies, i.e.,
skeletal muscle, visceral fat, and subcutaneous fat. This results in a
comprehensive whole-body CT dataset where almost every voxel within the body
possesses a semantic label. We refer to the enhanced TotalSeg dataset as
TotalSeg++. Details of the data curation process can be found in our
supplementary material. We comply with the official training and testing
partitioning, which uses 1139 CT scans for training CT-SAM3D and 65 CT scans
as the internal testing.

FLARE22 is an abdominal organ segmentation dataset proposed in the Fast and
Low-resource semi-supervised Abdominal oRgan sEgmentation challenge held at
MICCAI 2022 [37]. It contains CT scans of patients with liver, kidney, spleen,
or pancreas diseases in the test set. The labeled 13 organs include: liver,
spleen, pancreas, right kidney, left kidney, stomach, gallbladder, esophagus,
aorta, inferior vena cava, right adrenal gland, left adrenal gland, and
duodenum. We use the offline test set of 20 CT cases for the purpose of
external testing in our experiment.

BTCV is the Beyond the Cranial Vault abdomen challenge dataset [30] that
includes 30 abdominal CT scans with annotations for 13 organs. The labeled
organs of BTCV differs slightly from FLARE22 as it lacks duodenum but includes
the portal vein and splenic vein annotations. These 30 CT scans are also used
as our external testing set.

Chest CT is a private dataset of 99 esophageal cancer patients. Sternum is
labeled in each chest CT scan. Note that sternum is not annotated in the
TotalSeg++ dataset and we treat it as an unseen organ for external testing.

###  4.1 Evaluation Protocol

To maximize the reproducibility of the evaluation results, we follow the
practice in [63, 28, 57] to simulate the real-world interactive scenario.
Specifically, the first simulated click point is randomly sampled from the
foreground region, and the subsequent point is sampled iteratively using the
farthest point from the boundary of error regions. The subsequent point is
positive if false negative is bigger than false positive otherwise negative.
We measure 3D organ-wise Dice Similarity Coefficient (DSC) and Normalized
Surface Distance (NSD) [39] after $N$ point prompts, where
$N=\\{1,3,5,7,9\\}$. Hence, predictions from 2D SAM-derived methods need to be
heavily merged first before calculating the metrics. Note that we consider the
number of prompt points in the corresponding model space for the evaluation.
In other words, for a 2D SAM-derived model, $N$ indicates the number of
prompts used in each 2D CT slice, while $N$ represents the number of prompts
in a whole 3D CT scan for a 3D SAM-derived model. This evaluation protocol
actually favors the 2D models, since there are a total of $N\times$ organ-
slices prompts used to segment an organ. In comparison, only $N$ prompts are
utilized to segment an organ by the 3D models. Nevertheless, even under this
biased evaluation protocol, we show in our experiments that CT-SAM3D
significantly outperforms all other 2D SAM-derived works by a large margin of
at least $10\%$ DSC, although using a much smaller number of prompts.

Comparing methods. SAM [28] and several recent SAM-inspired medical image
segmentation models are reported and compared, including MedSAM [36], SAMed
[73], MA-SAM [4], SAM-Med2D [7], SAM-Med3D [63] and SegVol [13]. All these
methods provide the pre-trained networks so that we directly apply their
models to our testing datasets to evaluate and report the corresponding
quantitative segmentation performances. Note that the pre-trained models of
SAMed and MA-SAM disable the prompt encoding module. Thus, only a fixed number
of organs that have appeared in their training datasets can be segmented.

Implementation details. Our implementation is built upon PyTorch [47]. The
whole model is trained with AdamW optimizer [35] iteratively with an initial
$lr=1\mathrm{e}^{-4}$, default moment decay and weight decay values. The total
training process contains 1000 epochs: the first 100 epochs to linearly warm
up and the learning rate $lr$ is reduced by a factor of 10 at the 800th epoch.
Our model was trained on 8 A100 GPUs, with a batch size of 4 per GPU and a
sampling number of 8 per volume. The number of iterations per batch is set to
be 5. The first iteration only uses points as prompts, and the subsequent
iterations use both updated points and previous masks as prompts
simultaneously. For $\mathcal{L}_{seg}(\cdot)$, we use a combination of Focal
loss [32] and Dice loss [42] with coefficients of 0.2 and 0.8, respectively.
For data preprocessing, we process all data to have an isotropic spacing of
1.5 mm and apply normalization to scale it to the range of [0, 1]. The patch
size is set as $(64\times 64\times 64)$. Random cropping is employed to obtain
training samples. No additional data augmentation is used either in the
training or inference stages. CT-SAM3D is trained from scratch. To facilitate
the developing and validation of CT-SAM3D, we have also developed an
interactive 3D visualization and segmentation tool with quasi-real-time
responses, details of which are provided in the supplementary material.

Figure 3: Boxplot of DSC results reported by SAM-Med3D, SegVol and our
proposed CT-SAM3D model on the TotalSeg++ testing set. We group the 107 organs
into 12 groups and calculate the median and interquartile range of DSC within
each group.

capposition=top  Methods Liver Kidney_R Spleen Pancreas Aorta IVC RAG LAG
Gallbladder Esophagus Stomach Duodenum Kidney_L mDSC$\uparrow$ SAM [28]
86.0$\pm$5.5 87.6$\pm$8.7 84.5$\pm$8.9 53.4$\pm$10.6 77.5$\pm$16.1
44.5$\pm$16.3 19.4$\pm$14.0 33.9$\pm$15.0 52.4$\pm$16.4 35.2$\pm$6.8
68.0$\pm$11.2 44.4$\pm$12.4 82.6$\pm$11.3 59.2 MedSAM [36] 93.0$\pm$3.1
90.0$\pm$5.3 89.1$\pm$11.0 73.5$\pm$9.6 82.5$\pm$19.7 76.5$\pm$19.4
36.0$\pm$23.6 48.7$\pm$22.6 56.4$\pm$27.1 64.7$\pm$19.9 84.0$\pm$13.0
53.9$\pm$11.7 89.7$\pm$7.9 72.2 SAMed [73] 92.4$\pm$5.1 74.6$\pm$28.9
90.6$\pm$6.3 65.0$\pm$19.6 83.4$\pm$11.7 – – – 71.3$\pm$25.4 – 76.9$\pm$21.5 –
77.2$\pm$27.4 78.9* MA-SAM [4] 92.8$\pm$5.4 80.0$\pm$20.1 87.6$\pm$13.4
74.1$\pm$14.1 86.4$\pm$10.2 80.1$\pm$16.7 45.0$\pm$16.3 46.9$\pm$18.2
77.1$\pm$13.2 70.5$\pm$17.8 75.9$\pm$22.0 – 77.3$\pm$25.4 74.5* SAM-Med2D [7]
91.4$\pm$5.8 83.7$\pm$17.3 83.9$\pm$15.2 58.8$\pm$18.7 60.6$\pm$22.5
18.6$\pm$10.4 10.6$\pm$9.7 27.1$\pm$12.4 32.9$\pm$21.6 28.1$\pm$13.3
72.9$\pm$16.6 45.4$\pm$19.8 86.0$\pm$16.8 53.8 SAM-Med3D [63] 85.4$\pm$13.2
84.2$\pm$9.5 84.7$\pm$11.8 46.9$\pm$14.3 60.4$\pm$10.7 44.5$\pm$13.4
32.6$\pm$20.9 35.3$\pm$18.3 56.0$\pm$19.4 32.6$\pm$16.4 46.9$\pm$19.8
27.4$\pm$13.6 84.9$\pm$6.9 55.5 SegVol [13] 83.9$\pm$25.3 71.7$\pm$30.6
75.9$\pm$28.8 69.4$\pm$16.1 83.1$\pm$12.1 80.3$\pm$13.9 42.1$\pm$13.3
49.7$\pm$22.6 55.6$\pm$31.1 69.6$\pm$8.4 81.1$\pm$20.7 55.6$\pm$19.8
75.1$\pm$22.6 68.7 CT-SAM3D 95.6$\pm$2.0 95.0$\pm$1.8 96.1$\pm$4.4
83.6$\pm$12.0 94.5$\pm$2.8 91.8$\pm$4.7 78.4$\pm$18.0 82.5$\pm$4.0
88.4$\pm$8.1 82.9$\pm$18.1 92.3$\pm$4.4 73.2$\pm$16.8 94.8$\pm$1.4 88.4
Methods Liver Kidney_R Spleen Pancreas Aorta IVC RAG LAG Gallbladder Esophagus
Stomach Duodenum Kidney_L mNSD$\uparrow$ SAM [28] 61.4$\pm$9.6 73.2$\pm$18.2
67.8$\pm$15.4 57.0$\pm$11.1 77.8$\pm$20.0 33.6$\pm$13.7 40.0$\pm$22.3
53.8$\pm$15.5 46.2$\pm$20.4 42.3$\pm$9.6 50.4$\pm$13.6 46.8$\pm$13.0
62.8$\pm$21.4 54.9 MedSAM [36] 74.7$\pm$10.4 78.6$\pm$17.3 74.0$\pm$19.8
72.1$\pm$14.9 79.6$\pm$23.5 71.8$\pm$22.6 53.2$\pm$28.1 57.8$\pm$24.0
46.5$\pm$32.2 67.5$\pm$21.3 72.8$\pm$18.3 53.4$\pm$10.5 84.4$\pm$15.9 68.2
SAMed [73] 86.1$\pm$9.6 74.6$\pm$27.0 89.3$\pm$11.6 77.4$\pm$17.2
87.5$\pm$11.4 – – – 81.5$\pm$22.5 – 72.9$\pm$18.4 – 75.4$\pm$25.8 80.6* MA-SAM
[4] 87.6$\pm$10.7 80.3$\pm$18.1 87.9$\pm$14.8 85.9$\pm$10.8 89.8$\pm$10.2
87.7$\pm$13.6 59.6$\pm$13.6 62.0$\pm$18.9 88.1$\pm$13.1 85.7$\pm$17.5
71.4$\pm$19.7 – 79.6$\pm$24.0 80.5* SAM-Med2D [7] 79.5$\pm$18.3 72.9$\pm$27.2
72.0$\pm$27.2 59.1$\pm$22.8 47.6$\pm$24.7 15.0$\pm$6.7 16.1$\pm$11.6
41.5$\pm$16.5 24.6$\pm$20.2 29.4$\pm$15.7 58.8$\pm$21.5 43.2$\pm$18.8
81.8$\pm$25.1 49.3 SAM-Med3D [63] 76.3$\pm$19.2 82.8$\pm$11.6 84.5$\pm$14.3
59.4$\pm$14.5 70.3$\pm$8.7 62.2$\pm$11.0 66.6$\pm$28.6 70.0$\pm$23.1
76.3$\pm$12.8 58.2$\pm$17.9 48.8$\pm$19.2 40.7$\pm$15.2 83.5$\pm$10.0 67.7
SegVol [13] 79.7$\pm$25.0 74.5$\pm$29.4 77.1$\pm$26.6 83.3$\pm$13.6
91.7$\pm$9.1 92.2$\pm$10.4 75.6$\pm$12.8 76.6$\pm$23.2 68.9$\pm$33.4
91.1$\pm$7.1 84.3$\pm$21.6 69.6$\pm$19.7 78.4$\pm$23.8 80.2 CT-SAM3D
94.7$\pm$4.9 98.0$\pm$2.6 98.1$\pm$7.5 92.5$\pm$15.4 98.4$\pm$3.4 97.4$\pm$4.9
95.2$\pm$19.3 99.7$\pm$0.3 97.6$\pm$6.1 94.3$\pm$18.5 96.1$\pm$6.3
85.3$\pm$16.9 98.4$\pm$2.3 95.8

Table 1: Organ-specific DSC (%) and NSD (%) evaluation on FLARE22. The results
were obtained after 5 clicks. Abbreviations: “IVC”-Inferior Vena Cava,
“RAG”-Right Adrenal Gland, “LAG”-Left Adrenal Gland, “mDSC”-mean DSC,
“mNSD”-mean NSD. The asterisk symbol (*) signifies that the result gathers
exclusively valid values. Figure 4: Mean DSC (%) and NSD (%) under increasing
number of clicks on FLARE22. Figure 5: Qualitative results of different
methods on a subject who exhibits severe renal pathology (green region).
Displayed here are an axial slice in the first row, a coronal slice in the
second row, and a 3D volume rendering in the last row, respectively.

###  4.2 Quantitative Results of Internal & External Testing

The quantitative performance of CT-SAM3D in TotalSeg++ internal testing is
summarized in Fig. 3. We group the original 104 organs of TotalSeg dataset
into 9 groups (head, excretory organs, cardiovascular vessels, main chest
organs, main abdomen organs, vertebrae, ribs, muscles, and other bones) and
use our added three anatomies (curated skeletal muscles, visceral fat and
subcutaneous fat) as another 3 separate categories, and report the group-wise
DSC scores for 12 groups. Compared to other 3D SAM models, i.e., SAM-Med3D
[63] and SegVol [13], our CT-SAM3D yields substantially better performance
across all 12 organ groups. Among them, the ribs group and the newly added
three anatomies exhibit the largest performance improvements with median DSC
scores increased by $\sim$50% (skeletal muscles), $\sim$60% (visceral fat),
and $\sim$70% (subcutaneous fat and ribs). These results suggest that simply
replacing 2D operations in SAM with its 3D counterparts does not work well,
although the TotalSeg dataset is already included in the training set of SAM-
Med3D [63] and SegVol [13]. In contrast, CT-SAM3D uses only $1139$ CT scans
for training and achieves evidently superior performance than both SAM-Med3D
and SegVol, by proposing more suitable 3D SAM network architecture with two
effective novel prompt types.

The quantitative performance of CT-SAM3D and the comparing models on the
external testing set of FLARE22 is shown in Tab. 1 and Fig. 4. Tab. 1 presents
the detailed organ-wise segmentation results when the prompt number $N$=5,
while Fig. 4 illustrates the organ-averaged segmentation accuracy with respect
to the prompt number $N$, as $N$ varies from 1 to 9. As shown in Tab. 1 and
Fig. 4, the proposed CT-SAM3D generalizes well to the external testing and
performs significantly better as compared to other SAM-derived models with a
large margin of $>\sim$10% DSC and $>\sim$15% NSD. Several interesting
observations can be made. 1) The very recent 3D SAM models (SAM-Med3D [63] and
SegVol [13]) do not generalize well, although they are trained on $21K$ and
$6K$ medical images respectively. For instance, SAM-Med3D obtains much worse
performance even compared to those 2D adaptation methods [36, 4, 73] with at
least 17% DSC gap, not to mention the 32.9% DSC gap with our CT-SAM3D model.
2) Although SAMed [73] and MA-SAM [4] disable the prompt encoding module, they
achieve the 2nd and 3rd best accuracy (if not considering the missing
targets). This demonstrates the effectiveness of adaptation methods to some
degree. However, these adaptation methods target on segmenting only a small
fixed number of organs, limiting their general interactive segmentation
ability. 3) We see from Fig. 4 that with more point prompts, CT-SAM3D’s
segmentation accuracy continues to increase, illustrating the effectiveness
and efficiency of CT-SAM3D’s interactive segmentation capacity. In contrast,
with even 9 prompts per input sample, other methods still obtain undesirable
segmentation accuracy, ranging from 55% to 75% DSC or from 50% to 85% NSD. 4)
Not all methods improve consistently when the number of clicks increases,
e.g., MedSAM [36] starts to degrade when $N>3$, indicating its deficiency in
understanding/encoding user’s prompts. A qualitative example is shown in Fig.
5.

Methods | 1 click | 3 clicks
---|---|---
mDSC$\uparrow$ | mNSD$\uparrow$ | mDSC$\uparrow$ | mNSD$\uparrow$
SAM [28] | 14.2 | 6.8 | 39.3 | 31.3
MedSAM [36] | 64.2 | 61.8 | 67.9 | 67.8
SAM-Med2D [7] | 42.3 | 51.3 | 48.1 | 48.3
SAM-Med3D [63] | 41.9 | 51.5 | 47.5 | 61.2
SegVol [13] | 59.5 | 71.3 | 63.4 | 76.2
CT-SAM3D |  78.4 |  88.4 |  82.2 |  93.9

Figure 6: Mean DSC (%) and NSD (%) results on BTCV dataset.

Figure 7: Generalizability testing of CT-SAM3D on an unseen anatomical
structure (sternum) beyond our training label set.

The quantitative performance of organ segmentation on the external testing set
of BTCV is illustrated in Fig. 7. It can be seen that with only 1 click
prompt, CT-SAM3D has a mDSC of 78.4% and mNSD of 88.4%, substantially higher
than all other interactive SAM-derived models [28, 36, 7, 63]. When increasing
the number of clicks to 3, CT-SAM3D improves with $3.8\%$ mDSC (from 78.4% to
82.2%) and $5.5\%$ mNSD (from 88.4% to 93.9%) as compared to 1 click input.
The obtained performance (3 clicks per organ) is also comparable to the fully
supervised segmentation model trained on BTCV. These external testing results
demonstrate CT-SAM3D’s generalization ability to unseen datasets, indicating
the efficiency and effectiveness for potential real clinical usage.

Generalizability to unseen organ segmentation. External testing results on an
unseen organ (sternum) in Chest CT are shown in Fig. 7. As observed, a single
click input has a median DSC of $<60\%$ and the accuracy significantly
improves to $\sim$70% median DSC when applying 3 clicks. The performance
saturates after 5 clicks, resulting in $>75\%$ median DSC, approaching to a
previously reported fully supervised sternum segmentation model with mean DSC
of $81\%$ [17]. Considering that the sternum label never appears during
training, this indicates that CT-SAM3D has the potential to be an effective 3D
interactive segmentation tool when deployed to segment unseen anatomical
structures.

###  4.3 Ablation Study Results

The effectiveness of PSAP. To find an effective prompt encoding method, we
have carefully compared the performance of random Fourier features (RFF) [50,
58] and our proposed progressively and spatially aligned prompt (PSAP) on the
tasks of FLARE22 and BTCV. It is observed that, under the same image encoder
(ResT), PSAP achieves improvements of 20.7% and 21.1% in mDSC and mNSD,
respectively, compared to RFF on BTCV, as shown in Tab. 2. Similar
observations are obtained on FLARE22. Interestingly, when we replace the image
encoder with the UNet [9] structure, which is more common in medical image
analysis, we can still obtain very competitive results. This implies that an
effective prompt encoding mechanism is essentially crucial in 3D interactive
segmentation. It also reveals that PSAP can serve as a plug-and-play module to
be incorporated into different hierarchical backbone network structures to
construct a stronger interactive segmentation model.

The effectiveness of CPP. To investigate the effectiveness of cross-patch
prompt (CPP) on segmenting large organs, we conduct a 1-click experiment on
liver and aorta segmentation tasks (Tab. 3). As expected, the results are
quite disappointing without CPP due to large organ dimensions. Yet, a single
click with enabled CPP mechanism can boost mDSC performance of liver by 45.2%
and 42.4% on FLARE22 and BTCV, respectively. Even on a tubular-shaped aorta
structure, CPP can bring $\sim$10% gains both in mDSC and mNSD scores. It
should also be noted that there is still room for further improvement as the
number of clicks increases.

Table 2: Effectiveness of PSAP under 3 clicks. “I.Enc.” is short for image encoder. I.Enc. | w/ RFF | w/ PSAP | FLARE22 | BTCV
---|---|---|---|---
mDSC$\uparrow$ | mNSD$\uparrow$ | mDSC$\uparrow$ | mNSD$\uparrow$
ResT | ✓ |  | 73.1 | 81.4 | 61.5 | 72.8
ResT |  | ✓ |  87.5 |  94.8 |  82.2 |  93.9
UNet |  | ✓ | 86.4 | 93.8 | 79.9 | 92.0
Table 3: Effectiveness of CPP on large organs under 1 click. Organ | w/o CPP | w/ CPP | FLARE22 | BTCV
---|---|---|---|---
mDSC$\uparrow$ | mNSD$\uparrow$ | mDSC$\uparrow$ | mNSD$\uparrow$
Liver | ✓ |  | 41.9 | 31.5 | 38.1 | 29.7
Liver |  | ✓ |  87.1 |  77.1 |  80.5 |  70.3
Aorta | ✓ |  | 51.9 | 55.7 | 46.8 | 51.1
Aorta |  | ✓ |  61.6 |  64.1 |  57.7 |  61.9

##  5 Conclusion

In this work, we present a comprehensive, scalable and efficient 3D SAM model
on segmenting whole-body CT scans, named CT-SAM3D. Instead of adapting SAM, we
directly develop a 3D promptable segmentation model using a more complete
fully labeled dataset of CT scans (i.e., TotalSeg++). To train CT-SAM3D
effectively using 3D local image patches, we propose two key technical
developments to effectively encode the click prompt in local 3D space and
conduct the cross-patch prompt training to reduce clicks to handle segmenting
large organs. Using an enriched and curated dataset of 1204 CT scans
containing 107 whole-body anatomies, we train and validate CT-SAM3D, which
significantly outperforms all previous SAM-related models by large
quantitative performance margins yet with much fewer click inputs.

## References

  * [1] AlBadawy, E.A., Saha, A., Mazurowski, M.A.: Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Medical physics 45(3), 1150–1158 (2018)
  * [2] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. NeurIPS 33, 1877–1901 (2020)
  * [3] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV. pp. 9650–9660 (2021)
  * [4] Chen, C., Miao, J., Wu, D., Yan, Z., Kim, S., Hu, J., Zhong, A., Liu, Z., Sun, L., Li, X., et al.: Ma-sam: Modality-agnostic sam adaptation for 3d medical image segmentation. arXiv preprint arXiv:2309.08842 (2023)
  * [5] Chen, X., Zhao, Z., Zhang, Y., Duan, M., Qi, D., Zhao, H.: Focalclick: Towards practical interactive image segmentation. In: CVPR. pp. 1300–1309 (2022)
  * [6] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: CVPR. pp. 1290–1299 (2022)
  * [7] Cheng, J., Ye, J., Deng, Z., Chen, J., Li, T., Wang, H., Su, Y., Huang, Z., Chen, J., Jiang, L., Sun, H., He, J., Zhang, S., Zhu, M., Qiao, Y.: Sam-med2d. arXiv preprint arXiv:2308.16184 (2023)
  * [8] Chilamkurthy, S., Ghosh, R., Tanamala, S., Biviji, M., Campeau, N.G., Venugopal, V.K., Mahajan, V., Rao, P., Warier, P.: Deep learning algorithms for detection of critical findings in head ct scans: a retrospective study. The Lancet 392(10162), 2388–2396 (2018)
  * [9] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: MICCAI. pp. 424–432. Springer (2016)
  * [10] Deng, R., Cui, C., Liu, Q., Yao, T., Remedios, L.W., Bao, S., Landman, B.A., Wheless, L.E., Coburn, L.A., Wilson, K.T., et al.: Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging. arXiv preprint arXiv:2304.04155 (2023)
  * [11] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
  * [12] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [13] Du, Y., Bai, F., Huang, T., Zhao, B.: Segvol: Universal and interactive volumetric medical image segmentation. arXiv preprint arXiv:2311.13385 (2023)
  * [14] Ferré, E.M., Break, T.J., Burbelo, P.D., Allgäuer, M., et al.: Lymphocyte-driven regional immunopathology in pneumonitis caused by impaired central immune tolerance. Science translational medicine 11(495), eaav5597 (2019)
  * [15] Gong, S., Zhong, Y., Ma, W., Li, J., Wang, Z., Zhang, J., Heng, P.A., Dou, Q.: 3dsam-adapter: Holistic adaptation of sam from 2d to 3d for promptable medical image segmentation. arXiv preprint arXiv:2306.13465 (2023)
  * [16] Guo, D., Jin, D., Zhu, Z., Ho, T.Y., Harrison, A.P., Chao, C.H., Xiao, J., Lu, L.: Organ at risk segmentation for head and neck cancer using stratified learning and neural architecture search. In: CVPR. pp. 4223–4232 (2020)
  * [17] Guo, D., Ye, X., Ge, J., Di, X., Lu, L., Huang, L., Xie, G., Xiao, J., Lu, Z., Peng, L., et al.: Deepstationing: thoracic lymph node station parsing in ct scans using anatomical context encoding and key organ auto-search. In: MICCAI. pp. 3–12. Springer (2021)
  * [18] Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H.R., Xu, D.: Unetr: Transformers for 3d medical image segmentation. In: WACV. pp. 574–584 (2022)
  * [19] He, S., Bao, R., Li, J., Grant, P.E., Ou, Y.: Accuracy of segment-anything model (sam) in medical image segmentation tasks. arXiv preprint arXiv:2304.09324 (2023)
  * [20] Hesamian, M.H., Jia, W., He, X., Kennedy, P.: Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging 32, 582–596 (2019)
  * [21] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * [22] Huang, Y., Yang, X., Liu, L., Zhou, H., Chang, A., Zhou, X., Chen, R., Yu, J., Chen, J., Chen, C., et al.: Segment anything model for medical images? Medical Image Analysis 92, 103061 (2024)
  * [23] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [24] Iyer, K.S., Newell Jr, J.D., Jin, D., Fuld, M.K., Saha, P.K., Hansdottir, S., Hoffman, E.A.: Quantitative dual-energy computed tomography supports a vascular etiology of smoking-induced inflammatory lung disease. American journal of respiratory and critical care medicine 193(6), 652–661 (2016)
  * [25] Ji, Z., Shen, Y., Ma, C., Gao, M.: Scribble-based hierarchical weakly supervised learning for brain tumor segmentation. In: MICCAI. pp. 175–183. Springer (2019)
  * [26] Jin, D., Guo, D., Ge, J., Ye, X., Lu, L.: Towards automated organs at risk and target volumes contouring: Defining precision radiation therapy in the modern era. Journal of the National Cancer Center (2022)
  * [27] Jin, D., Guo, D., Ho, T.Y., Harrison, A.P., Xiao, J., Tseng, C.K., Lu, L.: Deeptarget: Gross tumor and clinical target volume segmentation in esophageal cancer radiotherapy. Medical Image Analysis 68, 101909 (2021)
  * [28] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: ICCV. pp. 4015–4026 (October 2023)
  * [29] Koohbanani, N.A., Jahanifar, M., Tajadin, N.Z., Rajpoot, N.: Nuclick: a deep learning framework for interactive segmentation of microscopic images. Medical Image Analysis 65, 101771 (2020)
  * [30] Landman, B., Xu, Z., Igelsias, J., Styner, M., Langerak, T., Klein, A.: Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge. In: Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge. vol. 5, p. 12 (2015)
  * [31] Lee, H.H., Bao, S., Huo, Y., Landman, B.A.: 3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation. arXiv preprint arXiv:2209.15076 (2022)
  * [32] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV. pp. 2980–2988 (2017)
  * [33] Liu, Q., Xu, Z., Bertasius, G., Niethammer, M.: Simpleclick: Interactive image segmentation with simple vision transformers. In: ICCV. pp. 22290–22300 (2023)
  * [34] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR. pp. 3431–3440 (2015)
  * [35] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [36] Ma, J., He, Y., Li, F., Han, L., You, C., Wang, B.: Segment anything in medical images. Nature Communications 15(1), 654 (2024)
  * [37] Ma, J., Zhang, Y., Gu, S., Ge, C., Ma, S., Young, A., Zhu, C., Meng, K., Yang, X., Huang, Z., et al.: Unleashing the strengths of unlabeled data in pan-cancer abdominal organ quantification: the flare22 challenge. arXiv preprint arXiv:2308.05862 (2023)
  * [38] Maciej, A., Haoyu, D., Hanxue, G., Jichen, Y., Nicholas, K., Yixin, Z.: Segment anything model for medical image analysis: an experimental study. arXiv preprint arXiv:2304.10517 2 (2023)
  * [39] Maier-Hein, L., Menze, B., et al.: Metrics reloaded: Pitfalls and recommendations for image analysis validation. arXiv preprint arXiv:2206.01653 (2022)
  * [40] Maleike, D., Nolden, M., Meinzer, H.P., Wolf, I.: Interactive segmentation framework of the medical imaging interaction toolkit. Computer methods and programs in biomedicine 96(1), 72–83 (2009)
  * [41] McKinney, S.M., Sieniek, M., Godbole, V., Godwin, J., Antropova, N., Ashrafian, H., Back, T., Chesus, M., Corrado, G.S., Darzi, A., et al.: International evaluation of an ai system for breast cancer screening. Nature 577(7788), 89–94 (2020)
  * [42] Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV). pp. 565–571. IEEE (2016)
  * [43] Mitani, A., Huang, A., Venugopalan, S., Corrado, G.S., Peng, L., Webster, D.R., Hammel, N., Liu, Y., Varadarajan, A.V.: Detection of anaemia from retinal fundus images via deep learning. Nature Biomedical Engineering 4(1), 18–27 (2020)
  * [44] Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., et al.: Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 (2023)
  * [45] Ouyang, W., Mueller, F., Hjelmare, M., Lundberg, E., Zimmer, C.: Imjoy: an open-source computational platform for the deep learning era. Nature methods 16(12), 1199–1200 (2019)
  * [46] Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis with spatially-adaptive normalization. In: CVPR. pp. 2337–2346 (2019)
  * [47] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. NeurIPS 32 (2019)
  * [48] Pieper, S., Halle, M., Kikinis, R.: 3d slicer. In: 2004 2nd IEEE international symposium on biomedical imaging: nano to macro (IEEE Cat No. 04EX821). pp. 632–635. IEEE (2004)
  * [49] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)
  * [50] Rahimi, A., Recht, B.: Random features for large-scale kernel machines. NeurIPS 20 (2007)
  * [51] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2), 3 (2022)
  * [52] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI. pp. 234–241. Springer (2015)
  * [53] Roth, H.R., Lu, L., Liu, J., Yao, J., Seff, A., Cherry, K., Kim, L., Summers, R.M.: Improving computer-aided detection using convolutional neural networks and random view aggregation. IEEE transactions on medical imaging 35(5), 1170–1181 (2015)
  * [54] Sakinis, T., Milletari, F., Roth, H., Korfiatis, P., Kostandy, P., Philbrick, K., Akkus, Z., Xu, Z., Xu, D., Erickson, B.J.: Interactive segmentation of medical images through fully convolutional neural networks. arXiv preprint arXiv:1903.08205 (2019)
  * [55] Shaker, A., Maaz, M., Rasheed, H., Khan, S., Yang, M.H., Khan, F.S.: Unetr++: delving into efficient and accurate 3d medical image segmentation. arXiv preprint arXiv:2212.04497 (2022)
  * [56] Shin, H.C., Roth, H.R., Gao, M., Lu, L., Xu, Z., Nogues, I., Yao, J., Mollura, D., Summers, R.M.: Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning. IEEE Transactions on Medical Imaging 35(5), 1285–1298 (2016)
  * [57] Sofiiuk, K., Petrov, I.A., Konushin, A.: Reviving iterative training with mask guidance for interactive segmentation. In: ICIP. pp. 3141–3145. IEEE (2022)
  * [58] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., Ng, R.: Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS 33, 7537–7547 (2020)
  * [59] Tang, Y., Yang, D., Li, W., Roth, H.R., Landman, B., Xu, D., Nath, V., Hatamizadeh, A.: Self-supervised pre-training of swin transformers for 3d medical image analysis. In: CVPR. pp. 20730–20740 (2022)
  * [60] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
  * [61] Wald, T., Roy, S., Koehler, G., Disch, N., Rokuss, M.R., Holzschuh, J., Zimmerer, D., Maier-Hein, K.: Sam. md: Zero-shot medical image segmentation capabilities of the segment anything model. In: Medical Imaging with Deep Learning, short paper track (2023)
  * [62] Wang, G., Li, W., Zuluaga, M.A., Pratt, R., Patel, P.A., Aertsen, M., Doel, T., David, A.L., Deprest, J., Ourselin, S., et al.: Interactive medical image segmentation using deep learning with image-specific fine tuning. IEEE transactions on medical imaging 37(7), 1562–1573 (2018)
  * [63] Wang, H., Guo, S., Ye, J., Deng, Z., Cheng, J., Li, T., Chen, J., Su, Y., Huang, Z., Shen, Y., Fu, B., Zhang, S., He, J., Qiao, Y.: Sam-med3d. arXiv preprint arXiv:2310.15161 (2023)
  * [64] Wang, X., Zhang, L., Roth, H., Xu, D., Xu, Z.: Interactive 3d segmentation editing and refinement via gated graph neural networks. In: International Workshop on Graph Learning in Medical Imaging. pp. 9–17. Springer (2019)
  * [65] Wang, X., Zhang, X., Cao, Y., Wang, W., Shen, C., Huang, T.: Seggpt: Segmenting everything in context. arXiv preprint arXiv:2304.03284 (2023)
  * [66] Wasserthal, J., Breit, H.C., Meyer, M.T., Pradella, M., Hinck, D., Sauter, A.W., Heye, T., Boll, D.T., Cyriac, J., Yang, S., et al.: Totalsegmentator: Robust segmentation of 104 anatomic structures in ct images. Radiology: Artificial Intelligence 5(5) (2023)
  * [67] Wu, J., Fu, R., Fang, H., Liu, Y., Wang, Z., Xu, Y., Jin, Y., Arbel, T.: Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304.12620 (2023)
  * [68] Xu, N., Price, B., Cohen, S., Yang, J., Huang, T.S.: Deep interactive object selection. In: CVPR. pp. 373–381 (2016)
  * [69] Ye, X., Guo, D., Ge, J., Yan, S., Xin, Y., Song, Y., Yan, Y., Huang, B.s., Hung, T.M., Zhu, Z., et al.: Comprehensive and clinically accurate head and neck cancer organs-at-risk delineation on a multi-institutional study. Nature Communications 13(1), 6137 (2022)
  * [70] Yue, W., Zhang, J., Hu, K., Xia, Y., Luo, J., Wang, Z.: Surgicalsam: Efficient class promptable surgical instrument segmentation. arXiv preprint arXiv:2308.08746 (2023)
  * [71] Yushkevich, P.A., Gao, Y., Gerig, G.: Itk-snap: An interactive tool for semi-automatic segmentation of multi-modality biomedical images. In: 2016 38th annual international conference of the IEEE engineering in medicine and biology society (EMBC). pp. 3342–3345. IEEE (2016)
  * [72] Zhang, J., Shi, Y., Sun, J., Wang, L., Zhou, L., Gao, Y., Shen, D.: Interactive medical image segmentation via a point-based interaction. Artificial Intelligence in Medicine 111, 101998 (2021)
  * [73] Zhang, K., Liu, D.: Customized segment anything model for medical image segmentation. arXiv preprint arXiv:2304.13785 (2023)
  * [74] Zhang, Q., Yang, Y.B.: Rest v2: simpler, faster and stronger. NeurIPS 35, 36440–36452 (2022)
  * [75] Zhao, F., Xie, X.: An overview of interactive medical image segmentation. Annals of the BMVA 2013(7), 1–22 (2013)
  * [76] Zhou, H.Y., Guo, J., Zhang, Y., Han, X., Yu, L., Wang, L., Yu, Y.: nnformer: Volumetric medical image segmentation via a 3d transformer. IEEE Transactions on Image Processing (2023)
  * [77] Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Gao, J., Lee, Y.J.: Segment everything everywhere all at once. arXiv preprint arXiv:2304.06718 (2023)

Appendix

##  Appendix 0.A Data Curation Process

Using an in-house developed muscle/fat segmentation model with manual
examination and curation, we enhance the TotalSeg [66] dataset by introducing
annotations of three important yet under-explored anatomies, i.e., skeletal
muscle, visceral fat, and subcutaneous fat. This results in a more complete
whole-body CT dataset where overall $\sim$83% of voxels within the body region
are semantically labeled. To accurately determine this annotated ratio, we
utilize a body region prediction model to exclude areas of background, defined
as regions with a Hounsfield Unit (HU) of -1024 and regions corresponding to
the CT table in the imaging data. Subsequently, we cumulate the count of
annotated pixels within the body across the entire dataset to ascertain the
overall proportion of intracorporeal annotations. More specifically on the
curation process of the newly added three anatomies, we use the in-house
muscle/fat segmentation model to predict the skeletal muscle, visceral fat,
and subcutaneous fat regions for all scans in TotalSeg dataset. The tricky
part is that we retain the original labels for those regions because a portion
of the muscles have already been annotated in TotalSeg, e.g., the iliopsoas.
Then, two radiologists with over 5 years of experience manually refine the
boundaries between the newly added annotations and existing structures. We
refer to the enhanced dataset as TotalSeg++, for CT-SAM3D developing and
evaluation.

##  Appendix 0.B Hyper-parameters and Network Details

We summarize the hyper-parameters and network details in Tab. 4. One thing
that needs a detailed explanation is how we obtain samples $U$ and $V$ with
overlapping regions during training. Concretely, we first sample a specific
anatomical label from the label set of the input scan. Then, from all the
foreground points of this anatomical structure, we randomly select an anchor
point. Centering on this point, we allow the centers of $U$ and $V$ to
randomly deviate from the anchor point by up to half the patch size in each
direction. As a result, the distance between the centers of $U$ and $V$ on any
axis falls within the range of $[0,64]$, when patch size is set as $(64\times
64\times 64)$. We also elaborate on the network architecture configuration and
the computational analysis in Tab. 4. It can be observed that the light-weight
CPP prediction network $\mathcal{P}$ adds only a minor computational overhead
(1.3G FLOPs) compared to the primary interactive segmentation network
$\mathcal{S}$ (124.4G FLOPs).

Table 4: Hyper-parameters and detailed computational parameters. Configuration | Value
---|---
AdamW $\beta$ | $(0.9,0.999)$
Weight decay | $1e^{-2}$
Initial learning rate | $1e^{-4}$
Warmup epochs | 100
Training epochs | 1000
Batch size on single GPU | 4
Number of GPUs | 8
Number of samples per volume | 8
Number of iterations per batch | 5
Patch size | $(64\times 64\times 64)$
Center distance range between $U$ and $V$ | $[0,64]$
#Params of segmentation network $\mathcal{S}$ | 72.5M
FLOPs of $\mathcal{S}$ | 124.4G
Embedding dimensions in four stages of $\mathcal{S}$ | [96, 192, 384, 768]
Number of ResT blocks in four stages of $\mathcal{S}$ | [1, 2, 6, 2]
#Params of CPP prediction network $\mathcal{P}$ | 4.8M
FLOPs of $\mathcal{P}$ | 1.3G
NSD Tolerance | 5

##  Appendix 0.C Quasi-Real-Time 3D Interactive Segmentation Tool

Figure 8: The diagram of our interactive segmentation tool.

Classic medical imaging interaction tools, such as ITK-SNAP [71] and 3D Slicer
[48], have been instrumental in propelling the development of the field.
Nonetheless, for 3D interactive segmentation tasks, these platforms often
necessitate meticulous slice-by-slice manipulation to obtain fine-grained
segmentation outcomes, indicating substantial potential for efficiency
enhancements. On the other hand, interactive models like SAM [28] are becoming
increasingly intelligent nowadays, and how to efficiently run such models on
these platforms remains an open problem. Therefore, we seek to build our 3D
interactive segmentation tool based on a flexible and open-source browser-
based platform, namely ImJoy [45], to facilitate the developing and validation
of our proposed CT-SAM3D. ImJoy can access various computational resources by
performing computations either in the browser itself or using “plugins” that
can run either locally or remotely. Building upon this, we embed the 3D image
and mask visualization directly within the browser, which also receives user
click prompts, while launching the model inference computation to GPU servers
for enhanced efficiency. An illustrative diagram is shown in Fig. 8. With this
efficient 3D interactive segmentation tool, users can experience response
times of less than one second for each interaction cycle.

##  Appendix 0.D Additional Experimental Results

###  0.D.1 Comparison with Leading Automatic Segmentation Methods

Several cutting-edge automatic segmentation methods are compared in this
section to provide a full picture about the positioning of CT-SAM3D in current
medical image segmentation field, as shown in Tab. 5. It is important to
clarify that we are not intending to claim superiority of interactive methods
over fully automatic ones. Indeed, an effective interactive method is expected
to utilize prompts and error regions to enhance segmentation refinement. We
wish to draw attention to a different perspective and the practical value that
emerges from this comparison: a promptable, human-in-the-loop collaborative
approach can further boost segmentation performance and step towards real-
world clinical applications.

Table 5: Quantitative comparison with leading automatic segmentation methods on FLARE22 dataset. Methods | mDSC$\uparrow$ | mNSD$\uparrow$
---|---|---
nnU-Net [23] | 83.6 | 88.5
UNETR [18] | 85.8 | 90.0
Swin UNETR [59] | 86.8 | 91.2
nnFormer [76] | 86.9 | 91.3
TotalSeg Model [66] | 86.5 | 94.1
CT-SAM3D |  88.4 |  95.8

###  0.D.2 Generalizability to Tumor Segmentation

Beyond anatomical structure segmentation tasks, we have applied CT-SAM3D to
more challenging endeavors, i.e., tumor segmentation, to investigate its zero-
shot capabilities. Two public tumor segmentation datasets from Medical
Segmentation Decathlon challenge held at MICCAI 2018 are tested: 1) pancreas
tumor segmentation dataset (MSD-Pancreas), and 2) colon cancer primaries
segmentation dataset (MSD-Colon). They contain 281 and 126 abdominal CT scans
respectively. A previous SAM adaptation method, 3DSAM-adapter [15], has also
reported results on these two datasets. 3DSAM-adapter are fine-tuned on each
dataset, followed by testing on a randomly selected 20% of the data. Our
significant distinction is that we do not conduct any fine-tuning at all,
resulting in a truly zero-shot testing. For fair comparison, we report results
on the test splits as in 3DSAM-adapter. The results are summarized in Tab. 6.
Some state-of-the-art automatic segmentation methods are also included. As
observed, a promptable, human-in-the-loop approach can greatly elevate the
upper-bound of challenging tumor segmentation results. The relative
improvements are even higher than organ segmentation tasks, e.g., CT-SAM3D
outperforms nnU-Net by 17.95% mDSC on pancreas tumor segmentation, primarily
benefiting from input prompts that directly indicate the location of the
tumors. When compared to the specifically fine-tuned 3DSAM-adapter, our CT-
SAM3D achieves comparable performance on both pancreas tumor segmentation and
colon cancer segmentation tasks under 10 clicks, maintaining a slight edge on
mDSC yet exhibiting a minor shortfall in mNSD, even under a more challenging
zero-shot setting. A qualitative comparison is shown in Fig. 11, where we
showcase results of SAM [28], 3DSAM-adapter [15], and our proposed CT-SAM3D
under gradually increased click prompts. This comparison further demonstrates
the good zero-shot capability of CT-SAM3D.

Table 6: Generalizability testing of CT-SAM3D on tumor segmentation tasks. The results of the compared automatic segmentation methods and the promptable methods, SAM and 3DSAM-adapter, are drawn from the paper of 3DSAM-adapter [15]. Methods | Pancreas Tumor | Colon Cancer
---|---|---
mDSC$\uparrow$ | mNSD$\uparrow$ | mDSC$\uparrow$ | mNSD$\uparrow$
nnU-Net [23] | 41.65 | 62.54 | 43.91 | 52.52
nnFormer [76] | 36.53 | 53.97 | 24.28 | 32.19
Swin UNETR [59] | 40.57 | 60.05 | 35.21 | 42.94
UNETR++ [55] | 37.25 | 53.59 | 25.36 | 30.68
3D UX-Net [31] | 34.83 | 52.56 | 28.50 | 32.73
SAM [28] | 30.55 | 32.91 | 39.14 | 42.70
3DSAM-adapter [15] | 57.47 |  79.62 | 49.99 |  65.67
CT-SAM3D |  59.60 | 77.93 |  50.68 | 64.14

###  0.D.3 Detailed Quantitative Results of TotalSeg++

Due to space limitations, it is infeasible to report every single organ
performance in the main text as there is a total of 107 anatomical structures
in TotalSeg++. Hence, we group the original 104 organs of TotalSeg into 9
groups (head, excretory organs, cardiovascular vessels, main chest organs,
main abdomen organs, vertebrae, ribs, muscles, and other bones) and use our
added three anatomies (curated skeletal muscles, visceral fat, and
subcutaneous fat) as another 3 separate categories, and report the group-wise
DSC in the main text. The label list of each group is summarized in Tab. 7. In
this supplementary material, we also provide a detailed complement to the
results for all 107 anatomic structures in the test set of TotalSeg++, as
shown in Fig. 9 and Fig. 10. As observed, our CT-SAM3D significantly
outperforms SegVol [13] (the overall second-best 3D SAM method) across all
structures, achieving superior results in both DSC and NSD metrics.

Table 7: Grouped label lists of TotalSeg++. Anatomy Group | Anatomy Labels
---|---
Head | 50, 93
Excretory Organs | 2, 3, 55, 57, 104
Cardiovascular Vessels | 7, 8, 9, 49, 51–54
Main Chest Organs | 13–17, 42–48
Main Abdomen Organs | 1, 4, 5, 6, 10, 11, 12, 56
Vertebrae | 18–41
Ribs | 58–81
Other Bones | 82–92
Muscles | 94–103
Skeletal Muscles (Curated) | 105
Visceral Fat (Curated) | 106
Subcutaneous Fat (Curated) | 107
Figure 9: DSC results of all 107 anatomic structures annotated in TotalSeg++.
Best viewed when rotated 90 degrees. Figure 10: NSD results of all 107
anatomic structures annotated in TotalSeg++. Best viewed when rotated 90
degrees. Figure 11: Qualitative visualizations of pancreas tumor and colon
cancer segmentation. We showcase results of SAM, 3DSAM-adapter, and our
proposed CT-SAM3D under gradually increased click prompts. This comparison
prominently demonstrates the good zero-shot capability of CT-SAM3D. Best
viewed in color. Figure 12: Qualitative results of CT-SAM3D on TotalSeg++
testing set. The first row shows ground truths and the second row shows our
merged prediction results. Overall, our method demonstrates promising
consistency with ground truths, except for a few discrepancies indicated by
golden arrows. Best viewed in color.

###  0.D.4 Additional Qualitative Results

We provide more qualitative results of our CT-SAM3D in Fig. 12. From this
figure, it is evident that our method can not only segment major organs
effectively but also handle extensive regions of skeletal muscle, visceral
fat, and subcutaneous fat with proficiency. This indicates that it can provide
fundamental support for downstream tasks such as body composition analysis and
sarcopenia assessment, playing a practical role in clinical settings.

##  Appendix 0.E Limitations

Although the proposed CT-SAM3D has achieved appealing results on a large
variety of anatomic structures and tumors, there still exist several
limitations.

Firstly, our model may have unsatisfactory results for extremely incomplete
structures presented in the scan. E.g., the 6th cervical vertebra
(Vertebrae_C6) and the right humerus (Humerus_R) show unstable performance in
Fig. 9. Upon further examination, it has been found that a substantial
percentage of the scans containing these two categories have corresponding
annotation regions fewer than 100 voxels, posing a great challenge for our
model.

Secondly, since our annotations do not contain different levels of
granularity, prompting anatomic structures with finer granularity may require
additional interactions. E.g., in the case of the liver segmentation, we only
have annotations for the entire liver to train our model. If a user wishes to
prompt a specific Couinaud segment of the liver, it would necessitate more
interactions.

Lastly, our current system is not yet capable of automatically extracting
semantic information of anatomic structures, which will be addressed in our
future work.