# MIS-FM: 3D Medical Image Segmentation using Foundation Models Pretrained on
a Large-Scale Unannotated Dataset

Guotai Wang1,2, Jianghao Wu1,2, Xiangde Luo1,2, Xinglong Liu3, Kang Li4,2,
Shaoting Zhang1,2
1School of Mechanical and Electrical Engineering, University of Electronic
Science and Technology of China
2Shanghai Artificial Intellgence Laboratory
3SenseTime Research, Beijing, China
4West China Biomedical Big Data Center, Sichuan University
<EMAIL_ADDRESS>
###### Abstract

Pretraining with large-scale 3D volumes has a potential for improving the
segmentation performance on a target medical image dataset where the training
images and annotations are limited. Due to the high cost of acquiring pixel-
level segmentation annotations on the large-scale pretraining dataset,
pretraining with unannotated images is highly desirable. In this work, we
propose a novel self-supervised learning strategy named Volume Fusion (VF) for
pretraining 3D segmentation models. It fuses several random patches from a
foreground sub-volume to a background sub-volume based on a predefined set of
discrete fusion coefficients, and forces the model to predict the fusion
coefficient of each voxel, which is formulated as a self-supervised
segmentation task without manual annotations. Additionally, we propose a novel
network architecture based on parallel convolution and transformer blocks that
is suitable to be transferred to different downstream segmentation tasks with
various scales of organs and lesions. The proposed model was pretrained with
110k unannotated 3D CT volumes, and experiments with different downstream
segmentation targets including head and neck organs, thoracic/abdominal organs
showed that our pretrained model largely outperformed training from scratch
and several state-of-the-art self-supervised training methods and segmentation
models. The code and pretrained model are available at
https://github.com/openmedlab/MIS-FM.

##  1 Introduction

Automatic segmentation of 3D medical images plays an important role in
computer-assisted diagnosis and treatment planning [38]. In recent years, deep
learning models such as Convolution Neural Networks (CNNs) [21] and
Transformers [18] have achieved state-of-the-art segmentation performance, and
some of them have obtained clinically applicable results in segmentation of
Organs-at-Risk (OAR) and brain lesions [49, 37]. Their high performance
heavily relies on supervised learning from a large set of training images for
the given segmentation targets with pixel-level manual annotations.

Figure 1: Overview of our proposed self-supervised pretraining method based on
volume fusion.

However, such a training paradigm is not applicable for a wide range of long-
tailed diseases due to challenges related to collection of data and
annotations [39]. First, the number of images may be limited in rare diseases
and non-routine imaging protocols, making it hard to collect a large set of
images for training deep learning models. Second, annotating 3D medical images
with pixel-level labels requires expert knowledge and is time-consuming, which
makes it expensive and difficult to obtain segmentation annotations for a
large training set. As a result, when the available training set is small,
which is the case for most diseases and targets of interest, the segmentation
performance of deep learning models is still limited.

To deal with this problem, one promising solution is to pretrain the
segmentation model on available large-scale datasets, such as public datasets
in the community and in-house datasets at hand that are easier to obtain than
images for the target segmentation task [48]. As pretraining with a large-
scale dataset can learn diverse features that are potentially general to
different downstream tasks, fine-tuning the model on a target dataset can not
only improve the convergence speed, but also reduce the risk of over-fitting,
leading to higher performance. For example, models pretrained on ImageNet have
been widely used for detection and segmentation models in computer vision [6,
9]. Recently, large-scale Contrastive Language-Image Pretraining (CLIP) models
[31] have also showed excellent generalizability in image recognition tasks.
However, these models pretrained on natural images may have a limited
performance on medical images due to their differences in contrast, image
styles and object categories. In addition, extracting 3D features is essential
for 3D medical image segmentation, while the pretrained models for general
computer vision only deal with 2D images. Therefore, it is desirable to
pretrain a model on large-scale 3D medical images to boost the performance in
downstream segmentation tasks.

Recently, some researches have tried to leverage public medical image datasets
with partial or full annotations to pretrain 3D segmentation models using
supervised learning [24, 20]. Despite their promising performance, the
supervised pretraining paradigm is limited by the high annotation cost and not
scalable to thousands of unannotated images. Instead, pretraining with Self-
Supervised Learning (SSL) is more appealing when the large-scale pretraining
dataset is not annotated, which is the common case in practice [40]. Most
existing SSL strategies are based on image/patch-level classification or
contrastive learning [7, 3, 12] to learn an effective feature extractor (i.e,
encoder), while segmentation models usually have an additional decoder that is
not used in the pretext tasks during pretraining, which may make the self-
supervised pretraining less effective. Using image restoration [52, 16, 10] as
the pretext task in SSL can alleviate this problem by pretraining the encoder
and encoder for pixel-level intensity prediction. However, the mismatch
between the pretraining and downstream tasks (i.e., regression vs
segmentation) may hinder the feature transferability and limit the
segmentation performance.

To address these issues, we propose a novel self-supervised pretraining
strategy named Volume Fusion (VF) for 3D medical image segmentation.
Differently from existing SSL methods that define pretext tasks as image-level
classification or image restoration, our method uses a pseudo-segmentation
pretext task that pretrains a 3D segmentation model directly. Specifically, it
fuses a foreground sub-volume from a 3D image with a background sub-volume
from another image based on a fusion coefficient map, and the coefficient of
each voxel takes values from a discrete set where each element is treated as a
class. The model to pretrain takes the fused sub-volume as input, and predicts
the class label of each voxel, which is exactly a supervised segmentation
pretext task, as illustrated in Fig. 1. It has several advantages over
existing pretraining approaches for medical image segmentation models. First,
compared with existing supervised learning-based pretraining [24, 20], it
generates paired input images and segmentation label maps automatically, which
does not require manual annotations, making it be able to easily leverage
unannotated images at a larger scale for pretraining at zero annotation cost.
Second, compared with methods that only pretrain a feature encoder [7, 12, 3],
it pretrains the encoder and decoder at the same time, leading the entire
segmentation model to be sufficiently pretrained. In addition, compared with
image restoration [52, 16, 10], the gap between pretraining and downstream
tasks are minimized, as both of them are formulated as segmentation tasks,
which makes the features learned during pretraining be more transferable to
downstream segmentation.

The contribution of this work is three-fold:

  * •

First, we propose Volume Fusion, a novel self-supervised learning method to
pretrain 3D medical image segmentation models. It is free of manual
annotation, and enables pretraining the entire segmentation model from large-
scale unannotated images by using a pseudo-segmentation pretext task to match
the downstream segmentation task.

  * •

Second, we propose a Parallel Convolution and Transformer Network (PCT-Net)
that combines local and long-range feature learning for medical image
segmentation. It has a strong feature learning ability with moderate model
size, which is suitable for pretraining and can be used on a common GPU
device.

  * •

Thirdly, we release a pretrained PCT-Net based on 110k unannotated 3D Computer
Tomography (CT) scans of various body parts. To the best of our knowledge,
this is the largest medical image dataset for pretraining so far, while the
pretraining datasets are much smaller in existing works (around 5k at most)
[40]. Effectiveness of our pretrained model is validated with several
downstream tasks for segmentation of head and neck, thoracic and abdominal
organs, respectively.

##  2 Related Works

###  2.1 Medical Image Segmentation Models

Deep learning models for medical image segmentation can be roughly divided
into CNN-based, Transformer-based and hybrid models that combine CNN and
Transformer. CNN-based models are dominated by U-Net-like neural networks [34,
1, 21] with encoder-decoder structures. Many new blocks have been introduced
to extend U-Net for better performance, such as residual connection [28],
nested connection [51], spatial and channel attentions [35, 13], multi-scale
feature learning [14] and 2.5D convolution [43].

Recently, Vision Transformers (ViT) [9, 25, 18] based on self-attention have
been increasingly investigated for segmentation. For 2D segmentation, He et
al. [19] proposed a Fully Transformer Network (FTN) to learn long-range
contextual information. MedT [41] uses a dual-branch Transformer with gated
axial attention to reduce the computational cost. Swin-UNet [2] is a U-Net-
like pure Transformer that uses swin Transformer based on shifted windows for
improving the efficiency. To segment 3D volumes, UNETR [15] uses a Transformer
as the encoder to learn sequence representations of the input volume that are
connected to a decoder for segmentation. UNETR++ [36] introduces parallel
attention modules with shared key-queries to efficiently learn enriched
spatial-channel feature representations.

To combine the advantages of CNNs and Transformers, TransUNet [4] replaces the
bottleneck of UNet with Transformers for enhancing its feature representation
ability. UTNet [11] uses a sequential combination of a residual convolutional
block and a Transformer in both the encoder and decoder. CoTr [45] uses a CNN-
based encoder to extract feature representations and an efficient deformable
Transformer to capture long-range dependency. The nnFormer [50] uses
interleaved convolution and self-attention, where local and global volume-
based self-attention mechanism is used to learn volume representations.

###  2.2 Pretrained Models for Segmentation

In early studies, supervised training with labeled images from computer vision
community has demonstrated improved performance on medical images after
transfer learning [26, 32]. To better learn features from medical images, Liu
et al. [24] trained a segmentation model from 3.4k public CT scans with
partial annotations based on CLIP, and demonstrated its stronger transfer
learning performance on novel tasks. Huang et al. [20] leveraged the
TotalSegmentator dataset [44] with annotations of 104 anatomical structures in
1.2k CT scans to train a scaled U-Net. However, acquiring accurate
segmentation annotations of multiple structures in 3D medical images is
extremely time-consuming and expensive, making these methods hardly applicable
to larger pretraining datasets. To get rid of the annotations, Model Genesis
[52] uses image restoration-based self-supervision to pretrain CNNs from 623
chest CT scans. Swin UNETR [40] uses self-supervised learning to pretrain swin
Transformers, making it possible to pretrain with a larger unannotated dataset
of 5k public CT scans. However, the pretraining datasets in these works are
relatively small compared with ours ($\approx$110k CT scans), and the Swin
UNETR only uses Transformers in the encoder, with limited long-range feature
learning ability in the decoder.

###  2.3 Self-Supervised Learning

Self-supervised learning has been widely used for visual representation
learning via pretext tasks without human annotations [22]. Pretext tasks can
be defined in various formulations. The first type is classification, such as
prediction of discrete rotation angles [12], jigsaw puzzle solving [29] and
self-distillation [3, 46]. The second type is contrastive learning [7, 17]
that maximizes the similarity between positive pairs and minimizes that
between negative pairs based on a contrastive loss. In addition, some image
restoration methods have been proposed, such as image colorization [47],
denoising [10], inpainting [30] and Masked Auto-Encoder (MAE) [16].

For 3D medical images, Chen et al. [5] proposed context restoration with
random swap of local patches. Zhu et al. [53] proposed Rubik’s Cube+ where the
pretext task involves cube ordering, cube rotating and cube masking. Zhou et
al. [50] combined non-linear transformation, local pixel shuffling and
outer/inner cutouts for image restoration. Tang et al. [40] combined
inpainting, contrastive learning and rotation prediction for SSL. Chen et al.
[8] proposed Masked Image Modeling (MIM) that is a 3D extension of MAE.
However, the pretext task of image restoration in these works is different
from the downstream segmentation task, where the gap may limit the transfer
learning performance.

##  3 Methods

Fig. 1 shows an overview of our proposed method for pretraining with
unannotated 3D medical images. We introduce a pretext task based on pseudo-
segmentation, where Volume Fusion (VF) is proposed to generate paired images
and segmentation labels to pretrain the 3D segmentation model, which can
better match the downstream task of segmentation than existing SSL methods.
The pretraining strategy is combined with our proposed PCT-Net to obtain a
pretrained model that is applied to segmentation of different objects from 3D
medical images after fine tuning with a small set of labeled data.

###  3.1 Volume Fusion for Self-Supervised Learning

The main idea of Volume Fusion-based SSL is to fuse two sub-volumes with
discrete voxel-level coefficients (i.e., different fusion categories), and
train a model taking the fused sub-volume as input to predict the fusion
category of each voxel, which is a supervised segmentation pretext task.

####  3.1.1 Volume Fusion

Let $I_{b}$ and $I_{f}$ denote two sub-volumes cropped from two different 3D
scans, respectively, and their size is denoted as $D\times H\times W$, where
$D$, $H$ and $W$ are the depth, height and width, respectively. We use $I_{b}$
as the background sub-volume, and $I_{f}$ as the foreground sub-volume, and
they are fused into a new sub-volume $X\in\mathbb{R}^{D\times H\times W}$:

$X=\bm{\alpha}I_{f}+(\bm{1}-\bm{\alpha})I_{b}$ (1)

where $\bm{\alpha}$ is the fusion coefficient map, and its $i$-th element
$\alpha_{i}$ represent the coefficient for voxel $i$. $\alpha_{i}$ takes
values from a discrete set $\mathcal{V}$ =$\\{0.0,1/K,2/K,...,(K-1)/K,1.0\\}$,
where $K$ is the number of non-zero fusion coefficients, and it corresponds to
the foreground class number in the fusion output.

To model the pretext task as a segmentation task, we take each element in
$\mathcal{V}$ as a class, leading to $C=K+1$ classes of fused voxels in $X$.
We use $Y$ to denote the class label map corresponding to $\bm{\alpha}$, and
the $i$-th voxel’s class label $y_{i}$ is 0, 1, 2, …, $K$ for $\alpha_{i}$ =
0.0, $1/K$, $2/K$, …, 1.0, respectively.

Input: Unannotated dataset $\mathcal{D}$. Learning rate $\eta$. Batch size
$B$. Foreground class number $K$.

Output: Optimized network parameter $\theta$.

1 Randomly initialize $\theta$

2 while  _Stop condition not reached_ do

3 Initialize training batch $\mathcal{B}$ = $\emptyset$.

4 for  _$b$ = $1$, $2$, …, $B$_ do

5 $I_{b}$, $I_{f}$ $\leftarrow$ Sub-volumes cropped from two different 3D
scans in $\mathcal{D}$.

6 Initializea $\bm{\alpha}$ as zero array

7 $M$ $\leftarrow$ Sampling from $U\sim(m_{0},m_{1})$

8 for  _$m$ = $1$, $2$, …, $M$_ do

9 $\mathbf{x}_{m}$ $\leftarrow$ A random local patch

10 $\alpha_{m}$ $\leftarrow$ Sampling from {1/K, 2/K, …, 1}

11 $\bm{\alpha}$[$\mathbf{x}_{m}$] = $\alpha_{m}$

12 end for

13 $X$ $\leftarrow$ $\bm{\alpha}I_{f}$ \+ $(\bm{1}-\bm{\alpha})I_{b}$

14 $Y$ $\leftarrow$ Covnert $\bm{\alpha}$ to a label map

15 $\mathcal{B}$ $\leftarrow$ $\mathcal{B}\cup\\{(X,Y)\\}$

16 end for

17 Forward pass with the batch $\mathcal{B}$

18

19 $L_{sup}$ $\leftarrow$ Calculate supervised loss using Eq. 2

20 $\theta$ $\leftarrow$ $\theta-$$\eta\nabla_{\theta}L_{sup}$

21 end while

Algorithm 1 SSL based on Volume Fusion Figure 2: 2D illustration of fused
images with different $K$ values. The colorful images are fusion coefficient
maps, and they are converted into label maps for the pretext segmentation
task.

####  3.1.2 Pretext Task based on Segmentation

During the pretraining process, for each pair of $I_{b}$ and $I_{f}$, we
generate the fusion coefficient map $\bm{\alpha}$ by setting the fusion
coefficient values of different local patches randomly. Specifically, assume
the foreground patch number in $\bm{\alpha}$ is $M\sim U(m_{0},m_{1})$, where
$m_{0}$ and $m_{1}$ are the minimal and maximal possible values of $M$,
respectively. We select $M$ foreground patches at random sizes sequentially,
and set the fusion coefficient in each patch as a random value from {$1/K$,
$2/K$, …, 1.0}. For voxels outside of these patches (i.e., the background
region), the $\alpha_{i}$ value is set to 0.0. Then the generated
$\bm{\alpha}$ is used to fuse $I_{b}$ and $I_{f}$ to obtain $X$, and the
corresponding label map $Y$ converted from $\bm{\alpha}$ is used as the
segmentation ground truth for $X$. Then a supervised segmentation loss
$L_{sup}$ based on Dice loss $L_{dice}$ and cross entropy loss $L_{ce}$ is
used to pretrain the segmentation model based on the paired $X$ and $Y$:

$L_{sup}=\frac{1}{2}(L_{dice}+L_{ce})$ (2)
$L_{dice}=1-\frac{1}{CV}\sum_{c=0}^{C-1}\sum_{i=0}^{V-1}\frac{2p_{i}^{c}y_{i}^{c}}{p_{i}^{c}+y_{i}^{c}+\epsilon}$
(3)
$L_{ce}=-\frac{1}{V}\sum_{i=0}^{V-1}\sum_{c=0}^{C-1}y_{i}^{c}log(p_{i}^{c})$
(4)

where $V=D\times H\times W$ is the voxel number. $p_{i}^{c}$ is the predicted
probability of voxel $i$ being class $c$, and $y_{i}^{c}$ is the corresponding
one-hot ground truth value obtained from $Y$. The pseudo code of SSL based on
Volume Fusion is shown in Algorithm 1.

Figure 3: Structure of our proposed PCT-Net.

####  3.1.3 Context Learning Ability of Volume Fusion

The effectiveness of pretraining is largely dependent on the feature
representations learned in the pretext task, and the ability to recognize
contextual semantics is essential for medical image segmentation [5, 52].
Therefore, a good pretraining strategy for segmentation models should have
high context learning ability. In SSL based on Volume Fusion, the model is
trained to recognize the fusion category of each voxel. For a given fused
voxel, as there are numerous possible combinations of the original foreground
and background intensities, one cannot directly find a mapping from a fused
voxel to the fusion category. Successful recognition of the voxel-level fusion
category requires being aware of the global context of the foreground and
background sub-volumes.

Fig. 2 shows an example of fused images with different $K$ values. In the case
of $K$ = 1, the model is trained for a binary segmentation of the foreground
patches from the background region. As both of the foreground and background
sub-volumes are CT scans with similar intensity distributions, the model needs
to find the structural discontinuity around the border of patches for the
segmentation, which forces it to have a context learning ability. Note that at
the center of a large patch, the structural discontinuity cannot be found
based on local semantics, and the model needs to be aware of the global
context to successfully recognize the category. When $K>1$, there are more
types of foreground patches with soft fusion coefficients, and the model needs
a higher contextual semantics recognition ability for the segmentation task.
Note that the foreground patches are randomly and sequentially generated at
various scales and aspect ratios, and the region for each class can have
irregular shapes due to overlapping of the sequential patches. Therefore, the
model is encouraged to learn semantics at multiple scales in our pretext task,
which is highly transferable to the downstream segmentation task for objects
at various scales.

####  3.1.4 PCT-Net

For medical image segmentation, CNNs are good at learning local features due
to the local convolution operations. Transformers based on self-attention can
better capture long-range dependency and global context. However, the patch
embedding operation in Transformers often reduces the image resolution largely
[9, 50] (i.e., by at least 4 times), which has a limited performance on tiny
structures in medical images. In addition, the self-attention operation and
Multi-Layer Perceptron (MLP) in Transformers are computationally expensive.
Therefore, it is desirable to combine CNNs and Transformers to design a 3D
segmentation model that has both local and global feature learning ability
with a moderate model size and computational cost.

The structure of our proposed PCT-Net is illustrated in Fig. 3. It has three
main components: a multi-scale feature embedding module, a pyramid Parallel
Convolution and Transformer (PCT) module, and a prediction head. The embedding
module uses convolutional layers to extract local features at two resolution
levels. Considering that medical images usually have an anisotropic 3D
resolution where the intra-plane pixel spacing is lower than inter-plane
spacing [43, 50], we use a 2D convolutional block at the highest resolution
level, which is followed by a 2D downsampling. After the 2D downsampling, the
3D resolution is less anisotropic, and a 3D convolution block is used in the
second resolution level. Each 2D/3D convolution block consists of two basic
convolutional layers, with each containing a Batch Normalization (BN), a
Parametric Rectified Linear Unit (PReLU) activation and a 2D/3D convolution
operation. In addition, before the 2D convolution block, a project layer based
on 2D convolution is used to convert the input image to a high-dimension
feature map.

The pyramid PCT module has three resolution levels with an encoder-decoder
structure equipped with PCT blocks. A PCT block has two branches. The first
branch uses convolutional layers to extract local features. It has a structure
similar to the 3D convolution block used in the embedding module, and the
difference is that a dropout layer is used before the second 3D convolution
operation. The second branch consists of two self-attention blocks to learn
long-range dependency. Considering the large voxel number in 3D volumes, we
use a 3D version of Swin Transformer [25] to reduce the cost of computation
and memory, i.e., Patch-based Multi-head Self-Attention (P-MSA) that applies
self attention at each of non-overlapping patches instead of the entire volume
. The first self-attention block has a Layer Norm (LN), a P-MSA, followed by
another LN and a MLP. The second self-attention block is similar to the first
one, and it uses a shifted version of the P-MSA (SP-MSA), where the shift size
is half of the patch size. The outputs of these two branches in a PCT block
are added together so that the complementary local and global features can be
leveraged.

The prediction head recovers high-resolution feature maps that is symmetric to
the embedding module. At each resolution level, a concatenation with
convolution is used to fuse the feature from the embedding module and that
upsampled from a lower resolution level, and it is followed by a 2D/3D
convolutional block. We use a pointwise convolution to predict a segmentation
map at each of the two resolution levels. In addition, two pointwise
convolutions are used at the first two resolution levels of the pyramid PCT
module respectively, so that PCT-Net generates predictions at four different
scales that are used for deep supervision.

##  4 Experiments and Results

###  4.1 Datasets

Pretraining Datasets: We collected 3D CT scans from several public datasets
and a private dataset to construct the pretraining images at three different
magnitudes: PData-1k, PData-10k and PData-110k where the 3D image number was
around 1k, 10k and 110k respectively.

PData-1k contains 1,065 public CT scans, where 298 head and neck CT scans are
from the public TCIA Head-Neck-PET-CT
dataset111https://wiki.cancerimagingarchive.net/display/Public/Head-Neck-PET-
CT, 267 lung CT scans are from the public LUNA dataset222https://luna16.grand-
challenge.org (subset 0-2), and 500 abdominal CT scans are from the public
FLARE22 dataset333https://flare22.grand-challenge.org/.

PData-10k contains 10,137 public CT scans each from a different patient: 2,037
head and neck CT scans are from TCIA Head-Neck-PET-CT, OPC-
Radiomics444https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=33948764,
HNSCC555https://wiki.cancerimagingarchive.net/display/Public/HNSCC, QIN-
HEADNECK666https://wiki.cancerimagingarchive.net/display/Public/QIN-HEADNECK
and TCGA-
HNSC777https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=11829589;
2,659 lung CT scans are from LUNA, TCIA
Covid19888https://wiki.cancerimagingarchive.net/display/Public/CT+Images+in+COVID-19
and LIDC-
IDRI999https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=1966254;
5,441 abdominal or whole-body CT scans are from FLARE22, ACRIN
6664101010https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=3539213,
AbdomenCT-1K [27] and TotalSegmentator [44].

Table 1:  Details of the 3D datasets used for pretraining.

Body part | Name | Pathology | Cases
---|---|---|---
Head-Neck | Head-Neck-PET-CT | Head and neck cancer | 298
| OPC-Radiomics | Oropharynx cancer | 606
| HNSCC | Head and neck cancer | 627
| QIN-HeadNeck | Head and neck cancer | 279
| TCGA-HNSC | Head and neck cancer | 227
Chest | LUNA | Lung nodule | 888
| TCIA Covid19 | Covid19 | 753
| LIDC-IDRI | Lung nodule / cancer | 1,018
Abdomen | FLARE22 | Abdominal lesions | 2,300
| AbdomenCT-1K | Abdominal lesions | 1,112
| ACRIN 6664 | Colon cancer | 825
Whole body | TotalSegmentator | Various pathologies | 1,204
Chest | Private | Lung diseases | 103k

PData-110k contains all the images in PData-10k and additional private set of
103k lung CT scans, as shown in Table 1. The private dataset was collected
from nine hospitals with different vendors and scanning sequences including
non-contrast CT, contrast-enhanced CT and CT pulmonary angiography. The
patient age ranged from 20 to 90 years. The slice thickness was 0.6$\sim$2.5
mm in 97% images, and $\geq$5.0 mm in the other 3% images. Around 80% of the
images have pathologies including lung nodules, pneumonedema, pneumonia,
emphysema, pleural effusion, lymphadenopathy, rib fracture and liver lesions,
etc.

Downstream Segmentation Datasets: We considered three different downstream
tasks for segmentation of head and neck organs, thoracic organs and abdominal
organs from CT scans, respectively. The datasets are:

1) MICCAI 2015 Head-Neck dataset [33] with 48 CT scans for segmentation of
Organs-at-Risks (OAR) in radiotherapy planning. The OARs include brain stem,
optic chiasm, mandible, bilateral optic nerves, bilateral parotid glands and
bilateral Sub-Mandibular (SM) glands. For each bilateral structure, we merged
the left and right instances as a single class, leading to 6 OARs for
segmentation. Each volume has 76$\sim$360 slices with a size of
512$\times$512\. The in-plane resolution ranged from 0.76 to 1.27 mm, and the
inter-slice spacing was 1.25$\sim$3.0 mm. The images were randomly split into
30, 6 and 12 for training, validation and testing, respectively.

2) Segmentation of THoracic Organs-at-Risk dataset (SegTHOR) [23]. The OARs
include heart, aorta, trachea and esophagus. The dataset consists of 60 CT
scans with an in-plane resolution of 0.90$\sim$1.37 mm and an inter-slice
spacing of 2.0$\sim$3.7 mm. Each volume has 150$\sim$284 slices, with a slice
size of 512$\times$512\. As the annotations for the official testing set (20
cases) are not publicly available, we used the remaining 40 cases for
experiments, and they were randomly split into 24, 6 and 10 for training,
validation and testing, respectively.

3) Synapse for multi-organ segmentation that includes 30 cases of abdominal CT
scans. Following existing works [4, 50], we deal with segmentation of 8
organs, i.e., aorta, gallbladder, left kidney, right kidney, liver, pancreas,
spleen and stomach. Each CT volume has 85$\sim$198 slices with a size of
512$\times$512\. The in-plane resolution was 0.54$\sim$0.98 mm, and the inter-
slice spacing was 2.5$\sim$5.0 mm. The 3D images were randomly split into 18,
4 and 8 for training, validation and testing, respectively.

###  4.2 Implementation Details

Our PCT-Net and Volume Fusion were implemented in
PyTorch111111https://pytorch.org and PyMIC121212https://github.com/HiLab-
git/PyMIC [42]. In PCT-Net, the convolutional kernel size was 3$\times$3 and
3$\times$3$\times$3 for the 2D and 3D convolutions, respectively. The channel
numbers at the five resolution levels were 24, 48, 128, 256 and 512
respectively. For Volume Fusion-based pretraining, the sub-volume size was
64$\times$128$\times$128\. To generate random fusion coefficient map
$\bm{\alpha}$, the foreground class number was set as $K=4$, and the patch
number was $M\sim U(10,40)$. The ranges of depth, height and width of the
local patches were 8$\sim$40, 8$\sim$80 and 8$\sim$80, respectively.

For pretraining, we used a batch size of 4 with the Adam optimizer, weight
decay of $10^{-5}$ and initial learning rate of $10^{-3}$. For PData-1k,
PData-10k and PData-110k, the maximal training iteration number was 100k, 200k
and 400k, respectively, and the learning rate was halved for every 20k, 40k
and 50k, respectively. For pretraining with PData-110k, a server with two
NVIDIA Tesla A100 GPUs (80 GB memory) was used, while the other pretraining
and downstream training/inference procedures were conducted on a Ubuntu
desktop that has two NVIDIA GTX 3090 GPUs (24 GB memory). The combination of
Dice loss and cross entropy loss was used for both the pretraining and
downstream training tasks. For a downstream task, the model was trained for
30k with a batch size of 2, and we set the initial learning rate to $10^{-3}$
that was halved at every 5k iterations. For quantitative evaluation, we used
the Dice similarity coefficient (Dice) and Average Symmetric Surface Distance
(ASSD) between segmentation results and the corresponding ground truths.

Figure 4: Performance of Volume Fusion with different $K$ values. 3D U-Net was
used as the segmentation model, and it was pretrained on PData-1K and then
transferred to SegTHOR dataset. Figure 5: Effect of the scale of pretraining
data used by Volume Fusion on the segmentation performance on SegTHOR dataset.
Table 2:  Quantitative comparison between models with different pretraining
strategies on the SegTHOR dataset. PData-1K was used as the pretraining
dataset, and the model architecture was 3D U-Net [1].

Method | Dice (%) | ASSD (mm)
---|---|---
| Esophagus | Heart | Trachea | Aorta | Average | Esophagus | Heart | Trachea | Aorta | Average
Scratch | 71.87$\pm$11.27 | 90.82$\pm$4.72 | 87.63$\pm$5.77 | 89.34$\pm$6.19 | 84.92$\pm$6.47 | 3.37$\pm$2.48 | 5.52$\pm$3.98 | 1.51$\pm$0.69 | 2.30$\pm$1.80 | 3.17$\pm$2.24
Patch Swapping [5] | 75.91$\pm$9.04 | 91.88$\pm$5.27 | 87.31$\pm$6.45 | 89.64$\pm$7.60 | 86.18$\pm$4.05 | 2.05$\pm$0.92 | 2.96$\pm$1.64 | 3.09$\pm$5.75 | 3.02$\pm$3.41 | 2.78$\pm$1.54
Model Genesis [52] | 76.98$\pm$8.29 | 92.78$\pm$3.13 | 87.87$\pm$6.28 | 89.64$\pm$7.74 | 86.81$\pm$4.00 | 2.15$\pm$1.00 | 2.81$\pm$1.43 | 2.86$\pm$6.00 | 3.02$\pm$3.26 | 2.71$\pm$1.58
MIM [8] | 76.29$\pm$8.91 | 91.85$\pm$2.82 | 87.52$\pm$5.43 | 92.19$\pm$ 2.98 | 86.97$\pm$3.50 | 2.05$\pm$ 0.77 | 4.65$\pm$ 2.25 | 3.20$\pm$4.61 | 2.22$\pm$ 1.66 | 3.03$\pm$1.55
Volume Fusion | 77.61$\pm$7.82 | 93.72$\pm$2.28 | 88.21$\pm$4.18 | 93.67$\pm$1.68 | 88.30$\pm$2.93 | 1.84$\pm$0.76 | 2.90$\pm$2.55 | 1.30$\pm$0.57 | 1.08$\pm$0.48 | 1.78$\pm$0.71

Figure 6: Visual comparison between results on SegTHOR dataset obtained by
models pretrained with different SSL methods.

###  4.3 Ablation Study

In this section, we conduct ablation studies to investigate the performance of
our Volume Fusion with different $K$ values and different scales of
pretraining data, where 3D UNet [1] and SegTHOR were employed as the
segmentation model and downstream segmentation dataset, respectively.

Effect of $K$. To observe the effect of different $K$ values on performance of
the pretrained model, we set it to 1, 2, 4, 8 and 16 respectively, and
pretrained 3D U-Net [1] on PData-1K before training with the SegTHOR dataset.
The pretrained models were compared with training from scratch. Quantitative
evaluation results on the SegTHOR testing images are shown in Fig. 4. It can
be observed that the average Dice value obtained by training from scratch was
84.92%, and all the variants of Volume Fusion outperformed it. Especially,
$K=1$ led to an average Dice of 88.08%, and increasing $K$ to 2, 4 and 8
further improved the performance, respectively. $K=16$ obtained an average
Dice of 87.94%, which was lower than those of the other $K$ values, but it was
still much better than learning from scratch. In addition, Fig. 4 shows that
$K=4$ achieved the lowest average ASSD value of 1.78 mm, compared with 3.17 mm
obtained by training from scratch.

Effect of the Scale of Pretraining data. We further compared pretraining with
PData-1K, PData-10K and PData-110K using 3D U-Net and PCT-Net, respectively.
The results on the SegTHOR dataset when $K=4$ are shown in Fig. 5. For 3D
U-Net, PData-1K, PData-10K and PData-110K improved the average Dice from
84.92% to 88.30%, 88.78% and 89.09% respectively. For PCT-Net, training from
scratch obtained an average Dice of 87.57%, and PData-1K, PData-10K and
PData-110K improved it to 88.56%, 89.39% and 89.56% respectively. The results
show that Volume Fusion works on different network structures, and its
performance can be improved by leveraging a larger unannotated pretraining
dataset.

Table 3:  Quantitative comparison between different methods for head and neck
OAR segmentation on MICCAI 2015 Head-Neck dataset.

| Method | Brain stem | Optic chiasm | Mandible | Optic nerves | Parotid Glands | SM glands | Average
---|---|---|---|---|---|---|---|---
Dice (%) | nnU-Net [21] | 89.27$\pm$2.42 | 57.47$\pm$24.24 | 90.12$\pm$5.84 | 72.50$\pm$7.85 | 87.53$\pm$3.43 | 75.06$\pm$12.58 | 78.66$\pm$5.42
| TransUNet [4] | 75.52$\pm$5.58 | 41.92$\pm$15.99 | 92.28$\pm$1.45 | 58.36$\pm$6.91 | 76.70$\pm$6.85 | 69.80$\pm$8.27 | 69.10$\pm$3.08
| nnFormer [50] | 80.02$\pm$3.53 | 52.72$\pm$14.70 | 87.96$\pm$2.27 | 57.34$\pm$7.72 | 75.31$\pm$7.24 | 68.25$\pm$5.53 | 70.27$\pm$4.19
| UNETR++ [36] | 87.26$\pm$2.13 | 60.44$\pm$22.49 | 93.99$\pm$1.30 | 75.19$\pm$ 5.85 | 84.61$\pm$3.89 | 80.74$\pm$ 4.53 | 80.37$\pm$3.94
| PCT-Net | 89.25$\pm$1.86 | 58.09$\pm$18.59 | 94.17$\pm$1.66 | 77.04$\pm$4.84 | 87.44$\pm$3.37 | 82.49$\pm$4.55 | 81.41$\pm$3.67
| PCT-Net + VF | 90.24$\pm$1.78 | 62.93$\pm$20.73 | 94.85$\pm$1.36 | 78.11$\pm$4.04 | 87.07$\pm$3.69 | 83.25$\pm$3.90 | 82.74$\pm$3.95
ASSD (mm) | nnU-Net [21] | 1.08$\pm$0.30 | 1.04$\pm$0.84 | 0.63$\pm$0.42 | 0.52$\pm$0.26 | 1.15$\pm$0.43 | 3.51$\pm$5.71 | 1.32$\pm$1.04
| TransUNet [4] | 2.00$\pm$0.40 | 4.11$\pm$2.66 | 1.46$\pm$1.13 | 2.58$\pm$0.76 | 2.65$\pm$1.06 | 4.10$\pm$1.81 | 2.82$\pm$0.76
| nnFormer [50] | 2.01$\pm$0.46 | 1.28$\pm$0.55 | 1.24$\pm$1.37 | 1.16$\pm$0.45 | 2.52$\pm$1.03 | 2.26$\pm$0.82 | 1.75$\pm$0.43
| UNETR++ [36] | 1.22$\pm$0.31 | 0.97$\pm$0.89 | 0.38$\pm$0.10 | 0.44$\pm$ 0.16 | 1.34$\pm$0.38 | 1.17$\pm$0.26 | 0.92$\pm$0.19
| PCT-Net | 1.14$\pm$0.29 | 1.00$\pm$0.70 | 0.41$\pm$0.17 | 0.40$\pm$0.14 | 1.14$\pm$0.50 | 1.67$\pm$2.03 | 0.96$\pm$0.33
| PCT-Net + VF | 0.92$\pm$0.26 | 0.87$\pm$0.72 | 0.32$\pm$0.08 | 0.38$\pm$0.12 | 1.09$\pm$0.29 | 1.03$\pm$0.26 | 0.77$\pm$0.16

Figure 7: Visual comparison between results of different methods on MICCAI
2015 Head-Neck dataset. The first two rows show results in two different axial
slices, and the last row presents 3D visualization. White arrows highlight the
mis-segmented regions.

###  4.4 Comparison with State-of-the-art Methods

Comparison of Different Pretraining Methods. Our proposed Volume Fusion was
compared with three existing self-supervised pretraining methods for medical
image segmentation models: 1) Patch Swapping [5] where random small patches in
an image are swapped for context restoration, 2) Model Genesis [52] that
combines non-linear transformation, local pixel shuffling and inter/outer
cutout for image restoration, and 3) Masked Image Modeling (MIM) [8] where
local patches are randomly masked at a high masking ratio for image
restoration.

In this experiment, we used PData-1K to pretrain 3D UNet [1] with these
different SSL methods respectively, and applied the pretrained models to the
SegTHOR dataset. The quantitative evaluation results are shown in Table 2.
Compared with learning from scratch, Patch Swapping [5], Model Genesis [52]
and MIM [8] improved the average Dice from 84.92% to 86.18%, 86.81% and
86.97%, respectively, and reduced the average ASSD value from 3.17 mm to 2.78
mm, 2.71 mm and 3.03 mm, respectively. Our proposed Volume Fusion outperformed
these three methods, with an average Dice and ASSD of 88.30% and 1.78 mm,
respectively. Fig. 6 shows a visual comparison between these methods. It can
be observed that learning from scratch had an over-segmentation of the heart
and esophagus, with an under-segmentation of the aorta. Patch Swapping [5] and
Model Genesis [52] obtained an over-segmentation of the aorta, while MIM [8]
obtained a poor segmentation of the heart. In contrast, Volume Fusion obtained
smoother results with less mis-segmentation than the other methods.

Table 4:  Quantitative comparison between different methods for thoracic OAR
segmentation on SegTHOR dataset.

Method | Dice (%) | ASSD (mm)
---|---|---
| Esophagus | Heart | Trachea | Aorta | Average | Esophagus | Heart | Trachea | Aorta | Average
nnU-Net [21] | 80.15$\pm$7.85 | 93.05$\pm$2.59 | 87.33$\pm$5.84 | 93.78$\pm$1.67 | 88.57$\pm$3.84 | 3.17$\pm$5.45 | 5.79$\pm$9.38 | 7.41$\pm$10.66 | 1.530$\pm$2.16 | 4.47$\pm$5.38
TransUNet [4] | 75.66$\pm$9.36 | 85.27$\pm$16.18 | 89.37$\pm$4.26 | 91.52$\pm$2.97 | 85.46$\pm$7.20 | 2.06$\pm$1.28 | 5.15$\pm$4.60 | 1.48$\pm$0.87 | 2.93$\pm$2.26 | 2.91$\pm$1.92
nnFormer [50] | 78.00$\pm$6.86 | 92.47$\pm$2.06 | 84.87$\pm$10.10 | 90.07$\pm$4.89 | 86.35$\pm$3.78 | 2.76$\pm$2.76 | 5.51$\pm$4.48 | 6.56$\pm$16.39 | 3.07$\pm$2.59 | 4.47$\pm$4.63
UNETR++ [36] | 76.24$\pm$10.43 | 92.00$\pm$4.79 | 88.61$\pm$4.49 | 92.48$\pm$ 2.32 | 87.33$\pm$3.67 | 1.90$\pm$ 0.99 | 3.84$\pm$3.93 | 2.65$\pm$3.58 | 1.19$\pm$0.56 | 2.39$\pm$1.44
PCT-Net | 82.08$\pm$6.19 | 88.47$\pm$11.18 | 88.11$\pm$4.43 | 91.65$\pm$3.67 | 87.58$\pm$4.39 | 1.59$\pm$1.01 | 3.40$\pm$2.75 | 1.70$\pm$0.95 | 1.26$\pm$0.66 | 1.99$\pm$1.01
PCT-Net + VF | 83.45$\pm$4.78 | 91.66$\pm$7.14 | 89.26$\pm$4.47 | 93.88$\pm$1.79 | 89.56$\pm$2.81 | 1.24$\pm$0.33 | 2.93$\pm$2.52 | 1.49$\pm$1.03 | 0.98$\pm$0.51 | 1.66$\pm$0.86

Table 5:  Comparison of Dice (%) obtained by different methods for abdominal
organ segmentation on Synapse dataset. ‘R’ and ‘L’ represent right and left,
respectively. Bold font and underline represent the first and second top
method in each column, respectively.

Method | Spleen | R Kidney | L Kidney | Gallbladder | Pancreas | Liver | Stomach | Aorta | Average
---|---|---|---|---|---|---|---|---|---
nnU-Net [21] | 94.00$\pm$4.26 | 91.89$\pm$7.72 | 93.30$\pm$4.17 | 78.17$\pm$18.48 | 83.27$\pm$3.98 | 94.33$\pm$3.89 | 79.30$\pm$19.89 | 89.26$\pm$3.38 | 87.94$\pm$5.26
TransUNet [4] | 92.00$\pm$7.15 | 92.48$\pm$4.11 | 92.30$\pm$4.77 | 74.21$\pm$12.31 | 72.18$\pm$16.12 | 94.73$\pm$4.11 | 75.72$\pm$15.54 | 90.67$\pm$4.16 | 85.54$\pm$5.40
nnFormer [50] | 92.25$\pm$5.83 | 92.86$\pm$2.11 | 93.84$\pm$1.45 | 73.56$\pm$14.48 | 72.02$\pm$6.22 | 95.31$\pm$1.28 | 80.77$\pm$10.29 | 90.47$\pm$3.54 | 86.39$\pm$2.81
UNETR++ [36] | 89.26$\pm$15.54 | 93.40$\pm$1.61 | 93.19$\pm$2.31 | 70.96$\pm$ 28.24 | 74.70$\pm$12.14 | 95.76$\pm$ 0.68 | 82.79$\pm$15.22 | 88.79$\pm$4..82 | 86.11$\pm$6.55
PCT-Net | 91.36$\pm$13.77 | 95.21$\pm$5.46 | 90.78$\pm$9.79 | 80.94$\pm$9.91 | 79.13$\pm$9.86 | 96.63$\pm$7.04 | 79.25$\pm$23.33 | 90.48$\pm$5.11 | 87.97$\pm$5.22
PCT-Net + VF | 91.38$\pm$12.97 | 95.31$\pm$0.55 | 92.17$\pm$8.00 | 80.79$\pm$13.58 | 83.24$\pm$3.97 | 96.70$\pm$6.30 | 82.46$\pm$15.99 | 90.86$\pm$4.10 | 89.11$\pm$4.43

Table 6: Comparison of ASSD (mm) obtained by different methods for abdominal
organ segmentation on Synapse dataset.

Method | Spleen | R Kidney | L Kidney | Gallbladder | Pancreas | Liver | Stomach | Aorta | Average
---|---|---|---|---|---|---|---|---|---
nnU-Net [21] | 4.82$\pm$7.90 | 3.56$\pm$8.00 | 4.96$\pm$9.78 | 2.32$\pm$2.57 | 1.92$\pm$1.24 | 3.11$\pm$4.39 | 6.43$\pm$11.02 | 3.42$\pm$3.36 | 3.82$\pm$5.33
TransUNet [4] | 6.34$\pm$11.64 | 2.69$\pm$3.77 | 4.20$\pm$7.37 | 4.73$\pm$6.31 | 3.55$\pm$1.69 | 2.17$\pm$2.16 | 6.65$\pm$8.28 | 2.91$\pm$3.19 | 4.15$\pm$5.16
nnFormer [50] | 6.07$\pm$9.21 | 1.70$\pm$2.24 | 1.25$\pm$1.61 | 6.63$\pm$9.88 | 4.35$\pm$3.00 | 2.27$\pm$2.29 | 7.84$\pm$13.15 | 1.88$\pm$1.89 | 4.00$\pm$5.04
UNETR++ [36] | 4.87$\pm$10.34 | 1.63$\pm$1.90 | 1.87$\pm$3.24 | 3.22$\pm$ 4.76 | 3.69$\pm$3.92 | 1.46$\pm$ 0.67 | 4.65$\pm$6.20 | 1.74$\pm$0.84 | 2.89$\pm$3.31
PCT-Net | 4.07$\pm$9.63 | 0.50$\pm$0.07 | 3.10$\pm$6.77 | 3.20$\pm$3.78 | 3.06$\pm$3.98 | 1.04$\pm$0.34 | 5.35$\pm$8.45 | 1.55$\pm$1.02 | 2.73$\pm$3.00
PCT-Net + VF | 4.04$\pm$9.20 | 0.47$\pm$0.06 | 2.05$\pm$4.22 | 1.77$\pm$1.98 | 1.52$\pm$0.65 | 1.01$\pm$0.29 | 3.64$\pm$3.81 | 1.85$\pm$2.06 | 2.04$\pm$2.09

Comparison of Different Models. Our PCT-Net pretrained with Volume Fusion (VF)
on PData-110k was compared with four state-of-the-art segmentation models:
nnU-Net [21], TransUNet [4], UNETR++ [36] and nnFormer [50]. They were
evaluated on MICCAI 2015 Head-Neck dataset, SegTHOR and Synapse datasets,
respectively.

Table 3 shows quantitative evaluation results on the MICCAI 2015 Head-Neck
dataset. The nnU-Net [21] achieved an average Dice of 78.66%, with the
mandible and optic chiasm being the easiest and hardest OARs to segment,
respectively. TransUNet and nnFormer failed to beat nnU-Net, with average Dice
of 69.10% and 70.27%, respectively. UNETR++ outperformed nnU-Net with average
Dice of 80.37%. Our PCT-Net achieved an average Dice of 81.41%, and performed
better than the existing networks. When PCT-Net was pretrained with Volume
Fusion, the average Dice was further improved to 82.74%. The average ASSD
obtained by PCT-Net + VF was 0.77 mm, which was better than those of the other
methods. Fig. 7 shows a visual comparison between the segmentation results
obtained by different methods. It can be observed that TransUNet and nnFormer
have a poor performance on tiny structures such as optic chiasm and optic
nerves. The nnFormer also has an over-segmentation of the brain stem in the
first row. UNETR++ has an over-segmentation in the mandible, as shown in the
second row. PCT-Net has an overall better performance than these methods, and
when pretrained with VF, its performance is further improved, especially for
optic nerves as shown in the first row.

Table 4 shows quantitative evaluation of different methods for thoracic OAR
segmentation on SegTHOR dataset. Among the existing methods, nnU-Net achieved
the highest average Dice (88.57%). However, the average ASSD value (4.47 mm)
obtained by nnU-Net was lower than that of TransUNet (2.91 mm) and UNETR++
(2.39 mm). PCT-Net obtained an average Dice of 87.58% and ASSD of 1.99 mm,
respectively. When pretrained with VF, the corresponding Dice and ASSD values
were 89.56% and 1.66 mm, respectively, which outperformed the other methods.

Figure 8: Visual comparison between results of different methods on Synapse
dataset for abdominal organ segmentation. The first two rows show results in
two different axial slices, and the last row presents 3D visualization.

Table 5 and 6 show the quantitative evaluation of different methods on the
Synapse dataset in terms of Dice and ASSD, respectively. The average Dice of
nnU-Net was 87.94%, which was better than that of TransUNet (85.54%), nnFormer
(86.39%) and UNETR++ (86.11%). The average Dice of PCT-Net (87.97%) was
slightly higher than that of nnU-Net, and its average ASSD value (2.73 mm) was
much better than that of nnU-Net (3.82 mm). When pretrained with VF, the
performance of PCT-Net was further improved, with average Dice and ASSD of
89.11% and 2.04 mm, respectively. Note that nnU-Net was a strong segmentation
framework. Despite that it is hard to defeat nnU-Net on all the organs, our
method outperformed nnU-Net on 5 out of the 8 organs in terms of Dice, and on
all the 8 organs in terms of ASSD. Fig. 8 presents a visual comparison between
the segmentation results of these different methods, which shows that PCT-Net
+ VF achieved the best performance overall compared with the other methods.

##  5 Conclusion

In this work, we present a self-supervised foundation model pretrained with
large-scale unannotated volumetric images for 3D medical image segmentation. A
volume Fusion method is proposed for self-supervised pretraining, where the
pretext task is formulated as a pseudo-segmentation task by forcing the model
to predict the discrete fusion coefficient of each voxel between two different
sub-volumes. It encourages the model to learn multi-scale contextual
information that is transferable to downstream segmentation tasks. A novel
Parallel Convolution and Transformer Network (PCT-Net) is also proposed for
better feature representation ability in 3D segmentation tasks. Our PCT-Net
was pretrained with Volume Fusion on 110k unannotated CT scans, and
experimental results showed that our pretrained model outperformed several
state-of-the-art segmentation methods on different downstream tasks for multi-
organ segmentation from CT images. In the future, it is of interest to
validate our pretrained model on downstream tasks for lesion and organ
segmentation from different modalities.

##  6 Acknowledgment

This work was supported by the National Natural Science Foundation of China
(62271115), National Key Research and Development Program of China
(2020YFB1711500) and the 1·3·5 project for disciplines of excellence, West
China Hospital, Sichuan University (ZYYC21004).

## References

  * [1] Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger.  3D U-Net: Learning dense volumetric segmentation from sparse annotation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 424–432, 2016.
  * [2] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation.  In ECCV workshop, pages 205–218, 2022.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging Properties in Self-Supervised Vision Transformers.  Proceedings of the IEEE International Conference on Computer Vision, pages 9630–9640, 2021.
  * [4] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou.  TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation.  arXiv preprint arXiv:2102.04306, pages 1–13, 2021.
  * [5] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert.  Self-supervised learning for medical image analysis using image context restoration.  Medical Image Analysis, 58:101539, 2019.
  * [6] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834 – 848, 2017.
  * [7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International Conference on Machine Learning, pages 1597–1607, 2020.
  * [8] Zekai Chen, Devansh Agarwal, Kshitij Aggarwal, Wiem Safta, Mariann Micsinai Balan, and Kevin Brown.  Masked Image Modeling Advances 3D Medical Image Analysis.  IEEE Winter Conference on Applications of Computer Vision, pages 1969–1979, 2023.
  * [9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In ICLR, pages 1–21, 2021.
  * [10] Yibo Gao, Huan Wang, Xinglong Liu, Ning Huang, Guotai Wang, and Shaoting Zhang.  A denoising self-supervised approach for COVID-19 pneumonia lesion segmentation with limited annotated CT images.  In Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pages 3705–3708. IEEE, 2021.
  * [11] Yunhe Gao, Mu Zhou, and Dimitris N. Metaxas.  UTNet: A Hybrid Transformer Architecture for Medical Image Segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, volume 1, pages 61–71. Springer International Publishing, 2021.
  * [12] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In ICLR, pages 1–16, 2018.
  * [13] Ran Gu, Guotai Wang, Tao Song, Rui Huang, Michael Aertsen, Jan Deprest, Sébastien Ourselin, Tom Vercauteren, and Shaoting Zhang.  CA-Net: Comprehensive attention convolutional neural networks for explainable medical image segmentation.  IEEE Transactions on Medical Imaging, 40(2):699–711, 2021.
  * [14] Zaiwang Gu, Jun Cheng, Huazhu Fu, Kang Zhou, Huaying Hao, Yitian Zhao, Tianyang Zhang, Shenghua Gao, and Jiang Liu.  CE-Net: Context Encoder Network for 2D Medical Image Segmentation.  IEEE Transactions on Medical Imaging, 38(10):2281–2292, 2019.
  * [15] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger Roth, and Daguang Xu.  UNETR: Transformers for 3D Medical Image Segmentation.  In IEEE Winter Conference on Applications of Computer Vision, pages 1748–1758, 2022.
  * [16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 15979–15988, 2022.
  * [17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum Contrast for Unsupervised Visual Representation Learning.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 9729–9738, 2020.
  * [18] Kelei He, Chen Gan, Zhuoyuan Li, Islem Rekik, Zihao Yin, Wen Ji, Yang Gao, Qian Wang, Junfeng Zhang, and Dinggang Shen.  Transformers in medical image analysis.  Intelligent Medicine, 3(1):59–78, 2023.
  * [19] Xinzi He, Ee Leng Tan, Hanwen Bi, Xuzhe Zhang, Shijie Zhao, and Baiying Lei.  Fully transformer network for skin lesion analysis.  Medical Image Analysis, 77:102357, 2022.
  * [20] Ziyuan Huang, Haoyu Wang, Zongying Deng, Jin Ye, Yanzhou Su, Hui Sun, Junjun He, Yun Gu, Lixu Gu, Shaoting Zhang, and Yu Qiao.  STU-Net : Scalable and transferable medical image segmentation models empowered by large-scale supervised pre-training.  arXiv:2304.06716, 2023.
  * [21] Fabian Isensee, Paul F. Jaeger, Simon A.A. Kohl, Jens Petersen, and Klaus H. Maier-Hein.  nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation.  Nature Methods, 18(2):203–211, 2021.
  * [22] Longlong Jing and Yingli Tian.  Self-Supervised Visual Feature Learning with Deep Neural Networks: A Survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(11):4037–4058, 2021.
  * [23] Zoe Lambert, Caroline Petitjean, Bernard Dubray, and Su Kuan.  SegTHOR: Segmentation of Thoracic Organs at Risk in CT images.  International Conference on Image Processing Theory, Tools and Applications (IPTA), (July 2019):1–6, 2020.
  * [24] Jie Liu, Yixiao Zhang, Jie-Neng Chen, Junfei Xiao, Yongyi Lu, Bennett A. Landman, Yixuan Yuan, Alan Yuille, Yucheng Tang, and Zongwei Zhou.  CLIP-driven universal model for organ segmentation and tumor detection.  arXiv:2301.00785, pages 1–20, 2023.
  * [25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.  In IEEE International Conference on Computer Vision, pages 9992–10002, 2021.
  * [26] Le Lu, Hoo-chang Shin, Holger R Roth, Mingchen Gao, Le Lu, Senior Member, Ziyue Xu, Isabella Nogues, Jianhua Yao, Daniel Mollura, and Ronald M Summers.  Deep Convolutional Neural Networks for Computer-Aided Detection : CNN Architectures , Dataset Characteristics and Transfer Learning.  IEEE Transactions on Medical Imaging, 35(5):1285–1298, 2016.
  * [27] Jun Ma, Yao Zhang, Song Gu, Cheng Zhu, Cheng Ge, Yichi Zhang, Xingle An, Congcong Wang, Qiyuan Wang, Xin Liu, Shucheng Cao, Qi Zhang, Shangqing Liu, Yunpeng Wang, Yuhui Li, Jian He, and Xiaoping Yang.  AbdomenCT-1K: Is Abdominal Organ Segmentation a Solved Problem?  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6695–6714, 2022.
  * [28] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi.  V-Net: Fully convolutional neural networks for volumetric medical image segmentation.  In International Conference on 3D Vision, pages 565–571, 2016.
  * [29] Mehdi Noroozi and Paolo Favaro.  Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles.  In European Conference on Computer Vision, pages 69–84, 2016.
  * [30] Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros.  Context Encoders: Feature Learning by Inpainting.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2536–2544, 2016.
  * [31] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, pages 1–36, 2021.
  * [32] Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio.  Transfusion: Understanding transfer learning for medical imaging.  In NeurIPS, pages 1–11, 2019.
  * [33] Patrik F. Raudaschl, Paolo Zaffino, Gregory C. Sharp, Maria Francesca Spadea, Antong Chen, Benoit M. Dawant, Thomas Albrecht, Tobias Gass, Christoph Langguth, Marcel Luthi, Florian Jung, Oliver Knapp, Stefan Wesarg, Richard Mannion-Haworth, Mike Bowes, Annaliese Ashman, Gwenael Guillard, Alan Brett, Graham Vincent, Mauricio Orbes-Arteaga, David Cardenas-Pena, German Castellanos-Dominguez, Nava Aghdasi, Yangming Li, Angelique Berens, Kris Moe, Blake Hannaford, Rainer Schubert, and Karl D. Fritscher.  Evaluation of segmentation methods on head and neck CT: Auto-segmentation challenge 2015.  Medical Physics, 44(5):2020–2036, 2017.
  * [34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234–241, 2015.
  * [35] Abhijit Guha Roy, Nassir Navab, and Christian Wachinger.  Recalibrating fully convolutional networks with spatial and channel ’squeeze and excitation’ blocks.  IEEE Transactions on Medical Imaging, 38(2):540–549, 2019.
  * [36] Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan.  UNETR++: Delving into efficient and accurate 3D medical image segmentation.  arXiv:2212.04497, 2022.
  * [37] Jonathan Shapey, Guotai Wang, Reuben Dorent, Alexis Dimitriadis, Wenqi Li, Ian Paddick, Neil Kitchen, Sotirios Bisdas, Shakeel R. Saeed, Sebastien Ourselin, Robert Bradford, and Tom Vercauteren.  An artificial intelligence framework for automatic segmentation and volumetry of vestibular schwannomas from contrast-enhanced T1-weighted and high-resolution T2-weighted MRI.  Journal of Neurosurgery, 134(1):171–179, 2019.
  * [38] Dinggang Shen, Guorong Wu, and Heung-Il Suk.  Deep learning in medical image analysis.  Annual Review of Biomedical Engineering, 19(1):221–248, 2017.
  * [39] Nima Tajbakhsh, Laura Jeyaseelan, Qian Li, Jeffrey Chiang, Zhihao Wu, and Xiaowei Ding.  Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation.  Medical Image Analysis, 63(2020):101693, 2019.
  * [40] Yucheng Tang, Dong Yang, Wenqi Li, Holger R. Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh.  Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis.  IEEE Conference on Computer Vision and Pattern Recognition, pages 20730–20740, 2022.
  * [41] Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, and Vishal M. Patel.  Medical Transformer: Gated axial-attention for medical image segmentation.  In International Conference on Medical Image Computing and Computer Assisted Intervention, pages 36–46, 2021.
  * [42] Guotai Wang, Xiangde Luo, Ran Gu, Shuojue Yang, Yijie Qu, Shuwei Zhai, Qianfei Zhao, Kang Li, and Shaoting Zhang.  PyMIC: A deep learning toolkit for annotation-efficient medical image segmentation.  Computer Methods and Programs in Biomedicine, 231:107398, 2023.
  * [43] Guotai Wang, Shuwei Zhai, Giovanni Lasio, Baoshe Zhang, Byong Yi, Shifeng Chen, Thomas J. Macvittie, Dimitris Metaxas, Jinghao Zhou, and Shaoting Zhang.  Semi-supervised segmentation of radiation-induced pulmonary fibrosis from lung CT scans with multi-scale guided dense attention.  IEEE Transactions on Medical Imaging, 41(3):531–542, 2022.
  * [44] Jakob Wasserthal, Manfred Meyer, Hanns-Christian Breit, Joshy Cyriac, Shan Yang, and Martin Segeroth.  TotalSegmentator: robust segmentation of 104 anatomical structures in CT images.  arXiv:2208.05868, 2022.
  * [45] Yutong Xie, Jianpeng Zhang, Chunhua Shen, and Yong Xia.  CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 171–180, 2021.
  * [46] Yiwen Ye, Jianpeng Zhang, Ziyang Chen, and Yong Xia.  DeSD: Self-supervised learning with deep self-distillation for 3D medical image segmentation.  In Medical Image Computing and Computer-Assisted Intervention, volume 3, pages 545–555, 2022.
  * [47] Richard Zhang, Phillip Isola, and Alexei A. Efros.  Colorful Image Colorization.  In European Conference on Computer Vision, pages 649–666, 2016\.
  * [48] Shaoting Zhang and Dimitris Metaxas.  On the challenges and perspectives of foundation models for medical image analysis.  arXiv:2306.05705, pages 1–8, 2023.
  * [49] Wei Zhang, Qing Zhou, Jingjie Zhou, Ying Wei, Ying Shao, Yanbo Chen, and Dinggang Shen.  Deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy.  Nature Communications, 13:6566, 2022.
  * [50] Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu.  nnFormer: Interleaved Transformer for Volumetric Segmentation.  arXiv preprint arXiv: 2109.03201, 2022.
  * [51] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang.  Unet++: A nested u-net architecture for medical image segmentation.  In MICCAI workshop on DLMIA, volume 11045, pages 3–11, 2018.
  * [52] Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B. Gotway, and Jianming Liang.  Models Genesis.  Medical Image Analysis, 67:101840, 2021.
  * [53] Jiuwen Zhu, Yuexiang Li, Yifan Hu, Kai Ma, S. Kevin Zhou, and Yefeng Zheng.  Rubik’s Cube+: A self-supervised feature learning framework for 3D medical image analysis.  Medical Image Analysis, 64:101746, 2020.