# Leveraging Unlabeled Data for 3D Medical Image Segmentation through Self-
Supervised Contrastive Learning

###### Abstract

Current 3D semi-supervised segmentation methods face significant challenges
such as limited consideration of contextual information and the inability to
generate reliable pseudo-labels for effective unsupervised data use. To
address these challenges, we introduce two distinct subnetworks designed to
explore and exploit the discrepancies between them, ultimately correcting the
erroneous prediction results. More specifically, we identify regions of
inconsistent predictions and initiate a targeted verification training
process. This procedure strategically fine-tunes and harmonizes the
predictions of the subnetworks, leading to enhanced utilization of contextual
information. Furthermore, to adaptively fine-tune the network’s
representational capacity and reduce prediction uncertainty, we employ a self-
supervised contrastive learning paradigm. For this, we use the network’s
confidence to distinguish between reliable and unreliable predictions. The
model is then trained to effectively minimize unreliable predictions. Our
experimental results for organ segmentation, obtained from clinical MRI and CT
scans, demonstrate the effectiveness of our approach when compared to state-
of-the-art methods. The codebase is accessible on GitHub.

Index Terms—  Semi-supervised, Contrastive Learning, Medical Imaging, 3D
Segmentation.

##  1 Introduction

Image segmentation, the process of partitioning an image into distinct
regions, is of great importance in various medical applications. Consequently,
it serves as a fundamental tool for automating clinical workflows, reducing
data processing time, and providing quantitative measures of organs or
pathologies [1, 2]. These capabilities greatly assist clinicians in making
more accurate diagnoses, assessing the response to therapy, and ultimately
improving patient care and outcomes. In this context, semantic segmentation is
essential for improving the quality of medical image segmentation, identifying
relevant regions of interest (ROIs), such as tumors or organs [3, 4], and
removing unwanted objects. However, conventional supervised approaches depend
on obtaining large-scale annotated data, which can be immensely costly in
real-world medical scenarios. To address this challenge, a promising solution
is semi-supervised semantic segmentation, where models are trained using a
limited number of labeled samples and an abundance of unlabeled data.
Effectively leveraging unlabeled data is a crucial in this context, driving
researchers to explore innovative approaches [5, 6, 7, 8].

One commonly used approach to address this challenge is the application of
pseudo-labeling [9]. Pseudo-labels are assigned to unlabeled pixels based on
predictions from a model trained on labeled data. These ”pseudo-labels” then
guide the training of a supervised model, improving its performance. In this
regard, Basak et al. [10] proposed a method for semantic segmentation that
combines contrastive learning (CL) and semi-supervised learning (SemiSL)
without requiring a specific initial task. Their method employs pseudo-labels
from SemiSL to enhance CL guidance, leading to more accurate multi-class
segmentation by learning discriminative class information. Chai et al. [11]
proposed a local contrastive loss approach to improve pixel-level feature
learning for segmentation tasks. Their method leverages semantic label
information obtained from pseudo-labels of unlabeled images, in conjunction
with a limited set of annotated images with ground truth labels. In another
study, Bai et al. [12] addresses empirical mismatch issues in semi-supervised
medical image segmentation by bidirectionally copying and pasting labeled and
unlabeled data in a mean teacher architecture. This promoted consistent
learning between labeled and unlabeled data, effectively reducing the
empirical distribution gap for improved segmentation performance. Wang et. al.
[7] introduced a two-stream network as a to address errors within each subnet.
This innovative approach considers the discrepancies between pseudo-labels and
predictions, effectively rectifying mistakes made by individual networks.
Similarly, Luo et al. [13] and Wu et al. [14] adopted a dual-task consistency
and mutual consistency training strategy to penalize incorrect predictions
made by the networks.

Despite the effectiveness of the pseudo-labeling paradigm, concerns remain
about the reliability of pseudo labels, which can lead to inaccurate mask
predictions. Previous research has attempted to mitigate this issue by
filtering out predictions with classification scores below a certain threshold
[15], but this approach may not be entirely effective in eliminating incorrect
predictions. Specifically, some wrong predictions can still exhibit high
classification scores, resulting in overconfidence or miscalibration phenomena
[16]. Additionally, setting a high threshold to remove unreliable predictions
can significantly reduce the number of generated pseudo-labels, limiting the
effectiveness of the semi-supervised learning algorithm. This reduction in
pseudo-labels can lead to categorically imbalanced training data, which can
cause problems like inaccurate assignment of pseudo-labels to pixels related
to a particular organ or tissue type. As a result, this imbalance can
negatively impact the overall segmentation performance.

As discussed before, directly using unreliable predictions as pseudo-labels
can adversely affect the model performance. To address this challenge, we thus
introduce here an innovative alternative approach that effectively leverages
unreliable pseudo-labels while overcoming the limitations of their direct
application. To this end, we propose a dual-stream network architecture in
which each subnetwork employs a 3D auto-encoder-decoder module to generate
segmentation maps for input images. A supervised loss function guides the
network in learning representations for each class, enabling precise and dense
predictions. We also propose a consistency regularization term to penalize
inaccurate predictions made by each network, using a set of confidence
predictions obtained from both pathways. This strategic approach allows the
model to adaptively update its feature representations, reducing the
occurrence of incorrect predictions from both paths. To effectively utilize
unlabeled data, we extend the concept of pseudo-labeling by distinguishing
between reliable and unreliable predictions. We then optimize the clustering
space using a contrastive method to align feature descriptions of unreliable
pixels with positive prototypes derived from trustworthy predictions.

Our contributions can be summarized as follows: (1) we introduce a consistency
regularization term to reduce false predictions; (2) we conceptualize a self-
supervised contrastive learning paradigm to decrease the number of unreliable
predictions; (3): we obtained the state-of-the-art (SOTA) results on 3D CT/MRI
segmentation datasets.

##  2 Proposed Method

###  2.1 Overview

Our goal is to train a semantic segmentation model using a combination of
labeled
($\mathcal{D}_{l}=\\{(\mathbf{x}_{i}^{l},\mathbf{y}_{i}^{l})\\}_{i=1}^{N_{l}}$)
and larger unlabeled data
($\mathcal{D}_{u}=\\{\mathbf{x}_{i}^{u}\\}_{i=1}^{N_{u}}$). To achieve this,
we employ two subnetworks, Subnet A ($f_{A}(x;\theta_{1})$) and Subnet B
($f_{B}(x;\theta_{2})$), each employing a 3D encoder-decoder architecture.
These subnetworks generate prediction maps (denoted as $Y_{A}$ and $Y_{B}$)
and corresponding feature representations for each voxel ($v$) in a
$D$-dimensional space.

During each training step, we randomly sample $b$ labeled images ($b_{l}$) and
$b$ unlabeled images ($b_{u}$). For the labeled images, our primary objective
is to minimize the standard cross-entropy loss and Dice loss as defined in
Equation 1:

$\displaystyle\mathcal{L}_{s}=\frac{1}{|\mathcal{B}_{l}|}\sum_{(\mathbf{x}_{i}^{l},\mathbf{y}_{i}^{l})\in\mathcal{B}_{l}}\ell_{ce}(f(\mathbf{x}_{i}^{l};\theta),\mathbf{y}_{i}^{l})+\text{Dice}({\hat{\mathbf{y_{i}}}^{l}},\mathbf{y_{i}}^{l}),$
(1)

where $\mathbf{y}_{i}^{l}$ represents the hand-annotated mask label for the
$i$-th labeled image. Additionally, to minimize false prediction, we introduce
consistency regularization term that considers the confidence predictions of
one network against the other. We further employ the contrastive loss function
to effectively leverage unreliable predictions through the training process.
Figure 1 illustrates the overall network process.

For the unlabeled images, we feed them through both networks to obtain
prediction maps, $Y_{a}$ and $Y_{b}$. We then apply a pixel-level entropy-
based filtering to exclude unreliable pixel-level pseudo-labels when
calculating the unsupervised loss defined in Equation 2:

$\mathcal{L}_{u}=\frac{1}{|\mathcal{B}_{u}|}\sum_{\mathbf{x}_{i}^{u}\in\mathcal{B}_{u}}\ell_{ce}(f(\mathbf{x}_{i}^{u};\theta),\hat{\mathbf{y}}_{i}^{u})+\text{Dice}(\hat{\mathbf{y}_{i}},\mathbf{\hat{\mathbf{y}}_{i}^{u}})+\mathcal{L}_{reg},$
(2)

where $\hat{\mathbf{y}}_{i}^{u}$ is the pseudo-label for the $i$-th unlabeled
image. To further perform error correction, we introduce a regularization
loss, $\mathcal{L}_{reg}$, to the $\mathcal{L}_{u}$, as detailed in Section
2.2. Finally, we employ a contrastive loss to exploit unreliable pixels
excluded from the unsupervised loss, as explained in Section Section 2.3. Our
optimization objective is to minimize the overall loss as follows:

$\mathcal{L}=\mathcal{L}_{s}+\lambda_{u}\mathcal{L}_{u}+\lambda_{c}\mathcal{L}_{c},$
(3)

Here, $\mathcal{L}_{s}$ and $\mathcal{L}_{u}$ represent the supervised and
unsupervised losses applied to labeled and unlabeled images, respectively,
while $\mathcal{L}_{c}$ is the contrastive loss for unreliable pseudo-labels.
The weights $\lambda_{u}$ and $\lambda_{c}$ control the contributions of the
unsupervised loss and contrastive loss, respectively. Both $\mathcal{L}_{s}$
and $\mathcal{L}_{u}$ are computed using the combination of cross-entropy (CE)
and Dice losses as shown in Figure Figure 1.

Fig. 1: An illustration of our suggested pipeline. In each itteration, we
utilize $\mathcal{L}_{s}$ for the labeled data and $\mathcal{L}_{u}$ for the
unlabeled data. When dealing with the unlabeled data, we adopt the prediction
of the network with the lower $\mathcal{L}_{s}$ as a pseudo label.

###  2.2 Consistency Regularization

We introduce a consistency regularization mechanism to refine the predictions
of our dual-subnet model. This module identifies areas where the subnetworks
(Subnet A and Subnet B) make discrepant predictions despite being highly
confident in their respective predictions, indicating potential
mispredictions. The goal is to rectify these incongruities. Mathematically, we
define the area of incorrect predictions between the softmax outputs
$\hat{Y}^{A}$ and $\hat{Y}^{B}$ as follows:

$M_{\text{diff}}=\arg_{\max}(\max(\hat{Y}^{u}_{A})>T)\neq\arg_{\max}(\max(\hat{Y}^{u}_{B})>T),$
(4)

where $M_{\text{diff}}$ represents the set of voxels in which Subnet A and
Subnet B generate different predictions with high confidence and T represents
the confidence threshold. We dynamically adjust the value of T during the
training process. We then define the $L1$ distance loss function as
regularization term to correct potential incorrect predictions by each of the
networks:

$\mathcal{L}_{reg}=\sum_{i=1}^{n}|(M_{\text{diff}}\odot\hat{Y}^{u}_{A})-(M_{\text{diff}}\odot\hat{Y}^{u}_{B})|,$
(5)

where $\odot$ denotes the Hadamard multiplication.

###  2.3 Contrastive loss

To mitigate uncertain predictions in our model, we incorporated the
contrastive loss function. Figure 2b illustrates a scenario where the
network’s predictions exhibit low confidence in categorizing certain voxels.
Our contrastive loss design aims to guide these uncertain voxels towards
aligning with their corresponding class prototypes, ultimately reducing
misclassifications and uncertainty rates. To achieve this, our approach first
computes the confidence of each voxel’s prediction. It then categorises the
predictions into two distinct sets: reliable and unreliable predictions. Next,
it defines prototypes for each category using the reliable set as a base. Each
prototype is computed as the mean vector of the reliable voxel
representations:

$\mathbf{c}_{k}=\frac{1}{\left|S_{k}\right|}\sum_{\left(\mathbf{v^{r}}_{i},y_{i}\right)\in
S_{k}}f(\mathbf{v^{r}}_{i}),$ (6)

where, $f(\mathbf{v^{r}}_{i})$ indicates the feature represenation of the
voxel corresponding to the reliable predictions. Our approach uses a distance
function, denoted as
$d:\mathbb{R}^{M}\times\mathbb{R}^{M}\rightarrow[0,+\infty)$, to compute a
distribution over classes for uncertain voxels $\mathbf{v^{u}}$. This
distribution is computed by applying a softmax operation to the distances
between the voxel’s representation in the embedding space and the class
prototypes:

$p_{\boldsymbol{\phi}}(y=k\mid\mathbf{v^{u}})=\frac{\exp\left(-d\left(f(\mathbf{v^{u}}),\mathbf{c}_{k}\right)\right)}{\sum_{k^{\prime}}\exp\left(-d\left(f(\mathbf{v^{u}}),\mathbf{c}_{k^{\prime}}\right)\right)}$
(7)

Our contrastive loss function aims to move uncertain voxels of the same class
towards their respective class prototype, while also pushing the prototypes of
each class away from each other to account for the distance between them.

Fig. 2: (a): Illustration of the regularization term and (b) contrastive loss
effects on prediction refinement.

##  3 Experimental Results

###  3.1 Experimental Setup

In our study, we developed and implemented our model using the PyTorch
framework on a single RTX 3090 GPU. Following [14, 7] we choose Resnet and
V-Net for the two stream network for fair comparision. To optimize the
parameters of our network, we employed the SGD optimizer with a weight decay
factor of 0.0001 and a momentum coefficient of 0.9. We set our initial
learning rate to 0.01 and implemented a dynamic learning rate schedule that
reduced the learning rate by a factor of 10 after every 2500 iterations, for a
total of 6000 iterations. During our training process, we included both
labeled and unlabeled samples in each iteration, maintaining a consistent
batch size of two for both categories. In Equation 3 we use $\lambda_{u}=1.0$
and $\lambda_{c}=0.1*e^{4(1-t/t_{max})^{2}}$, where $t$ and $t_{max}$ denote
the current and maximum itterations, respectively. Additionally, for a robust
assessment of our model’s performance, we adopted the K-fold cross-validation
method recommended by [7].

###  3.2 Dataset

Left Atrial Dataset (LA): The LA dataset [17] consists of 1003 3D gadolinium-
enhanced MR imaging volumes with manual left atrial annotations, featuring an
anisotropic resolution of 0.625 × 0.625 × 0.625 mm³. We preprocessed the data
according to [7], initially applying volume normalization to standardize the
data. During training, we employed random cropping to achieve model input
dimensions of 112 × 112 × 80. For inference, we used a sliding window approach
with the same dimensions and a stride of 18 × 18 × 4.

NIH Pancreas Dataset: The NIH Pancreas Dataset [18] comprises 82 abdominal CT
volumes with manual pancreas annotations. The CT volumes have dimensions of
512×512×D, where D represents the number of CT slices, which ranges from 181
to 466. Our preprocessing method, similar to [13, 7], involves applying a soft
tissue CT window with Hounsfield Units (HU) from -120 to 240. We then align
the CT scans to the pancreas region and expand the margins by 25 voxels.
During training, we perform random cropping, resulting in volumes with
dimensions of 96×96×96. For inference, we use a stride of 16×16×16.

###  3.3 Results

Table 1: Comparison of results using the LA dataset (MRI).

Method | Dice(%)$\uparrow$ | Jaccard(%)$\uparrow$ | 95HD(voxel)$\downarrow$ | ASD(voxel)$\downarrow$
---|---|---|---|---
MT [6] | 85.89 $\pm$ 0.024 | 76.58 $\pm$ 0.027 | 12.63 $\pm$ 5.741 | 3.44 $\pm$ 1.382
UA-MT [19] | 85.98 $\pm$ 0.014 | 76.65 $\pm$ 0.017 | 9.86 $\pm$ 2.707 | 2.68 $\pm$ 0.776
SASSNet [20] | 86.21 $\pm$ 0.023 | 77.15 $\pm$ 0.024 | 9.80 $\pm$ 1.842 | 2.68 $\pm$ 0.416
DTC [13] | 86.36 $\pm$ 0.023 | 77.25 $\pm$ 0.020 | 9.02 $\pm$ 1.015 | 2.40 $\pm$ 0.223
MC-Net [14] | 87.65 $\pm$ 0.011 | 78.63 $\pm$ 0.013 | 9.70 $\pm$ 2.361 | 3.01 $\pm$ 0.700
MCF [7] | 88.71 $\pm$ 0.018 | 80.41 $\pm$ 0.022 | 6.32 $\pm$ 0.800 | 1.90 $\pm$ 0.187
Our Method | 89.10 $\pm$ 0.012 | 81.62 $\pm$ 0.024 | 6.30 $\pm$ 0.850 | 1.80 $\pm$ 0.020

|  |  |
---|---|---|---
|  |  |
(a) Ground Truth | (b) MC-Net | (c) MCF | (e) Proposed Method

Fig. 3: Visual comparison of segmentation results: the first and the second
rows show the left atrium (LA) and pancreas, respectively.

The comparison of our proposed method with SOTA techniques on the left atrial
dataset is provided in Table 1. Our method shows significant improvements in
all metrics, with a substantial enhancement in organ voxel detection,
specifically DSC and Jacard. Compared to MCF, our proposed method exhibits
noteworthy enhancements, with an increase in DSC from 88.71 to 89.10 and
Jaccard index from 80.41 to 81.62. Furthermore, our approach maintains low-
performance variance, contributing to its stability and reliability. Figure 3
presents the visual results of our proposed method compared to other methods
for left atrial segmentation. These visual results showcase higher overlap
with ground truth labels and fewer false segmentations, highlighting the finer
details captured by our approach. Our method showcases strong performance on
the Pancreas dataset as well, as presented in Table 2. Figure 3 provides
additional insights into the segmentation results, underscoring the impact of
the suggested modules for enhancing the overall segmentation quality. In
greater detail, our approach generates sharper edges and more precise boundary
separation then the MCF and MC-Net methods. This highlights its effectiveness
in improving the reliability of object boundary predictions and distinguishing
the organ of interest from the background.

We also performed an ablation study on the LA dataset to thoroughly assess the
impact of the regularization and contrastive loss components of our method.
Notably, removing the regularization module resulted in a substantial decrease
of 0.5 in the DSC. Similarly, removing the contrastive loss resulted in a more
significant drop of 0.73 in the DSC score. These findings underscore the
critical roles played by both the regularization and contrastive loss
components in enhancing the performance of our method.

Table 2: Comparison of results using the Pancreas dataset (CT).

Method | Dice(%)$\uparrow$ | Jaccard(%)$\uparrow$ | 95HD(voxel)$\downarrow$ | ASD(voxel)$\downarrow$
---|---|---|---|---
MT [6] | 74.43 $\pm$ 0.024 | 60.53 $\pm$ 0.030 | 14.93 $\pm$ 2.000 | 4.61 $\pm$ 0.929
UA-MT [19] | 74.01 $\pm$ 0.029 | 60.00 $\pm$ 3.031 | 17.00 $\pm$ 3.031 | 5.19 $\pm$ 1.267
SASSNet [20] | 73.57 $\pm$ 0.017 | 59.71 $\pm$ 0.020 | 13.87 $\pm$ 1.079 | 3.53 $\pm$ 1.416
DTC [13] | 73.23 $\pm$ 0.024 | 59.18 $\pm$ 0.027 | 13.20 $\pm$ 2.241 | 3.81 $\pm$ 0.953
MC-Net [14] | 73.73 $\pm$ 0.019 | 59.19 $\pm$ 0.021 | 13.65 $\pm$ 3.902 | 3.92 $\pm$ 1.055
MCF [7] | 75.00 $\pm$ 0.026 | 61.27 $\pm$ 0.030 | 11.59 $\pm$ 1.611 | 3.27 $\pm$ 0.919
Our Method | 76.40 $\pm$ 0.018 | 62.96 $\pm$ 0.027 | 10.69 $\pm$ 1.603 | 2.79 $\pm$ 0.0954

##  4 Conclusion

This paper presents a novel dual-stream network for semi-supervised semantic
segmentation that leverages labeled and unlabeled imaging data. Our approach
focuses on reducing the problem of unreliable predictions by integrating
contrastive learning and error correction mechanisms. We outperformed SOTA
techniques on CT and MRI images.

## References

  * [1] Reza Azad, Ehsan Khodapanah Aghdam, Amelie Rauland, Yiwei Jia, Atlas Haddadi Avval, Afshin Bozorgpour, Sanaz Karimijafarbigloo, Joseph Paul Cohen, Ehsan Adeli, and Dorit Merhof,  “Medical image segmentation review: The success of u-net,”  arXiv preprint arXiv:2211.14830, 2022.
  * [2] Bobby Azad, Reza Azad, Sania Eskandari, Afshin Bozorgpour, Amirhossein Kazerouni, Islem Rekik, and Dorit Merhof,  “Foundational models in medical imaging: A comprehensive survey and future vision,”  arXiv preprint arXiv:2310.18689, 2023.
  * [3] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.,  “The medical segmentation decathlon,”  Nature communications, vol. 13, no. 1, pp. 1–13, 2022.
  * [4] Abhishek Srivastava, Debesh Jha, Elif Keles, Bulent Aydogan, Mohamed Abazeed, and Ulas Bagci,  “An efficient multi-scale fusion network for 3d organ at risk (oar) segmentation,”  arXiv preprint arXiv:2208.07417, 2022.
  * [5] Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang,  “Semi-supervised semantic segmentation with cross pseudo supervision,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2021.
  * [6] Antti Tarvainen and Harri Valpola,  “Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results,”  in Advances in Neural Information Processing Systems, 2017.
  * [7] Yongchao Wang, Bin Xiao, Xiuli Bi, Weisheng Li, and Xinbo Gao,  “Mcf: Mutual correction framework for semi-supervised medical image segmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15651–15660.
  * [8] Yassine Ouali, Céline Hudelot, and Myriam Tami,  “Semi-supervised semantic segmentation with cross-consistency training,”  in IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [9] Dong-Hyun Lee et al.,  “Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks,”  in International Conference on Machine Learning Workshop, 2013, vol. 3, p. 896.
  * [10] Hritam Basak and Zhaozheng Yin,  “Pseudo-label guided contrastive learning for semi-supervised medical image segmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19786–19797.
  * [11] Krishna Chaitanya, Ertunc Erdil, Neerav Karani, and Ender Konukoglu,  “Local contrastive loss with pseudo-label based self-training for semi-supervised medical image segmentation,”  Medical Image Analysis, vol. 87, pp. 102792, 2023.
  * [12] Yunhao Bai, Duowen Chen, Qingli Li, Wei Shen, and Yan Wang,  “Bidirectional copy-paste for semi-supervised medical image segmentation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11514–11524.
  * [13] Xiangde Luo, Jieneng Chen, Tao Song, and Guotai Wang,  “Semi-supervised medical image segmentation through dual-task consistency,”  in Proceedings of the AAAI conference on artificial intelligence, 2021, vol. 35, pp. 8801–8809.
  * [14] Yicheng Wu, Minfeng Xu, Zongyuan Ge, Jianfei Cai, and Lei Zhang,  “Semi-supervised left atrium segmentation with mutual consistency training,”  in Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24. Springer, 2021, pp. 297–306.
  * [15] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki,  “Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling,”  in Advances in Neural Information Processing Systems, 2021, vol. 34.
  * [16] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger,  “On calibration of modern neural networks,”  in International Conference on Machine Learning. PMLR, 2017, pp. 1321–1330.
  * [17] Zhaohan Xiong, Qing Xia, Zhiqiang Hu, Ning Huang, Cheng Bian, Yefeng Zheng, Sulaiman Vesal, Nishant Ravikumar, Andreas Maier, Xin Yang, et al.,  “A global benchmark of algorithms for segmenting the left atrium from late gadolinium-enhanced cardiac magnetic resonance imaging,”  Medical image analysis, vol. 67, pp. 101832, 2021.
  * [18] Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers,  “Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation,”  in MICCAI 2015. Springer, 2015, pp. 556–564.
  * [19] Lequan Yu, Shujun Wang, Xiaomeng Li, Chi-Wing Fu, and Pheng-Ann Heng,  “Uncertainty-aware self-ensembling model for semi-supervised 3d left atrium segmentation,”  in MICCAI 2019. Springer, 2019, pp. 605–613.
  * [20] Shuailin Li, Chuyu Zhang, and Xuming He,  “Shape-aware semi-supervised 3d semantic segmentation for medical images,”  in MICCAI 2020. Springer, 2020.