11institutetext:  Division of Medical Image Computing (MIC), German Cancer
Research Center (DKFZ), Heidelberg, Germany 22institutetext: Pattern Analysis
and Learning Group, Department of Radiation Oncology, Heidelberg University
Hospital 33institutetext: Faculty of Mathematics and Computer Science,
Heidelberg University, Germany 44institutetext: Helmholtz Imaging, German
Cancer Research Center, Heidelberg, Germany 55institutetext: National Center
for Tumor Diseases (NCT), NCT Heidelberg, A partnership between DKFZ and
University Medical Center Heidelberg 66institutetext: Interactive Machine
Learning Group, German Cancer Research Center, Heidelberg, Germany

# MedNeXt: Transformer-driven Scaling of ConvNets for Medical Image
Segmentation

Saikat Roy 1133 Gregor Koehler 11 Constantin Ulrich 1155 Michael Baumgartner
113344 Jens Petersen 11 Fabian Isensee 1144 Paul F. Jaeger 4466 Klaus Maier-
Hein 1122

###### Abstract

There has been exploding interest in embracing Transformer-based architectures
for medical image segmentation. However, the lack of large-scale annotated
medical datasets make achieving performances equivalent to those in natural
images challenging. Convolutional networks, in contrast, have higher inductive
biases and consequently, are easily trainable to high performance. Recently,
the ConvNeXt architecture attempted to modernize the standard ConvNet by
mirroring Transformer blocks. In this work, we improve upon this to design a
modernized and scalable convolutional architecture customized to challenges of
data-scarce medical settings. We introduce MedNeXt, a Transformer-inspired
large kernel segmentation network which introduces – 1) A fully ConvNeXt 3D
Encoder-Decoder Network for medical image segmentation, 2) Residual ConvNeXt
up and downsampling blocks to preserve semantic richness across scales, 3) A
novel technique to iteratively increase kernel sizes by upsampling small
kernel networks, to prevent performance saturation on limited medical data, 4)
Compound scaling at multiple levels (depth, width, kernel size) of MedNeXt.
This leads to state-of-the-art performance on 4 tasks on CT and MRI modalities
and varying dataset sizes, representing a modernized deep architecture for
medical image segmentation. Our code is made publicly available at:
https://github.com/MIC-DKFZ/MedNeXt.

###### Keywords:

Medical Image Segmentation Transformers MedNeXt Large Kernels ConvNeXt

##  1 Introduction

Transformers [30, 7, 21] have seen wide-scale adoption in medical image
segmentation as either components of hybrid architectures [3, 9, 33, 2, 8, 31]
or standalone techniques [34, 25, 15] for state-of-the-art performance. The
ability to learn long-range spatial dependencies is one of the major
advantages of the Transformer architecture in visual tasks. However,
Transformers are plagued by the necessity of large annotated datasets to
maximize performance benefits owing to their limited inductive bias. While
such datasets are common to natural images (ImageNet-1k [6], ImageNet-21k
[26]), medical image datasets usually suffer from the lack of abundant high
quality annotations [19]. To retain the inherent inductive bias of
convolutions while taking advantage of architectural improvements of
Transformers, the ConvNeXt [22] was recently introduced to re-establish the
competitive performance of convolutional networks for natural images. The
ConvNeXt architecture uses an inverted bottleneck mirroring that of
Transformers, composed of a depthwise layer, an expansion layer and a
contraction layer (Sec. 2.1), in addition to large depthwise kernels to
replicate their scalability and long-range representation learning. The
authors paired large kernel ConvNeXt networks with enormous datasets to
outperform erstwhile state-of-the-art Transformer-based networks. In contrast,
the VGGNet [28] approach of stacking small kernels continues to be the
predominant technique for designing ConvNets in medical image segmentation.
Out-of-the-box data-efficient solutions such as nnUNet [13], using variants of
a standard UNet [5], have still remained effective across a wide range of
tasks.

The ConvNeXt architecture marries the scalability and long-range spatial
representation learning capabilities of Vision [7] and Swin Transformers [21]
with the inherent inductive bias of ConvNets. Additionally, the inverted
bottleneck design allows us to scale width (increase channels) while not being
affected by kernel sizes. Effective usage in medical image segmentation would
allow benefits from – 1) learning long-range spatial dependencies via large
kernels, 2) less intuitively, simultaneously scaling multiple network levels.
To achieve this would require techniques to combat the tendency of large
networks to overfit on limited training data. Despite this, there have been
recent attempts to introduce large kernel techniques to the medical vision
domain. In [18], a large kernel 3D-UNet [5] was used by decomposing the kernel
into depthwise and depthwise dilated kernels for improved performance in organ
and brain tumor segmentation – exploring kernel scaling, while using constant
number of layers and channels. The ConvNeXt architecture itself was utilized
in 3D-UX-Net [17], where the Transformer of SwinUNETR [8] was replaced with
ConvNeXt blocks for high performance on multiple segmentation tasks. However,
3D-UX-Net only uses these blocks partially in a standard convolutional
encoder, limiting their possible benefits.

In this work, we maximize the potential of a ConvNeXt design while uniquely
addressing challenges of limited datasets in medical image segmentation. We
present the first fully ConvNeXt 3D segmentation network, MedNeXt, which is a
scalable Encoder-Decoder network, and make the following contributions:

  * •

We utilize an architecture composed purely of ConvNeXt blocks which enables
network-wide advantages of the ConvNeXt design. (Sec. 2.1)

  * •

We introduce Residual Inverted Bottlenecks in place of regular up and
downsampling blocks, to preserve contextual richness while resampling to
benefit dense segmentation tasks. The modified residual connection in
particular improves gradient flow during training. (Sec. 2.2)

  * •

We introduce a simple but effective technique of iteratively increasing kernel
size, UpKern, to prevent performance saturation on large kernel MedNeXts by
initializing with trained upsampled small kernel networks. (Sec. 2.3)

  * •

We propose applying Compound Scaling [29] of multiple network parameters owing
to our network design, allowing orthogonality of width (channels), receptive
field (kernel size) and depth (number of layers) scaling. (Sec. 2.4)

MedNeXt achieves state-of-the-art performance against baselines consisting of
Transformer-based, convolutional and large kernel networks. We show
performance benefits on 4 tasks of varying modality (CT, MRI) and sizes
(ranging from 30 to 1251 samples), encompassing segmentation of organs and
tumors. We propose MedNeXt as a strong and modernized alternative to standard
ConvNets for building deep networks for medical image segmentation.

##  2 Proposed Method

((a)) MedNeXt macro and block architecture

((b)) UpKern Initialization

((c)) Leaderboard Scores

Figure 1: (a) Architectural design of the MedNeXt. The network has 4 Encoder
and Decoder layers each, with a bottleneck layer. MedNeXt blocks are present
in Up and Downsampling layers as well. Deep Supervision is used at each
decoder layer, with lower loss weights at lower resolutions. All residuals are
additive while convolutions are padded to retain tensor sizes. (b) Upsampled
Kernel (UpKern) initialization of a pair of MedNeXt architectures with similar
configurations ($\theta$) except kernel size ($k_{1},k_{2}$). (c) MedNeXt-L
($5\times 5\times 5$) leaderboard performance.

###  2.1 Fully ConvNeXt 3D Segmentation Architecture

In prior work, ConvNeXt [22] distilled architectural insights from Vision
Transformers [7] and Swin Transformers [21] into a convolutional architecture.
The ConvNeXt block inherited a number of significant design choices from
Transformers, designed to limit computation costs while scaling the network,
which demonstrated performance improvements over standard ResNets [10]. In
this work, we leverage these strengths by adopting the general design of
ConvNeXt as the building block in a 3D-UNet-like [5] macro architecture to
obtain the MedNeXt. We extend these blocks to up and downsampling layers as
well (Sec. 2.2), resulting in the first fully ConvNeXt architecture for
medical image segmentation. The macro architecture is illustrated in Figure
1(a). MedNeXt blocks (similar to ConvNeXt blocks) have 3-layers mirroring a
Transformer block and are described for a $C$-channel input as follows:

  1. 1.

Depthwise Convolution Layer: This layer contains a Depthwise Convolution with
kernel size $k\times k\times k$, followed by normalization, with $C$ output
channels. We use channel-wise GroupNorm [32] for stability with small batches
[27], instead of the original LayerNorm. The depthwise nature of convolutions
allow large kernels in this layer to replicate a large attention window of
Swin-Transformers, while simultaneously limiting compute and thus delegating
the “heavy lifting" to the Expansion Layer.

  2. 2.

Expansion Layer: Corresponding to a similar design in Transformers, this layer
contains an overcomplete Convolution Layer with $CR$ output channels, where
$R$ is the expansion ratio, followed by a GELU [12] activation. Large values
of $R$ allow the network to scale width-wise while $1\times 1\times 1$ kernel
limits compute. It is important to note that this layer effectively decouples
width scaling from receptive field (kernel size) scaling in the previous
layer.

  3. 3.

Compression Layer: Convolution layer with $1\times 1\times 1$ kernel and $C$
output channels performing channel-wise compression of the feature maps.

MedNeXt is convolutional and retains the inductive bias inherent to Conv-Nets
that allows easier training on sparse medical datasets. Our fully ConvNeXt
architecture also enables width (more channels) and receptive field (larger
kernels) scaling at both standard and up/downsampling layers. Alongside depth
scaling (more layers), we explore these 3 orthogonal types of scaling to
design a compound scalable MedNeXt for effective medical image segmentation
(Sec.2.4).

###  2.2 Resampling with Residual Inverted Bottlenecks

The original ConvNeXt design utilizes separate downsampling layers which
consist of standard strided convolutions. An equivalent upsampling block would
be standard strided transposed convolutions. However, this design does not
implicitly take advantage of width or kernel-based ConvNeXt scaling while
resampling. We improve upon this by extending the Inverted Bottleneck to
resampling blocks in MedNeXt. This is done by inserting the strided
convolution or transposed convolution in the first Depthwise Layer for
Downsampling and Upsampling MedNeXt blocks respectively. The corresponding
channel reduction or increase is inserted in the last compression layer of our
MedNeXt $2\times$ Up or Down block design as in Fig. 1(a). Additionally, to
enable easier gradient flow, we add a residual connection with $1\times
1\times 1$ convolution or transposed convolution with stride of 2. In doing
so, MedNeXt fully leverages the benefits from Transformer-like inverted
bottlenecks to preserve rich semantic information in lower spatial resolutions
in all its components, which should benefit dense medical image segmentation
tasks.

Config. # Blocks (B) Exp. Rat. (R) S $B_{all}=2$ $R_{all}$ = 2 B
$R_{1}=R_{9}=2$ $R_{2}=R_{8}=3$ $R_{3-7}=4$ M $B_{1}=B_{9}=3$ $B_{2-8}=4$ L
$B_{1}=B_{9}=3$ $B_{2}=B_{8}=4$ $B_{3-7}=8$ $R_{1}=R_{9}=3$ $R_{2}=R_{8}=4$
$R_{3-7}=8$

Network Variants BTCV AMOS22 DSC SDC DSC SDC MedNeXt-B Resampling 84.01 86.77
89.14 92.10 Standard Up and Downsampling 83.13 85.64 88.96 91.86 MedNeXt-B
($\mathbf{5\times 5\times 5})$ \+ UpKern 84.23 87.06 89.38 92.36 MedNeXt-B
($5\times 5\times 5$) from scratch 84.03 86.71 89.12 92.10 MedNeXt-B
($\mathbf{5\times 5\times 5})$ \+ UpKern 84.23 87.06 89.38 92.36 MedNeXt-B
($3\times 3\times 3$) trained $2\times$ 84.00 86.85 89.18 92.16 nnUNet (non-
MedNeXt baseline) 83.56 86.07 88.88 91.70

Table 1: (Left) MedNeXt configurations from scaling Block Counts ($B$) and
Expansion Ratio ($R$) as in Figure 1(a). (Right) MedNext-B ablations (Sec.
4.1) .

###  2.3 UpKern: Large Kernel Convolutions without Saturation

Large convolution kernels approximate the large attention windows in
Transformers, but remain prone to performance saturation. ConvNeXt
architectures in classification of natural images, despite the benefit of
large datasets such as ImageNet-1k and ImageNet-21k, are seen to saturate at
kernels of size $7\times 7\times 7$ [22]. Medical image segmentation tasks
have significantly less data and performance saturation can be a problem in
large kernel networks. To propose a solution, we borrow inspiration from Swin
Transformer V2 [20] where a large-attention-window network is initialized with
another network trained with a smaller attention window. Specifically, Swin
Transformers use a bias matrix $\hat{B}\in\mathbb{R}^{(2M-1)\times(2M-1)}$ to
store learnt relative positional embeddings, where $M$ is the number of
patches in an attention window. On increasing the window size, $M$ increases
and necessitates a larger $\hat{B}$. The authors proposed spatially
interpolating an existing bias matrix to the larger size as a pretraining
step, instead of training from scratch, which demonstrated improved
performance. We propose a similar approach but customized to convolutions
kernels, as seen in Figure 1(b), to overcome performance saturation. UpKern
allows us to iteratively increase kernel size by initializing a large kernel
network with a compatible pretrained small kernel network by trilinearly
upsampling convolutional kernels (represented as tensors) of incompatible
size. All other layers with identical tensor sizes (including normalization
layers) are initialized by copying the unchanged pretrained weights. This
leads to a simple but effective initialization technique for MedNeXt which
helps large kernel networks overcome performance saturation in the
comparatively limited data scenarios common to medical image segmentation.

###  2.4 Compound Scaling of Depth, Width and Receptive Field

Compound scaling [29] is the idea that simultaneous scaling on multiple levels
(depth, width, receptive field, resolution etc) offers benefits beyond that of
scaling at one single level. The computational requirements of indefinitely
scaling kernel sizes in 3D networks quickly becomes prohibitive and leads us
to investigate simultaneous scaling at different levels. Keeping with Figure
1(a), our scaling is tested for block count ($B$), expansion ratio ($R$) and
kernel size ($k$) – corresponding to depth, width and receptive field size. We
use 4 model configurations of the MedNeXt to do so, as detailed in Table 1
(Left). The basic functional design (MedNeXt-S) uses number of channels ($C$)
as 32, $R=2$ and $B=2$. Further variants increase on just $R$ (MedNeXt-B) or
both $R$ and $B$ (MedNeXt-M). The largest 70-MedNext-block architecture uses
high values of both $R$ and $B$ (MedNeXt-L) and is used to demonstrate the
ability of MedNeXt to be significantly scaled depthwise (even at standard
kernel sizes). We further explore large kernel sizes and experiment with
$k=\\{3,5\\}$ for each configuration, to maximize performance via compound
scaling of the MedNeXt architecture.

##  3 Experimental Design

###  3.1 Configurations, Implementation and Baselines

We use PyTorch [24] for implementing our framework. We experiment with 4
configurations of the MedNeXt with 2 kernel sizes as detailed in Section 2.4.
The GPU memory requirements of scaling are limited via – 1) Mixed precision
training with PyTorch AMP, 2) Gradient Checkpointing. [4]. Our experimental
framework uses the nnUNet [13] as a backbone - where the training schedule
(epochs=1000, batches per epoch=250), inference (50% patch overlap) and data
augmentation remain unchanged. All networks, except nnUNet, are trained with
AdamW [23] as optimizer. The data is resampled to $1.0$ mm isotropic spacing
during training and inference (with results on original spacing), using input
patch size of $128\times 128\times 128$ and $512\times 512$, and batch size 2
and 14, for 3D and 2D networks respectively. The learning rate for all MedNeXt
models is $0.001$, except kernel:5 in KiTS19, which uses $0.0001$ for
stability. For baselines, all Swin models and 3D-UX-Net use $0.0025$, while
ViT models use $0.0001$. We use Dice Similarity Coefficient (DSC) and Surface
Dice Similarity (SDC) at 1.0mm tolerance for volumetric and surface accuracy.
5-fold cross-validation (CV) mean performance for supervised training using
80:20 splits for all models are reported. We also provide test set DSC scores
for a 5-fold ensemble of MedNeXt-L (kernel: $5\times 5\times 5$) without
postprocessing. Our extensive baselines consist of a high-performing
convolutional network (nnUNet [13]), 4 convolution-transformer hybrid networks
with transformers in the encoder (UNETR [9], SwinUNETR [8]) and in
intermediate layers (TransBTS [31], TransUNet [3]), a fully transformer
network (nnFormer [34]) as well as a partially ConvNeXt network (3D-UX-Net
[17]). TransUNet is a 2D network while the rest are 3D networks. The uniform
framework provides a common testbed for all networks, without incentivizing
one over the other on aspects of patch size, spacing, augmentations, training
and evaluation.

###  3.2 Datasets

We use 4 popular tasks, encompassing organ as well as tumor segmentation
tasks, to comprehensively demonstrate the benefits of the MedNeXt architecture
– 1) Beyond-the-Cranial-Vault (BTCV) Abdominal CT Organ Segmentation [16], 2)
AMOS22 Abdominal CT Organ Segmentation [14] 3) Kidney Tumor Segmentation
Challenge 2019 Dataset (KiTS19) [11], 4) Brain Tumor Segmentation Challenge
2021 (BraTS21) [1]. BTCV, AMOS22 and KiTS19 datasets contain 30, 200 and 210
CT volumes with 13, 15 and 2 classes respectively, while the BraTS21 dataset
contains 1251 MRI volumes with 3 classes. This diversity shows the
effectiveness of our methods across imaging modalities and training set sizes.

##  4 Results and Discussion

Networks Cat. BTCV AMOS22 KiTS19 BraTS21 AVG DSC SDC DSC SDC DSC SDC DSC SDC
DSC SDC nnUNet Baselines 83.56 86.07 88.88 91.70 89.88 86.88 91.23 90.46 88.39
88.78 UNETR 75.06 75.00 81.98 82.65 84.10 78.05 89.65 88.28 82.36 81.00
TransUNet 76.72 76.64 85.05 86.52 80.82 72.90 89.17 87.78 82.94 80.96 TransBTS
82.35 84.33 86.52 88.84 87.03 83.53 90.66 89.71 86.64 86.60 nnFormer 80.76
82.37 84.20 86.38 89.09 85.08 90.42 89.83 86.12 85.92 SwinUNETR 80.95 82.43
86.83 89.23 87.36 83.09 90.48 89.56 86.41 86.08 3D-UX-Net 80.76 82.30 87.28
89.74 88.39 84.03 90.63 89.63 86.77 86.43 MedNeXt-S kernel: 3 83.90 86.60
89.03 91.97 90.45 87.80 91.27 90.46 88.66 89.21 MedNeXt-B 84.01 86.77 89.14
92.10 91.02 88.24 91.30 90.51 88.87 89.41 MedNeXt-M 84.31 87.34 89.27 92.28
90.78 88.22 91.57 90.78 88.98 89.66 MedNeXt-L 84.57 87.54 89.58 92.62 90.61
88.08 91.57 90.81 89.08 89.76 MedNeXt-S kernel: 5 83.92 86.80 89.27 92.26
90.08 87.04 91.40 90.57 88.67 89.17 MedNeXt-B 84.23 87.06 89.38 92.36 90.30
87.40 91.48 90.70 88.85 89.38 MedNeXt-M 84.41 87.48 89.58 92.65 90.87 88.15
91.49 90.67 89.09 89.74 MedNeXt-L 84.82 87.85 89.87 92.95 90.71 87.85 91.46
90.73 89.22 89.85

Table 2: 5-fold CV results of MedNeXt at kernel sizes: $\\{3,5\\}$
outperforming 7 baselines – consisting of convolutional, transformer and large
kernel networks.
val (bold): better than or equal to ($\geq$) top baseline
val (underline): better than ($>$) kernel: $3$ of same configuration

###  4.1 Performance ablation of architectural improvements

We ablate the MedNeXt-B configuration on AMOS22 and BTCV datasets to highlight
the efficacy of our improvements and demonstrate that a vanilla ConvNeXt is
unable to compete with existing segmentation baselines such as nnUNet. The
following are observed in ablation tests in Table 1 (Right) –

  1. 1.

Residual Inverted Bottlenecks, specifically in Up and Downsampling layers,
functionally enables MedNeXt (MedNeXt-B Resampling vs Standard Resampling) for
medical image segmentation. In contrast, absence of these modified blocks lead
to considerably worse performance. This is possibly owing to preservation of
semantic richness in feature maps while resampling.

  2. 2.

Training large kernel networks for medical image segmentation is a non-trivial
task, with large kernel MedNeXts trained from scratch failing to perform in
seen in MedNeXt-B (UpKern vs From Scratch). UpKern improves performance in
kernel $5\times 5\times 5$ on both BTCV and AMOS22, whereas large kernel
performance is indistinguishable from small kernels without it.

  3. 3.

The performance boost in large kernels is seen to be due to the combination of
UpKern with a larger kernel and not merely a longer effective training
schedule (Upkern vs Trained $2\times$), as a trained MedNeXt-B with kernel
$3\times 3\times 3$ retrained again is unable to match its large kernel
counterpart.

This highlights that the MedNeXt modifications successfully translate the
ConvNeXt architecture to medical image segmentation. We further establish the
performance of the MedNeXt architecture against our baselines – comprising of
convolutional, transformer-based and large kernel baselines – on all 4
datasets. We discuss the effectiveness of the MedNeXt on multiple levels.

###  4.2 Performance comparison to baselines

There are 2 levels at which MedNeXt successfully overcomes existing baselines
- 5 fold CV and public testset performance. In 5-fold CV scores in Table 2,
MedNeXt, with $3\times 3\times 3$ kernels, takes advantage of depth and width
scaling to provide state-of-the-art segmentation performance against every
baseline on all 4 datasets with no additional training data. MedNeXt-L
outperforms or is competitive with smaller variants despite task heterogeneity
(brain and kidney tumors, organs), modality (CT, MRI) and training set size
(BTCV: 18 samples vs BraTS21: 1000 samples), establishing itself as a powerful
alternative to established methods such as nnUNet. With UpKern and $5\times
5\times 5$ kernels, MedNeXt takes advantage of full compound scaling to
improve further on its own small kernel networks, comprehensively on organ
segmentation (BTCV, AMOS22) and in a more limited fashion on tumor
segmentation (KiTS19, BraTS21).

Furthermore, in leaderboard scores on official testsets (Fig. 1(c)), 5-fold
ensembles for MedNeXt-L (kernel: $5\times 5\times 5$) and nnUNet, its
strongest competitor are compared – 1) BTCV: MedNeXt beats nnUNet and, to the
best of our knowledge, is one of the leading methods with only supervised
training and no extra training data (DSC: 88.76, HD95: 15.34), 2) AMOS22:
MedNeXt not only surpasses nnUNet, but is also Rank 1 (date: 09.03.23)
currently on the leaderboard (DSC: 91.77, NSD: 84.00), 3) KITS19: MedNeXt
exceeds nnUNet performance (DSC: 91.02), 4) BraTS21: MedNeXt surpasses nnUNet
in both volumetric and surface accuracy (DSC: 88.01, HD95: 10.69). MedNeXt
attributes its performance solely to its architecture without leveraging
techniques like transfer learning (3D-UX-Net) or repeated 5-fold ensembling
(UNETR, SwinUNETR), thus establishing itself as the state-of-the-art for
medical image segmentation.

##  5 Conclusion

In comparison to natural image analysis, medical image segmentation lacks
architectures that benefit from scaling networks due to inherent domain
challenges such as limited training data. In this work, MedNeXt is presented
as a scalable Transformer-inspired fully-ConvNeXt 3D segmentation architecture
customized for high performance on limited medical image datasets. We
demonstrate MedNeXt’s state-of-the-art performance across 4 challenging tasks
against 7 strong baselines. Additionally, similar to ConvNeXt for natural
images [22], we offer the compound scalable MedNeXt design as an effective
modernization of standard convolution blocks for building deep networks for
medical image segmentation.

## References

  * [1] Baid, U., Ghodasara, S., Mohan, S., Bilello, M., Calabrese, E., Colak, E., Farahani, K., Kalpathy-Cramer, J., Kitamura, F.C., Pati, S., et al.: The rsna-asnr-miccai brats 2021 benchmark on brain tumor segmentation and radiogenomic classification. arXiv preprint arXiv:2107.02314 (2021)
  * [2] Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., Wang, M.: Swin-unet: Unet-like pure transformer for medical image segmentation. arXiv preprint arXiv:2105.05537 (2021)
  * [3] Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021)
  * [4] Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016)
  * [5] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. pp. 424–432. Springer (2016)
  * [6] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [7] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [8] Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images. In: International MICCAI Brainlesion Workshop. pp. 272–284. Springer (2022)
  * [9] Hatamizadeh, A., Tang, Y., Nath, V., Yang, D., Myronenko, A., Landman, B., Roth, H.R., Xu, D.: Unetr: Transformers for 3d medical image segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 574–584 (2022)
  * [10] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [11] Heller, N., Isensee, F., Maier-Hein, K.H., Hou, X., Xie, C., Li, F., Nan, Y., Mu, G., Lin, Z., Han, M., et al.: The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge. Medical Image Analysis p. 101821 (2020)
  * [12] Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)
  * [13] Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods 18(2), 203–211 (2021)
  * [14] Ji, Y., Bai, H., Yang, J., Ge, C., Zhu, Y., Zhang, R., Li, Z., Zhang, L., Ma, W., Wan, X., et al.: Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. arXiv preprint arXiv:2206.08023 (2022)
  * [15] Karimi, D., Vasylechko, S.D., Gholipour, A.: Convolution-free medical image segmentation using transformers. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, Proceedings, Part I 24. pp. 78–88. Springer (2021)
  * [16] Landman, B., Xu, Z., Igelsias, J., Styner, M., Langerak, T., Klein, A.: Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge. In: Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault Challenge. vol. 5, p. 12 (2015)
  * [17] Lee, H.H., Bao, S., Huo, Y., Landman, B.A.: 3d ux-net: A large kernel volumetric convnet modernizing hierarchical transformer for medical image segmentation. arXiv preprint arXiv:2209.15076 (2022)
  * [18] Li, H., Nan, Y., Del Ser, J., Yang, G.: Large-kernel attention for 3d medical image segmentation. Cognitive Computation pp. 1–15 (2023)
  * [19] Litjens, G., Kooi, T., Bejnordi, B.E., Setio, A.A.A., Ciompi, F., Ghafoorian, M., Van Der Laak, J.A., Van Ginneken, B., Sánchez, C.I.: A survey on deep learning in medical image analysis. Medical image analysis 42, 60–88 (2017)
  * [20] Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., et al.: Swin transformer v2: Scaling up capacity and resolution. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12009–12019 (2022)
  * [21] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10012–10022 (2021)
  * [22] Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 11976–11986 (2022)
  * [23] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [24] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)
  * [25] Peiris, H., Hayat, M., Chen, Z., et al.: A robust volumetric transformer for accurate 3d tumor segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 162–172. Springer (2022)
  * [26] Ridnik, T., Ben-Baruch, E., Noy, A., Zelnik-Manor, L.: Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 (2021)
  * [27] Roy, S., Kügler, D., Reuter, M.: Are 2.5 d approaches superior to 3d deep networks in whole brain segmentation? In: International Conference on Medical Imaging with Deep Learning. pp. 988–1004. PMLR (2022)
  * [28] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [29] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning. pp. 6105–6114. PMLR (2019)
  * [30] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017)
  * [31] Wang, W., Chen, C., Ding, M., Yu, H., et al.: Transbts: Multimodal brain tumor segmentation using transformer. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 109–119. Springer (2021)
  * [32] Wu, Y., He, K.: Group normalization. In: Proceedings of the European conference on computer vision (ECCV). pp. 3–19 (2018)
  * [33] Xie, Y., Zhang, J., Shen, C., Xia, Y.: Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation. In: International conference on medical image computing and computer-assisted intervention. pp. 171–180. Springer (2021)
  * [34] Zhou, H.Y., Guo, J., Zhang, Y., Yu, L., Wang, L., Yu, Y.: nnformer: Interleaved transformer for volumetric segmentation. arXiv preprint arXiv:2109.03201 (2021)

##  6 Supplementary Material

###  6.1 UpKern: Algorithm

UpKern Initialization can be condensed into 20 lines of PyTorch pseudo-code.
While a 3D architecture is used in this work, the technique is general enough
to be translated to 2D by changing ‘trilinear’ interpolation to ‘bilinear’.

Algorithm 1 PyTorch code in 20 lines for Upsampled Kernel Initialization
(UpKern) for MedNeXt with large kernels ($\geq 5$)

⬇

1import torch.functional as F

2

3def upkern_init_load_weights(network, pretrained_net):

4

5 pretrn_dct = pretrained_net.state_dict()

6 model_dct = network.state_dict()

7 for k in model_dct.keys():

8 # Only common keys and core algorithm in Demo code

9 if k in model_dct.keys() and k in pretrn_dct.keys():

10 inc1, outc1, *spt_dims1 = model_dct[k].shape

11 inc2, outc2, *spt_dims2 = pretrn_dct[k].shape

12 if spt_dims1 == spt_dims2: # standard init

13 model_dct[k] = pretrn_dct[k]

14 else: # Upsampled kernel init

15 model_dct[k] = F.interpolate(

16 pretrn_dct[k],

17 size=spt_dims1,

18 mode=’trilinear’)

19 network.load_state_dict(model_dct)

20 return network

###  6.2 Detailed Model Configuration

Configs # Blocks (B) Exp. Ratio (R) C Params (Mi) GFLOPs secs/Train Epoch S
$B_{all}=2$ $R_{all}$ = 2 32 5.6, 5.9 130, 169 117, 364 B $R_{1}=R_{9}=2$
$R_{2}=R_{8}=3$ $R_{3-7}=4$ 10.5, 11.0 170, 208 120, 381 M $B_{1}=B_{9}=3$
$B_{2-8}=4$ 17.6, 18.3 248, 308 213, 668 L $B_{1}=B_{9}=3$ $B_{2}=B_{8}=4$
$B_{3-7}=8$ $R_{1}=R_{9}=3$ $R_{2}=R_{8}=4$ $R_{3-7}=8$ 61.8, 63.0 500, 564
267, 735

Table 3: Detailed configurations, GLOPs and training time at kernel={3, 5} of
MedNeXt architectures (left) in this work. GFLOPs is calculated for a single
$128\times 128\times 128$ patch. Training time is provided of a single epoch
with batch size = 2 and 250 batches.

##  7 Tabulation of Results on 4 Test Sets

Technique | Dataset | DSC
---|---|---
MedNeXt | BTCV | 88.76
AMOS22 | 91.77
KiTS19 | 91.02
BraTS21 | 88.01
Table 4: Tabulation of results on public validation sets from Figure 1(c) of
the 4 datasets – BTCV, AMOS22, KiTS19 and BraTS21 – used in this work. A
single 5-fold ensemble of MedNeXt-L using $5\times 5\times 5$ kernels,
initialized using UpKern were used for all datasets. The 3d_fullres mode of
the nnUNet pipeline was used and the cascade mode was not utilized.