# Beyond Self-Attention: Deformable Large Kernel Attention for Medical Image
Segmentation

Reza Azad1 Leon Niggemeier1 Michael Hüttemann1 Amirhossein Kazerouni2
Ehsan Khodapanah Aghdam3 Yury Velichko4 Ulas Bagci4 Dorit Merhof
111Corresponding author. Email: dorit.merhof@ur.de.${}^{\;\,,5,6}$
1 Faculty of Electrical Engineering and Information Technology, RWTH Aachen
University,
Aachen, Germany
2 School of Electrical Engineering, Iran University of Science and Technology,
Tehran, Iran
3 Department of Electrical Engineering, Shahid Beheshti University, Tehran,
Iran
4 Department of Radiology, Northwestern University, Chicago, USA
5 Faculty of Informatics and Data Science, University of Regensburg,
Regensburg, Germany
6 Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany

###### Abstract

Medical image segmentation has seen significant improvements with transformer
models, which excel in grasping far-reaching contexts and global contextual
information. However, the increasing computational demands of these models,
proportional to the squared token count, limit their depth and resolution
capabilities. Most current methods process D volumetric image data slice-by-
slice (called pseudo 3D), missing crucial inter-slice information and thus
reducing the model’s overall performance. To address these challenges, we
introduce the concept of Deformable Large Kernel Attention (D-LKA Attention),
a streamlined attention mechanism employing large convolution kernels to fully
appreciate volumetric context. This mechanism operates within a receptive
field akin to self-attention while sidestepping the computational overhead.
Additionally, our proposed attention mechanism benefits from deformable
convolutions to flexibly warp the sampling grid, enabling the model to adapt
appropriately to diverse data patterns. We designed both 2D and 3D adaptations
of the D-LKA Attention, with the latter excelling in cross-depth data
understanding. Together, these components shape our novel hierarchical Vision
Transformer architecture, the D-LKA Net. Evaluations of our model against
leading methods on popular medical segmentation datasets (Synapse, NIH
Pancreas, and Skin lesion) demonstrate its superior performance. Our code
implementation is publicly available at the GitHub.

##  1 Introduction

Medical image segmentation plays a crucial role in computer-assisted
diagnostics, aiding medical professionals in analyzing complex medical images.
This process not only reduces the laboriousness of manual tasks and the
dependence on medical expertise but also enables faster and more accurate
diagnoses. The automation of segmentation offers the potential for faster and
more accurate diagnostic outcomes, facilitating appropriate treatment
strategies and enabling the execution of image-guided surgical procedures.
Thus, the imperative to create rapid and precise segmentation algorithms
serves as a driving force behind the motivation for this research.

Since the mid-2010s, Convolutional Neural Networks (CNNs) have become the
preferred technique for many computer vision applications. Their ability to
automatically extract complex feature representations from raw data without
the need for manual feature engineering has generated significant interest
within the medical image analysis community. Many successful CNN architectures
such as U-Net [48], Fully Convolutional Networks [44], DeepLab [16], or
SegCaps (segmentation capsules) [38] have been developed. These architectures
have achieved great success in the task of semantic segmentation and have
outperformed state-of-the-art (SOTA) methods previously [3, 35, 36].

The problem of identifying objects at different scales is a key concern in
computer vision research [41, 34]. In CNNs, the size of a detectable object is
closely linked to the receptive field dimensions of the corresponding network
layer. If an object extends beyond the boundaries of this receptive field,
this may lead to under-segmentation outcomes. Conversely, using excessively
large receptive fields compared to an object’s actual size can limit
recognition, as background information may exert undue influence on
predictions [27].

A promising approach to address this issue involves employing multiple kernels
with distinct sizes in parallel, similar to the mechanism of an Inception
Block [53]. However, increasing the kernel sizes to accommodate larger objects
is limited in practice due to the exponential increase in parameters and
computational requirements [27]. Consequently, various strategies, including
pyramid pooling techniques [28] and dilated convolutions [63] at varying
scales, have emerged to capture multi-scale contextual information.

Another intuitive concept entails directly incorporating multi-scale image
pyramids or their associated feature representations into the network
architecture. Yet, this approach poses challenges, particularly concerning the
feasibility of managing training and inference times [41]. The use of Encoder-
Decoder networks, such as U-Net, has proven advantageous in this context. Such
networks encode appearance and location in shallower layers, while deeper
layers capture higher semantic information and context from the broader
receptive fields of neurons [40]. Some methods combine features from different
layers or predict features from layers of different sizes to use information
from multiple scales [10]. Also, forecasting features from layers of varied
scales have emerged, effectively enabling the integration of insights across
multiple scales [41]. However, most Encoder-Decoder structures face a
challenge: they frequently fail to maintain consistent features across
different scales and mainly use the last decoder layer to generate
segmentation results [3, 37, 27].

Semantic segmentation is a task that involves predicting the semantic category
for each pixel in an image based on a predefined label set. This task requires
extracting high-level features while preserving the initial spatial resolution
[46, 42]. CNNs are well suited to capture local details and low-level
information, albeit at the expense of overlooking global context. This gap in
handling global information has been a focus of the vision transformer (ViT)
architecture, which has achieved remarkable success in several computer vision
tasks, including semantic segmentation.

The cornerstone of the ViT is the attention mechanism, which facilitates the
aggregation of information across the entire input sequence. This capability
empowers the network to incorporate long-range contextual cues beyond the
limited receptive field sizes of CNNs [24, 52]. However, this strategy usually
limits the ability of ViTs to effectively model local information [9]. This
limitation can impede their ability to detect local textures, which is crucial
for various diagnostic and prognostic tasks. This lack of local representation
can be attributed to the particular way ViT models process images. ViT models
partition an image into a sequence of patches and model their dependencies
using self-attention mechanisms. This approach may not be as effective as the
convolution operations employed by CNN models for extracting local features
within receptive fields. This difference in image processing methods between
ViT and CNN models may explain the superior performance of CNN models in local
feature extraction [22, 6]. In recent years, innovative approaches have been
developed to address the insufficient representation of local textures within
Transformer models. One such approach involves integrating CNN and ViT
features through complementary methods to combine their strengths and mitigate
any local representation shortcomings [15]. TransUNet [15] is an early example
of this approach, incorporating Transformer layers within the CNN bottleneck
to model both local and global dependencies. HiFormer [29] proposes a solution
that combines a Swin Transformer module and a CNN-based encoder to generate
two multi-scale feature representations that are integrated via a Double-Level
Fusion module. UNETR [26] employs a Transformer-based encoder and a CNN
decoder for 3D medical image segmentation. CoTr [61] and TransBTS [57] bridge
the CNN encoder and decoder with the Transformer to enhance segmentation
performance at low-resolution stages.

An alternate strategy to enhance local feature representation is to redesign
the self-attention mechanism within pure Transformer models. In this vein,
Swin-Unet [13] integrates a Swin Transformer [43] block of linear
computational complexity within a U-shaped structure as a multi-scale
backbone. MISSFormer [32] employs the Efficient Transformer [60] to address
parameter issues in vision transformers by incorporating a non-invertible
downsampling operation on input blocks. D-Former [59] introduces a pure
transformer-based pipeline featuring a double attention module to capture
fine-grained local attention and interaction with diverse units in a dilated
manner. Nonetheless, certain specific limitations remain. These include
computational inefficiency, as evidenced in the TransUNet model, the heavy
dependence on a CNN backbone, as observed in HiFormer, and the lack of
consideration of multi-scale information. Additionally, current segmentation
architectures often take a slice-by-slice approach to process 3D input
volumes, inadvertently disregarding potential correlations between neighboring
slices. This oversight limits the comprehensive use of volumetric information,
consequently compromising both localization accuracy and context integration.
Furthermore, it is crucial to recognize that lesions in the medical domain
often exhibit deformations in their shape. Therefore, any learning algorithm
intended for medical image analysis must be endowed with the ability to
capture and comprehend these deformations. Simultaneously, the algorithm
should maintain computational efficiency to facilitate the processing of 3D
volumetric data.

Our contributions: To address the challenges outlined above, we propose a
solution in the form of the Deformable-LKA module (❶), which serves as a
fundamental building block within our network design. This module is
explicitly designed to effectively handle contextual information while
simultaneously preserving local descriptors. This balance between the two
aspects within our architecture enhances its ability to achieve precise
semantic segmentation. Notably, our model introduces a dynamic adaptation of
receptive fields based on the data, diverging from the conventional fixed
filter masks found in traditional convolutional operations. This adaptive
approach allows us to overcome the inherent limitations associated with static
methods. This innovative approach extends to the development of both 2D and 3D
versions of the D-LKA Net architecture (❷). In the case of the 3D model, the
D-LKA mechanism is tailored to suit a 3D context, thus enabling the seamless
exchange of information across different volumetric slices. (❸). Finally, our
contribution is further emphasized by its computational efficiency. We achieve
this through a design that leans solely on the D-LKA concept, resulting in a
remarkable performance on various segmentation benchmarks that establish our
method as a new SOTA approach.

Figure 1: Proposed network architecture of the 3D D-LKA model on the left and
the 2D D-LKA model on the right.

##  2 Method

In this section, we begin by providing an overview of the methodology. First,
we revisit the concept of Large Kernel Attention (LKA) as introduced by Guo et
al. [23]. Then, we introduce our innovative exploration of the deformable LKA
module. Built up on this, we introduce both 2D and 3D network architectures
for segmentation tasks.

###  2.1 Large Kernel Attention

Large convolution kernels provide a similar receptive field as the self-
attention mechanism. A large convolution kernel can be constructed with much
less parameters and computation by using a depth-wise convolution, a depth-
wise dilated convolution, and a $1\times 1$ convolution. The equations for the
kernel sizes of the depthwise convolution and the depthwise dilated
convolution to construct a $K\times K$ kernel for an input of dimension
$H\times W$ and channels $C$ are:

$DW=\left(2d-1\right)\times\left(2d-1\right),$ (1)
$DW\text{-}D=\left\lceil\frac{K}{d}\right\rceil\times\left\lceil\frac{K}{d}\right\rceil,$
(2)

with a kernel size of $K$ and a dilation rate of $d$. The number of parameters
$P\left(K,d\right)$ and floating-point operations (FLOPs) $F\left(K,d\right)$
is calculated by:

$P\left(K,d\right)=C\left(\left\lceil\frac{K}{d}\right\rceil^{2}+\left(2d-1\right)^{2}+3+C\right),$
(3) $F\left(K,d\right)=P\left(K,d\right)\times H\times W.$ (4)

The number of FLOPs grows linearly with the size of the input image. The
number of parameters increases quadratically with the number of channels and
kernel size. However, as both are usually so small, they are not restricting
factors.

To minimize the number of parameters for a fixed kernel size $K$, the
derivative of equation 3 with respect to dilation rate $d$ can be set to zero:

$\frac{d}{d\hat{d}}P\left(K,\hat{d}\right)\overset{!}{=}0=C\cdot\left(8\hat{d}+\frac{2K^{2}}{\hat{d}^{3}}-4\right).$
(5)

For example, when the kernel size is $K=21$, this results in $d\approx 3.37$.
Extending the formulas to the three-dimensional case is straightforward. For
an input of size $H\times W\times D$ and channels $C$, then the equations for
the number of parameters $P_{3d}\left(K,d\right)$ and FLOPs
$F_{3d}\left(K,d\right)$ are:

$P_{3d}\left(K,d\right)=C\left(\left\lceil\frac{K}{d}\right\rceil^{3}+\left(2d-1\right)^{3}+3+C\right),$
(6) $F_{3d}\left(K,d\right)=P_{3d}\left(K,d\right)\times H\times W\times D,$
(7)

with kernel size $K$ and dilation $d$.

###  2.2 Deformable Large Kernel Attention

The concept of utilizing Large Kernels for medical image segmentation is
extended by incorporating Deformable Convolutions [20]. Deformable
Convolutions enable adjusting the sampling grid with whole-numbered offsets
for free deformation. An additional convolutional layer learns The deformation
from the feature maps, which creates an offset field. Learning the deformation
based on the features itself results in an adaptive convolution kernel. This
flexible kernel shape can improve the representation of lesions or organ
deformations, resulting in an enhanced definition of object boundaries. The
convolutional layer responsible for calculating offsets follows the kernel
size and dilation of its corresponding convolutional layer. Bilinear
interpolation is employed to compute pixel values for offsets that are not
found on the image grid. As shown in Figure 2, the D-LKA module can be
formulated as:

Attention $\displaystyle=\text{Conv}{1\times
1}(\text{DDW}\text{-D-}\text{Conv}(\text{DDW}\text{-}\text{Conv}(\text{F}^{\prime}))),$
(8) Output $\displaystyle=\text{Conv}{1\times
1}(\text{Attention}\otimes\text{F}^{\prime})+\text{F},$

where the input feature is denoted by $F\in\mathbb{R}^{C\times H\times W}$ and
$F^{\prime}=GELU(Conv(F))$. The attention component $\in$ $\mathbb{R}^{C\times
H\times W}$ is represented as an attention map, with each value indicating the
relative importance of corresponding features. The operator $\otimes$ denotes
an element-wise product operation. Notably, LKA departs from conventional
attention methods by not requiring additional normalization functions such as
sigmoid or Softmax. According to [56], these normalization functions tend to
neglect high-frequency information, thereby decreasing the performance of
self-attention-based methods.

In the 2D version of the approach, the convolution layers are substituted with
deformable convolutions because deformable convolutions improve the ability to
capture objects characterized by irregular shapes and sizes. Such objects are
commonly found in medical image data, making this augmentation especially
significant.

However, extending the concept of deformable LKA to the 3D domain presents
certain challenges. The primary constraint arises from the additional
convolution layer needed for offset generation. In contrast to the 2D case,
this layer cannot be executed in a depth-wise manner due to the nature of
input and output channels. In the 3D context, input channels correspond to
features, and the output channels enlarge to $3\cdot k\times k\times k$, with
a kernel size of $k$. The intricate nature of large kernels leads to an
expansion of channel count along the third dimension, causing a substantial
rise in parameters and FLOPs. Consequently, an alternative approach is
implemented for the 3D scenario. A sole deformable convolution layer is
introduced into the existing LKA framework, following the depth-wise
convolutions. This strategic design adaptation aims to mitigate the challenges
posed by the extension to three dimensions.

###  2.3 2D D-LKA Net

Figure 2: Architecture of the deformable LKA module.

The architecture of the 2D network is illustrated in Figure 1. The first
variant uses a MaxViT [54] as the encoder component for efficient feature
extraction, while the second variant incorporates deformable LKA layers for
more refined, superior segmentation.

In a more formal description, the encoder generates four hierarchical output
representations. A convolutional stem first reduces the dimensions of the
input image to $\frac{H}{4}\times\frac{W}{4}\times C$. Subsequently, feature
extraction is carried out through four stages of MaxViT blocks, each followed
by downsampling layers. As the process advances to the decoder, four stages of
D-LKA layers are implemented, each stage encompassing two D-LKA blocks. Patch-
expanding layers are then applied to achieve resolution upsampling while also
decreasing channel dimensions. Finally, a linear layer is responsible for
generating the ultimate output.

The structure of the 2D D-LKA block comprises LayerNorm, deformable LKA, and a
Multi-Layer Perceptron (MLP). The integration of residual connections ensures
effective feature propagation, even across deeper layers. This arrangement can
be mathematically represented as:

$x_{1}=D\text{-}LKA\text{-}Attn(LN(x_{in}))+x_{in},$ (9)
$x_{out}=MLP(LN(x_{1}))+x_{1},$ (10)
$MLP=Conv_{1}(GeLU(Conv_{d}(Conv_{1}(x)))),$ (11)

with input features $x_{in}$, layer normalization $LN$, deformable LKA
attention $D\text{-}LKA\text{-}Attn$, depthwise convolution $Conv_{d}$, linear
layers $Conv_{1}$ and GeLU activation function $GeLU$.

###  2.4 3D D-LKA Net

The 3D network architecture, depicted in Figure 1, is structured
hierarchically using an encoder-decoder design. Initially, a patch embedding
layer reduces the input image dimensions from $(H\times W\times D)$ to
$(\frac{H}{4}\times\frac{W}{4}\times\frac{D}{2})$. Within the encoder, a
sequence of three D-LKA stages is employed, each containing three D-LKA
blocks. After each stage, a downsampling step reduces spatial resolution by
half while doubling the channel dimension. The central bottleneck includes
another set of two D-LKA blocks. The decoder structure is symmetric to that of
the encoder. In order to double the feature resolution while reducing channel
count, transpose convolutions are utilized. Each decoder stage employs three
D-LKA blocks to promote long-range feature dependencies. The final
segmentation output is produced by a $3\times 3\times 3$ convolutional layer,
succeeded by a $1\times 1\times 1$ convolutional layer to match class-specific
channel requirements. To establish a direct connection between the input image
and the segmentation output, a skip connection is formed using convolutions.
Additional skip connections perform a fusion of features from other stages
based on simple addition. The ultimate segmentation map is produced through a
combination of $3\times 3\times 3$ and $1\times 1\times 1$ convolutional
layers.

The 3D D-LKA block includes layer normalization followed by D-LKA Attention,
with residual connections applied. The subsequent section employs a $3\times
3\times 3$ convolutional layer, followed by a $1\times 1\times 1$
convolutional layer, both accompanied by residual connections. This entire
process can be summarized as follows:

$x_{1}=DAttn(LN(x_{in}))+x_{in},$ (12)
$x_{out}=Conv_{1}(Conv_{3}(x_{1}))+x_{1},$ (13)

with input features $x_{in}$, layer normalization $LN$, deformable LKA
$DAttn$, convolutional layer $Conv_{1}$, and output features $x_{out}$.
$Conv_{3}$ refers to a feed forward network, with two convolutional layers and
activation functions.

##  3 Experiments

###  3.1 Experimental Setup

We have implemented both 2D and 3D models using the PyTorch framework and
performed training on a single RTX 3090 GPU. For the 2D method, a batch size
of 20 was used, along with Stochastic Gradient Descent (SGD) employing a base
learning rate of 0.05, a momentum of 0.9, and a weight decay of 0.0001. The
training process consisted of 400 epochs, employing a combination of cross-
entropy and Dice loss, represented as follows:

$\mathcal{L}_{total}=0.6\cdot\mathcal{L}_{dice}+0.4\cdot\mathcal{L}_{ce}.$
(14)

Consistent with [15], identical data augmentation techniques were applied. For
the 3D model, a batch size of 2 was chosen, and stochastic gradient descent
was employed with a base learning rate of 0.01 and a weight decay of
$3e^{-5}$. The input images were in the form of patches sized $128\times
128\times 64$. The training process consisted of 1000 epochs, with 250 patches
utilized per epoch. Data augmentation techniques consistent with nnFormer [64]
and UNETR++ [51] were employed.

###  3.2 Datasets

Synapse Multi-Organ Segmentation: First, we evaluate the performance of our
method using the well-established synapse multi-organ segmentation dataset
[14]. This dataset consists of 30 cases with a total of 3779 axial abdominal
clinical CT images. Each CT volume comprises a range of $85$ to $198$ slices,
each with dimensions of $512\times 512$ pixels. The voxel spatial resolution
varies in the range of $([0.54\sim 0.54]\times[0.98\sim 0.98]\times[2.5\sim
5.0])$ $\mathrm{mm}^{3}$. Our evaluation follows the setting presented in [15,
51] for 2D and 3D versions.

Skin Lesion Segmentation: Our comprehensive experiments also extend to skin
lesion segmentation datasets. Specifically, we leverage the ISIC 2017 dataset
[19], which contains 2000 dermoscopic images for training, 150 for validation,
and 600 for testing. Furthermore, we adopt the division scheme described in
previous literature [2, 5, 1, 21] for the ISIC 2018 dataset [18].
Additionally, the $\mathrm{PH}^{2}$ dataset [45] is used as a dermoscopic
image repository designed for both segmentation and classification tasks. This
dataset consists of 200 dermoscopic images containing 160 nevi and 40
melanomas.

NIH Pancreas Segmentation: The publicly available NIH Pancreas dataset
consists of 82 contrast-enhanced 3D abdominal CT volumes, each accompanied by
manual annotations [49]. In our configuration, we use 62 samples for training
and reserve the remaining samples for testing.

###  3.3 Quantitative and Qualitative Results

Table 1: Comparison results of the proposed method evaluated on the Synapse
dataset. Blue indicates the best result, and red displays the second-best.
Parameters are reported in millions (M) and FLOPS in billions (G). DSC is
presented for abdominal organs spleen (Spl), right kidney (RKid), left kidney
(LKid), gallbladder (Gal), liver (Liv), stomach (Sto), aorta (Aor), and
pancreas (Pan).

Methods | Params (M) | FLOPs (G) | Spl | RKid | LKid | Gal | Liv | Sto | Aor | Pan | Average
---|---|---|---|---|---|---|---|---|---|---|---
DSC $\uparrow$ | HD95 $\downarrow$
TransUNet [15] | 96.07 | 88.91 | 85.08 | 77.02 | 81.87 | 63.16 | 94.08 | 75.62 | 87.23 | 55.86 | 77.49 | 31.69
Swin-UNet [11] | 27.17 | 6.16 | 90.66 | 79.61 | 83.28 | 66.53 | 94.29 | 76.60 | 85.47 | 56.58 | 79.13 | 21.55
LeViT-UNet-384 [62] | 52.17 | 25.55 | 88.86 | 80.25 | 84.61 | 62.23 | 93.11 | 72.76 | 87.33 | 59.07 | 78.53 | 16.84
MISSFormer [33] | 42.46 | 9.89 | 91.92 | 82.00 | 85.21 | 68.65 | 94.41 | 80.81 | 86.99 | 65.67 | 81.96 | 18.20
ScaleFormer [31] | 111.6 | 48.93 | 89.40 | 83.31 | 86.36 | 74.97 | 95.12 | 80.14 | 88.73 | 64.85 | 82.86 | 16.81
HiFormer-B[30] | 25.51 | 8.045 | 90.99 | 79.77 | 85.23 | 65.23 | 94.61 | 81.08 | 86.21 | 59.52 | 80.39 | 14.70
DAEFormer [4] | 48.07 | 27.89 | 91.82 | 82.39 | 87.66 | 71.65 | 95.08 | 80.77 | 87.84 | 63.93 | 82.63 | 16.39
TransDeepLab [7] | 21.14 | 16.31 | 89.00 | 79.88 | 84.08 | 69.16 | 93.53 | 78.40 | 86.04 | 61.19 | 80.16 | 21.25
PVT-CASCADE [47] | 35.28 | 6.40 | 90.1 | 80.37 | 82.23 | 70.59 | 94.08 | 83.69 | 83.01 | 64.43 | 81.06 | 20.23
[HTML]C8FFFD LKA Baseline | 85.82 | 13.62 | 91.45 | 81.93 | 84.93 | 71.05 | 94.87 | 83.71 | 87.48 | 66.76 | 82.77 | 17.42
[HTML]C8FFFD 2D D-LKA Net | 101.64 | 19.92 | 91.22 | 84.92 | 88.38 | 73.79 | 94.88 | 84.94 | 88.34 | 67.71 | 84.27 | 20.04

####  3.3.1 2D results

Synapse Dataset: In Table 3.3, we present a comprehensive comparison of the
leading performances achieved by other SOTA techniques in contrast to our
proposed approach.

The results in terms of the Dice Similarity Coefficient (DSC) reveal that
D-LKA Net exhibits superiority over the previously established SOTA methods.
Specifically, it outperforms ScaleFormer [31] by 1.41%, DAEFormer [4] by
1.64%, and by even larger margins when compared to other approaches. Notably,
significant improvements are seen in the segmentation of specific anatomical
regions such as the right kidney, left kidney, stomach, and pancreas. Notably,
the pancreas achieves a significantly improved segmentation result, showing an
impressive 2.04% improvement over the second-best performing method. Given
that the segmentation of smaller abdominal organs, such as the gallbladder or
pancreas, has historically been a challenge for existing SOTA approaches, this
notable performance improvement represents a significant step forward in
achieving more accurate segmentation results. A qualitative comparison of
different methods is shown in Figure 3. Compared to DAE-Former [4], our
approach results in less misclassifications for the stomach. While Unet [48]
and Swin-UNet [12] sometimes classify distant tissue as parts of the liver,
gallbladder or stomach, our approach reduces the misclassifications and better
represents the shape of the organs.

Figure 3: Qualitative comparison of the results from UNet [48], Swin UNet
[12], DAEFormer [4] and D-LKA Net.

Skin Lesion Segmentation Results: The comparative outcomes for skin lesion
segmentation benchmarks, including ISIC 2017, ISIC 2018, and
$\mathrm{PH}^{2}$, in contrast to leading methods, are detailed in Table 2.
Notably, our D-LKA Net consistently outperforms its competitors across various
evaluation metrics. This consistent superiority observed across different
datasets underscores D-LKA Net’s robust generalization capabilities.

Table 2: Performance comparison of the proposed method against the SOTA
approaches on skin lesion segmentation benchmarks.

Methods | ISIC 2017 | ISIC 2018 | $\mathbf{PH^{2}}$
---|---|---|---
DSC | SE | SP | ACC | DSC | SE | SP | ACC | DSC | SE | SP | ACC
U-Net [48] | 0.8159 | 0.8172 | 0.9680 | 0.9164 | 0.8545 | 0.8800 | 0.9697 | 0.9404 | 0.8936 | 0.9125 | 0.9588 | 0.9233
Att-UNet [50] | 0.8082 | 0.7998 | 0.9776 | 0.9145 | 0.8566 | 0.8674 | 0.9863 | 0.9376 | 0.9003 | 0.9205 | 0.9640 | 0.9276
DAGAN [39] | 0.8425 | 0.8363 | 0.9716 | 0.9304 | 0.8807 | 0.9072 | 0.9588 | 0.9324 | 0.9201 | 0.8320 | 0.9640 | 0.9425
TransUNet [15] | 0.8123 | 0.8263 | 0.9577 | 0.9207 | 0.8499 | 0.8578 | 0.9653 | 0.9452 | 0.8840 | 0.9063 | 0.9427 | 0.9200
MCGU-Net [2] | 0.8927 | 0.8502 | 0.9855 | 0.9570 | 0.8950 | 0.8480 | 0.9860 | 0.9550 | 0.9263 | 0.8322 | 0.9714 | 0.9537
MedT [55] | 0.8037 | 0.8064 | 0.9546 | 0.9090 | 0.8389 | 0.8252 | 0.9637 | 0.9358 | 0.9122 | 0.8472 | 0.9657 | 0.9416
FAT-Net [58] | 0.8500 | 0.8392 | 0.9725 | 0.9326 | 0.8903 | 0.9100 | 0.9699 | 0.9578 | 0.9440 | 0.9441 | 0.9741 | 0.9703
TMU-Net [8] | 0.9164 | 0.9128 | 0.9789 | 0.9660 | 0.9059 | 0.9038 | 0.9746 | 0.9603 | 0.9414 | 0.9395 | 0.9756 | 0.9647
Swin-Unet [11] | 0.9183 | 0.9142 | 0.9798 | 0.9701 | 0.8946 | 0.9056 | 0.9798 | 0.9645 | 0.9449 | 0.9410 | 0.9564 | 0.9678
DeepLabv3+ (CNN) [17] | 0.9162 | 0.8733 | 0.9921 | 0.9691 | 0.8820 | 0.8560 | 0.9770 | 0.9510 | 0.9202 | 0.8818 | 0.9832 | 0.9503
HiFormer-B [29] | 0.9253 | 0.9155 | 0.9840 | 0.9702 | 0.9102 | 0.9119 | 0.9755 | 0.9621 | 0.9460 | 0.9420 | 0.9772 | 0.9661
[HTML]C8FFFD LKA (Baseline) | 0.9229 | 0.9192 | 0.9824 | 0.9704 | 0.9118 | 0.8984 | 0.9802 | 0.9632 | 0.9409 | 0.9312 | 0.9759 | 0.9612
[HTML]C8FFFD 2D D-LKA Net | 0.9254 | 0.9327 | 0.9793 | 0.9705 | 0.9177 | 0.9164 | 0.9773 | 0.9647 | 0.9490 | 0.9430 | 0.9775 | 0.9659

A qualitative comparison of the results is presented in Figure 4. In
comparison to the baseline method, D-LKA Net better follows the complex
outline of the lesions. In contrast to Swin-UNet and HiFormer-B, which tend to
over- or under-segment certain areas, our approach achieves a more accurate
segmentation.

Figure 4: 2D visualizations on the ISIC 2018 [18] dataset.

####  3.3.2 3D results

Synapse Dataset: We compare our 3D approach with previous SOTA methods on the
Synapse dataset. The results are presented in Table 3.

Table 3: Results on the Synapse dataset using 3D models. Blue indicates the
best result, and red displays the second-best.

Methods | Params (M) | FLOPs (G) | Spl | RKid | LKid | Gal | Liv | Sto | Aor | Pan | Average
---|---|---|---|---|---|---|---|---|---|---|---
DSC $\uparrow$ | HD95 $\downarrow$
UNETR [26] | 92.49 | 75.76 | 85.00 | 84.52 | 85.60 | 56.30 | 94.57 | 70.46 | 89.80 | 60.47 | 78.35 | 18.59
Swin-UNETR [25] | 62.83 | 384.2 | 95.37 | 86.26 | 86.99 | 66.54 | 95.72 | 77.01 | 91.12 | 68.80 | 83.48 | 10.55
nnFormer [64] | 150.5 | 213.4 | 90.51 | 86.25 | 86.57 | 70.17 | 96.84 | 86.83 | 92.04 | 83.35 | 86.57 | 10.63
UNETR++ [51] | 42.96 | 47.98 | 95.77 | 87.18 | 87.54 | 71.25 | 96.42 | 86.01 | 92.52 | 81.10 | 87.22 | 7.53
[HTML]C8FFFD LKA Baseline | 28.94 | 48.79 | 90.49 | 87.54 | 87.57 | 63.81 | 96.96 | 84.89 | 93.22 | 82.46 | 85.87 | 14.35
[HTML]C8FFFD D-LKA Net | 42.35 | 66.96 | 95.88 | 88.50 | 87.64 | 72.14 | 96.25 | 85.03 | 92.87 | 81.64 | 87.49 | 9.57

We achieve a 0.27% improvement in DSC over the previous SOTA methods UNETR++
[51]. Compared to nnFormer [64], an improvement of 0.92% is achieved. For the
HD95 metric, D-LKA Net reaches the second-best result. In comparison to
UNETR++, a small performance increase is observed for spleen, left kidney and
aorta. A significant increase is reported of right kidney and the small organs
gallbladder and pancreas. An increase in the segmentation performance of these
small organs is especially important. In terms of parameters, we have the
lowest number of parameters with only 42.35 M while still achieving excellent
segmentation performance. The number of FLOPs is 66.96 G, the second lowest.
Only UNETR++ has less FLOPs. Compared to SOTA approaches like Swin-UNETR and
nnFormer, we need only about 17% and 31% of the computations while achieving a
better performance. Figure 5 shows qualitative results on the Synapse dataset.
In comparison to the nnFormer, we better capture the aorta as a whole and
don’t confuse other tissue as the organ. In comparison to UNETR++, we better
segment the pancreas, whereas UNETR++ tends to under-segment. Also, our
approach segments the liver and stomach more accurately than UNETR++, which
tends to over-segment these organs.

Figure 5: Qualitative results on the Synapse dataset. The boxes mark regions
of failure.

Pancreas Dataset:  The results of the NIH Pancreas dataset are presented in
Table 4. Our approach achieves the best performance in all four metrics. In
comparison to the closest competitor UNETR++, an increase of 0.63% in DSC,
0.82% in Jaccard, and a decrease of 1.04 in HD95 and 0.26 in ASD can be noted.
D-LKA Net also has the lowest number of Parameters, with 62.07 M.

Table 4: Results on the Pancreas dataset. The dice score, Jaccard index, HD95,
and ASD are reported.

Methods | Params (M) | FLOPs (G) | DSC $\uparrow$ | Jaccard $\uparrow$ | HD95 $\downarrow$ | ASD$\downarrow$
---|---|---|---|---|---|---
UNETR [26] | 92.45 | 63.53 | 77.42 | 63.95 | 15.07 | 5.09
UNETR++ [51] | 96.77 | 102.44 | 80.59 | 68.08 | 8.63 | 2.25
[HTML]C8FFFD D-LKA Net | 62.07 | 166.63 | 81.22 | 68.90 | 7.59 | 1.99

Figure 6 shows a qualitative comparison of different approaches. UNETR fails
to segment the Pancreas as a single object. UNETR++ has smaller artifacts in
the segmentation result. Our approach follows the highly irregular shape of
the organ better than the other approaches.

Figure 6: Qualitative results on the Pancreas Dataset.

###  3.4 Ablations Studies

Robustness. To enhance the robustness of our evaluation and to analyze
statistical significance, we undertake 5 training runs for each method on the
Synapse 2D version. This practice not only ensures a more comprehensive
assessment but also enables us to visualize the variations in performance
(Please refer to the supplementary file for the visualizations). In our
evaluation, we observed a stable increase in performance for the aorta,
gallbladder, left and right kidney, liver, pancreas, and stomach, where the
median performance is higher than other SOTA methods. Solely the spleen
segmentation performance is slightly worse. Furthermore, significant
performance improvements are achieved for the gallbladder, pancreas, and
stomach.

Deformable LKA Influence. We continue our ablation study by determining the
effectiveness of D-LKA. For this purpose, we construct a network employing 3D
LKA without a deformable layer and another version using 3D LKA with an extra
convolutional layer instead of the deformable one. The results of this
analysis are presented in Table 5. Introducing the additional 3D convolutional
layer results in a notable improvement in the performance of 0.99% in DSC when
compared to the 3D LKA baseline. However, this modification also increases the
number of parameters within the network. Replacing the 3D convolutional layer
with a deformable convolutional layer leads to an additional performance
boost, as indicated by a 0.63% increase in DSC. This alteration, similar to
the previous one, also introduces more parameters and FLOPs into the network.
Since the network’s size remains relatively small, the increase in these
metrics is acceptable.

Table 5: Results for the experiments of the influence of the deformable
convolution layer with the Synapse Dataset.

Methods | Params | FLOPs | DSC $\uparrow$ | HD95 $\downarrow$
---|---|---|---|---
3D LKA | 28.94 | 48.79 | 85.87 | 14.35
3D LKA + 3D conv. | 37.73 | 58.64 | 86.86 | 12.69
3D LKA + 3D deform. conv. | 42.35 | 66.96 | 87.49 | 9.57

Skip Connections. Lastly, we assess the effect of the skip connections on the
segmentation process. The results are shown in Table 6. We remove all skip
connections and gradually add them to the network, starting with the highest-
level skip connection. The results yield that skip connections are crucial for
obtaining optimal segmentation performance. Additionally, we highlight that
the highest-level skip connection is vital for achieving the best segmentation
result, improving the DSC performance by 0.42%.

Table 6: Results for the experiments of the influence of the number of skip connections with the Synapse Dataset. Metrics | 0 | 1 | 2 | 3 | 4
---|---|---|---|---|---
DSC $\uparrow$ | 84.78 | 85.99 | 86.56 | 87.07 | 87.49
HD95 $\downarrow$ | 11.18 | 12.21 | 14.89 | 9.25 | 9.57

##  4 Conclusion

In this paper, we propose a novel hierarchical hybrid Vision Transformer and
CNN architecture using Deformable Large Kernel Attention (D-LKA Net). This
attention mechanism enables the network to learn a deformation grid for
accessing more relevant information than conventional attention strategies.
Furthermore, the Large Kernel Attention mechanism can aggregate global
information similar to that of self-attention to overcome the local
restrictions of CNN mechanisms. Further, we present a 3D version of our
proposed network, which includes cross-slice feature extraction for an even
stronger representation capability. Our models receive SOTA results on several
publicly available segmentation datasets. Overall, we believe that the
proposed D-LKA Net is a robust and powerful choice for medical image
segmentation.

## References

  * [1] Ehsan Khodapanah Aghdam, Reza Azad, Maral Zarvani, and Dorit Merhof.  Attention swin u-net: Cross-contextual attention mechanism for skin lesion segmentation.  arXiv preprint arXiv:2210.16898, 2022.
  * [2] Maryam Asadi-Aghbolaghi, Reza Azad, Mahmood Fathy, and Sergio Escalera.  Multi-level context gating of embedded collective knowledge for medical image segmentation.  arXiv preprint arXiv:2003.05056, 2020.
  * [3] Reza Azad, Ehsan Khodapanah Aghdam, Amelie Rauland, Yiwei Jia, Atlas Haddadi Avval, Afshin Bozorgpour, Sanaz Karimijafarbigloo, Joseph Paul Cohen, Ehsan Adeli, and Dorit Merhof.  Medical image segmentation review: The success of u-net.  arXiv preprint arXiv:2211.14830, 2022.
  * [4] Reza Azad, René Arimond, Ehsan Khodapanah Aghdam, Amirhosein Kazerouni, and Dorit Merhof.  Dae-former: Dual attention-guided efficient transformer for medical image segmentation.  arXiv preprint arXiv:2212.13504, 2022.
  * [5] Reza Azad, Maryam Asadi-Aghbolaghi, Mahmood Fathy, and Sergio Escalera.  Bi-directional convlstm u-net with densley connected convolutions.  In ICCV 2019, IEEE International Conference on Computer Vision 2019, 2019.
  * [6] Reza Azad, Abdur R Fayjie, Claude Kauffmann, Ismail Ben Ayed, Marco Pedersoli, and Jose Dolz.  On the texture bias for few-shot cnn segmentation.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2674–2683, 2021.
  * [7] Reza Azad, Moein Heidari, Moein Shariatnia, Ehsan Khodapanah Aghdam, Sanaz Karimijafarbigloo, Ehsan Adeli, and Dorit Merhof.  Transdeeplab: Convolution-free transformer-based deeplab v3+ for medical image segmentation.  In Predictive Intelligence in Medicine, pages 91–102. Springer Nature Switzerland, 2022.
  * [8] Reza Azad, Moein Heidari, Yuli Wu, and Dorit Merhof.  Contextual attention network: Transformer meets u-net.  In Machine Learning in Medical Imaging: 13th International Workshop, MLMI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 18, 2022, Proceedings, pages 377–386. Springer, 2022.
  * [9] Reza Azad, Amirhossein Kazerouni, Moein Heidari, Ehsan Khodapanah Aghdam, Amirali Molaei, Yiwei Jia, Abin Jose, Rijo Roy, and Dorit Merhof.  Advances in medical image analysis with vision transformers: A comprehensive review.  arXiv preprint arXiv:2301.03505, 2023.
  * [10] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation, 2016.
  * [11] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation, 2021\.
  * [12] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In European conference on computer vision, pages 205–218. Springer, 2022.
  * [13] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang.  Swin-unet: Unet-like pure transformer for medical image segmentation.  In Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part III, pages 205–218. Springer, 2023.
  * [14] MICCAI 2015 Multi-Atlas Abdomen Labeling Challenge.  Synapse multi-organ segmentation dataset.  https://www.synapse.org/#!Synapse:syn3193805/wiki/217789, 2015.  Accessed: 2022-04-20.
  * [15] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou.  Transunet: Transformers make strong encoders for medical image segmentation.  arXiv preprint arXiv:2102.04306, 2021.
  * [16] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L. Yuille.  Semantic image segmentation with deep convolutional nets and fully connected crfs, 2016.
  * [17] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In Proceedings of the European conference on computer vision (ECCV), pages 801–818, 2018.
  * [18] Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al.  Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic).  arXiv preprint arXiv:1902.03368, 2019.
  * [19] Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al.  Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic).  In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pages 168–172. IEEE, 2018.
  * [20] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 764–773, 2017.
  * [21] Sania Eskandari and Janet Lumpp.  Inter-scale dependency modeling for skin lesion segmentation with transformer-based networks.  In Medical Imaging with Deep Learning, short paper track, 2023.
  * [22] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  In International Conference on Learning Representations.
  * [23] Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu.  Visual attention network.  arXiv preprint arXiv:2202.09741, 2022.
  * [24] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao.  A survey on vision transformer.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):87–110, jan 2023.
  * [25] Ali Hatamizadeh, Vishwesh Nath, Yucheng Tang, Dong Yang, Holger R Roth, and Daguang Xu.  Swin unetr: Swin transformers for semantic segmentation of brain tumors in mri images.  In International MICCAI Brainlesion Workshop, pages 272–284. Springer, 2021.
  * [26] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu.  Unetr: Transformers for 3d medical image segmentation.  In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574–584, 2022.
  * [27] Junjun He, Zhongying Deng, and Yu Qiao.  Dynamic multi-scale filters for semantic segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  IEEE transactions on pattern analysis and machine intelligence, 37(9):1904–1916, 2015.
  * [29] Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah Aghdam, Julien Cohen-Adad, and Dorit Merhof.  Hiformer: Hierarchical multi-scale representations using transformers for medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6202–6212, 2023.
  * [30] Moein Heidari, Amirhossein Kazerouni, Milad Soltany, Reza Azad, Ehsan Khodapanah Aghdam, Julien Cohen-Adad, and Dorit Merhof.  Hiformer: Hierarchical multi-scale representations using transformers for medical image segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6202–6212, 2023.
  * [31] Huimin Huang, Shiao Xie, Lanfen Lin, Yutaro Iwamoto, Xianhua Han, Yen-Wei Chen, and Ruofeng Tong.  Scaleformer: revisiting the transformer-based backbones from a scale-wise perspective for medical image segmentation.  arXiv preprint arXiv:2207.14552, 2022.
  * [32] Xiaohong Huang, Zhifang Deng, Dandan Li, Xueguang Yuan, and Ying Fu.  Missformer: An effective transformer for 2d medical image segmentation.  IEEE Transactions on Medical Imaging, pages 1–1, 2022.
  * [33] Xiaohong Huang, Zhifang Deng, Dandan Li, Xueguang Yuan, and Ying Fu.  Missformer: An effective transformer for 2d medical image segmentation.  IEEE Transactions on Medical Imaging, 2022.
  * [34] Abin Jose, Rijo Roy, Dennis Eschweiler, Ina Laube, Reza Azad, Daniel Moreno-Andrés, and Johannes Stegmaier.  End-to-end classification of cell-cycle stages with center-cell focus tracker using recurrent neural networks.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [35] Sanaz Karimijafarbigloo, Reza Azad, Amirhossein Kazerouni, Saeed Ebadollahi, and Dorit Merhof.  Mmcformer: Missing modality compensation transformer for brain tumor segmentation.  In Medical Imaging with Deep Learning, 2023.
  * [36] Sanaz Karimijafarbigloo, Reza Azad, Amirhossein Kazerouni, and Dorit Merhof.  Ms-former: Multi-scale self-guided transformer for medical image segmentation.  In Medical Imaging with Deep Learning, 2023.
  * [37] Sanaz Karimijafarbigloo, Reza Azad, and Dorit Merhof.  Self-supervised few-shot learning for semantic segmentation: An annotation-free approach.  In MICCAI 2023 workshop, 2023.
  * [38] Rodney LaLonde, Ziyue Xu, Ismail Irmakci, Sanjay Jain, and Ulas Bagci.  Capsules for biomedical image segmentation.  Medical image analysis, 68:101889, 2021.
  * [39] Baiying Lei, Zaimin Xia, Feng Jiang, Xudong Jiang, Zongyuan Ge, Yanwu Xu, Jing Qin, Siping Chen, Tianfu Wang, and Shuqiang Wang.  Skin lesion segmentation via generative adversarial networks with dual discriminators.  Medical Image Analysis, 64:101716, 2020.
  * [40] Di Lin, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang.  Multi-scale context intertwining for semantic segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
  * [41] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection, 2017.
  * [42] Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang, Zhongchao Shi, Jianping Fan, and Zhiqiang He.  A survey of visual transformers.  IEEE Transactions on Neural Networks and Learning Systems, pages 1–21, 2023.
  * [43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021.
  * [44] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [45] Teresa Mendonça, Pedro M Ferreira, Jorge S Marques, André RS Marcal, and Jorge Rozeira.  Ph 2-a dermoscopic image database for research and benchmarking.  In 2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC), pages 5437–5440. IEEE, 2013\.
  * [46] Yujian Mo, Yan Wu, Xinneng Yang, Feilin Liu, and Yujun Liao.  Review the state-of-the-art technologies of semantic segmentation based on deep learning.  Neurocomputing, 493, 01 2022.
  * [47] Md Mostafijur Rahman and Radu Marculescu.  Medical image segmentation via cascaded attention decoding.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6222–6231, 2023.
  * [48] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In International Conference on Medical image computing and computer-assisted intervention, pages 234–241. Springer, 2015.
  * [49] Holger R Roth, Le Lu, Amal Farag, Hoo-Chang Shin, Jiamin Liu, Evrim B Turkbey, and Ronald M Summers.  Deeporgan: Multi-level deep convolutional networks for automated pancreas segmentation.  In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part I 18, pages 556–564. Springer, 2015.
  * [50] Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert.  Attention gated networks: Learning to leverage salient regions in medical images.  Medical image analysis, 53:197–207, 2019.
  * [51] Abdelrahman Shaker, Muhammad Maaz, Hanoona Rasheed, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan.  Unetr++: delving into efficient and accurate 3d medical image segmentation.  arXiv preprint arXiv:2212.04497, 2022.
  * [52] Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and Huazhu Fu.  Transformers in medical imaging: A survey, 2022.
  * [53] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions, 2014.
  * [54] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxvit: Multi-axis vision transformer.  In European conference on computer vision, pages 459–479. Springer, 2022.
  * [55] Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, and Vishal M Patel.  Medical transformer: Gated axial-attention for medical image segmentation.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 36–46. Springer, 2021.
  * [56] Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang.  Anti-oversmoothing in deep vision transformers via the fourier domain analysis: From theory to practice.  In International Conference on Learning Representations, 2022.
  * [57] Wenxuan Wang, Chen Chen, Meng Ding, Hong Yu, Sen Zha, and Jiangyun Li.  Transbts: Multimodal brain tumor segmentation using transformer.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24, pages 109–119. Springer, 2021.
  * [58] Huisi Wu, Shihuai Chen, Guilian Chen, Wei Wang, Baiying Lei, and Zhenkun Wen.  Fat-net: Feature adaptive transformers for automated skin lesion segmentation.  Medical Image Analysis, 76:102327, 2022.
  * [59] Yixuan Wu, Kuanlun Liao, Jintai Chen, Jinhong Wang, Danny Z Chen, Honghao Gao, and Jian Wu.  D-former: A u-shaped dilated transformer for 3d medical image segmentation.  Neural Computing and Applications, pages 1–14, 2022.
  * [60] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  Advances in Neural Information Processing Systems, 34:12077–12090, 2021.
  * [61] Yutong Xie, Jianpeng Zhang, Chunhua Shen, and Yong Xia.  Cotr: Efficiently bridging cnn and transformer for 3d medical image segmentation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part III 24, pages 171–180. Springer, 2021.
  * [62] Guoping Xu, Xingrong Wu, Xuan Zhang, and Xinwei He.  Levit-unet: Make faster encoders with transformer for medical image segmentation.  arXiv preprint arXiv:2107.08623, 2021.
  * [63] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * [64] Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu.  nnformer: Interleaved transformer for volumetric segmentation.  arXiv preprint arXiv:2109.03201, 2021.

## Supplementary Material

The additional materials provided encompass an extended ablation study, which
serves to demonstrate the robustness and efficacy of our method in addressing
semantic segmentation tasks. Furthermore, we present supplementary
visualizations and discussions that accentuate the impactful role played by
the D-LKA module that we have proposed.

##  Appendix A Computationl complexity of the D-LKA

A comparison of the number of parameters for normal convolution and the
constructed convolution is shown in table 7. While the numbers of the standard
convolution explode for a larger number of channels, the parameters for
decomposed convolution are lower in general and do not increase as fast.
Deformable decomposed convolution adds a lot of parameters in comparison to
decomposed convolution but is still significantly smaller than standard
convolution. The main amount of parameters for deformable convolution is
created by the offset network. Here, we assumed a kernel size of (5,5) for the
deformable depth-wise convolution and (7,7) for the deformable depth-wise
dilated convolution. This results in the optimal number of parameters for a
large kernel of size $21\times 21$. A more efficient way to generate the
offsets would greatly reduce the number of parameters.

Table 7: The number of parameters for standard convolution and decomposed convolution. The kernel size is $21\times 21$. Adapted from [23]. # Channels | Std. Conv. | Decomp. Conv. | Deform. Decom. Conv. | Offset DDW-Conv. | Offset DDW-D Conv.
---|---|---|---|---|---
$C=32$ | $451,584$ | $3,392$ | $197,204$ | $40,050$ | $153,762$
$C=64$ | $1,806336$ | $8,832$ | $396,308$ | $80,050$ | $307,426$
$C=128$ | $7,225344$ | $25,856$ | $800,660$ | $160,050$ | $614,754$
$C=256$ | $28,901,376$ | $84,480$ | $1,633,940$ | $320,050$ | $1,229,410$
$C=512$ | $115,605,504$ | $300,032$ | $3,398,804$ | $640,050$ | $2,458,722$

It is worth noting that the introduction of the deformable LKA does indeed
introduce additional parameters and floating-point operations per second
(FLOPS) to the model. However, it’s important to emphasize that this increase
in computational load does not impact the overall inference speed of our
model. Instead, for batch sizes $>1$, we even observe a reduction in inference
time, shown in Figure 7. For instance, based on our extensive experiments, we
have observed that for a batch size of 16, the inference times with and
without deformable convolution are only 8.01ms and 17.38ms, respectively. We
argue that this is due to the efficient implementation of the deformable
convolution in 2D. To measure the times, a random input of size $(b\times
3\times 224\times 224)$ is used. The network is inferred 1000 times after a
GPU warm-up period of 50 iterations. The measurements are done on an NVIDIA
RTX 3090 GPU.

Figure 7: The inferences time in ms for an input image of size $3\times
224\times 224$ on the 2D methods. The times are already calculated for a
single image for better comparison.

##  Appendix B Performance vs Efficiency

To leverage the performance vs. parameter tradeoff, we visualize the
performances on the Synapse 2D dataset, reported in DSC and HD, and the memory
consumption based on the number of parameters in Figure 8.

Figure 8: Performance vs memory chart to compare the performance of SOTA
approaches, including ScaleFormer [31], MISSFormer [33], SwinUnet [11],
DAEFormer [4], with our the proposed 2D D-LKA Net on Synapse dataset. DSC, HD,
and Memory values are normalized using min-max normalization for improved
visibility and comparability.

The D-LKA Net induces a rather large amount of parameters with approximately
101M. This is less than the second best-performing method, the ScaleFormer
[31], which used 111.6M parameters. Compared to the more light-weight
DAEFormer [4] model, we, however, achieve a better performance justifying the
parameter increase. The majority of the parameters are from the MaxViT
encoder; thus, replacing the encoder with a more efficient one can reduce the
model parameters. It’s also worth noting that in this visualization, we
initially normalized both the HD and memory values within the [0, 100] range.
Subsequently, we scaled them down from 100 to enhance the representation of
higher values.

##  Appendix C Qualitative results on the Synapse dataset

To further visualize our model capability, we provide a different perspective
of the 3D organ segmentation of the Synapse dataset in Figure 9 and Figure 10.
We neglect the visualization of the liver and the stomach so partly occluded
organs get a better visibility.

Figure 9: Additional qualitative results on the Synapse Dataset. Liver and
Stomach are not shown for improved visibility of smaller occluded organs.
Figure 10: Additional qualitative results on the Synapse Dataset.

To gain a better understanding of the limitations associated with the 2D
approach, it is advisable to expand our perspective into the 3D domain. As
illustrated in Figure 11, we can observe inconsistencies among the slices.
These discrepancies can be attributed to the absence of information exchange
between neighboring slices in a 2D network. In contrast, our 3D network
successfully mitigates these limitations.

Figure 11: Additional qualitative results of the 2D D-LKA-Former on the
Synapse dataset, visualized in 3D. The comparison to the 3D D-LKA Net is
shown. Here, it is visible that the 3D version creates less noise due to the
inter-slice dependencies.

##  Appendix D Limitations on the Skin dataset

Figure 12 shows a qualitative visualization of ISIC 2018 samples, where our
approach fails. However, it is also visible that the segmentation is either
noisy or quite primitive. Since this is also present in the training data,
this could hinder the network from learning accurate segmentations.

Figure 12: Additional qualitative results of the 2D D-LKA-Former on the ISIC
2018 dataset.

##  Appendix E Robustness Visualization

In line with the ablation study presented in the main paper, we conducted a
thorough evaluation of various methods on the Synapse 2D dataset. To ensure
the robustness of our findings, we executed each model five times and reported
their statistical significance. This detailed analysis is visually represented
in Figure 13.

Figure 13: Statistical evaluation of single organ performance of Synapse
dataset comparing state-of-the-art methods, including TransUnet [15],
MISSFormer [33], SwinUnet [11], DAEFormer [4], with our the proposed 2D D-LKA
Net. Visualized are the results on the Synapse dataset with the performance
reported in DSC.