# Optimizing Universal Lesion Segmentation: State Space Model-Guided
Hierarchical Networks with Feature Importance Adjustment

Kazi Shahriar Sanjid, Md. Tanzim Hossain, Md. Shakib Shahariar Junayed and M.
Monir Uddin This work was funded by the North South University Conference
Travel and Research Grants (NSU-CTRG) under Grant No. CTRG-22-SEPS-06The first
three authors contributed equally to this work. Asterisk indicates
corresponding authorKazi Shahriar Sanjid, Md. Tanzim Hossain and Md. Shakib
Shahariar Junayed are with the Department of Electrical & Computer
Engineering, North South University, Dhaka-1229, Bangladesh (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS>shakib.junayed@northsouth.edu)∗M. Monir Uddin is with the Department of
Mathematics & Physics, North South University, Dhaka-1229, Bangladesh (e-mail:
<EMAIL_ADDRESS>0009-0001-6845-8881 0000-0002-4776-7220
0009-0009-8755-6370 0000-0002-9817-6156

###### Abstract

Deep learning has revolutionized medical imaging by providing innovative
solutions to complex healthcare challenges. Traditional models often struggle
to dynamically adjust feature importance, resulting in suboptimal
representation, particularly in tasks like semantic segmentation crucial for
accurate structure delineation. Moreover, their static nature incurs high
computational costs. To tackle these issues, we introduce Mamba-Ahnet, a novel
integration of State Space Model (SSM) and Advanced Hierarchical Network
(AHNet) within the MAMBA framework, specifically tailored for semantic
segmentation in medical imaging.Mamba-Ahnet combines SSM’s feature extraction
and comprehension with AHNet’s attention mechanisms and image reconstruction,
aiming to enhance segmentation accuracy and robustness. By dissecting images
into patches and refining feature comprehension through self-attention
mechanisms, the approach significantly improves feature resolution.
Integration of AHNet into the MAMBA framework further enhances segmentation
performance by selectively amplifying informative regions and facilitating the
learning of rich hierarchical representations. Evaluation on the Universal
Lesion Segmentation dataset demonstrates superior performance compared to
state-of-the-art techniques, with notable metrics such as a Dice similarity
coefficient of approximately 98% and an Intersection over Union of about 83%.
These results underscore the potential of our methodology to enhance
diagnostic accuracy, treatment planning, and ultimately, patient outcomes in
clinical practice. By addressing the limitations of traditional models and
leveraging the power of deep learning, our approach represents a significant
step forward in advancing medical imaging technology.

###### Index Terms:

mamba, lesion segmentation, magnetic resonance imaging, semantic segmentation,
deep learning

##  I Introduction

The total number of CT exams performed each year has been rising recently [1],
which has increased radiologists’ workloads [2]. Oncological radiology is
anticipated to play a significant role in the anticipated increase in workload
in the future, given the predicted 47% rise in the global cancer burden from
2020 to 2040 [3]. This is because patients with cancer frequently undergo
multiple imaging exams over a longer period to track the progression of their
disease.

Deeplesion | Pancreas | Bone
---|---|---
Image | Mask | Image | Mask | Image | Mask
|  |  |  |  |
|  |  |  |  |
Figure 1: Example of cross-sectional MRI pictures and corresponding labels
ranging from normal sizes to those occupying only a few pixels.

Assessing disease advancement and evaluating treatment efficacy in
longitudinal CT scans frequently necessitate manual measurements of lesions
[4] along either the long or short axes. Typically, these measurements adhere
to the Response Evaluation Criteria In Solid Tumors (RECIST) guidelines [5]
were formulated to streamline and standardize this procedure. These guidelines
prescribe a cap on the number of lesions to be measured, restricting it to a
maximum of five ”target lesions” across various organs or structures through
which the overall response is estimated.

To alleviate the time constraints associated with the automated segmentation
of lesions from CT scans holds significant importance across various computer-
aided diagnosis (CAD) endeavors, including pathology identification [6], tumor
progression tracking [7] and quantitative analysis of disease advancement [8].
In the case of annotating lesions in oncological scans, automatic segmentation
models offer a solution by extracting information with minimal input from a
radiologist. Considerable efforts have been dedicated to the advancement of
automated lesion segmentation techniques to alleviate the workload of
radiologists and enhance diagnostic precision [9, 10, 11, 12, 13, 14]. This
guidance may entail a single click within the lesion by a radiologist or
utilizing a bounding box prediction from a detection model. Employing
segmentation of the lesion volume yields supplementary data that can be
exploited to calculate more comprehensive lesion volumes or characteristics.
Additionally, registration algorithms can facilitate the propagation of
segmented lesions [15], leading to substantial time efficiencies during
follow-up examinations.

In recent times, significant advancements have been achieved in the realm of
AI owing to the rapid evolution of convolutional neural networks (CNNs),
particularly fully convolutional networks (FCNs) [16]. Significant progress
has been achieved in automatic segmentation models tailored for tumors.
Specifically, medical challenges centered around liver, kidney, or lung tumors
have witnessed substantial enhancements in performance concerning lesion
segmentation. Nevertheless, in clinical settings, a pressing demand exists for
versatile and robust models capable of swiftly segmenting the myriad types of
lesions found within the thorax-abdomen region.

The efficacy of semantic segmentation models [17, 18, 19, 20], underscores the
pivotal role of meticulously annotated training datasets, such as Cityscapes
[21] and ADE20K [22], in driving performance enhancements. However, in
contrast to natural images, acquiring medical images poses significant
challenges due to their high privacy and confidentiality concerns.
Furthermore, annotating medical images is labor-intensive and costly and
demands substantial clinical expertise, resulting in a scarcity of publicly
available data for segmentation tasks. Consequently, the primary obstacle in
training a precise lesion segmentation model is the absence of a comprehensive
dataset.

Moreover, aside from the limited scale of available data, another issue arises
from the specificity of existing medical datasets [23, 24], which typically
focus on particular lesion types such as liver, and kidney tumors [23], breast
masses [24, 25], or lung nodules [26]. This limitation poses a challenge in
developing a comprehensive lesion segmentation framework. Consequently,
prevailing lesion segmentation models often concentrate on segmenting a single
lesion type from its corresponding anatomical region. However, in reality,
various types of lesions exhibit interconnections. For instance, metastases
can disseminate to different body parts [27, 28, 29] via the lymphatic system
or bloodstream.

To address the above problems, we use The Universal Lesion Segmentation ’23
Challenge dataset. The dataset consists of 6,514 fully 3D annotated lesions of
multiple body parts. Although prior efforts have been devoted to ULS [10, 30,
31] development, most research in this domain has heavily relied upon a single
partially annotated dataset [32], which solely encompasses the long-and short-
axis diameters on a solitary axial slice.

Lesions of various types typically manifest diverse characteristics in size,
shape, and appearance. For instance, depicted in Figure 1 are lesions ranging
from normal sizes to those occupying only a few pixels. Additionally, a
multitude of shapes and appearances of lesions are observable. Consequently,
employing existing segmentation techniques directly [33, 34, 35] for universal
lesion segmentation is deemed suboptimal. Numerous initiatives have emerged to
automate the process of measuring lesion size. Notably, deep convolutional
neural networks have been effectively deployed to delineate tumors within
various anatomical regions such as the brain [36], lungs [37, 38], pancreas
[39], liver [28, 40] and enlarged lymph nodes [41, 42]. While many of these
methodologies are tailored to specific lesion types, the ideal lesion size
measurement tool should be able to manage a diverse array of lesions
encountered in clinical practice.

Recently, several deep learning (DL) algorithms aimed at universal lesion
detection (ULD) or segmentation (ULS) in CT images have demonstrated notable
success in aiding the diagnosis of various medical conditions [4, 43, 44, 45].
These investigations have revealed that ULS algorithms, drawing from diverse
datasets encompassing various organs, exhibit enhanced efficacy and
scalability compared to those tailored for singular organ lesion segmentation.
Furthermore, in response to the challenges posed by the laborious and costly
nature of manual image annotations in medical contexts, ULS algorithms trained
on diverse ultrasound (US) image datasets spanning multiple organs have shown
promise in enhancing the quality of annotations in novel datasets through
transfer learning. This approach effectively mitigates the burdens associated
with the demanding requirements for both quantity and quality of medical image
annotations.

To address the problems discussed earlier and to design a better universal
lesion segmentation model, we propose a unique methodology that merges to
address semantic universal lesion segmentation tasks. The methodology
incorporates the Advanced Hierarchical Network (AHNet) model into the
selective state space model (MAMBA) framework [46], inspired by the
integration of HUNet into MAMBA as described by [47], incorporating attention
mechanisms and residual blocks to enhance semantic segmentation performance.
Firstly, the AHNet model, a customized UNet architecture is seamlessly
integrated into the MAMBA architecture, extending its capabilities in medical
image segmentation. The integration process between the state space model
Mamba and the AHNet architecture occurred by incorporating Mamba blocks
alongside attention gates and residual blocks within the AHNet model’s
upsampling pathway.

To conduct a thorough comparative analysis between our methodology and
previous approaches, we have constructed an exhaustive benchmark encompassing
established medical and semantic image segmentation techniques. Our evaluation
employs four widely recognized metrics in medical image segmentation. The
experimental findings affirm that our approach exhibits superior performance
compared to prior state-of-the-art methods, thereby establishing itself as a
robust benchmark for future research endeavors. This comprehensive benchmark
serves as a valuable resource for future investigations and underscores our
contributions. We summarize our contributions as follows:

  * •

HUNet in [48] is enhanced by AHNet with upsampling attention and residual
blocks, resulting in improved segmentation accuracy in medical imaging and
computer vision tasks.

  * •

The attention-enhanced upsampling block, derived from Hunet, is introduced in
AHNet, enabling dynamic feature importance adjustment and boosting semantic
segmentation accuracy, surpassing standard upsampling blocks in segmentation
tasks with enhanced feature representation and semantic information
propagation.

  * •

The ability of AHNet to capture intricate patterns is further enhanced by
integrating Mamba blocks, leading to enhanced segmentation performance in
biomedical image analysis.

  * •

Input features are selectively emphasized by upsampling attention gates,
enhancing reconstruction fidelity in tasks such as image segmentation.

  * •

Fine details are preserved during reconstruction by upsampling residual
blocks, contributing to enhanced performance in image segmentation tasks.

  * •

The superior performance of Mamba-AHNet with image reconstruction is
highlighted by achieving the best result on the ULS23 dataset. This outcome is
further clarified and supported in Section VII-A.

  * •

Segmentation accuracy is improved by integrating upsampling image
reconstruction with upsampling attention mechanisms, emphasizing relevant
image regions.

  * •

Developed a user-friendly website with API integration that facilitates
efficient segmentation using the Mamba-AHNet with image reconstruction,
ensuring quick analysis turnaround times in clinical settings for improved
diagnosis and treatment planning.

The rest of the paper is structured into several sections and subsections to
provide a clear presentation of the research. It begins with an examination of
related works (Section II), followed by an exploration of preliminaries
(Section III) and a detailed explanation of the methodology used (Section IV).
The experiments conducted are outlined in Section V, with specific attention
given to the dataset (Subsection V-A), data preprocessing (Subsection V-B),
and evaluation metrics (Subsection V-C). Computational aspects, including
computation time, are discussed in Section VI. The numerical analysis of
results is presented in Section VII, with subsections dedicated to presenting
the outcomes (Subsection VII-A) and discussing their implications (Subsection
VII-B). The paper concludes with a summary of findings in Section VIII,
followed by acknowledgment of each author’s contributions in Section Authors
Contribution.

##  II Related Works

Recently, CNNs, the Vision-language model, and Attention mechanisms have been
used for medical image segmentation. Most of these methodologies are tailored
to specific lesion types.

Semantic segmentation can be understood as classifying each pixel in an image,
akin to pixel-level image classification. Consequently, numerous image
classification networks have been extended to facilitate semantic segmentation
[16, 49, 50, 51]. Notably, FCN is widely recognized as the initial end-to-end
pixel-to-pixel network for semantic segmentation. Among these approaches,
U-Net stands out as a seminal contribution to medical image segmentation [34].
Building upon this foundation, UNet++ [52] has enhanced the skip connection
mechanism of U-Net. Nonetheless, many of these methods exhibit sensitivity to
the data’s quantity and quality, thereby constraining their generalization
performance. To address this limitation, several approaches have investigated
the utility of semi-supervised learning across diverse domains [53, 54, 55].
Moreover, the challenge posed by insufficient data and annotations can be
alleviated by incorporating multiple modalities into the learning models.

CLIP [56] stands as a groundbreaking venture into the realm of large-scale
Vision-Language Pretraining (VLP) models. It employs contrastive learning to
imbibe image representations from a vast dataset comprising 400 million pairs
of images and corresponding text descriptions. To streamline visual input
processing, ViLT [57] capitalizes on interaction layers to handle visual
features without necessitating a separate deep visual embedder. Consequently,
many studies have emerged, focusing on enhancing image segmentation by
integrating textual information. For instance, Ding et al. [58] devised a
Vision-Language Transformer (VLT) framework tailored for referring
segmentation tasks. This framework fosters deep interactions among multimodal
data streams, harmonizing linguistic and visual features. In contrast, the
Language-Aware Vision Transformer (LAVT) framework [59] employs an early
fusion approach, leveraging a pixel-word attention mechanism to blend
linguistic and visual features seamlessly. While the influence of VLP extends
to medical image analysis [60, 61, 62], the unique characteristics of medical
images pose distinct challenges. Unlike natural images, medical images often
exhibit blurred boundaries between different regions, rendering precise
segmentation formidable. Consequently, the direct application of vision-
language models to medical image analysis proves impractical. VLT and LAVT,
tailored for reference segmentation in natural images, rely on explicit
alignment between image and text, a feat difficult to achieve in the context
of medical images due to inherent disparities. Similarly, ViLT, geared towards
multimodal tasks like Visual Question Answering (VQA), may not seamlessly
translate to the nuances of medical image analysis.

Beginning with the advent of attention mechanisms in computer vision as
evidenced by RAN [63], researchers have made strides in integrating them into
this domain. Authors in [64] introduced the widely recognized Convolutional
Block Attention Module (CBAM), which employs spatial and channel attention to
refine features dynamically. Furthermore, alongside traditional attention
mechanisms [65], self-attention mechanisms [66] have made their foray into
computer vision. Nevertheless, since self-attention was initially tailored for
addressing Natural Language Processing (NLP) tasks, it confronts several
hurdles, notably elevated computational overhead and the oversight of local
image features.

[43] propose a novel model integrating segmentation assistance alongside
conventional detection, employing a mutual-mining strategy between the two
branches to jointly identify lesions and mitigate adverse effects through
selective loss and exclusion of suspicious lesions from fine-tuning. [4]
present an end-to-end framework structured around an enhanced Mask RCNN,
incorporating detection, tagging, and segmentation branches. Additionally,
[67] develop a universal lesion detection algorithm, starting with a framework
that learns from diverse lesion datasets concurrently and utilizes proposal
fusion, followed by methodologies to extract absent annotations from datasets
with incomplete labeling through clinical prior knowledge and knowledge
transfer across datasets. These advancements collectively contribute to more
effective lesion detection and segmentation in medical imaging.

##  III Preliminaries

###  III-A State Space Models

The SSMs are a class of systems that relate a one-dimensional function or
sequence $u(t)\longmapsto y(t)\in\mathcal{R}$ and can be expressed as a linear
time-invariant (LTI) continuous-time system of the form 1

$\displaystyle\dot{x}(t)$ $\displaystyle=\textbf{A}x(t)+\textbf{B}u(t),$ (1)
$\displaystyle y(t)$ $\displaystyle=\textbf{C}x(t)+\textbf{D}u(t),$

where $\textbf{A}\in\mathbb{R}^{n\times n}$ and
$\textbf{B}\in\mathbb{R}^{n\times 1}$ and $\textbf{C}\in\mathbb{R}^{1\times
n}$, are representing the system, input and output matrices respectively, of
the system. The vectors $x(t)$, $u(t)$, and $y(t)$, respectively, are known as
the state, input, and output of the system. These models possess advantageous
features like linear computational complexity per time step and parallelized
computation, facilitating efficient training. Conventional SSMs require more
memory than equivalent convolutional neural networks (CNNs). They frequently
face challenges such as vanishing gradients during training, limiting their
broader utility in general sequence modeling.

The S4 [68] enhances conventional SSMs by introducing structured patterns in
the state matrix $A$ and employing a proficient algorithm. In particular, the
state matrix is constructed and initialized using the High-Order Polynomial
Projection Operator (HIPPO) [69], creating deep sequence models with enhanced
capabilities for efficient long-range reasoning. S4, as an innovative network
architecture, has outperformed Transformers [66] by a considerable margin in
the demanding Long Range Arena Benchmark [70]. Mamba introduces notable
advancements in discrete data modeling using SSMs, particularly in domains
such as text and genomics. Two primary enhancements distinguish Mamba from
traditional SSMs approaches. Firstly, it incorporates an input-dependent
selection mechanism, unlike conventional SSMs, invariant to time and input,
enabling efficient information extraction tailored to specific inputs. This is
achieved by parameterizing the SSMs parameters based on the input data.
Secondly, Mamba introduces a hardware-aware algorithm that scales linearly
with sequence length, facilitating recurrent model computation through
scanning, thereby outperforming previous methods on modern hardware. The Mamba
architecture integrates SSM blocks with linear layers, is notably simpler, and
has exhibited state-of-the-art performance across various long-sequence
domains such as language and genomics. This underscores its significant
computational efficiency in both the training and inference stages.

###  III-B Discrete-time SSM: The Recurrent Representation

In discretizing 1, the granularity of the input is indicated by the step size
$\Delta$. This is required when working with a discrete input sequence
$(u_{0},u_{1},\ldots)$ rather than a continuous function $u(t)$. As
$u_{k}=u(k\Delta)$, the inputs are essentially samples of an underlying
continuous signal $u(t)$.

As demonstrated in earlier research, including [71], we use the bilinear
approach to discretize the continuous-time State Space Model (SSM). The state
matrix A is converted into an approximation $\mathbf{\overline{A}}$ using this
procedure. The discrete SSM is

$\displaystyle\mathbf{\overline{A}}$
$\displaystyle=(\mathbf{\overline{I}}-\Delta/2.\mathbf{\overline{A}})^{-1}(\mathbf{\overline{I}}+\Delta/2.\mathbf{\overline{A}})$
(2) $\displaystyle\mathbf{\overline{B}}$
$\displaystyle=(\mathbf{\overline{I}}-\Delta/2.\mathbf{\overline{A}})^{-1}\Delta\mathbf{\overline{B}}$
$\displaystyle\mathbf{\overline{C}}$ $\displaystyle=\mathbf{\overline{C}}$
$\displaystyle y_{k}$ $\displaystyle=\mathbf{\overline{C}}x_{k}$
$\displaystyle x_{k}$
$\displaystyle=\mathbf{\overline{A}}x_{k}+\mathbf{\overline{B}}u_{k}$

Rather than treating $u_{k}$ in 2 as a function that maps from sequence-to-
sequence, we now consider it as a sequence-to-sequence mapping to $y_{k}$. The
discrete State-Space Model (SSM) may now be computed similarly to a Recurrent
Neural Network (RNN) because the state equation also recurs in $x_{k}$. To be
more precise, a hidden state $x_{k}\in\mathbb{R}^{N}$ has a transition matrix
$\mathbf{\overline{A}}$.

In this study, the discretized SSM matrices are denoted as
$\mathbf{\overline{A}},\mathbf{\overline{B}},\ldots$, which are found using 2.
It is important to remember that these matrices depend on a step size $\Delta$
in addition to A. When this reliance is acknowledged, we leave it out of our
notation explicitly for the sake of concision.

Figure 2: An illustration of AHNet architecture

###  III-C Training SSMs: The Convolutional Representation

Since the recurrent SSM is sequential, training on modern hardware presents
practical issues 2. Yet, a well-established connection has been identified
between continuous convolutions and linear time-invariant (LTI) SSMs such as
1. This allows for the rewriting of 2 as a discrete convolution.

To simplify, assume the initial state is $x_{-1}=0$. Then unrolling 2
explicitly yields

$\displaystyle x_{0}$ $\displaystyle=\mathbf{\overline{B}}u_{0}$
$\displaystyle x_{1}$
$\displaystyle=\mathbf{\overline{AB}}u_{0}+\mathbf{\overline{B}}u_{1}$
$\displaystyle x_{2}$
$\displaystyle=\mathbf{\overline{A}}^{2}\mathbf{\overline{B}}u_{0}+\mathbf{\overline{AB}}u_{1}+\mathbf{\overline{B}}u_{2}$
$\displaystyle\vdots$ $\displaystyle y_{0}$
$\displaystyle=\mathbf{\overline{CB}}u_{0}$ $\displaystyle y_{1}$
$\displaystyle=\mathbf{\overline{CAB}}u_{0}+\mathbf{\overline{CB}}u_{1}$
$\displaystyle y_{2}$
$\displaystyle=\mathbf{\overline{CA}}^{2}\mathbf{\overline{B}}u_{0}+\mathbf{\overline{CAB}}u_{1}+\mathbf{\overline{CB}}u_{2}$
$\displaystyle\vdots$

This can be vectorized into a convolution 7 with an explicit formula for the
convolution kernel 4.

$\displaystyle y_{k}$
$\displaystyle=\mathbf{\overline{CA}}^{k}\mathbf{\overline{B}}u_{0}+\mathbf{\overline{CA}}^{k-1}\mathbf{\overline{B}}u_{1}+\ldots+\mathbf{\overline{CAB}}u_{k-1}+\mathbf{\overline{CB}}u_{k},$
(3) $\displaystyle y$ $\displaystyle=\mathbf{\overline{K}}*u.$

$\displaystyle\mathbf{\overline{K}}\in\mathbb{R}^{L}$
$\displaystyle:=\mathcal{K}_{L}(\mathbf{\overline{A}},\mathbf{\overline{B}},\mathbf{\overline{C}})$
(4)
$\displaystyle:=(\mathbf{\overline{CA}}^{i}\mathbf{\overline{B}})_{i\in[L]}$
$\displaystyle=(\mathbf{\overline{CB}},\mathbf{\overline{CAB}}\ldots,\mathbf{\overline{CA}}^{L-1}\mathbf{\overline{B}}).$

Put otherwise, given $\mathbf{\overline{K}}$, 7 indicates a solitary (non-
circular) convolution that can be computed well with FFTs. Nonetheless,
$\mathbf{\overline{K}}$ determination in 4 is intricate and serves as the
foundation for our technological advances in Section IV. We call the SSM
convolution kernel or filter $\mathbf{\overline{K}}$.

###  III-D Vision Mamba Layer (VM Layer):

The original SSM block is improved by the VM Layer for the extraction of deep
semantic features from images. In particular, it nearly eliminates the need
for new parameter introduction and computing complexity by leveraging
sophisticated residual connections and adjustment factors to improve the long-
range spatial modeling capabilities of SSM significantly. Given the input deep
feature $M_{in}^{l}\in\mathbb{R}^{L\times C}$, as shown in Figure 3(a), the VM
Layer first uses LayerNorm and then the Vision State Space Model (VSSM) to
capture spatial long-range dependencies. For improved performance, it then
uses an adjustment factor $s\in\mathbb{R}^{C}$ in the residual connection
[72]. This process can be represented mathematically as follows:

$\widetilde{M^{l}}=VSSM(L_{norm}(M_{in}^{l}))+s.M_{in}^{l}$ (5)

where $L_{norm}$ is the Layer Norm. The RVM Layer then uses a second LayerNorm
to normalize $\widetilde{M^{l}}$. Finally, it uses a projection layer to
transform $\widetilde{M^{l}}$ into a more profound feature. The above process
can be formulated as:

$M_{out}^{l}=Projection(L_{norm}(\widetilde{M^{l}}))$ (6)

###  III-E Vision State-Space Module (VSS Module):

For long-range spatial modeling, 3(b) is comparable to the methodology
described in [73]. The feature $W_{in}^{l}\in\mathbb{R}^{L\times C}$ is fed
into two parallel branches by the VSS Module. The VSS Module uses a linear
layer in the first branch to expand the feature channels to $\lambda\times C$,
where $\lambda$ is a pre-defined channel expansion factor. It then applies the
SSM and LayerNorm, then a DWConv and SiLU activation function [74]. The SiLU
activation function is applied after the VSS Module uses a linear layer to
increase the feature channels $\lambda\times C$ in the second branch. The
output $W_{out}$ has the same form as the input $W_{in}$ because the VSS
Module then uses the Hadamard product to combine the features from the two
branches and projects the channel number back to $C$. The above process can be
formulated as:

$\displaystyle W_{1}$
$\displaystyle=L_{norm}(SSM(SiLU(DWConv(Linear(W_{in})))))$ (7) $\displaystyle
W_{2}$ $\displaystyle=SiLU(Linear(W_{in}))$ $\displaystyle W_{out}$
$\displaystyle=Linear(W_{1}\odot W_{2})$

##  IV Methodology

Figure 3: An illustration of Mamba-AHNet with image reconstruction
architecture

Our methodology merges the innovative SSM with the CNN-based architecture,
integrated into the MAMBA framework to address semantic segmentation tasks.
The VSSM methodology begins with dissecting images into patches, then
progressing through VSSLayer encoder and decoder modules. Within these layers,
the SS2D mechanism orchestrates self-attention, complemented by residual
connections and stochastic depth regularization, refining the model’s
comprehension. This iterative process of compression and expansion enhances
the feature resolution significantly.

Simultaneously, the methodology incorporates the AHNet model into the MAMBA
framework, incorporating attention mechanisms and residual blocks to enhance
semantic segmentation performance. Firstly, the AHNet model, a customized
HUNet architecture [48], is seamlessly integrated into the MAMBA architecture,
extending its capabilities in medical image segmentation.

The integration involves adapting the AHNet model’s architecture to fit the
MAMBA framework while preserving unique features such as attention mechanisms
and residual blocks. Attention mechanisms selectively amplify informative
regions in feature maps by computing attention weights based on the
interaction between feature maps. This mechanism enhances the model’s ability
to focus on salient features while suppressing irrelevant information,
improving segmentation accuracy. Additionally, residual blocks play a crucial
role in feature extraction and propagation. By introducing skip connections
and residual learning, these blocks facilitate the learning of rich
hierarchical representations, enabling effective feature propagation across
different scales and depths within the network architecture. This adaptive
adjustment further enhances the model’s capacity to capture and propagate
features, improving segmentation performance. Combining the AHNet model with
attention mechanisms and residual blocks within the MAMBA framework enhances
semantic segmentation accuracy. It contributes to the model’s
interpretability, robustness, and scalability across diverse medical imaging
datasets and segmentation tasks.

###  IV-A Ahnet:

The AHNet model in Figure 2 introduces several innovative features to enhance
feature importance adjustment and learning capabilities. Feature Importance
Adjustment in the context of AHNet refers to the dynamic modulation of the
relevance or importance of different features extracted from medical images
during the segmentation process. Firstly, it incorporates upsampling attention
blocks, which dynamically adjust the importance of features at different
spatial locations. This attention mechanism is achieved by computing the
attention map through convolutional operations between the input and
corresponding features from the contracting path. This allows the model to
focus on relevant features while discarding irrelevant ones, enhancing feature
representation. The model uses an upsample block, which concatenates the
upsampled feature maps with the feature maps from the contracting path to
provide fine-grained details and spatial context. Additionally, the model
utilizes upsampling residual blocks to facilitate learning complex feature
representations through residual connections and multiple convolutional
layers. These residual blocks improve gradient flow and training stability and
enable the model to capture intricate patterns and details in the data.
Furthermore, the model employs standard convolutional and pooling layers for
downsampling followed by transposed convolutional layers for upsampling,
ensuring efficient feature extraction and reconstruction. The AHNet model
improves upon the traditional UNet architecture by addressing its limitations.
While UNet has been widely used and proven effective for semantic segmentation
tasks, it lacks explicit mechanisms to focus on relevant features and capture
intricate details in the data. Traditional UNet architectures rely solely on
skip connections and transposed convolutions for upsampling. This may lead to
the loss of spatial information and inadequate feature representation,
especially in regions with complex structures or subtle variations.
Additionally, UNet does not incorporate attention mechanisms to dynamically
adjust feature importance based on spatial context, potentially resulting in
suboptimal segmentation performance, particularly in scenarios with
significant variances in object sizes or shapes. By integrating upsampling
attention blocks and residual blocks, the AHNet model overcomes these
limitations by enhancing feature extraction, capturing fine details, and
improving segmentation accuracy, making it a more robust and versatile
architecture for various medical imaging and computer vision tasks.

###  IV-B Intregating Mamba with Ahnet:

The integration process between the state space model Mamba and the AHNet
architecture occurred by incorporating Mamba blocks alongside attention gates
and residual blocks within the AHNet model’s upsampling pathway. Initially,
the transformation parameters of the SSM were manipulated to ensure efficient
computation and modeling of long-range dependencies. These matrices underwent
discretization to convert continuous parameters to discrete ones, facilitating
resolution invariance and proper model normalization. Subsequently, within the
AHNet architecture, the upsampling blocks were augmented with Mamba blocks,
attention gates, and residual blocks to enhance the model’s ability to capture
intricate patterns and relationships within the feature maps. This integration
process enabled the AHNet model to effectively recover spatial details while
simultaneously capturing complex patterns and relationships across different
spatial scales, leading to improved segmentation performance in biomedical
image analysis tasks. The architecture is shown in Figure 3.

Input: Input tensor $x$, Skip connection tensor $t$, Number of filters $f$,
Kernel size $k$, Padding method $p$ and Strides $s$.

Output: Tensor $x_{up}$.

1

2Upsample the input tensor $x$ using transpose convolution with $f$, $k$, $p$,
and $s$

3 Apply batch normalization to the upsampled tensor $x_{up}$

4 Apply ReLU activation function element-wise to the normalized tensor
$x_{up}$

5 Compute an attention map $A(x,g,d)$ by applying an attention mechanism to
$x_{up}$, the skip connection tensor $t$, and $\frac{filters}{2}$

6 Concatenate the upsampled tensor $x_{up}$ with the skip connection tensor

7 Apply a convolutional layer to the concatenated tensor, using $f$, $k$, and
$p$, and ReLU activation

8 Apply batch normalization to the resulting tensor

9 Apply ReLU activation function element-wise to the normalized tensor

10 Return the resulting tensor $x_{up}$ as the output of the upsampling block

Algorithm 1 Attention-Enhanced Upsampling Block

Input: Input tensor $x$, Number of filters $f$, Kernel size $k$, Padding
method $p$ and Strides $s$.

Output: Residual tensor $r$.

1 Upsample the input tensor $x$ using transpose convolution with $f$, $k$,
$p$, and $s$

2 Apply batch normalization to the upsampled tensor $x$

3 Apply ReLU activation function element-wise to the normalized tensor $x$

4 Compute an attention map $A(x,g)$ by applying an attention mechanism to $x$,
the skip connection tensor, and $\frac{f}{2}$

5 Concatenate the upsampled tensor $x$ with the skip connection tensor

6 Apply a convolutional layer to the concatenated tensor, using $f$, $k$, and
$p$, and ReLU activation

7 Apply batch normalization to the resulting tensor

8 Apply ReLU activation function element-wise to the normalized tensor

9 Return the resulting tensor as the output of the upsampling block

Algorithm 2 Upsampling Residual Block Function

Input: Input tensors $x$, $g$, Number of channels $c$, Attended feature tensor
$\varrho$.

Output: Output tensor $att_{x}$.

1

2Calculate Shape of $x$ and $g$

3 Calculate Offset for Height

4 Calculate Offset for Width

5 Crop $g$ to Match $x$ Shape as $g_{cropped}$

6 Theta Transformation:
$\theta_{x}\leftarrow\text{Conv2D}(x,c,(1,1),(1,1),\text{padding='same'})$

7 Batch Normalize $\theta_{x}$

8 Phi Transformation:

9
$\phi_{g}\leftarrow\text{Conv2D}(g_{cropped},c,(1,1),(1,1),\text{padding='same'})$

10 Batch Normalize $\phi_{g}$

11 Calculate Attention Map:

12 $f\leftarrow\text{ReLU}(\theta_{x}+\phi_{g})$
$\psi_{f}\leftarrow\text{Conv2D}(f,1,(1,1),(1,1),\text{padding='same'})$

13 Batch Normalize $\psi_{f}$

14 $rate\leftarrow\text{Sigmoid}(\psi_{f})$

15 Apply Attention to Input Features:

16 $att_{x}\leftarrow x\odot rate$

return Attended Features $att_{x}$

Algorithm 3 Upsampling Attention Gate

###  IV-C Attention-Enhanced Upsampling Block

The attention-enhanced upsampling block described in Algorithm 1 is a
modification from the upsampling block in HUNet, integrates attention
mechanisms to bolster feature selection during the expansive path of the
network. This modification becomes imperative for image segmentation tasks
because it can selectively focus on relevant spatial locations from the
contracting path. By dynamically adjusting the importance of different
features based on their contextual relevance, the attention mechanism enables
the model to capture intricate details crucial for accurate segmentation.
Additionally, integrating attention mechanisms enhances the semantic
information flow between different resolutions, facilitating the propagation
of essential features throughout the network. Overall, this modification aims
to improve feature representation and selection capabilities, thereby
enhancing the segmentation performance of the model and making it more adept
at handling various complexities inherent in image segmentation tasks.

The attention-enhanced upsampling block significantly improves HUnet’s
upsampling block by incorporating an attention mechanism. This mechanism
allows the model to selectively focus on relevant spatial locations from the
contracting path, dynamically adjusting the importance of different features
based on their contextual relevance. The upsampling block facilitates enhanced
feature selection by integrating attention, ensuring the model captures
intricate details crucial for accurate segmentation. Furthermore, the
attention mechanism enhances the semantic information flow between different
resolutions, enabling better propagation of essential features throughout the
network. This improves feature representation and selection capabilities,
leading to superior segmentation performance. In contrast, the standard
upsampling block cannot selectively attend to informative regions, potentially
limiting its ability to capture fine details and contextual information
essential for accurate segmentation. Thus, the attention-enhanced upsampling
block represents a significant advancement in semantic segmentation
architectures, offering improved performance and robustness in handling
various complexities inherent in image segmentation tasks.

###  IV-D Upsampling Residual Block Function:

The upsampling residual block function described in Algorithm 2 is designed to
be employed in the decoder (upsampling) portion of a neural network
architecture where it plays a critical role in reconstructing high-resolution
feature maps. This block includes convolutional layers followed by batch
normalization, activations, residual connections, upsampling operations, and
concatenation with skip connections from the encoder. Integrating skip
connections effectively leverages the rich spatial information captured in the
encoder, allowing for the recovery of fine details lost during downsampling.
The residual connections aid in the propagation of features and gradients,
fostering stable and efficient training. Consequently, the upsampling residual
block contributes to the preservation of spatial details and the overall
performance enhancement in tasks such as image segmentation or image
generation.

In essence, the upsampling residual block outperforms a typical residual block
in scenarios where the reconstruction of high-resolution feature maps is
crucial. Its incorporation of upsampling operations and utilization of skip
connections make it particularly effective in tasks that demand the
preservation of fine details and spatial information, ultimately leading to
superior performance in tasks like image segmentation and image generation.

Input: Output feature map from the last convolutional layer of the contraction
path, denoted as $conv_{\text{last}}\in\mathbb{R}^{H\times W\times C}$, where
$H$ is the height, $W$ is the width, and $C$ is the number of channels.

Output: Reconstructed image tensor, denoted as $re_{out}\in\mathbb{R}^{H\times
W\times N}$ where $N$ is the no. of classes.

1 Convolutional Layer Operation: $\displaystyle re_{out}$
$\displaystyle=\text{Conv2D}(conv_{\text{last}},N,(3,3),\text{softmax},\text{Same})$
$\displaystyle=\sigma\left(W_{\text{conv}}*conv_{\text{last}}+b_{\text{conv}}\right)$
where $\sigma$ denotes the Softmax activation function, $W_{\text{conv}}$ is
the set of convolutional filters, $*$ denotes the convolution operation, and
$b_{\text{conv}}$ is the bias term. The ”Same” padding option ensures that the
output feature map has the exact spatial dimensions as the input feature map.
Padding with zeros is applied to the input feature map to achieve this,
preserving the spatial information during convolution.

Algorithm 4 Reconstruction Output Branch

###  IV-E Upsampling Image reconstruction Branch:

The reconstruction branch in Ahnet first receives feature maps from different
scales of the encoder-decoder architecture used for segmentation. These
feature maps undergo an attention mechanism, fusing with corresponding
features to focus on relevant regions. This attention-enhanced information is
then propagated through residual blocks to extract hierarchical
representations. The reconstruction branch integrates skip connections to
preserve detailed information from the encoder. After passing through
convolutional layers, the reconstructed output is generated to faithfully
recreate the input image while emphasizing regions crucial for segmentation.
During training, the reconstruction branch is jointly optimized with the
segmentation task, incorporating a combined loss function to balance both
objectives and improve overall model performance effectively.

The upsampling image reconstruction branch in the Ahnet differs from a
standalone reconstruction model’s tailored integration within the segmentation
framework. Unlike a typical reconstruction model, which focuses solely on
faithfully reproducing the input image, the reconstruction branch here is
designed to complement the segmentation task by emphasizing relevant regions
for segmentation while maintaining overall image fidelity. By incorporating
attention mechanisms, feature fusion from multiple scales, and joint
optimization with segmentation objectives, the reconstruction branch not only
reconstructs the input image but also enhances the segmentation process by
providing additional guidance and regularization. This integration improves
the model’s ability to capture meaningful representations, mitigate
overfitting, and achieve better segmentation performance by ensuring that
reconstructed images emphasize features crucial for accurate segmentation.
Thus, this adapted reconstruction branch within the segmentation model
framework offers improved interpretability, regularization, and segmentation
accuracy compared to standalone reconstruction models.

###  IV-F Upsampling Attention Gate Function:

The upsampling attention gate described in Algorithm 3 operates by selectively
attending to different regions of the input features while enhancing spatial
details during upsampling tasks within neural network architectures.
Initially, the input and context features undergo separate transformations via
convolutional layers, followed by batch normalization to stabilize training.
These transformed features are then combined and processed through activation
functions to compute an attention map, indicating the relative importance of
each spatial location. Crucially, the attention map is upsampled to match the
spatial resolution of the input features, ensuring that attention weights are
applied appropriately across all spatial dimensions. By multiplying the input
features with the upsampled attention map, the upsampling attention gate
selectively amplifies or suppresses features, effectively highlighting
relevant regions and preserving spatial details during the upsampling process.
This mechanism facilitates the reconstruction of high-resolution feature maps
with enhanced fidelity, making it particularly advantageous for tasks such as
image segmentation, where maintaining fine details is critical for
performance. Attention mechanisms allow for selective feature emphasis,
enhancing the model’s ability to focus on relevant image regions while
suppressing noise and irrelevant information. Additionally, upsampling
residual blocks facilitates better gradient flow and feature propagation
through skip connections, enabling the model to capture richer hierarchical
representations and learn more discriminative features. Furthermore, AHNet
dynamically adjusts the number of filters within residual blocks to match the
dimensionality of the feature maps, enhancing adaptability and improving
feature representation learning. Through these enhancements, AHNet performs
superior medical image segmentation tasks by capturing long-range dependencies
and contextual information more effectively, resulting in improved
segmentation accuracy, robustness, and generalization across diverse datasets
compared to traditional UNet architectures.

The key difference between the upsampling attention gate and a normal
attention gate lies in their applicability and effectiveness in tasks
involving upsampling operations within neural network architectures. While a
normal attention gate focuses on selectively attending to different regions of
the input features, the upsampling attention gate is specifically tailored to
enhance upsampling tasks by effectively capturing and propagating spatial
information at higher resolutions. By incorporating upsampling operations to
match the size of the input features, the upsampling attention gate ensures
that attention weights are applied at the exact spatial resolution of the
input features, which is crucial for reconstructing high-resolution feature
maps. This approach allows the upsampling attention gate to preserve spatial
details better and improve the performance of tasks such as image
segmentation, where maintaining fine details is essential. Thus, the
upsampling attention gate offers a more targeted and effective solution for
upsampling tasks compared to a normal attention gate.

##  V Experiments

###  V-A Dataset

The data is stored in the NIfTI (Neuroimaging Informatics Technology
Initiative) format and comprises volumes of interest (VOIs) cropped around
individual annotated lesions with a largest axial diameter of $5mm$ or
greater. Each VOI is centered on a randomly selected lesion foreground voxel
and is sized $256\times 256\times 128$ voxels in the original scan spacing.
Padding was applied before cropping, using the minimum intensity value of the
volume minus one.

A new data annotation approach, ULS23_DeepLesion3D, was employed. In this
method, trained biomedical students utilized measurement information from
DeepLesion for 3D segmentation in the axial plane. Each lesion was segmented
three times, and most masks were adopted as the final label. Lesions were
chosen based on their performance compared to reference measurements from
DeepLesion, with lesions exhibiting the poorest performance included. The
selection aimed to represent the entire thorax-abdomen area, encompassing
abdominal, bone, kidney, liver, lung, mediastinal, and assorted lesions.

Additionally, two specific datasets for bone and pancreas were derived from
studies at Radboudumc Hospital in Nijmegen, The Netherlands. These datasets
contain VOIs selected based on radiological reports mentioning bone or
pancreas disease. An experienced radiologist identified and segmented the
lesions in 3D. The bone dataset encompasses both sclerotic and lytic bone
lesions.

###  V-B Data Preprocessing

First, we extract image slices from volumetric medical image data encapsulated
within NIfTI files. This task involves a series of intricate steps:
normalization of each slice, conversion to grayscale PNG format, and resizing
to a standardized dimension of $256\times 256$ pixels using the Lanczos
interpolation technique.

Subsequently, attention shifts to harmonizing image and label pairs, a pivotal
step in the preprocessing pipeline. Then seamlessly amalgamate corresponding
image and label data by meticulously matching filenames from distinct
directories based on a shared identifier. The resulting fusion is encapsulated
within HDF5 files, offering a structured and efficient means of storage
conducive to the demands of subsequent deep learning endeavors.

In tandem, we orchestrate the systematic processing of images in batches, a
strategy essential for handling large-scale datasets with finesse. Leveraging
dictionaries as organizational scaffolds, the code systematically categorizes
images by a unique identifier extracted from filenames. This methodical
approach ensures the seamless association of each image with its pertinent
metadata, thereby facilitating subsequent analysis and interpretation.

Lastly, we use patient identities to organize the data. The new HDF5 files are
customized to each patient’s unique ID by navigating the combined dataset
repeatedly. Image and label data relating to specific patients reside in these
custom archives, providing a well-organized and cohesive framework that is
beneficial for further research and model training efforts.

###  V-C Evaluation Metrics

The evaluation of segmentation algorithms in this study involved metrics such
as the dice similarity coefficient (DSC), intersection over union (IoU), mean
Hausdorff distance (MHD), relative absolute volume difference (RAVD), average
surface distance (ASD) and area under receiver operating characteristic (ROC)
curve (AUC).

We utilized the DSC as an area-based metric to quantify the degree of spatial
overlap between the predicted mask and the ground truth. This metric offers a
precise measure of segmentation accuracy. Also, we present the IoU (Jaccard
Index) for the segmentation tasks, to ensure that the annotations that are
delivered have a mean $IoU$ ${\geq X}$ (where ${X=0.5}$) concerning the “good”
segmentation.

$DSC=\frac{2\times TP}{2\times TP+FP+FN}$ $\displaystyle IoU$
$\displaystyle=\frac{Intersection\>Area}{Union\>Area}$
$\displaystyle=\frac{TP}{TP+FP+FN}$

where, TP (True Positives) represents the number of pixels correctly
classified as belonging to the region of interest, FP (False Positives)
represents the number of pixels in the predicted mask that do not correspond
to the region of interest, FN (False Negatives) represents the number of
pixels in the ground truth mask that were missed by the segmentation
algorithm. Besides, AUC is a crucial segmentation metric that summarizes the
model’s overall performance across all possible threshold settings. A perfect
classifier would have an AUC of 1, indicating perfect discrimination between
positive and negative samples. Specifically, AUC is calculated from the ROC
curve, a graphical plot that illustrates the true positive rate (sensitivity)
against the false positive rate (1-specificity) at various threshold settings.
It provides insights into the segmentation model’s performance in
distinguishing positive and negative regions.

MHD provides insight into the average maximum distance between points in the
predicted mask and ground truth. This metric offers a comprehensive evaluation
of boundary discrepancies which is followed by,

$MHD=\frac{1}{N}\sum_{i=1}^{N}\max_{j}\min_{k}d(P_{i},T_{j})$

where $N$ represents the total number of data points, $P_{i}$ is a point in
the predicted mask, $T_{j}$ is a point in the ground truth and
$d(P_{i},T_{j})$ calculates the distance between $P_{i}$ and $T_{j}$. The
metric RAVD is particularly relevant for understanding volume-related
disparities.

$RAVD=\frac{|V_{P}-V_{T}|}{V_{T}}$

Here $V_{P}$ represents the volume of the predicted mask and $V_{T}$
represents the volume of the ground truth. Also, ASD calculates the average
distance between the predicted mask and ground truth surfaces, focusing on
boundary accuracy.

$ASD=\frac{1}{N}\sum_{i=1}^{N}\frac{1}{|P_{i}|}\sum_{p\in P_{i}}\min_{q\in
T}d(p,q)$

where $N$ represents the total number of data points, $P_{i}$ is the predicted
mask for data point $i$, $|P_{i}|$ denotes the number of points on $P_{i}$,
$p$ is a point on $P_{i}$, $T$ represents the ground truth and $d(p,q)$
calculates the distance between point $p$ on the predicted mask and point $q$
on the ground truth.

##  VI Computational Time

This section analyzes the computational time associated with implementing the
proposed Mamba-AHNet with image reconstruction. The experiments were conducted
on a machine equipped with the following specifications:

  * •

Processors: AMD Ryzen Threadripper 1920X 12-Core processor

  * •

Clock Speed: 3.5 GHz

  * •

RAM: 32 GB

  * •

Environment: Python 3.9 with Tensorflow framework

Dataset | Samples | Model |  | Training
---
time (s)
(per epoch)
|  | HUNet | 1997.1423
|  | AHNet | 3678.7891
|  |  | AHNet
---
with image
reconstruction
3712.1234
|  | Mamba-AHNet | 2162.8765
Deeplesion |  24580 (Train) 6145 (Test) 3414 (Val) |  | Mamba-AHNet
---
with image
reconstruction
2302.3456
|  | HUNet | 5295.8901
|  | AHNet | 10934.5678
|  |  | AHNet
---
with image
reconstruction
12987.2345
|  | Mamba-AHNet | 5483.9012
ULS Bone |  62339 (Train) 15585 (Test) 8659 (Val) |  | Mamba-AHNet
---
with image
reconstruction
5892.6789
|  | HUNet | 746.9876
|  | AHNet | 1728.4567
|  |  | AHNet
---
with image
reconstruction
1918.7890
|  | Mamba-AHNet | 859.1098
ULS Pancreas |  9760 (Train) 2440 (Test) 1356 (Val) |  | Mamba-AHNet
---
with image
reconstruction
1025.8764
TABLE I: Computational time and CPU usage for both training and testing phases
for each dataset using HUNet, AHNet, AHNet with image reconstruction, Mamba-
AHNet, and Mamba-AHNet with image reconstruction (times are measured in
seconds)

Across all datasets, the training time for Mamba-AHNet with image
reconstruction was recorded and compared against other models in Table I.
Notably, training times varied significantly depending on the neural network’s
architecture and image reconstruction techniques’ presence. For instance, in
the Deeplesion dataset, HUNet exhibited the shortest training time per epoch
at 1997.1423 seconds, while AHNet with image reconstruction required the
longest time at 3712.1234 seconds. Similarly, in the ULS Bone dataset, HUNet’s
training time per epoch was comparatively shorter than AHNet with image
reconstruction.

A notable trade-off emerges when considering the incorporation of image
reconstruction techniques. While these techniques may enhance the model’s
performance, they often incur a computational cost. This is evident in the
case of AHNet with image reconstruction and Mamba-AHNet with image
reconstruction, both of which demonstrated longer training times than their
counterparts without image reconstruction.

Despite the substantial computational requirements associated with
incorporating image reconstruction techniques, Mamba-AHNet with image
reconstruction showed competitive performance in training time and model
effectiveness.

It is observed that Mamba-AHNet with image reconstruction incurred longer
training times than its counterpart without image reconstruction. However,
this increase in computational overhead was offset by the superior performance
achieved by the model.

##  VII Numerical Analysis

###  VII-A Results

Test Image | Test Label | Predicted Label
---|---|---
|  |
|  |
Figure 4: Output segmentation results for deeplesion dataset Test Image | Test Label | Predicted Label
---|---|---
|  |
|  |
Figure 5: Output segmentation results for pancreas dataset Test Image | Test Label | Predicted Label
---|---|---
|  |
|  |
Figure 6: Output segmentation results for bone dataset Study |  DSC$\uparrow$ |  RAVD$\downarrow$ |  ASD$\downarrow$ |  MHD$\downarrow$ |  AUC$\uparrow$ |  IoU$\uparrow$
---|---|---|---|---|---|---
MULAN [4] |  0.7534 |  0.7120 |  0.1269 |  0.5839 |  0.9601 |  0.6785
ULD [43] |  0.7987 |  0.5784 |  0.1070 |  0.5169 |  0.9641 |  0.6943
ULS4US [67] |  0.8912 |  0.4031 |  0.0641 |  0.4206 |  0.9781 |  0.7310
C2FVL [75] |  0.8907 |  0.3893 |  0.0767 |  0.4594 |  0.9774 |  0.7116
DCAU-Net [76] |  0.8164 |  0.4135 |  0.0839 |  0.4798 |  0.9685 |  0.6995
HUNet |  0.8789 |  0.3766 |  0.0669 |  0.4122 |  0.9750 |  0.7016
AHNet |  0.9676 |  0.2776 |  0.0467 |  0.3192 |  0.9879 |  0.7944
AHNet with image reconstruction |  0.9821 |  0.2574 |  0.0319 |  0.2167 |  0.9891 |  0.8144
Mamba-AHNet |  0.9874 |  0.2688 |  0.0265 |  0.2128 |  0.9952 |  0.8092
Mamba-AHNet with image reconstruction |  0.9896 |  0.2452 |  0.0140 |  0.1999 |  0.9963 |  0.8265
TABLE II: Performance of different models on Deeplesion dataset. $\uparrow$ denotes higher is better and $\downarrow$ denotes lower is better Study |  DSC$\uparrow$ |  RAVD$\downarrow$ |  ASD$\downarrow$ |  MHD$\downarrow$ |  AUC$\uparrow$ |  IoU$\uparrow$
---|---|---|---|---|---|---
MULAN [4] |  0.7398 |  0.7354 |  0.1405 |  0.5911 |  0.9562 |  0.6605
ULD [43] |  0.8059 |  0.5521 |  0.1008 |  0.5362 |  0.9613 |  0.6832
ULS4US [67] |  0.8831 |  0.3845 |  0.0738 |  0.4098 |  0.9752 |  0.7423
C2FVL [75] |  0.8655 |  0.4036 |  0.0925 |  0.4512 |  0.9721 |  0.7306
DCAU-Net [76] |  0.8276 |  0.4059 |  0.0813 |  0.4845 |  0.9663 |  0.7063
HUNet |  0.8637 |  0.3675 |  0.0898 |  0.4603 |  0.9687 |  0.7128
AHNet |  0.9635 |  0.2664 |  0.0489 |  0.2976 |  0.9855 |  0.8063
AHNet with image reconstruction |  0.9776 |  0.2558 |  0.0337 |  0.2335 |  0.9873 |  0.8225
Mamba-AHNet |  0.9802 |  0.2498 |  0.0286 |  0.2034 |  0.9935 |  0.8261
Mamba-AHNet with image reconstruction |  0.9820 |  0.2365 |  0.0178 |  0.1876 |  0.9947 |  0.8339
TABLE III: Performance of different models on ULS Bone dataset. $\uparrow$ denotes higher is better and $\downarrow$ denotes lower is better Study |  DSC$\uparrow$ |  RAVD$\downarrow$ |  ASD$\downarrow$ |  MHD$\downarrow$ |  AUC$\uparrow$ |  IoU$\uparrow$
---|---|---|---|---|---|---
MULAN [4] |  0.7456 |  0.7296 |  0.1348 |  0.5672 |  0.9573 |  0.6659
ULD [43] |  0.8014 |  0.5623 |  0.0945 |  0.5267 |  0.9627 |  0.6889
ULS4US [67] |  0.8775 |  0.3887 |  0.0695 |  0.4156 |  0.9765 |  0.7378
C2FVL [75] |  0.8702 |  0.3998 |  0.0862 |  0.4392 |  0.9732 |  0.7251
DCAU-Net [76] |  0.8221 |  0.4097 |  0.0776 |  0.4813 |  0.9671 |  0.7024
HUNet |  0.8585 |  0.4133 |  0.0716 |  0.4215 |  0.9716 |  0.7203
AHNet |  0.9712 |  0.2701 |  0.0512 |  0.3065 |  0.9863 |  0.8017
AHNet with image reconstruction |  0.9868 |  0.2613 |  0.0356 |  0.2256 |  0.9882 |  0.8178
Mamba-AHNet |  0.9852 |  0.2531 |  0.0307 |  0.2073 |  0.9941 |  0.8229
Mamba-AHNet with image reconstruction |  0.9891 |  0.2397 |  0.0193 |  0.1912 |  0.9952 |  0.8306
TABLE IV: Performance of different models on ULS Pancreas dataset. $\uparrow$
denotes higher is better and $\downarrow$ denotes lower is better Figure 7:
Workflow diagram illustrating the process of data collection, unifications,
augmentation, and model deployment in the development of Mamba-AHNet with
image reconstruction model for enhanced dataset analysis and integration into
a website through an API

In Table II, we comprehensively compare various models’ performance on the
Deeplesion3D dataset. Upon scrutiny, it becomes apparent that Mamba-AHNet with
image reconstruction outperforms other models, including HUNet, AHNet, AHNet
with image reconstruction, MAMBA-AHNet, and baseline models [75, 76, 43, 67,
4]. Mamba-AHNet with image reconstruction achieves a DSC score of 0.9896,
indicating a superior overlap between predicted and ground truth segmentations
compared to the baseline models. Additionally, it demonstrates lower values
for RAVD, ASD, and MHD, indicating improved segmentation accuracy and spatial
alignment. Furthermore, Mamba-AHNet with image reconstruction attains an
impressive AUC of 0.9963 and an IoU of 0.8265 on the Deeplesion3D dataset,
underscoring its exceptional capability in accurately delineating lesions
within medical images.

Moving to Table III, which presents findings on the ULS Bone dataset, Mamba-
AHNet with image reconstruction continues to demonstrate superiority over the
baseline models. While the baseline models show DSC scores as low as 0.7398,
Mamba-AHNet with image reconstruction achieves a remarkable DSC of 0.9820,
indicating a substantial enhancement in bone structure segmentation accuracy.
Furthermore, Mamba-AHNet with image reconstruction exhibits lower values for
RAVD, ASD, and MHD than the baseline models, emphasizing its effectiveness in
precisely delineating bone structures in medical images. Additionally, it
achieves an AUC of 0.9947 and an IoU of 0.8339, further highlighting its
proficiency in accurately segmenting bone structures.

Likewise, Table IV provides insights into the performance comparison on the
ULS Pancreas dataset, where Mamba-AHNet with image reconstruction showcases
remarkable performance. While the baseline models achieve minimal DSC scores,
Mamba-AHNet with image reconstruction achieves a noteworthy DSC of 0.9891,
significantly improving pancreas segmentation accuracy. Moreover, Mamba-AHNet
with image reconstruction demonstrates lower values for RAVD, ASD, MHD, and
IoU than the baseline models, confirming its efficacy in precisely delineating
pancreas structures in medical images. Additionally, it achieves an AUC of
0.9952 and an IoU of 0.8306, further emphasizing its effectiveness in
accurately delineating pancreas structures within medical images. High AUC and
IoU values underscore the precise segmentation and localization of pancreas
regions achieved by Mamba-AHNet with image reconstruction.

To complement the quantitative analysis, visual representations of the
segmentation results for each dataset were provided in Figure 4, 5 and 6.
These visualizations offer a qualitative assessment of the segmentation
performance of Mamba-AHNet with image reconstruction compared to the baseline
models, further supporting the conclusions drawn from the quantitative
analysis.

###  VII-B Discussion

The results obtained from the comparative analysis demonstrate the superior
performance of Mamba-AHNet with image reconstruction in medical image
segmentation tasks across multiple datasets. Across all three datasets, Mamba-
AHNet with image reconstruction consistently outperformed the baseline models
in terms of DSC, RAVD, ASD, MHD, AUC, and IoU. These findings underscore the
efficacy and robustness of Mamba-AHNet with image reconstruction in accurately
delineating anatomical structures in medical images, thus addressing the
primary objective of the research.

Interpreting the results reveals several key insights. Firstly, the
significantly higher DSC scores, AUC, and IoU achieved by Mamba-AHNet with
image reconstruction indicate a better overlap between the predicted and
ground truth segmentations, emphasizing its superior segmentation accuracy.
Additionally, the lower values for RAVD, ASD and MHD further confirm the
enhanced precision and spatial alignment of Mamba-AHNet with image
reconstruction segmentations compared to the baseline models. These
interpretations highlight the effectiveness of the proposed model in
addressing the challenges associated with medical image segmentation, such as
accurate localization and delineation of anatomical structures.

The implications of these results are profound for both clinical practice and
research. The superior performance of Mamba-AHNet with image reconstruction
holds great promise for various medical applications, including disease
diagnosis, treatment planning, and monitoring. Accurate and precise
segmentation of medical images is crucial for clinical decision-making, and
the efficacy of Mamba-AHNet with image reconstruction signifies its potential
to improve patient outcomes and streamline healthcare workflows. Moreover, the
robustness of Mamba-AHNet with image reconstruction across different datasets
underscores its versatility and generalizability, further enhancing its
utility in diverse clinical scenarios.

Additionally, to facilitate the utilization of Mamba-AHNet with image
reconstruction in clinical settings, a user-friendly website has been
developed. This website is connected via API, providing seamless integration
with existing medical imaging systems. Radiologists can easily access the
functionality of the model through this interface, enabling efficient
segmentation of medical images. Furthermore, due to the lightweight nature of
the model, the response from the API is fast, ensuring quick turnaround times
for image analysis. Also, the workflow of the developed system in Figure 7,
highlights the integration of Mamba-AHNet with image reconstruction into the
medical imaging pipeline and its interaction with the user-friendly website
via API.

##  VIII Conclusion

In conclusion, integrating the SSM with the AHNet within the MAMBA framework,
as proposed in our methodology named Mamba-AHNet, demonstrates remarkable
efficacy in addressing the challenges of semantic segmentation in medical
imaging. Through a combination of SSM’s feature extraction capabilities,
AHNet’s attention mechanisms and residual blocks, our approach enhances
segmentation accuracy and robustness. The comprehensive evaluation of various
datasets, including Deeplesion3D, ULS Bone, and ULS Pancreas, consistently
illustrates the superiority of Mamba-AHNet with image reconstruction over
baseline models in terms of metrics such as DSC, RAVD, ASD, MHD, AUC, and IoU.
These findings highlight the model’s ability to accurately delineate
anatomical structures in medical images, which is crucial for diagnosis,
treatment planning, and monitoring in clinical practice. Moreover, the
qualitative assessment through visual representations reinforces the
quantitative results, providing further evidence of the superior segmentation
performance of Mamba-AHNet with image reconstruction. These findings’
implications extend to clinical practice and research, promising advancements
in disease diagnosis, treatment planning, and patient care. The versatility
and generalizability of Mamba-AHNet with image reconstruction across diverse
datasets underscore its potential to streamline healthcare workflows and
improve patient outcomes. The proposed methodology represents a significant
advancement in medical image segmentation, offering a robust and effective
solution to the challenges encountered in clinical settings.

## Authors Contribution

Conceptualization: K S Sanjid, M T Hossain, M S S Junayed. Data curation: M T
Hossain, K S Sanjid. Formal Analysis: M S S Junayed, K S Sanjid, M T Hossain.
Methodology: K S Sanjid, M T Hossain, M S S Junayed. Supervision: M M Uddin.
Writing – original draft: M T Hossain, M S S Junayed, K S Sanjid, M M Uddin.
Funding acquisition: M M Uddin. K S Sanjid, M S S Junayed, and M T Hossain all
made equal contributions and are entitled to have their names listed first on
their CVs. All authors played a role in the article’s development and approved
the submitted version.

## References

  * [1] G. W. Boland, A. S. Guimaraes, and P. R. Mueller, “The radiologist’s conundrum: benefits and costs of increasing ct capacity and utilization,” _European radiology_ , vol. 19, pp. 9–11, 2009.
  * [2] R. J. McDonald, K. M. Schwartz, L. J. Eckel, F. E. Diehn, C. H. Hunt, B. J. Bartholmai, B. J. Erickson, and D. F. Kallmes, “The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload,” _Academic radiology_ , vol. 22, no. 9, pp. 1191–1198, 2015.
  * [3] H. Sung, J. Ferlay, R. L. Siegel, M. Laversanne, I. Soerjomataram, A. Jemal, and F. Bray, “Global cancer statistics 2020: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries,” _CA: a cancer journal for clinicians_ , vol. 71, no. 3, pp. 209–249, 2021.
  * [4] K. Yan, Y. Tang, Y. Peng, V. Sandfort, M. Bagheri, Z. Lu, and R. M. Summers, “Mulan: multitask universal lesion analysis network for joint lesion detection, tagging, and segmentation,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2019: 22nd International Conference, Shenzhen, China, October 13–17, 2019, Proceedings, Part VI 22_. Springer, 2019, pp. 194–202.
  * [5] E. A. Eisenhauer, P. Therasse, J. Bogaerts, L. H. Schwartz, D. Sargent, R. Ford, J. Dancey, S. Arbuck, S. Gwyther, M. Mooney _et al._ , “New response evaluation criteria in solid tumours: revised recist guideline (version 1.1),” _European journal of cancer_ , vol. 45, no. 2, pp. 228–247, 2009.
  * [6] Y. Peng, K. Yan, V. Sandfort, R. M. Summers, and Z. Lu, “A self-attention based deep learning method for lesion attribute detection from ct reports,” in _2019 IEEE international conference on healthcare informatics (ICHI)_. IEEE, 2019, pp. 1–5.
  * [7] A. Katzmann, A. Muehlberg, M. Sühling, D. Noerenberg, J. W. Holch, V. Heinemann, and H.-M. Groß, “Predicting lesion growth and patient survival in colorectal cancer patients using deep neural networks,” in _Medical Imaging with Deep Learning_ , 2022.
  * [8] K. Lu, R. Bascom, R. P. Mahraj, and W. E. Higgins, “Quantitative analysis of the central-chest lymph nodes based on 3d mdct image data,” in _Medical Imaging 2009: Computer-Aided Diagnosis_ , vol. 7260. SPIE, 2009, pp. 257–265.
  * [9] V. Agarwal, Y. Tang, J. Xiao, and R. M. Summers, “Weakly supervised lesion co-segmentation on ct scans,” in _2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI)_. IEEE, 2020, pp. 203–206.
  * [10] J. Cai, Y. Tang, L. Lu, A. P. Harrison, K. Yan, J. Xiao, L. Yang, and R. M. Summers, “Accurate weakly-supervised deep lesion segmentation using large-scale clinical annotations: Slice-propagated 3d mask generation from 2d recist,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part IV 11_. Springer, 2018, pp. 396–404.
  * [11] L. Chunfeng, R. Su, T. Denœux _et al._ , “Joint tumor segmentation in pet-ct images using co-clustering and fusion based on belief functions [j],” _IEEE Transactions on Image Processing_ , vol. 28, no. 2, pp. 755–766, 2019.
  * [12] S. Nikan, K. Van Osch, M. Bartling, D. G. Allen, S. A. Rohani, B. Connors, S. K. Agrawal, and H. M. Ladak, “Pwd-3dnet: a deep learning-based fully-automated segmentation of multiple structures on temporal bone ct scans,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 739–753, 2020.
  * [13] Y. Tang, J. Cai, L. Lu, A. P. Harrison, K. Yan, J. Xiao, L. Yang, and R. M. Summers, “Ct image enhancement using stacked generative adversarial networks and transfer learning for lesion segmentation improvement,” in _Machine Learning in Medical Imaging: 9th International Workshop, MLMI 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018, Proceedings 9_. Springer, 2018, pp. 46–54.
  * [14] Z. Wang, L. Wei, L. Wang, Y. Gao, W. Chen, and D. Shen, “Hierarchical vertex regression-based segmentation of head and neck ct images for radiotherapy planning,” _IEEE Transactions on Image Processing_ , vol. 27, no. 2, pp. 923–937, 2017.
  * [15] A. Hering, F. Peisen, T. Amaral, S. Gatidis, T. Eigentler, A. Othman, and J. H. Moltz, “Whole-body soft-tissue lesion tracking and segmentation in longitudinal ct imaging studies,” in _Medical Imaging with Deep Learning_. PMLR, 2021, pp. 312–326.
  * [16] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [17] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 801–818.
  * [18] Z. Tian, T. He, C. Shen, and Y. Yan, “Decoders matter for semantic segmentation: Data-dependent decoding enables flexible feature aggregation,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 3126–3135.
  * [19] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang, “Denseaspp for semantic segmentation in street scenes,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 3684–3692.
  * [20] Y. Yuan, L. Huang, J. Guo, C. Zhang, X. Chen, and J. Wang, “Ocnet: Object context network for scene parsing,” _arXiv preprint arXiv:1809.00916_ , 2018.
  * [21] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 3213–3223.
  * [22] B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba, “Semantic understanding of scenes through the ade20k dataset,” _International Journal of Computer Vision_ , vol. 127, pp. 302–321, 2019.
  * [23] N. Heller, N. Sathianathen, A. Kalapara, E. Walczak, K. Moore, H. Kaluzniak, J. Rosenberg, P. Blake, Z. Rengel, M. Oestreich _et al._ , “The kits19 challenge data: 300 kidney tumor cases with clinical context, ct semantic segmentations, and surgical outcomes,” _arXiv preprint arXiv:1904.00445_ , 2019.
  * [24] I. C. Moreira, I. Amaral, I. Domingues, A. Cardoso, M. J. Cardoso, and J. S. Cardoso, “Inbreast: toward a full-field digital mammographic database,” _Academic radiology_ , vol. 19, no. 2, pp. 236–248, 2012.
  * [25] R. S. Lee, F. Gimenez, A. Hoogi, K. K. Miyake, M. Gorovoy, and D. L. Rubin, “A curated mammography data set for use in computer-aided detection and diagnosis research,” _Scientific data_ , vol. 4, no. 1, pp. 1–9, 2017.
  * [26] S. G. Armato III, G. McLennan, L. Bidaut, M. F. McNitt-Gray, C. R. Meyer, A. P. Reeves, B. Zhao, D. R. Aberle, C. I. Henschke, E. A. Hoffman _et al._ , “The lung image database consortium (lidc) and image database resource initiative (idri): a completed reference database of lung nodules on ct scans,” _Medical physics_ , vol. 38, no. 2, pp. 915–931, 2011.
  * [27] H. Cao, H. Liu, E. Song, C.-C. Hung, G. Ma, X. Xu, R. Jin, and J. Lu, “Dual-branch residual network for lung nodule segmentation,” _Applied Soft Computing_ , vol. 86, p. 105934, 2020.
  * [28] P. F. Christ, F. Ettlinger, F. Grün, M. E. A. Elshaera, J. Lipkova, S. Schlecht, F. Ahmaddy, S. Tatavarty, M. Bickel, P. Bilic _et al._ , “Automatic liver and tumor segmentation of ct and mri volumes using cascaded fully convolutional neural networks,” _arXiv preprint arXiv:1702.05970_ , 2017.
  * [29] X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes,” _IEEE transactions on medical imaging_ , vol. 37, no. 12, pp. 2663–2674, 2018.
  * [30] Y. Tang, K. Yan, J. Xiao, and R. M. Summers, “One click lesion recist measurement and segmentation on ct scans,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part IV 23_. Springer, 2020, pp. 573–583.
  * [31] Y. Tang, K. Yan, J. Cai, L. Huang, G. Xie, J. Xiao, J. Lu, G. Lin, and L. Lu, “Lesion segmentation and recist diameter prediction via click-driven attention and dual-path connection,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24_. Springer, 2021, pp. 341–351.
  * [32] K. Yan, X. Wang, L. Lu, and R. M. Summers, “Deeplesion: automated mining of large-scale lesion annotations and universal lesion detection with deep learning,” _Journal of medical imaging_ , vol. 5, no. 3, pp. 036 501–036 501, 2018.
  * [33] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz _et al._ , “Attention u-net: Learning where to look for the pancreas,” _arXiv preprint arXiv:1804.03999_ , 2018.
  * [34] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_. Springer, 2015, pp. 234–241.
  * [35] H. Zhang, K. Dana, J. Shi, Z. Zhang, X. Wang, A. Tyagi, and A. Agrawal, “Context encoding for semantic segmentation,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2018, pp. 7151–7160.
  * [36] M. Havaei, A. Davy, D. Warde-Farley, A. Biard, A. Courville, Y. Bengio, C. Pal, P.-M. Jodoin, and H. Larochelle, “Brain tumor segmentation with deep neural networks,” _Medical image analysis_ , vol. 35, pp. 18–31, 2017.
  * [37] D. Jin, Z. Xu, Y. Tang, A. P. Harrison, and D. J. Mollura, “Ct-realistic lung nodule simulation from 3d conditional generative adversarial networks for robust lung segmentation,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11_. Springer, 2018, pp. 732–740.
  * [38] S. Wang, M. Zhou, Z. Liu, Z. Liu, D. Gu, Y. Zang, D. Dong, O. Gevaert, and J. Tian, “Central focused convolutional neural networks: Developing a data-driven model for lung nodule segmentation,” _Medical image analysis_ , vol. 40, pp. 172–183, 2017.
  * [39] L. Zhang, Y. Shi, J. Yao, Y. Bian, K. Cao, D. Jin, J. Xiao, and L. Lu, “Robust pancreatic ductal adenocarcinoma segmentation with multi-institutional multi-phase partially-annotated ct scans,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part IV 23_. Springer, 2020, pp. 491–500.
  * [40] G. Chlebus, A. Schenk, J. H. Moltz, B. van Ginneken, H. K. Hahn, and H. Meine, “Automatic liver tumor segmentation in ct with fully convolutional neural networks and object-based postprocessing,” _Scientific reports_ , vol. 8, no. 1, p. 15497, 2018.
  * [41] I. Nogues, L. Lu, X. Wang, H. Roth, G. Bertasius, N. Lay, J. Shi, Y. Tsehay, and R. M. Summers, “Automatic lymph node cluster segmentation using holistically-nested neural networks and structured optimization in ct images,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2016, pp. 388–397.
  * [42] Z. Zhu, D. Jin, K. Yan, T.-Y. Ho, X. Ye, D. Guo, C.-H. Chao, J. Xiao, A. Yuille, and L. Lu, “Lymph node gross tumor volume detection and segmentation via distance-based gating using 3d ct/pet imaging in radiotherapy,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2020, pp. 753–762.
  * [43] F. Lyu, B. Yang, A. J. Ma, and P. C. Yuen, “A segmentation-assisted model for universal lesion detection with partial labels,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24_. Springer, 2021, pp. 117–127.
  * [44] Y. Tang, J. Cai, K. Yan, L. Huang, G. Xie, J. Xiao, J. Lu, G. Lin, and L. Lu, “Weakly-supervised universal lesion segmentation with regional level set loss,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24_. Springer, 2021, pp. 515–525.
  * [45] K. Yan, J. Cai, Y. Zheng, A. P. Harrison, D. Jin, Y. Tang, Y. Tang, L. Huang, J. Xiao, and L. Lu, “Learning from multiple datasets with heterogeneous and partial labels for universal lesion detection in ct,” _IEEE Transactions on Medical Imaging_ , vol. 40, no. 10, pp. 2759–2770, 2020.
  * [46] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” _arXiv preprint arXiv:2312.00752_ , 2023.
  * [47] K. S. Sanjid, M. T. Hossain, M. S. S. Junayed, and D. M. M. Uddin, “Integrating mamba sequence model and hierarchical upsampling network for accurate semantic segmentation of multiple sclerosis legion,” 2024.
  * [48] K. S. Sanjid, M. S. S. Junayed, M. T. Hossain, Y.-L. Wang, M. M. Uddin, and S. A. Haque, “From pixels to pathology: A novel dual-pathway multi-scale hierarchical upsampling network for mri-based prostate zonal segmentation,” _SSRN_ , 2023. [Online]. Available: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4725376
  * [49] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 40, no. 4, pp. 834–848, 2017.
  * [50] F. Xu, L. Lin, Z. Li, Q. Hong, K. Liu, Q. Wu, Q. Li, Y. Zheng, and J. Tian, “Mrdff: A deep forest based framework for ct whole heart segmentation,” _Methods_ , vol. 208, pp. 48–58, 2022.
  * [51] F. Xu, L. Lin, D. Li, Q. Hong, K. Liu, Q. Wu, Q. Li, Y. Zheng, and J. Tian, “A multi-resolution deep forest framework with hybrid feature fusion for ct whole heart segmentation,” in _2021 IEEE International Conference on Bioinformatics and Biomedicine (BIBM)_. IEEE, 2021, pp. 1119–1124.
  * [52] Z. Zhou, M. M. Rahman Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: A nested u-net architecture for medical image segmentation,” in _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop, DLMIA 2018, and 8th International Workshop, ML-CDS 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018, Proceedings 4_. Springer, 2018, pp. 3–11.
  * [53] Q. Liu, L. Yu, L. Luo, Q. Dou, and P. A. Heng, “Semi-supervised medical image classification with relation-driven self-ensembling model,” _IEEE transactions on medical imaging_ , vol. 39, no. 11, pp. 3429–3440, 2020.
  * [54] Z. Li, W. Chen, Z. Wei, X. Luo, and B. Su, “Semi-wtc: A practical semi-supervised framework for attack categorization through weight-task consistency,” _arXiv preprint arXiv:2205.09669_ , 2022.
  * [55] Y. Xia, D. Yang, Z. Yu, F. Liu, J. Cai, L. Yu, Z. Zhu, D. Xu, A. Yuille, and H. Roth, “Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation,” _Medical image analysis_ , vol. 65, p. 101766, 2020.
  * [56] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark _et al._ , “Learning transferable visual models from natural language supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 8748–8763.
  * [57] W. Kim, B. Son, and I. Kim, “Vilt: Vision-and-language transformer without convolution or region supervision,” in _International conference on machine learning_. PMLR, 2021, pp. 5583–5594.
  * [58] H. Ding, C. Liu, S. Wang, and X. Jiang, “Vlt: Vision-language transformer and query generation for referring segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [59] Z. Yang, J. Wang, Y. Tang, K. Chen, H. Zhao, and P. H. Torr, “Lavt: Language-aware vision transformer for referring image segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 155–18 165.
  * [60] R. Bhalodia, A. Hatamizadeh, L. Tam, Z. Xu, X. Wang, E. Turkbey, and D. Xu, “Improving pneumonia localization via cross-attention on medical images and reports,” in _Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part II 24_. Springer, 2021, pp. 571–581.
  * [61] P. Müller, G. Kaissis, C. Zou, and D. Rueckert, “Joint learning of localized representations from medical images and reports,” in _European Conference on Computer Vision_. Springer, 2022, pp. 685–701.
  * [62] N. K. Tomar, D. Jha, U. Bagci, and S. Ali, “Tganet: Text-guided attention for improved polyp segmentation,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2022, pp. 151–160.
  * [63] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, “Residual attention network for image classification,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 3156–3164.
  * [64] S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, “Cbam: Convolutional block attention module,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 3–19.
  * [65] C. Huang, Y. Lan, G. Xu, X. Zhai, J. Wu, F. Lin, N. Zeng, Q. Hong, E. Ng, Y. Peng _et al._ , “A deep segmentation network of multi-scale feature fusion based on attention mechanism for ivoct lumen contour,” _IEEE/ACM Transactions on computational biology and bioinformatics_ , vol. 18, no. 1, pp. 62–69, 2020.
  * [66] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [67] X. Wu, Y. Jiang, H. Xing, W. Song, P. Wu, X.-w. Cui, and G. Xu, “Uls4us: universal lesion segmentation framework for 2d ultrasound images,” _Physics in Medicine & Biology_, vol. 68, no. 16, p. 165009, 2023.
  * [68] A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” _arXiv preprint arXiv:2111.00396_ , 2021.
  * [69] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré, “Hippo: Recurrent memory with optimal polynomial projections,” _Advances in neural information processing systems_ , vol. 33, pp. 1474–1487, 2020.
  * [70] Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, “Long range arena: A benchmark for efficient transformers,” _arXiv preprint arXiv:2011.04006_ , 2020.
  * [71] A. Tustin, “A method of analysing the behaviour of linear systems in terms of time series,” _Journal of the Institution of Electrical Engineers-Part IIA: Automatic Regulators and Servo Mechanisms_ , vol. 94, no. 1, pp. 130–142, 1947.
  * [72] Z. Chen, Y. Zhang, J. Gu, L. Kong, and X. Yang, “Recursive generalization transformer for image super-resolution,” _arXiv preprint arXiv:2303.06373_ , 2023.
  * [73] Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, and Y. Liu, “Vmamba: Visual state space model,” _arXiv preprint arXiv:2401.10166_ , 2024.
  * [74] N. Shazeer, “Glu variants improve transformer,” _arXiv preprint arXiv:2002.05202_ , 2020.
  * [75] D. Shan, Z. Li, W. Chen, Q. Li, J. Tian, and Q. Hong, “Coarse-to-fine covid-19 segmentation via vision-language alignment,” in _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2023, pp. 1–5.
  * [76] W. Yuan, Y. Peng, Y. Guo, Y. Ren, and Q. Xue, “Dcau-net: dense convolutional attention u-net for segmentation of intracranial aneurysm images,” _Visual computing for industry, biomedicine, and art_ , vol. 5, no. 1, p. 9, 2022.