# Identifying and Interpreting Non-Aligned Human Conceptual Representations
using Language Modeling

Wanqian Bao & Uri Hasson
Center for Mind/Brain Sciences (CIMeC)
University of Trento
Rovereto, 38068, Italy
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

The question of whether people’s experience in the world shapes conceptual
representation and lexical semantics is longstanding. Word-association,
feature-listing and similarity rating tasks are methods that aim to address
this question but ultimately require a subjective interpretation of the latent
dimensions or clusters identified. In this study, we introduce a supervised
representational-alignment method that (i) determines whether two groups of
individuals share the same basis of a certain category, and (ii) explains in
what respects they differ. In applying this method, we show that congenital
blindness induces conceptual reorganization in both a-modal and sensory-
related verbal domains, and we identify the associated semantic shifts. We
first apply supervised feature-pruning to a language model (GloVe) to optimize
prediction accuracy of human similarity judgments from word embeddings.
Pruning identifies one subset of retained GloVe features that optimizes
prediction of judgments made by sighted individuals and another subset that
optimizes judgments made by blind. A linear probing analysis then interprets
the latent semantics of these feature-subsets by learning a mapping from the
retained GloVe features to 65 interpretable semantic dimensions. We applied
this approach to seven semantic domains, including verbs related to motion,
sight, touch, and amodal verbs related to knowledge acquisition. We find that
blind individuals more strongly associate social and cognitive meanings to
verbs related to motion or those communicating non-speech vocal utterances
(e.g., whimper, moan). Conversely, for amodal verbs, they demonstrate much
sparser information. Finally, for some verbs, representations of blind and
sighted are highly similar. The study presents a formal approach for studying
interindividual differences in word meaning, and the first demonstration of
how blindness impacts conceptual representation of everyday verbs.

##  1 Introduction

###  1.1 Conceptual representation in congenitally blind

The question of whether words capture similar meanings across different
languages or among different speakers of the same language is central to
understanding the relation between language and conceptual representation.
Thompson et al. (2020) used a computational modeling approach to show that
representations of non-specialized, everyday words can substantially differ
across cultures. The question also strongly applies to speakers of the same
language. Acquiring sub-cultural lexicons (e.g., slang terms) clearly requires
learning and is associated with changes in meaning over time (e.g., Friendly &
Glucksberg, 1970). But does one’s personal experience produce different
meanings for everyday words such as ‘touch,’ ‘see,’ and ‘learn’? One approach
to tackling this issue has been to study the nature of word meaning in blind
individuals, directly evaluating whether sensory experiences in the world can
impact knowledge of word-meaning. Indeed, this question has interested
cognitive psychologists for decades (for recent reviews, see Mamus et al.,
2023; Bedny et al., 2019). This work has produced substantial evidence that
the blind use language in a way indicative of having knowledge of the visual
world, and of the use of vision-related words.

However, the question of how blind and sighted differ in conceptual
organization is still open. Multiple approaches have been used to quantify
semantic knowledge in the blind, including the production of definitions
(e.g., Landau & Gleitman, 1985), word-associations (e.g., Lenci et al., 2013),
event-description Mamus et al. (2023) and similarity judgments for colors
(e.g., Shepard & Cooper, 1992), among others. More recently, Bedny et al.
(2019) obtained similarity judgments for different types of verbs, and
examined the data using representational similarity analysis and other
descriptive techniques. An important finding was that for some verb types,
there was a resemblance in similarity judgments between the two groups. In
particular, verbs related to visual stimulation (e.g., ‘sparkle’, ‘blink’)
were judged in a consistent manner. Interestingly, Lewis et al. (2019)
subsequently showed that these similarity judgments were well predicted by
(proxies for) similarity values produced from a language model (word2vec;
Mikolov et al., 2013). The authors suggested that blind and sighted learn
similar representations because of similar exposure to language. These prior
studies however could not formally and precisely explicate how the conceptual
representations of blind and sighted differ when representing everyday words.

###  1.2 Current approach: supervised learning of representation

Figure 1: Analysis workflow consisting of supervised pruning and supervised
probing. Solid lines indicate input datasets and dashed lines indicate
products of analyses

The current study aims to determine whether blind and sighted individuals use
similar information when representing verb meanings. To address this, we adopt
a two-step computational modeling approach, each verified using cross-
validation. In step 1 we use supervised-pruning to fine-tune a language model
to predict human similarity judgments. In step 2, we examine the information
latent in the fine-tuned model’s feature space, studying both the retained
vectors (word embeddings) and the information contained in them, using a
probing task. In sum, we investigate both the basis vectors and the
representations formed in (subspaces of) language-models fine-tuned to predict
blind or sighted behavior.

We present an informal description of the logic here. Details are provided in
Methods, and a more formal algorithmic presentation is given in the Appendix.
The workflow is summarized in Figure 1, and we refer to elements within that
figure here. To begin, we first prune GloVe word embeddings (1) to improve
prediction of human similarity judgments (2). This produces an embedding
matrix based on a smaller set of retained features (3). This is done
separately for judgments provided by congenitally blind and sighted controls,
across seven different categories of verb types (anticipating the results,
these produced divergent retained feature sets for blind and sighted). After
pruning, we examine information in the retained feature sets using a probing
task (Belinkov, 2022). The probing task is based on a Partial Least Squares
Regression (PLSR) function, and takes as its target a dataset provided by
Binder et al. (2016), which includes 535 words rated on 65 dimensions (5). It
has been shown (Chersoni et al., 2021) that it is possible to predict the
human ratings for these 535 words from their GloVe features, using PLSR. We
instead learn a mapping between the the embeddings of these 535 words using
only the retained features (4) and the 65 human annotated dimensions (5),
which produces predictions of the 65 dimensions for each word (6). This step
is performed separately using the features retained for blind and sighted.
Crucially, by comparing the predicted values (6) to the ground-truth values
(5), the probing task allows quantifying the relative salience of 65 semantic
dimensions in the retained feature sets used in the mapping. In addition,
using the probing task we can also determine which of the 65 Binder features
are differentially predicted for blind and sighted. We use pruning and probing
after showing they perform effectively in cross-validation.

##  2 Method

###  2.1 Datasets and word embeddings

For similarity judgments we used data made publicly available by Bedny et al.
(2019). Twenty-five congenitally blind and 22 sighted individuals rated
semantic similarity of verb-pairs on a scale of 1 to 7 (7 being most similar).
The materials were 119 verbs divided into seven categories (we follow the
original terminology). Each category contained 15 verbs apart from one
category that contained 14. These verb categories were: Perception Sight
(e.g., see, look), Perception Touch (e.g., pat, rub), Perception Amodal (e.g.,
learn, discover), Emission of Light (e.g., twinkle, glow), Emission of Animate
Sounds (e.g., groan, growl), Emission of Inanimate Sounds (e.g., buzz, beep),
and manner of Motion verbs (e.g., float, bounce). A complete list is given in
the Appendix. Similarity ratings were normalized within participants to create
similarity matrices, with matrices calculated separately for each verb
category and averaged for the group. Furthermore, in our analysis we excluded
three participants who did not complete the rating task. This ultimately
produced data for 24 blind and 20 sighted participants.

For the probing task we used semantic annotations collected by Binder et al.
(2016). Using an online survey, they obtained human feature-ratings for 535
words. Participants indicated the extent to which each word was linked to 65
distinct semantic attributes, covering meanings across 14 overarching domains.
For certain words, some attributes were inapplicable. For instance, the
dimensions of time and duration were not applicable to words like ‘shoe’. In
these cases participants indicated zero. The original dataset contained 535
words by 65 dimensions from 14 categories, However, the word ‘used’ appeared
twice: once as verb and once as adjective. In our analyses, the results did
not change whether one or the other were used and so we deleted one and
analyzed the remaining 534 words.

For word embeddings we used GloVe (Pennington et al., 2014) as its embeddings
perform well in predicting word similarity judgments (e.g., Richie & Bhatia,
2021). We used the 300-dimensional GloVe vectors sourced from the Wikipedia
2014 + Gigaword 5 corpus, which contains approximately 400,000 uncased
vocabulary entries, with a total 6 billion tokens. We extracted the embeddings
of those words that were used in Bedny et al. (2019) and Binder et al. (2016).

###  2.2 Supervised pruning

####  2.2.1 Pruning algorithm

We used a supervised pruning algorithm, which is a wrapper-based sequential
feature selection procedure optimizing the alignment between a human
similarity matrix and one generated from an embedding space. Its objective is
to identify a reduced subset of features, so that when that subset is used to
generate the $Word\times Word$ similarity matrix, the resulting matrix
produces a maximal fit to the human similarity judgments. Pairwise similarity
between words was computed from word embeddings using cosine similarity, while
the fit between the two similarity matrices was computed using the Spearman’s
Rho rank correlation coefficient ($\rho$), following Richie & Bhatia (2021).
Pruning was applied in two ways. In the first approach, pruning was applied by
considering the entire set of similarity judgments associated with each verb
category. This identifies the subset of features that optimizes prediction of
similarity judgments for the verbs in that category. For instance, if a
category consisted of 15 verbs, there were 105 unique similarity judgments
that supervised the pruning procedure. The second approach used cross-
validation to demonstrate that pruning indeed identifies meaningful
dimensions, as indicated by out-of-sample generalization. In each fold, one
word was designated as the target, and the test set only included those
similarity judgments involving the target word. The training set consisted of
all other verb-pairs within the category. For example, in a verb category with
15 words, the test set comprised the 14 word-pairs where the target word
appeared. Baseline performance for the test set was established by determining
the prediction accuracy for matched human data using the complete set of 300
GloVe features. This baseline was compared to performance when using only the
features retained from the training set. As an additional control, in each
case we also constructed a random feature-set, matched in size to that of the
retained set, and used that for the test-set. Both approaches were applied to
similarity matrices associated with each of the seven verb types in Bedny et
al. (2019), as produced by blind and sighted (i.e., 14 different times in
total).

####  2.2.2 Overlap between feature sets retained by pruning

Because pruning identifies a subset of features, it is possible, for any two
results of the pruning algorithm, to compare the similarity between the two
sets of retained features. This computation operates on the indices of the
feature sets retained via pruning. As an indicator of association we used the
Dice coefficient of two sets $U,V$: Dice(U,V) = $(2*|U\cap V|)/(|U|+|V|)$.
Given that we compared sets that often varied in size, in such instances, we
adjusted the larger set to match the size of the smaller set, by selecting the
$n$ top-ranking features from the larger set where $n$ is the size of the
smaller set. This assured that both sets contributed equally to the
denominator term.

It is important to keep in mind that two distinct semantic domains (e.g.,
domains A and B) with ostensibly different meanings (i.e., low similarity
between words across the two domains) can still converge on the exact same
feature sets through pruning. This can occur as long as the retained features
consistently have different values for words in domains A and B. In other
words, pruning identifies features that form a basis set for characterizing
the structure of a domain, while the similarity ratings are determined by the
values on those features across objects. To illustrate, consider words in two
domains, A and B (e.g., two types of nouns), where pruning retains only
features a, b, c, d for both domains. For words in domain A, the values of
(a,b) are consistently higher than (c,d), with the remaining variance
accounting for similarity within the domain. Conversely, for words in domain
B, the values of (a,b) are consistently lower than (c,d). Although the
similarity of objects within A and within B is greater than the similarity
across domains, both domains rely on the exact same features. Thus, whether
the same features are required to account for the similarity structure of two
domains is independent of the domains’ similarity.

###  2.3 Supervised probing

Following previous work (e.g., Chersoni et al., 2021; Utsumi, 2020; Flechas
Manrique et al., 2023), we used Partial Least Squares Regression (PLSR) to
predict the 65 Binder dimensions for a dataset of 535 words for which we also
extracted GloVe embeddings. We follow the exact same procedure, applying it
separately to each of the 14 retained feature sets (seven verb types for
blind/sighted) found via supervised pruning. We emphasize that for each verb
category, we only use the GloVe features that were identified for that
category in the pruning step (i.e., we do not use all 300 features). The
outcome of this prediction allows determining if sets retained by different
domains encode different semantics. The retained features used for PLSR were
produced taking into account the entire set of human similarity judgements
(i.e., outside a cross-validation context).

We applied PLSR to the retained sets, employing a leave-one-out cross-
validation procedure (following Chersoni et al., 2021). Binder’s dataset
included 535 words, with 534 remaining after removing the dual use of the word
‘use’. For each fold, we designated 533 words as the training set and the
single left-out word as the test set. PLSR was trained on the training set,
and the model was then applied to the test word, predicting its 65 feature
values. Note that when applied to all 534 words this produces a $534\times 65$
stacked matrix, including all words, which we refer to as a leave-one-out-
cross-validation (LOOCV) Stacked Matrix 65; $LSM_{65}$. The stacked matrix was
produced by 534 different regressions. We further note that in some analyses
we condense the $534\times 65$ matrix into $534\times 14$ by merging the 65
dimensions into the 14 broader semantic areas, producing $LSM_{14}$. The
merging of the 65 features to 14 domains was implemented according to an
assignment specified in Binder et al. (2016) (see their Tables 1, 2).

The data in the prediction matrices serve as the basis for evaluating
differences in the semantic content of embeddings pruned for blind and
sighted, and for different verb types. We perform three analyses: First,
considering $LSM_{65}$ by column (feature), we correlated the predicted values
of each column ($n=534$) with Binder’s ground truth data. This is an indicator
for the accuracy by which each feature was predicted, used by Chersoni et al.
(2021). These data were were also averaged into the 14 larger semantic
domains. Second, considering $LSM_{65}$ by row (word), we correlated the
predicted values of each row with the ground truth data. Third, to determine
whether there were differences in the prediction-accuracy from pruned
embeddings from blind and sighted, we also defined accuracy as the absolute
deviation between the predicted value and ground-truth value for each cell in
$LSM_{65}$. Taking words ($N=534$) as unit of analysis, these cell-wise
prediction errors allow comparing prediction accuracy, for each of the 14
domains, for blind and sighted. We use paired T-tests, with words as the unit
of analysis.

##  3 Results

###  3.1 Preliminary evaluation: out of sample generalization of pruning
solution

Table 1 shows prediction for test-set data when using all GloVe features
(baseline) or those learned via pruning of the training set (‘retained’).
Pruning often generalized successfully: the subsets of features learned via
pruning either exceeded baseline performance, or maintained baseline
performance using only between 20-30% of the total 300 features. In three
cases we did not find generalization: for Emission-InanimateSounds (sighted
and blind) and for Emission-AnimateSounds (for blind). In the latter case the
performance from full set, retained set and random set were all very similar
($\approx$ 0.27). Note that all data reported in Table 1 are based on
prediction-performance computed for test-folds partitions consisting of 14
data points, which may limit sensitivity. In the following analysis we apply
and evaluate pruning on the same, complete dataset.

| CB Base | CB Retained | CB F. | S Base | S Retained | S F.
---|---|---|---|---|---|---
Perc. Amodal | 0.41 |  0.39 {0.32} | 74 | 0.47 |  0.46 {0.37} | 84
Perc. Sight | 0.56 |  0.63 {0.48} | 118 | 0.64 |  0.68 {0.50} | 113
Perc. Touch | 0.16 |  0.40 {0.19} | 76 | 0.14 |  0.38 {0.08} | 58
Emission. Light | 0.56 |  0.62 {0.54} | 99 | 0.42 |  0.54 {0.41} | 80
Emission. Animate Sound | 0.25 |  0.26 {0.27} | 67 | 0.33 |  0.37 {0.28} | 59
Emission. Inanimate Sound | 0.10 | 0.09 {0.08} | 46 | 0.13 | 0.06 {0.07} | 57
Motion | 0.35 | 0.29 {0.22} | 75 | 0.33 |  0.40 {0.18} | 60
Table 1: Out of sample generalization of supervised pruning. Base (Baseline)
indicates prediction when using all 300 GloVe features. Retained indicates
prediction using features learned from training set. CB = Congenitally Blind ;
S = Sighted. F. (Features) indicates average number of features retained from
training set. Numbers in curly brackets in the ‘Retained’ columns define
chance and indicate prediction performance when using randomly selected
feature sets matched in size to those retained from the training set. Values
in bold indicate better prediction than baseline, or similar prediction to
baseline that surpasses the chance value.

###  3.2 Correspondence between feature-sets retained by pruning

Table 2 presents the outcomes of the pruning process applied to individual
verb datasets, conducted outside the context of a cross-validation framework.
While this may be treated as over-fitting from a machine learning perspective,
our aim here is to obtain the most effective predictor of a specific set of
human similarity judgments. This pruning established the retained feature-sets
used in all subsequent analyses. As is evident from the table, the impact of
pruning in this case was substantial, in some cases increasing prediction
accuracy four- or six-fold.

The data also indicate that prediction of human similarity judgments from
pruned GloVe embeddings can approach the noise ceiling. Bedny et al. (2019)
provided a ceiling estimation: they obtained similarity ratings from a
separate (third) group of sighted individuals, and evaluated the correlation
between that group’s ratings and those of the experimental sighted and blind
groups. In some cases, such as for Light Emission verbs, high correlations
were observed (0.91, 0.93 for sighted and blind) indicating high reliability
of similarity ratings across different groups. In this case, supervised
pruning achieved correlations of 0.85 and 0.88 respectively. Conversely, for
Motion verbs, lower behavioral correlations were found (0.75, 0.72), with
supervised pruning resulting in 0.73 and 0.68. This suggests that for specific
semantic categories, prediction from pruned features approached the ceiling.

Beyond this, we note the following. First, retaining more features did not
necessarily produce better prediction accuracy: For Perception-Amodal verbs
and Emission-Light verbs the number of retained features strongly differed
between blind and sighted, but prediction accuracy (both baseline and
retained) was similar. Second, pruning solutions that retained more features
were not sparser. We compressed each retained embedding matrix and compared it
to its original size (Ziv & Lempel, 1977). We found that across all solutions,
the compression ratio was almost exactly 0.4 (range: $0.39-0.42$). This
suggests that the different retained feature sets contained similar
information, per dimension. And consequently, retained sets with larger
numbers of features contain more information.

| CB Base | CB Retained | S Base | S Retained | CB F. | S F.
---|---|---|---|---|---|---
Perc. Amodal | 0.43 | 0.74 | 0.48 | 0.77 | 25 | 126
Perc. Sight | 0.56 | 0.81 | 0.69 | 0.85 | 113 | 149
Perc. Touch | 0.13 | 0.64 | 0.14 | 0.69 | 60 | 59
Emission. Light | 0.64 | 0.88 | 0.49 | 0.85 | 55 | 94
Emission. Animate Sound | 0.25 | 0.69 | 0.32 | 0.76 | 88 | 62
Emission. Inanimate Sound | 0.10 | 0.60 | 0.16 | 0.64 | 32 | 32
Motion | 0.36 | 0.68 | 0.35 | 0.73 | 57 | 32
Table 2: Comparison of prediction accuracy (Spearman’s rho) when using full
feature set (baseline) or retained features sets (retained). In this
application, there was no test set.

The Dice coefficient analysis (Figure 2(a)) indicated that the highest
coefficients were found in the diagonal, which indexes cases of the same verb
type, across sighted and blind. The highest value (0.89) was found for light-
emission verbs (S_e_l, CB_e_l). Notably however, for Perception-Amodal,
Motion, and Emission-InanimateSounds, the overlap was weaker. There was one
case of strong overlap across verb types, seen in high Dice value for
Perception-Amodal and Perception-Sight verbs, but only for sighted
($Dice=0.42$). Importantly, for blind, this was not the case, with these verb
types showing effectively no overlap ($Dice=0.04$). This suggests that for
sighted, Perception-Amodal verbs are associated with a stronger visual
component. Many Dice coefficient were low, in the range of 0 to 0.05, with one
reaching zero. Motion and Emission-InanimateSounds show particularly low
values. To conclude, analyzing the overlap of features retained by pruning
strongly suggests that different verb categories map onto different embedding
dimensions.

(a)

(b)

Figure 2: Panel A: Dice coefficients for different combinations of pruned
sets. The size of the larger feature set was always matched to the size of the
smaller one prior to computing the coefficient. p_a, p_s, p_t indicate
Perception Action, Sight, and Touch verbs. e_l, e_sa, e_si indicate Emission
of Light, Animate Sounds, and Inanimate Sounds. ’m’ indicates Motion verbs.
Panel B: The number of GloVe features ($sum=300$) that appeared in zero or
more retained sets. No feature appeared in more than nine of the 14 sets.

Figure 2(b) shows there was no stable core of features that tended to be
retained across the pruning solutions. For each of the 300 GloVe features, we
computed how often it appeared across the 14 different retained sets. As the
Figure indicates, no feature appeared in more than nine of the 14 solutions,
and only 16 of the 300 appeared in 7-9 solutions. Thus, there is little
support for a core set of features that is relevant for all verb categories.
Interestingly, 19 of GloVe’s 300 features were not retained in any of the 14
pruning applications. To understand the meanings they contain, we identified
the top-20 words, from the dataset of Binder et al. (2016) for which these
features produced the highest activation sum. The obtained words were:
election, minister, banker, voter, arrested, politician, scream, party,
airport, gunshot, diplomat, spoke, rally, businessman, glass, cathedral,
driver, megaphone, commander, jury (with the first five scoring much higher
than the rest). As can be appreciated, most of these are terms related to
society, law and order, groups and social situations, which are indeed
unrelated to the verb categories we used.

###  3.3 Probing: Information in retained feature sets

As described in Section 2.3, from each set of pruned embeddings we learned a
PLSR solution and used LOOCV to produce a prediction of all Binder’s 534 words
on 65 semantic attributes. These 65 attributes were condensed (via averaging)
into 14 main semantic domains. As first indicator of quality of prediction,
for each semantic dimension we evaluated the correlation between the 534
predictions and the ground-truth ratings. Prediction was often very good, and
Figure 3(a) identifies several findings that speak to differences in the
information selected by pruning from blind and sighted data.

First, a methodological point is that accuracy of out-of-sample prediction of
Binder et al.’s data was not solely determined by the number of features in
the retained set. For example, Perception-Touch verbs selected an almost
identical number of retained features for blind and sighted (60, 59; see Table
2), but predictions based on features pruned for the blind provided better
correlations with ground-truth data almost across the board, and particularly
for the cognition domain. This means that similarity judgments for Perception-
Touch verbs made by blind select for GloVe features that more strongly
emphasize cognition-related information. Similarly, for sighted, Motion verbs
and Emission-InanimateSounds verbs retained the same number of features
($n=32$), with Motion better predicting the causal domain, and Emission-
InanimateSounds better predicting the social and cognition domain. A second
observation is that Perception-Sight verbs provided very good predictions,
across the board, for both blind and sighted. Finally, the bottom row in the
Figure (300d) presents prediction accuracy when using all 300 GloVe features
as reference. It can be seen that while none of the retained feature-subsets
reach these levels, the features retained by Perception-Sight verbs (for blind
and sighted; S_p_s; CB_p_s) and those retained by Perception-Amodal verbs (for
sighted) approximate this upper bound.

(a)

(b)

Figure 3: Panel A: Correlations between predicted and ground truth ratings
when retained feature sets were used as regressors for predicting Binder’s
words. p_a, p_s, p_t indicate Perception Action, Sight, and Touch verbs. e_l,
e_sa, e_si indicate Emission of Light, Animate Sounds, and Inanimate Sounds.
‘m’ indicates Motion verbs. The top row (300d) shows prediction accuracy when
all 300 GloVe dimensions are used in the PLSR model. Panel B: Clustering of
experimental conditions by prediction accuracy for the 65 Binder features.
Only in the case of Emission-InanimateSounds are congenitally blind and
sighted positioned in adjacent terminal leaves.

Table 3 shows which semantic domains were better predicted for sighted or
blind, per verb category. Note that these results do not reflect over-fitting
because they are based on predictions for out of sample data. For Perception-
Amodal and Perception-Sight verbs, predictions for all semantic domain were
more accurate when predicted from features retained for sighted. Controlling
for multiple comparisons within each verb type ($p<.003,n=14$ tests) we find
that for Perception-Amodal verbs all differences were significant and for
Perception-Sight, 10 of the 14.

For Perception-Touch verbs, five domains were better predicted for blind, but
none were better predicted for sighted. This suggests that for blind, touch-
related verbs are more strongly associated with other sensory functions, and
also cognitive properties. Verbs of Emission-AnimateSounds also selected for
much more information in blind, across both sensory and cognitive domains. We
note that these verbs included words that are often indicative of particular
mental states or event-types, e.g., moan, whimper, groan, growl, wail and it
may be that for the blind these types of auditory signals serve as a more
relevant way for communicating information than for sighted. This is in marked
contrast to Emission-InanimateSounds (e.g., sizzle, buzz, creak, crackle) for
which such attributions are less relevant. Indeed, for the latter we found
only one difference between the groups, where sighted better predicted the
motor domain.

As indicated above, the PLSR method produced predictions for all 65 semantic
attributes (see Appendix Figure 1). This provides an additional way to
understand the information contained in the retained features: given the 14
pruned datasets (7 verb types for blind/sighted) one can apply clustering
directly to the produced $14\times 65$ prediction matrix. The clustering
solution revealed that the terminal leaves rarely consisted of pairings of
Blind and Sighted (see Figure 3(b)). Only for inanimate sounds were blind and
sighted results positioned in adjacent terminal leaves (S_e_si, CB_e_si).
These results suggest that the information contained in the retained
embeddings, as assessed by the probing task, is not necessarily shared between
congenitally blind and sighted for a given verb type.

Domain | P. Amodal | P. Sight | P. Touch | E. Light | E. A.S | E. I.A.S | Motion
---|---|---|---|---|---|---|---
Vision | 14.03 | 4.56 | -3.75 | 5.10 | -5.90 | -0.36 | -4.13
Somatic | 8.82 | 4.50 | -2.32 | 2.94 | -5.33 | 0.93 | -0.57
Audition | 9.67 | 3.17 | 1.12 | 2.75 | -2.79 | 0.77 | -3.58
Gustation | 8.86 | 4.09 | -3.19 | 4.69 | -6.29 | 1.63 | -2.72
Olfaction | 9.13 | 3.56 | -3.04 | 4.09 | -5.30 | -0.44 | -3.85
Motor | 9.72 | 4.13 | -1.81 | 6.94 | -4.23 | 4.90 | 0.20
Spatial | 8.78 | 3.34 | -2.41 | 6.49 | -3.64 | 2.92 | -2.20
Temporal | 7.80 | 3.39 | -3.18 | 3.07 | -1.56 | -0.57 | -0.89
Causal | 9.00 | 2.51 | -1.96 | 4.45 | -4.57 | -2.00 | -2.70
Social | 9.42 | 4.17 | -2.39 | 1.53 | -3.05 | -2.20 | -4.76
Cognition | 6.55 | 2.93 | -4.13 | 1.12 | -5.81 | -1.93 | -3.46
Emotion | 8.65 | 2.88 | 1.47 | 5.92 | -4.89 | 0.68 | -3.15
Drive | 7.08 | 2.20 | -2.11 | 2.25 | -2.18 | 0.12 | -5.04
Attention | 7.69 | 3.00 | -0.68 | 2.84 | -3.85 | 0.86 | -3.23
Table 3: Differences in prediction accuracy of humanly derived semantic
dimensions from retained GloVe features applied to the Binder et al. dataset.
Each column corresponds to an analysis performed on features retained from
similarity judgments of one verb type (7 in total). Numeric values indicate T
statistics from t-tests comparing prediction accuracy from sighted and blind.
For each semantic dimension (rows), positive values indicate greater
prediction accuracy for sighted, and negative values greater accuracy for
blind. Values in bold are statistically significant (Bonferroni corrected for
14 contrasts in each column). P, Perception; E, Emission; E. A.S, Emission
Animate Sounds; E. I.A.S., Emission Inanimate Sounds

##  4 Discussion

By examining representations of different verb types from different domains,
we find there are some domains for which the two populations share a common
understanding, and some for which they strongly diverge. Our main contribution
is identifying the semantic dimensions of verb meanings that are differently
salient for blind and sighted.

Pruning presented good out of sample generalization which licensed the
analysis of the retained feature sets and their use in the probing task. In
most cases of out-of-sample testing, pruning matched or exceeded baseline
accuracy level with 30-150 of GloVe’s original 300 features. Dice analyses
revealed that for certain verb categories, pruning from blind and sighted
behavior retained similar features, but for others it retained different sets.
We then used probing to evaluate whether these different feature sets
described similar representational (latent) spaces. We used PLSR to predict 65
semantic properties for each left-out-word. This allowed us to evaluate the
correlations between word-level predictions and ground-truth ratings (Figures
3(a), 3(b)), as well as the raw distances (absolute error) between predictions
and ground-truth ratings (Table 3). We find that the two groups produced very
different prediction-accuracy profiles for Motion verbs and Perception-Amodal
verbs. Specifically, the blind’s representation of Motion verbs, as
operationalized by the set of retained features, performed much better than
sighted’s in predicting higher-level cognitive concepts related to Causality,
Cognition and Social information. Conversely, for Amodal verbs, sighted
individuals drew on richer information than blind. This was confirmed the
analysis of absolute prediction errors, which indicated that sighted
individuals link more semantic contents (compared to blind) with Amodal verbs,
whereas blind associate more information (compared to sighted) with Motion
verbs.

The study has several methodological implications. First, supervised pruning
allowed comparing representations of words for which pairwise similarity
ratings were not obtained. Whereas RSA relates different Representational
Dissimilarity Matrices (RDMs), it requires that the two RDMs are based on the
same set of words (or objects more generally). Second, to our knowledge, this
is the first application of a probing task to study interindividual
differences in knowledge representation. We note that a probing task differs
from canonical correlation and analysis and centered kernel alignment, methods
commonly used to assess representation similarity (for review, see Klabunde et
al., 2023). These methods quantify the relative similarity of representations
within two matrices. We could have directly applied these techniques to the
pruned word-embedding matrices (per verb type) to determine the similarity of
information captured by sighted and blind, without using any probing task
(i.e., without relying on data of Binder et al., 2016). However, the advantage
of probing is that it not only indicates for which verb types do the
representations of blind and sighted differ strongly or weakly, but also shows
which semantic dimensions are more accurately predicted by information from
one group or the other. It has been shown that differential predictions in
probing tasks are mathematically linked to similarity metrics obtained through
canonical correlation analysis (see Feng et al., 2020). An alternative
methodology could have involved having both blind and sighted individuals rate
verbs on experimenter-defined features (e.g., Kerr & Johnson, 1991). However a
limitation of this approach is that there is no guarantee that all chosen
features are cognitively relevant for representing the category. This means
that, in the limit, blind and sighted may provide very different feature-
ratings for a set of words (producing different representations in this
feature space), but still judge the similarity of the words in the same manner
because many of the features are less relevant.

A more general consideration relates to how (or whether) we should think of
word-embeddings as analogs of human semantic knowledge. The current study
suggests that word-embedding features do not capture a general
representational space as a result of their training. Instead, relatively
modular subspaces in the language model seem to match with different human
concepts. These subspaces are described by limited subsets of features, which
are highly sufficient for accounting for the geometry of particular human
categories.

Because of the in-depth analysis applied to each set of verbs, our study
relied on specific computational models and datasets, which produce several
limitations. First, our ability to study differences in saliency of semantic
dimensions was limited to those 65 dimensions made explicit in the probing
task. Using additional datasets for the probing task could offer a more
comprehensive view of differences between the blind and sighted groups.
Second, we present results derived from a single variant of a pruning
algorithm. Other algorithms may produce even more efficient pruning. Third, we
used GloVe as a language model, pre-trained from a specific corpus. There are
numerous other combinations of corpora and models trained on them, and they
tend to produce different representational spaces (Lenci et al., 2022).
However, GloVe is competitive with other word-embedding algorithms in
predicting human similarity judgments, and in any case, differences between
models appear minor (see Richie & Bhatia, 2021, their Figure 1).

In conclusion, we show that combining machine learning and behavioral methods
allows studying inter-individual differences in word meaning for speakers of
the same language. More specifically with respect to blindness, we find that
it introduces important changes in both the amount of information associated
with certain verb types, and the nature of the semantic dimensions related to
verb meaning. Interestingly, this occurs for some verb categories but not
others indicating that the organization of representation is domain or
concept-specific.

## References

  * Bedny et al. (2019) Marina Bedny, Jorie Koster-Hale, Giulia Elli, Lindsay Yazzolino, and Rebecca Saxe.  There’s more to “sparkle” than meets the eye: Knowledge of vision and light verbs among congenitally blind and sighted individuals.  _Cognition_ , 189:105–115, 2019.
  * Belinkov (2022) Yonatan Belinkov.  Probing classifiers: Promises, shortcomings, and advances.  _Computational Linguistics_ , 48(1):207–219, 2022.
  * Binder et al. (2016) Jeffrey R Binder, Lisa L Conant, Colin J Humphries, Leonardo Fernandino, Stephen B Simons, Mario Aguilar, and Rutvik H Desai.  Toward a brain-based componential semantic representation.  _Cognitive neuropsychology_ , 33(3-4):130–174, 2016.
  * Chersoni et al. (2021) Emmanuele Chersoni, Enrico Santus, Chu-Ren Huang, and Alessandro Lenci.  Decoding word embeddings with brain-based semantic features.  _Computational Linguistics_ , 47(3):663–698, 2021.
  * Feng et al. (2020) Yunzhen Feng, Runtian Zhai, Di He, Liwei Wang, and Bin Dong.  Transferred discrepancy: Quantifying the difference between representations.  _arXiv preprint arXiv:2007.12446_ , 2020.
  * Flechas Manrique et al. (2023) Natalia Flechas Manrique, Wanqian Bao, Aurelie Herbelot, and Uri Hasson.  Enhancing interpretability using human similarity judgements to prune word embeddings.  In _Proceedings of BlackboxNLP at EMNLP 2023_ , October 2023.
  * Friendly & Glucksberg (1970) Michael L Friendly and Sam Glucksberg.  On the description of subcultural lexicons: A multidimensional approach.  _Journal of Personality and Social Psychology_ , 14(1):55, 1970.
  * Kerr & Johnson (1991) Nancy H Kerr and Thomas H Johnson.  Word norms for blind and sighted subjects: Familiarity, concreteness, meaningfulness, imageability, imagery modality, and word associations.  _Behavior Research Methods, Instruments, & Computers_, 23:461–485, 1991.
  * Klabunde et al. (2023) Max Klabunde, Tobias Schumacher, Markus Strohmaier, and Florian Lemmerich.  Similarity of neural network models: A survey of functional and representational measures.  _arXiv preprint arXiv:2305.06329_ , 2023.
  * Landau & Gleitman (1985) Barbara Landau and Lila Gleitman.  _Language and experience: Evidence from the blind child_.  Harvard University Press, Cambridge, MA, 1985.
  * Lenci et al. (2013) Alessandro Lenci, Marco Baroni, Giulia Cazzolli, and Giovanna Marotta.  Blind: A set of semantic feature norms from the congenitally blind.  _Behavior Research Methods_ , 45:1218–1233, 2013.
  * Lenci et al. (2022) Alessandro Lenci, Magnus Sahlgren, Patrick Jeuniaux, Amaru Cuba Gyllensten, and Martina Miliani.  A comparative evaluation and analysis of three generations of distributional semantic models.  _Language resources and evaluation_ , 56(4):1269–1313, 2022.
  * Lewis et al. (2019) Molly Lewis, Martin Zettersten, and Gary Lupyan.  Distributional semantics as a source of visual knowledge.  _Proceedings of the National Academy of Sciences_ , 116(39):19237–19238, 2019.
  * Mamus et al. (2023) Ezgi Mamus, Laura J Speed, Lilia Rissman, Asifa Majid, and Aslı Özyürek.  Lack of visual experience affects multimodal language production: Evidence from congenitally blind and sighted people.  _Cognitive Science_ , 47(1):e13228, 2023.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  _Advances in Neural Information Processing Systems_ , 26, 2013.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pp. 1532–1543, 2014.
  * Richie & Bhatia (2021) Russell Richie and Sudeep Bhatia.  Similarity judgment within and across categories: A comprehensive model comparison.  _Cognitive Science_ , 45(8):e13030, 2021.
  * Shepard & Cooper (1992) Roger N Shepard and Lynn A Cooper.  Representation of colors in the blind, color-blind, and normally sighted.  _Psychological Science_ , 3(2):97–104, 1992.
  * Thompson et al. (2020) Bill Thompson, Seán G Roberts, and Gary Lupyan.  Cultural influences on word meanings revealed through large-scale semantic alignment.  _Nature Human Behaviour_ , 4(10):1029–1038, 2020.
  * Utsumi (2020) Akira Utsumi.  Exploring what is encoded in distributional word vectors: A neurobiologically motivated analysis.  _Cognitive Science_ , 44(6):e12844, 2020.
  * Ziv & Lempel (1977) Jacob Ziv and Abraham Lempel.  A universal algorithm for sequential data compression.  _IEEE Transactions on information theory_ , 23(3):337–343, 1977.

##  Appendix A Appendix

###  A.1 Verbs in each category from Bedny et al. 2019

  * •

Perception Sight verbs ($N=14$): These were based on vision: gawk, gaze,
glance, glimpse, leer, look, peek, peer, scan, see, spot, stare, view, watch.

  * •

Perception Touch verbs ($N=15$): These involved touching using the hands:
caress, dab, feel, grip, nudge, pat, pet, pinch, prod, rub, scrape, stroke,
tap, tickle, touch.

  * •

Perception Amodal verbs ($N=15$): These involved cognitive activities related
to knowledge acquisition: characterize, classify, discover, examine, identify,
investigate, learn, note, notice, perceive, question, recognize, scrutinize,
search, study.

  * •

Emission (of) light verbs ($N=15$): Refer to events in the environment sensed
through vision only: blaze, blink , flare, flash, flicker, gleam, glimmer,
glint, glisten, glitter, glow, shimmer, shine, sparkle, twinkle. These are all
associated with inanimate objects.

  * •

Emission (of) Animate Sounds verbs ($N=15$): Refer to events in the
environment sensed through audition only and that are produced by humans or
animals: bark, bellow, groan, growl, grumble, grunt, howl, moan, mutter,
shout, squawk, wail, whimper, whisper, yelp.

  * •

Emission (of) Inanimate Sounds verbs ($N=15$): Refer to events in the
environment sensed through audition only and that are produced by inanimate
objects: beep, boom, buzz, chime, clang, clank, click, crackle, creak, crunch,
gurgle, hiss, sizzle, squeak, twang.

  * •

Manner of Motion verbs ($N=15$): refer to acts involving physically moving or
changing position: bounce, float, glide, hobble, roll, saunter, scurry, skip,
slither, spin, strut, trot, twirl, twist, waddle.

###  A.2 Formal description of pruning and probing

Matrix $M$ represents the embedding of $n$ words onto $d$ features within a
language model ($M\in\mathbb{R}^{n\times d}$). The word-pair similarity
matrix, $Z$, is derived by computing the cosine similarity for each pair of
rows in $M$ using $Z_{ij}=\frac{M_{i}\cdot M_{j}}{\|M_{i}\|\cdot\|M_{j}\|}$,
with our focus being the upper triangle of matrix $Z$.

Our predictive target is the upper triangle of the human similarity judgment
matrix $H$. We assume that a better approximation of matrix $H$ can be
achieved by using a subset of columns in $M$, that is, by reducing the number
of features ($d^{\prime}<d$).

We therefore aim to identify matrix $M^{\prime}$, which constitutes a
submatrix of $M$, containing only selected columns. These selected columns are
denoted as $S$, where $S\subseteq\\{1,2,\ldots,d\\}$.
$M^{\prime}\in\mathbb{R}^{n\times|S|}$ represents this submatrix. A similarity
matrix $Z^{\prime}$ derived from $M^{\prime}$ is expressed as
$Z^{\prime}_{ij}=\frac{(M^{\prime}_{i})\cdot(M^{\prime}_{j})}{\|M^{\prime}_{i}\|\cdot\|M^{\prime}_{j}\|}$.

The primary objective of the pruning algorithm is to find the feature-subset
$S^{*}$ that maximizes the correlation, measured using Spearman Rho, between
the predicting matrix $Z^{\prime}(S^{*})$ and the target matrix $H$. To
achieve this, the algorithm identifies a subset $S^{*}$ that maximizes the
fit. The objective function is:

$\arg\max\,\text{Spearman}\big{(}\text{vec}_{\text{upper}}(Z^{\prime}(S)),\text{vec}_{\text{upper}}(H)\big{)}.$

As detailed in the main text, the pruning algorithm is supervised by
similarity judgments from two distinct matrices representing Congenitally
Blind and Sighted individuals. This yields two separate feature sets, denoted
as $S_{\text{blind}}^{*}$ and $S_{\text{sighted}}^{*}$, which optimize the
objective function. To evaluate whether these two sets of features define
different representational spaces we use the logic of a probing task. Here,
similarity between two representations is operationalized based on their
performance in a downstream task. Two similar representations should perform
similarly on a downstream prediction task.

Consider matrix $B$ as the representation of 534 target words in a
300-dimensional GloVe embedding space. The initial step involves creation of
two submatrices: $B_{\text{blind}}$, utilizing exclusively
$S_{\text{blind}}^{*}$, and $B_{\text{sighted}}$, utilizing exclusively
$S_{\text{sighted}}^{*}$. (We follow here the terminology of Feng et al.
(2020).) The downstream task is specified by matrix $Y$
($Y\in\mathbb{R}^{534\times 65}$) where $Y_{(i,:)}$ represents the 65 human
annotations for each word. This is the matrix to be predicted from the
submatrices of $B$.

Let $p_{W}(B_{\text{blind}})$ and
$p^{\prime}_{W^{\prime}}(B_{\text{sighted}})$ be two PLSR linear models
applied to $B_{\text{blind}}$ and $B_{\text{sighted}}$ respectively, with $W$
and $W^{\prime}$ representing the associated PLSR weights.

Using the least squares loss function $\ell(\hat{y},y)$, the parameters $W$
and $W^{\prime}$ are obtained by minimizing the following expressions, with
$n=534$:

$\displaystyle\hat{W}$
$\displaystyle=\underset{W}{\arg\min}\,\frac{1}{n}\,\sum_{i=1}^{n}\ell\left(p_{W}(B_{\text{blind}_{i}}),y_{i}\right)$
$\displaystyle\hat{W^{\prime}}$
$\displaystyle=\underset{W^{\prime}}{\arg\min}\,\frac{1}{n}\,\sum_{i=1}^{n}\ell\left(p^{\prime}_{W^{\prime}}(B_{\text{sighted}_{i}}),y_{i}\right)$

After obtaining $\hat{W}$ and $\hat{W^{\prime}}$, the best fitting parameters
for the two models (blind and sighted, respectively), it is possible to
compare the two PLSR models $p_{W}(B_{\text{blind}})$ and
$p^{\prime}_{W^{\prime}}(B_{\text{sighted}})$. This is done by quantifying by
how strongly their predictions diverge, a property which Feng at al. refer to
as Transferred Discrepancy. Two similar representations should produce very
similar predictions. Note that this approach identifies similar
representations even if the underlying vectors forming their bases exhibit
substantial differences. That is, even if the columns selected in
$S_{sighted}^{*}$ and $S_{blind}^{*}$ are different, as long as they reflect
the same representation, they should produce similar predictions.

As introduced by Feng et al. (2020), the Transferred Discrepancy between the
two representations is quantified by a divergence function $d()$, measuring
the distance between the predictions derived from the two PLSR models
(Equation 1).

$TD(B_{\text{blind}},B_{\text{sighted}};\,Y)=\frac{1}{n}\,\sum_{i=1}^{n}\,d(p_{W}(B_{\text{blind}_{i}}),p^{\prime}_{W^{\prime}}(B_{\text{sighted}_{i}}))$
(1)

We follow this approach with two minor modifications. First, rather than
defining $TD$ based on a single (point) mean statistic, we compare the two-
models’ predictions using a test of differences between distributions,
considering the entire set of predictions made by each model (534 predictions
in each case). To this end we use paired-sample T-tests with words $n=534$ as
unit of analysis. Second, we define the divergence function $d()$ as the
absolute distances between the predicted value of each model and the true
values. These absolute magnitudes are compared in order to understand whether
one produces more accurate predictions than the other. The results are
reported in Table 3.

Algorithm 1 Pruning

1: Inputs:

2: $SM_{HM}$: Similarity Matrix of human similarity judgments

3: $SM_{DNN}$: Similarity Matrix of similarity estimations derived from the
DNN by computing the cosine similarity between the embeddings of two words

4: Step 1: Compute baseline

5: Compute baseline $\mathit{\rho(SM_{HM},SM_{DNN})}$, from the full set of
features

6: Step 2: Rank features

7: Substep 1: Remove the first feature from all the original embeddings and
compute the corresponding similarity matrix $\mathit{SM_{DNNRED}}$

8: Substep 2: Compute the difference
$\mathit{D=\rho(SM_{HM},SM_{DNN})-\rho(SM_{HM},SM_{DNNRED})}$. $\rho$ is
Spearman’s rank correlation. Higher positive values for $\mathit{D}$ indicate
that the removed feature was important

9: Substep 3: Repeat the step above for all the possible $\mathit{N-1}$
feature subsets (where $\mathit{N=4096}$)

10: Substep 4: Rank the features based on $\mathit{D}$

11: Step 3: Construct pruned embeddings

12: Substep 1: Starting from an empty set of features, construct pruned
embeddings by iteratively reinserting one feature at a time, in descending
order of importance

13: Substep 2: Compute Spearman $\rho$ after each feature reinsertion and
store the values in the array $\mathit{a}$

14: Substep 3: Compute the maximum of $\mathit{a}$. Its position (index)
within the array delimits the set of features to be included in the pruned
embeddings

###  A.3 Prediction accuracy from pruning solutions

Figure 1: Prediction accuracy for each of the 65 features in Binder et al.,
from each of the 14 retained features sets obtained via pruning


# Integrating Mamba Sequence Model and Hierarchical Upsampling Network for
Accurate Semantic Segmentation of Multiple Sclerosis Legion

Kazi Shahriar Sanjid Md. Tanzim Hossain Md. Shakib Shahariar Junayed Dr.
Mohammad Monir Uddin Department of Mathematics and Physics, North South
University, Dhaka-1229, Bangladesh

###### Abstract

Integrating components from convolutional neural networks and state space
models in medical image segmentation presents a compelling approach to enhance
accuracy and efficiency. We introduce Mamba-HUNet, a novel architecture
tailored for robust and efficient segmentation tasks. Leveraging strengths
from Mamba-UNet and the lighter version of Hierarchical Upsampling Network
(HUNet), Mamba-HUNet combines convolutional neural networks local feature
extraction power with state space models long-range dependency modeling
capabilities. We first converted HUNet into a lighter version, maintaining
performance parity and then integrated this lighter HUNet into Mamba-HUNet,
further enhancing its efficiency. The architecture partitions input grayscale
images into patches, transforming them into 1-D sequences for processing
efficiency akin to Vision Transformers and Mamba models. Through Visual State
Space blocks and patch merging layers, hierarchical features are extracted
while preserving spatial information. Experimental results on publicly
available Magnetic Resonance Imaging scans, notably in Multiple Sclerosis
lesion segmentation, demonstrate Mamba-HUNet’s effectiveness across diverse
segmentation tasks. The model’s robustness and flexibility underscore its
potential in handling complex anatomical structures. These findings establish
Mamba-HUNet as a promising solution in advancing medical image segmentation,
with implications for improving clinical decision-making processes.

††footnotetext:  ∗Corresponding author<EMAIL_ADDRESS>
†These authors contributed equally to this work Keywords: Medical image
segmentation; State space model; Mamba; Multiple sclerosis; Convolutional
neural networks

## 1\. Introduction

Medical image segmentation, essential for diagnosing illnesses and planning
treatments, is fundamental to various biomedical applications. Traditionally,
manual segmentation has been labor-intensive, prompting a demand for automatic
methods. Deep learning has revolutionized segmentation methodologies in recent
years, significantly enhancing accuracy and efficiency across biomedical tasks
[1, 2, 3].

Deep learning-based networks, like U-Net and DeepLab, excel in hierarchical
feature extraction and are more parameter-efficient than fully connected
networks [4], particularly the U-Net architecture, have exhibited superior
performance in this domain [5]. U-Net’s symmetrical encoder-decoder style and
skip connections facilitate effective feature extraction at different levels,
enhancing information transformation efficiency [6]. Various enhancements to
U-Net, incorporating advanced techniques like dense connections, residual
blocks and attention mechanisms, have been explored, leading to modified
versions for segmenting Computerized Tomography (CT), Magnetic Resonance
Imaging (MRI) and ultrasound images [7, 8, 9, 10, 11, 12].

Initially designed for natural language processing, transformers have been
successfully adapted for image tasks, offering unique capabilities in
capturing global information [13]. Recent strides in computer vision owe much
to the advancements inspired by natural language processing, notably
exemplified by the Vision Transformer (ViT) and its introduction of a self-
attention mechanism for image recognition, adept at modeling long-range
dependencies [14]. Further adaptations like shift windows have expanded ViT’s
utility, leading to the emergence of the SwinTransformer, particularly suited
for dense prediction tasks in computer vision [15, 16, 17, 18]. Within medical
image segmentation, the fusion of ViT with U-Net architectures has birthed
hybrid and pure ViT-based U-Nets, as evidenced by notable examples such as
TransUNet and UNETR, harnessing ViT’s feature learning prowess in various
configurations [19, 20]. Moreover, transformers, initially tailored for
natural language processing, have found successful translation into image
tasks, offering unique capabilities in capturing global information. This
adaptability is strikingly evident in recent forays into 3D medical image
segmentation, where UNETR incorporates ViT as its encoder. At the same time,
SwinUNETR leverages the SwinTransformer, achieving promising outcomes through
multi-scale feature extraction and fusion [20]. Nonetheless, the computational
demands and reduced speed performance posed by the typically high resolution
of 3D medical images remain noteworthy challenges for transformer-based
methodologies.

Despite Transformers’ proficiency in capturing long-range dependencies, their
computational cost, especially for high-resolution biomedical images, remains
a challenge [21, 22]. State Space Models (SSMs), particularly Structured SSMs
(S4), offer a promising solution with their efficiency in processing long
sequences [23, 24, 25]. Enhanced models like Mamba and VMamba further extend
S4’s applicability to computer vision tasks [26, 27]. Due to the quadratic
scaling of the self-attention mechanism with input size, the computational
intensity of Transformers remains a challenge, particularly for high-
resolution biomedical images. The SSMs, notably S4 and its enhanced versions
like Mamba, have shown promise in efficiently modeling long sequences [28].
Mamba’s selective mechanism and hardware optimization have surpassed
Transformers on dense modalities such as language and genomics. SSMs offer a
potential solution, exhibiting linear complexity concerning input size while
establishing long-distance dependencies. Models like U-Mamba and SegMamba
present hybrid SSM-CNN architectures for medical image segmentation tasks.

This paper proposes Mamba-HUNet, a novel biomedical image segmentation
approach that integrates a hybrid dual-pathway multi-scale hierarchical
upsampling network and a state space model. Mamba-HUNet is designed to
effectively capture localized fine-grained features and long-range
dependencies in biomedical images while ensuring linear scaling in feature
size, thereby avoiding the quadratic complexity often associated with
Transformers. Its self-configuring capability allows seamless adaptation to
various datasets, enhancing scalability and flexibility across diverse
biomedical segmentation tasks. Quantitative and qualitative evaluations on
three distinct datasets demonstrate Mamba-HUNet’s superior performance,
surpassing Transformer-based networks significantly, thus paving the way for
future advancements in network designs to model long-range dependencies in
biomedical imaging efficiently. Additionally, we introduce a lightweight model
suitable for deployment as a web application, an open API that accepts input
images and returns disease labels, insights into the disease’s etiology and
suggested subsequent actions. This facilitates automatic annotation of new
instances, benefiting the research community and represents a fundamental
exploration into the potential of pure SSM-based models in medical image
segmentation.

Key Contributions:

  1. 1.

Developed a lightweight version of the HUNet model [29], maintaining
performance parity, thereby enhancing computational efficiency in medical
image segmentation tasks.

  2. 2.

Integrated the lightweight HUNet model into the Mamba architecture, resulting
in superior performance compared to existing methods, as demonstrated through
experimental evaluations.

  3. 3.

Achieved highly accurate predictions in multiple sclerosis lesion
segmentation, a critical task in medical radiology, highlighting the practical
significance and efficacy of the proposed Mamba-HUNet model in clinical
applications. Additionally, we developed a user-friendly website with an API,
facilitating seamless integration of the Mamba-HUNet model into radiologists
workflow for improved diagnosis and treatment planning.

## 2\. Preliminaries

### 2.1. State Space Models

The SSMs are a class of systems that relate a one-dimensional function or
sequence $u(t)\longmapsto y(t)\in\mathcal{R}$ and can be expressed as a linear
time-invariant (LTI) continuous-time system of the form [30]

$\displaystyle\dot{x}(t)$ $\displaystyle=\textbf{A}x(t)+\textbf{B}u(t),$ (1)
$\displaystyle y(t)$ $\displaystyle=\textbf{C}x(t)+\textbf{D}u(t),$

where $\textbf{A}\in\mathbb{R}^{n\times n}$ and
$\textbf{B}\in\mathbb{R}^{n\times 1}$ and $\textbf{C}\in\mathbb{R}^{1\times
n}$, are representing the system, input and output matrices respectively, of
the system. The vectors $x(t)$, $u(t)$ and $y(t)$, respectively known as
state, input and output of the system. These models possess advantageous
features like linear computational complexity per time step and parallelized
computation, which facilitate efficient training. Nonetheless, conventional
SSMs tend to require more memory compared to equivalent convolutional neural
networks (CNNs) and frequently face challenges such as vanishing gradients
during the training process, limiting their broader utility in general
sequence modeling.

The S4 [31] enhances conventional SSMs by introducing structured patterns in
the state matrix $A$ and employing a proficient algorithm. In particular, the
state matrix is constructed and initialized using the High-Order Polynomial
Projection Operator (HIPPO) [32], creating deep sequence models with enhanced
capabilities for efficient long-range reasoning. S4, as an innovative network
architecture, has outperformed Transformers [33] by a considerable margin in
the demanding Long Range Arena Benchmark [34]. Mamba introduces notable
advancements in discrete data modeling using SSMs, particularly in domains
such as text and genomics. Two primary enhancements distinguish Mamba from
traditional SSMs approaches. Firstly, it incorporates an input-dependent
selection mechanism, unlike conventional SSMs, invariant to time and input,
enabling efficient information extraction tailored to specific inputs. This is
achieved by parameterizing the SSMs parameters based on the input data.
Secondly, Mamba introduces a hardware-aware algorithm that scales linearly
with sequence length, facilitating recurrent model computation through
scanning, thereby outperforming previous methods on modern hardware. The Mamba
architecture, which integrates SSMs blocks with linear layers, is notably
simpler and has exhibited state-of-the-art performance across various long-
sequence domains such as language and genomics. This underscores its
significant computational efficiency in both the training and inference
stages.

### 2.2. Discrete-time SSM: The Recurrent Representation

When dealing with a discrete input sequence $(u_{0},u_{1},\ldots)$ instead of
a continuous function $u(t)$, it’s necessary to discretize equation 1 using a
step size $\Delta$, which denotes the granularity of the input. Essentially,
the inputs $u_{k}$ can be seen as samples taken from an underlying continuous
signal $u(t)$, where $u_{k}=u(k\Delta)$.

To discretize the continuous-time State Space Model (SSM), we adopt the
bilinear method, as established in previous studies such as [35]. This method
transforms the state matrix A into an approximation $\mathbf{\overline{A}}$.
The discrete SSM is

$\displaystyle x_{k}$
$\displaystyle=\mathbf{\overline{A}}x_{k}+\mathbf{\overline{B}}u_{k}$ (2)
$\displaystyle\mathbf{\overline{A}}$
$\displaystyle=(\mathbf{\overline{I}}-\Delta/2.\mathbf{\overline{A}})^{-1}(\mathbf{\overline{I}}+\Delta/2.\mathbf{\overline{A}})$
$\displaystyle y_{k}$ $\displaystyle=\mathbf{\overline{C}}x_{k}$
$\displaystyle\mathbf{\overline{B}}$
$\displaystyle=(\mathbf{\overline{I}}-\Delta/2.\mathbf{\overline{A}})^{-1}\Delta\mathbf{\overline{B}}$
$\displaystyle\mathbf{\overline{C}}$ $\displaystyle=\mathbf{\overline{C}}$

In Equation 2, we now treat $u_{k}$ as a sequence-to-sequence mapping to
$y_{k}$, departing from its previous function-to-function interpretation.
Additionally, the state equation now recurs in $x_{k}$, enabling the discrete
State-Space Model (SSM) to be computed like a Recurrent Neural Network (RNN).
Specifically, $x_{k}\in\mathbb{R}^{N}$ serves as a hidden state with a
transition matrix denoted by $\mathbf{\overline{A}}$.

Throughout this paper, we denote the discretized SSM matrices by
$\mathbf{\overline{A}},\mathbf{\overline{B}},\ldots$, which are determined by
Equation 2. It’s worth noting that these matrices depend not only on A but
also on a step size $\Delta$. For the sake of brevity, we omit explicit
mention of this dependence in our notation when it’s understood.

### 2.3. Training SSMs: The Convolutional Representation

The recurrent SSM referenced as 2 poses practical challenges for training on
contemporary hardware because of its sequential nature. However, there exists
a widely acknowledged link between linear time-invariant (LTI) SSMs like 1 and
continuous convolutions. Consequently, 2 can be reformulated as a discrete
convolution.

To simplify, assume the initial state is $x_{-1}=0$. Then unrolling 2
explicitly yields

$\displaystyle x_{0}$ $\displaystyle=\mathbf{\overline{B}}u_{0}$
$\displaystyle x_{1}$
$\displaystyle=\mathbf{\overline{AB}}u_{0}+\mathbf{\overline{B}}u_{1}$
$\displaystyle x_{2}$
$\displaystyle=\mathbf{\overline{A}}^{2}\mathbf{\overline{B}}u_{0}+\mathbf{\overline{AB}}u_{1}+\mathbf{\overline{B}}u_{2}$
$\displaystyle\ldots$ $\displaystyle y_{0}$
$\displaystyle=\mathbf{\overline{CB}}u_{0}$ $\displaystyle y_{1}$
$\displaystyle=\mathbf{\overline{CAB}}u_{0}+\mathbf{\overline{CB}}u_{1}$
$\displaystyle y_{2}$
$\displaystyle=\mathbf{\overline{CA}}^{2}\mathbf{\overline{B}}u_{0}+\mathbf{\overline{CAB}}u_{1}+\mathbf{\overline{CB}}u_{2}$
$\displaystyle\ldots$

This can be vectorized into a convolution 3 with an explicit formula for the
convolution kernel 4.

$\displaystyle y_{k}$
$\displaystyle=\mathbf{\overline{CA}}^{k}\mathbf{\overline{B}}u_{0}+\mathbf{\overline{CA}}^{k-1}\mathbf{\overline{B}}u_{1}+\ldots+\mathbf{\overline{CAB}}u_{k-1}+\mathbf{\overline{CB}}u_{k},$
(3) $\displaystyle y$ $\displaystyle=\mathbf{\overline{K}}*u.$

$\displaystyle\mathbf{\overline{K}}\in\mathbb{R}^{L}:=\mathcal{K}_{L}(\mathbf{\overline{A}},\mathbf{\overline{B}},\mathbf{\overline{C}}):=(\mathbf{\overline{CA}}^{i}\mathbf{\overline{B}})_{i\in[L]}=(\mathbf{\overline{CB}},\mathbf{\overline{CAB}}\ldots,\mathbf{\overline{CA}}^{L-1}\mathbf{\overline{B}}).$
(4)

In other words, equation 3 represents a solitary (non-circular) convolution
and can be efficiently calculated using FFTs, as long as
$\mathbf{\overline{K}}$ is provided. However, determining
$\mathbf{\overline{K}}$ in 4 is complex and forms the core of our technical
advancements in Section 3. We refer to $\mathbf{\overline{K}}$ as the SSM
convolution kernel or filter.

Figure 1: Overview of the Mamba-HUNet architecture. Figure 2: VSS block
architecture

## 3\. Mamba-HUNet

The selective space architecture (SSA) is a foundational component in the
model, orchestrating a series of operations to prepare input tensors and
parameters for selective scan processing. The SSA optimally structures the
data to leverage selective attention mechanisms efficiently through chunking,
convolution and reshaping. Discretizing the continuous-time selective SSM
involves converting the model’s representation from a continuous to a
discrete-time domain. This process samples the continuous-time signal at
specific intervals, resulting in a sequence of discrete-time states. This
discretization allows the model to approximate its state matrix, enabling
sequence-to-sequence mapping and recurrent computations like those performed
by recurrent neural networks (RNNs). Essentially, it empowers the model to
process input sequences over time and maintain internal states across
different time steps, mirroring the behavior of RNNs. The selective scan
operation within the 2D-selective-scan (SS2D) model further enhances this
process, enabling selective aggregation of informative features while managing
computational complexity effectively. By integrating bidirectional processing
information and selectively attending to key spatial locations and feature
channels, the selective scan operation enriches the model’s ability to capture
intricate patterns and dependencies within the data. This selective
aggregation mechanism enhances contextual understanding and boosts
computational efficiency by minimizing redundant computations. The integration
of selective scan with the HUNet layer in the subsequent stages of the model
ensures seamless feature integration across different architectural components

Incorporating a lighter version of the HUNet model into the methodology
involves adapting the architecture to reduce computational complexity and
enhance efficiency without compromising performance. This consists of
modifying the original HUNet architecture by implementing strategies such as
reducing the number of parameters, adjusting dropout rates and simplifying
convolutional layers while maintaining accuracy in segmentation tasks. By
integrating insights from the lighter HUNet version, the Mamba-HUNet
architecture can effectively handle diverse datasets and complex anatomical
structures while offering improved computational efficiency and streamlined
processing. This adaptation seamlessly integrates into the website API,
providing a robust solution for medical image segmentation tasks with enhanced
efficiency and performance, as detailed in Figure 3.

The proposed lighter Mamba-HUNet architecture integrates features from the
Mamba-UNet [36] and HUNet models to address the complexities of medical image
segmentation tasks. The architecture begins by partitioning the input 2D grey-
scale image into patches. These patches are then transformed into a 1-D
sequence, facilitating efficient processing. An initial linear embedding layer
adapts the feature dimensions, after which the data flows through multiple
Visual State Space (VSS) blocks and patch merging layers. Inspired by Mamba-
UNet, the VSS blocks leverage depth-wise convolution and softmax activation
for feature extraction, avoiding positional embedding to maintain a
streamlined structure. The encoder generates hierarchical features at
different resolutions, preserving spatial information while reducing the token
count and doubling feature dimensions through patch merging layers. Each
encoder level utilizes skip connections to blend multi-scale features and
maintain spatial detail. The decoder mirrors the encoder’s structure,
employing patch-expanding layers for upsampling and enhancing spatial
resolution. The encoder and decoder incorporate VSS blocks to facilitate
feature learning and reconstruction. The final segmentation masks are produced
through convolutional layers, with the number of output channels corresponding
to the classes to be segmented. Batch normalization and dropout regularization
are applied throughout the architecture to enhance model stability and prevent
overfitting. Incorporating insights from HUNet, the Mamba-HUNet architecture
integrates additional convolutional layers and transposed convolutional layers
within the upsampling block, bolstering the model’s ability to capture
intricate features and facilitate accurate segmentation. These enhancements
empower the model to handle diverse datasets and complicated anatomical
structures adeptly. By combining the strengths of both architectures, Mamba-
HUNet offers a robust solution for medical image segmentation tasks, with the
flexibility to handle diverse datasets and complex anatomical structures
effectively.

Figure 3: Overall workflow

## 4\. Experimental Procedures

### 4.1. Dataset Description

Figure 4: Example of cross-sectional brain MRI pictures and corresponding
masks from an individual with MS

We have used $2$ publicly available datasets for the experiments. The first
dataset [37] comprises MRI scans from $60$ individuals diagnosed with Multiple
Sclerosis (MS). Three experts in neurology-radiology conducted the lesions’
segmentation manually, ensuring consensus across T1-weighted, T2-weighted and
FLAIR MRI sequences. As mentioned, the dataset consists of 3D FLAIR MRI scans
obtained from the patients. Among them, $46$ are female and $14$ are male,
with an average age of $33$ years, ranging from $15$ to $56$ years. These
scans were conducted between $2019$ and $2020$ using a $1.5$ Tesla scanner
across $20$ different medical centers. The second dataset [38] consists of
FLAIR images from $100$ individuals with MS. These images were obtained from
$15$ different MRI scanners, with six operating at $1.5T$ and nine at $3T$.
Among these scanners were three from GE, six from Philips and six from
Siemens. The images exhibit size and voxel spacing variations, standardized to
the dataset’s median spacing of $0.977\times 0.977\times 0.530\>mm^{3}$ before
model training. Each patient underwent two scans, spaced $1$–$3$ years apart,
resulting in $200$ images. Annotations focused solely on new lesions
identified at the second time point, omitting delineation of pre-existing
lesions or changes in size. Four neuroradiologists independently annotated
each patient, with a consensus that a new lesion mask was generated for model
training and evaluation purposes.

### 4.2. Data Processing Steps

The initial processing involved converting volumetric MRI data into individual
2D slices, facilitating focused analysis per slice and isolating potential
lesions.

Standardizing the size of 2D slices to $256\times 256$ pixels was crucial to
address variations in image sizes resulting from different acquisition
parameters and equipment.

Image sharpening and intensity normalization were integrated into the
preprocessing pipeline to enhance MRI data. This technique accentuated edges
and subtle abnormalities while maintaining data integrity and mitigating
artifacts and noise.

The dataset underwent partitioning, distributing $70\%$ of the images to the
training set, while $15\%$ each was designated for validation and testing
purposes.

Collaboration with medical professionals specializing in MRI data provided
valuable insights into radiological nuances. Their expertise guided
preprocessing steps such as noise reduction and intensity normalization,
ensuring data reliability for deep learning-based lesion segmentation. This
interdisciplinary approach enhances the model’s understanding of MRI
intricacies and improves segmentation accuracy.

### 4.3. Evaluation Metrics

Within the scope of this study, the evaluation of segmentation algorithms
extended to encompass several critical metrics, Intersection over Union
($IoU$), Dice Similarity Coefficient ($DSC$), Hausdorff distance at $95\%$
percentile ($HD_{95}$), $Sensitivity$ and $Specificity$. These metrics
collectively provided a comprehensive assessment of the algorithm’s
performance.

In accordance with assessing whether the model was prone to overfitting,
underfitting, or achieving an optimal fit, the results featured the
presentation of the $IoU$ for prostate segmentation tasks.

$\displaystyle IoU$ $\displaystyle=\frac{Intersection\>Area}{Union\>Area}$
$\displaystyle=\frac{TP}{TP+FP+FN}$

where, Intersection Area is equivalent to True Positive ($TP$) and Union Area
is equivalent to $TP$ \+ False Positive ($FP$) + False Negative ($FN$).

$DSC$, a metric based on area overlap, quantified the agreement between manual
and algorithmic segmentations:

$DSC=\frac{2TP}{2TP+FP+FN}$

$HD_{95}$ is a metric used to quantify the similarity between two sets of
points, often employed in image processing and computer vision tasks such as
image registration or object recognition. It measures the maximum distance
from each point in one set to the nearest point in the other set, considering
only the closest $95\%$ of points. Mathematically, it can be represented as:

$HD_{95}(A,B)=max_{a\in A}\\{min_{b\in B}\\{d(a,b)\\}\\}$

where $A$ and $B$ are sets of points, $a$ and $b$ are individual points in $A$
and $B$ respectively and $d(a,b)$ is the distance between point $a$ and point
$b$. The $95\%$ indicates that only the distances corresponding to the $95$
percentile of the distances are considered in this calculation.

$Sensitivity$ measures the proportion of actual positive cases that a
diagnostic test or model correctly identifies. It is calculated as the ratio
of true positive cases to the sum of true positive and false negative cases.
Mathematically, $Sensitivity$ can be represented as:

$Sensitivity=\frac{TP}{TP+FN}$

$Specificity$ measures the proportion of actual negative cases correctly
identified by a diagnostic test or model. It is calculated as the ratio of
true negative cases to the sum of true negative and false positive cases.
Mathematically, $Specificity$ can be represented as:

$Specificity=\frac{TN}{TN+FP}$

where $TN$ is the number of true negatives.

## 5\. Result and Discussion

The evaluation conducted on the test dataset reveals that Mamba-HUNet
outperforms all other models across multiple performance metrics. Notably, it
achieves an $IoU$ of $0.8536$, a $HD_{95}$ of $2.2518$, a $DSC$ of $0.9287$, a
$Sensitivity$ of $0.9294$ and a $Specificity$ of $0.9865$. These results
highlight the proposed architecture’s superior segmentation accuracy and
robustness, indicating its potential for advancing medical image analysis.

The success of Mamba-HUNet can be attributed to its unique design, which
integrates features from both Mamba-UNet and HUNet models. By leveraging the
linear scaling advantage of Mamba and the global context understanding of
HUNet, the architecture effectively captures both local features and long-
range dependencies within medical images. Furthermore, the meticulous
architecture design, including the Contraction and Expansive Paths, allows for
efficient feature extraction and spatial resolution restoration, enhancing
segmentation outcomes.

Accurate segmentation of anatomical structures and lesions is crucial for
diagnosis and treatment planning in medical imaging. The robust performance of
Mamba-HUNet across diverse datasets and complex anatomical structures
underscores its potential for real-world applications in clinical settings.
Moreover, qualitative results of lesion segmentation demonstrate the model’s
ability to accurately delineate lesions in medical images, providing further
evidence of its efficacy in capturing intricate features and facilitating
precise segmentation. Overall, the study’s results showcase the effectiveness
of Mamba-HUNet as a robust solution for medical image segmentation tasks, with
implications for advancing the field of medical image analysis and improving
clinical decision-making processes.

Additionally, Table 1 summarizes the quantitative performance metrics of
various segmentation models, clearly comparing Mamba-HUNet with other state-
of-the-art architectures. Furthermore, Figure 5 showcases sample predicted
images with segmentation masks, offering visual insights into the model’s
segmentation accuracy and demonstrating its potential for clinical
applications. These supplementary materials augment the discussion, providing
comprehensive evidence of the efficacy and potential impact of Mamba-HUNet in
medical image analysis.

Test Image | Test Label | Predicted Label
---|---|---
|  |
|  |
|  |
|  |
Figure 5: Output segmentation results of the proposed method Model | IoU$\uparrow$ |  HD95 $\downarrow$ | DSC$\uparrow$ | Sensitivity$\uparrow$ | Specificity$\uparrow$
---|---|---|---|---|---
U-Net | $0.8170$ |  $3.2158$ | $0.8597$ | $0.9045$ | $0.9395$
Swin-Unet | $0.7947$ |  $4.5927$ | $0.8437$ | $0.8921$ | $0.9186$
Mamba-UNet | $0.8322$ |  $2.3875$ | $0.8761$ | $0.9178$ | $0.9787$
VM-Unet | $0.8039$ |  $2.9006$ | $0.8749$ | $0.9135$ | $0.9531$
Mamba-HUNet | $0.8536$ |  $2.2518$ | $0.9287$ | $0.9294$ | $0.9865$
Table 1: Performance of different models on the test dataset. $\uparrow$
denotes higher is better and $\downarrow$ denotes lower is better

## 6\. Conclusion

This paper introduced Mamba-HUNet, a novel architecture for robust and
efficient medical image segmentation tasks. By integrating components from
CNNs and SSMs, Mamba-HUNet harnesses the strengths of both approaches to
achieve superior segmentation accuracy and robustness.

Inspired by Mamba-UNet and HUNet models, the architecture combines CNNs’ local
feature extraction power with SSMs’ long-range dependency modeling
capabilities. Through careful design, including partitioning input images into
patches and leveraging VSS blocks, Mamba-HUNet effectively captures both local
details and global contexts within medical images. Skip connections and HUNet-
inspired upsampling blocks enhance feature blending and spatial resolution
restoration, contributing to superior segmentation outcomes.

Evaluation of MRI scans, particularly in Multiple Sclerosis lesion
segmentation, demonstrates Mamba-HUNet’s effectiveness across diverse datasets
and complex anatomical structures. The model outperforms state-of-the-art
architectures, achieving remarkable performance metrics such as $IoU$, $DSC$,
$HD_{95}$, $Sensitivity$ and $Specificity$. These results underscore the
potential of Mamba-HUNet in advancing medical image analysis, with
implications for improving clinical decision-making processes.

The success of Mamba-HUNet lies in its superior performance and potential for
real-world applications in clinical settings. Accurate segmentation of
anatomical structures and lesions is crucial for diagnosis and treatment
planning and the architecture demonstrates promising results in this regard.
Furthermore, qualitative assessments highlight the model’s ability to
accurately delineate lesions in medical images, further affirming its efficacy
in capturing intricate features and facilitating precise segmentation.

## References

  * [1] P. Bilic, P. Christ, H. B. Li, E. Vorontsov, A. Ben-Cohen, G. Kaissis, A. Szeskin, C. Jacobs, G. E. H. Mamani, G. Chartrand _et al._ , “The liver tumor segmentation benchmark (lits),” _Medical Image Analysis_ , vol. 84, p. 102680, 2023.
  * [2] N. Heller, F. Isensee, K. H. Maier-Hein, X. Hou, C. Xie, F. Li, Y. Nan, G. Mu, Z. Lin, M. Han _et al._ , “The state of the art in kidney and kidney tumor segmentation in contrast-enhanced ct imaging: Results of the kits19 challenge,” _Medical image analysis_ , vol. 67, p. 101821, 2021.
  * [3] M. T. Hossain, M. S. S. Junayed, K. S. Sanjid, A. H. Rohan, M. A. Khan, S. A. Haque, and M. M. Uddin, “Automated segmentation of multiple sclerosis lesions using deep learning,” in _2023 26th International Conference on Computer and Information Technology (ICCIT)_. IEEE, 2023, pp. 1–6.
  * [4] Y. LeCun, Y. Bengio _et al._ , “Convolutional networks for images, speech, and time series,” _The handbook of brain theory and neural networks_ , vol. 3361, no. 10, p. 1995, 1995.
  * [5] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [6] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18_. Springer, 2015, pp. 234–241.
  * [7] N. Ibtehaz and M. S. Rahman, “Multiresunet: Rethinking the u-net architecture for multimodal biomedical image segmentation,” _Neural networks_ , vol. 121, pp. 74–87, 2020.
  * [8] X. Li, H. Chen, X. Qi, Q. Dou, C.-W. Fu, and P.-A. Heng, “H-denseunet: hybrid densely connected unet for liver and tumor segmentation from ct volumes,” _IEEE transactions on medical imaging_ , vol. 37, no. 12, pp. 2663–2674, 2018.
  * [9] O. Oktay, J. Schlemper, L. L. Folgoc, M. Lee, M. Heinrich, K. Misawa, K. Mori, S. McDonagh, N. Y. Hammerla, B. Kainz _et al._ , “Attention u-net: Learning where to look for the pancreas,” _arXiv preprint arXiv:1804.03999_ , 2018.
  * [10] Z. Wang, Z. Zhang, and I. Voiculescu, “Rar-u-net: a residual encoder to attention decoder by residual connections framework for spine segmentation under noisy labels,” in _2021 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2021, pp. 21–25.
  * [11] Y. Zhang, L. Yuan, Y. Wang, and J. Zhang, “Sau-net: efficient 3d spine mri segmentation using inter-slice attention,” in _Medical Imaging with Deep Learning_. PMLR, 2020, pp. 903–913.
  * [12] Z. Zhou, M. M. R. Siddiquee, N. Tajbakhsh, and J. Liang, “Unet++: Redesigning skip connections to exploit multiscale features in image segmentation,” _IEEE transactions on medical imaging_ , vol. 39, no. 6, pp. 1856–1867, 2019.
  * [13] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need in advances in neural information processing systems, 2017,” _Search PubMed_ , pp. 5998–6008.
  * [14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [15] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, J. Ning, Y. Cao, Z. Zhang, L. Dong _et al._ , “Swin transformer v2: Scaling up capacity and resolution,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 12 009–12 019.
  * [16] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 10 012–10 022.
  * [17] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 3202–3211.
  * [18] Z. Xie, Y. Lin, Z. Yao, Z. Zhang, Q. Dai, Y. Cao, and H. Hu, “Self-supervised learning with swin transformers,” _arXiv preprint arXiv:2105.04553_ , 2021.
  * [19] J. Chen, Y. Lu, Q. Yu, X. Luo, E. Adeli, Y. Wang, L. Lu, A. L. Yuille, and Y. Zhou, “Transunet: Transformers make strong encoders for medical image segmentation,” _arXiv preprint arXiv:2102.04306_ , 2021.
  * [20] A. Hatamizadeh, Y. Tang, V. Nath, D. Yang, A. Myronenko, B. Landman, H. R. Roth, and D. Xu, “Unetr: Transformers for 3d medical image segmentation,” in _Proceedings of the IEEE/CVF winter conference on applications of computer vision_ , 2022, pp. 574–584.
  * [21] J. Ma, F. Li, and B. Wang, “U-mamba: Enhancing long-range dependency for biomedical image segmentation,” _arXiv preprint arXiv:2401.04722_ , 2024.
  * [22] Z. Xing, T. Ye, Y. Yang, G. Liu, and L. Zhu, “Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation,” _arXiv preprint arXiv:2401.13560_ , 2024.
  * [23] A. Gu, _Modeling Sequences with Structured State Spaces_. Stanford University, 2023.
  * [24] H. Mehta, A. Gupta, A. Cutkosky, and B. Neyshabur, “Long range language modeling via gated state spaces,” _arXiv preprint arXiv:2206.13947_ , 2022.
  * [25] J. Wang, W. Zhu, P. Wang, X. Yu, L. Liu, M. Omar, and R. Hamid, “Selective structured state-spaces for long-form video understanding,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 6387–6397.
  * [26] A. Gu and T. Dao, “Mamba: Linear-time sequence modeling with selective state spaces,” _arXiv preprint arXiv:2312.00752_ , 2023.
  * [27] Y. Liu, Y. Tian, Y. Zhao, H. Yu, L. Xie, Y. Wang, Q. Ye, and Y. Liu, “Vmamba: Visual state space model,” _arXiv preprint arXiv:2401.10166_ , 2024.
  * [28] A. Gu, I. Johnson, K. Goel, K. Saab, T. Dao, A. Rudra, and C. Ré, “Combining recurrent, convolutional, and continuous-time models with linear state space layers,” _Advances in neural information processing systems_ , vol. 34, pp. 572–585, 2021.
  * [29] K. S. Sanjid, M. S. S. Junayed, M. T. Hossain, Y.-L. Wang, M. M. Uddin, and S. A. Haque, “From pixels to pathology: A novel dual-pathway multi-scale hierarchical upsampling network for mri-based prostate zonal segmentation,” _Available at SSRN 4725376_.
  * [30] A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” _CoRR_ , vol. abs/2111.00396, 2021. [Online]. Available: https://arxiv.org/abs/2111.00396
  * [31] A. Gu, K. Goel, and C. Ré, “Efficiently modeling long sequences with structured state spaces,” _arXiv preprint arXiv:2111.00396_ , 2021.
  * [32] A. Gu, T. Dao, S. Ermon, A. Rudra, and C. Ré, “Hippo: Recurrent memory with optimal polynomial projections,” _Advances in neural information processing systems_ , vol. 33, pp. 1474–1487, 2020.
  * [33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [34] Y. Tay, M. Dehghani, S. Abnar, Y. Shen, D. Bahri, P. Pham, J. Rao, L. Yang, S. Ruder, and D. Metzler, “Long range arena: A benchmark for efficient transformers,” _arXiv preprint arXiv:2011.04006_ , 2020.
  * [35] A. Tustin, “A method of analysing the behaviour of linear systems in terms of time series,” _Journal of the Institution of Electrical Engineers-Part IIA: Automatic Regulators and Servo Mechanisms_ , vol. 94, no. 1, pp. 130–142, 1947.
  * [36] Z. Wang, J.-Q. Zheng, Y. Zhang, G. Cui, and L. Li, “Mamba-unet: Unet-like pure visual mamba for medical image segmentation,” _arXiv preprint arXiv:2402.05079_ , 2024.
  * [37] A. M. Muslim, S. Mashohor, G. Al Gawwam, R. Mahmud, M. binti Hanafi, O. Alnuaimi, R. Josephine, and A. D. Almutairi, “Brain mri dataset of multiple sclerosis with consensus manual lesion segmentation and patient meta information,” _Data in Brief_ , vol. 42, p. 108139, 2022.
  * [38] _MSSEG-2 challenge proceedings: Multiple sclerosis new lesions segmentation challenge using a data management and processing infrastructure_ , Strasbourg, France, Sep. 2021. [Online]. Available: https://inria.hal.science/hal-03358968