0

# I Learn to Diffuse, or Data Alchemy 101: a Mnemonic Manifesto

Victor Schetinger
TU Wien  e-mail<EMAIL_ADDRESS>Velitchko Filipov
TU Wien  Ignacio Pérez-Messina
TU Wien  Ethan Smith
University of Florida  e-mail<EMAIL_ADDRESS>Rodrigo Oliveira de
Oliveira
Unochapecó  e-mail<EMAIL_ADDRESS>
###### Abstract

In this manifesto, we put forward the idea of data alchemy as a narrative
device to discuss storytelling and transdisciplinarity in visualization. If
data is the prima materia of modern science, how does one perform the Great
Work? We use text-to-image diffusion-based generative art to develop the
concept, and structure our argument in ten propositions, as if they were ten
issues of a comic novel on data alchemy: Ad Disco Diffusionem. To follow the
argument, the reader must immerse themselves in our miro board, and navigate a
multimedia semiotic topology that includes comics, videos, code demos, and
ergotic literature in a true alchemic sense. By accessing this paradigm one
might find new sources of inspiration for scientific inquiry in familiar
places, or get lost in the creative exploration of the unknown. Our colorful,
sometimes poetic, exposition should not distract the reader from the
seriousness of the ideas discussed, but ultimately it is about the journey.

‘The Metamorphosis of Data, as it permeates different strata and arrives at us
through many channels. In overlay, our interpretations, models, and
representations that try to create meaning from the patterns, effectively
projecting and actualizing them, beautiful detailed, hyperrealism, by Escher,
Kandinsky, Pollock”

Introduction

The paper “Perception! Immersion! Empowerment! Superpowers as Inspiration for
Visualization”[45], which was among the best papers on IEEE VIS 2021, proposes
a very interesting argumentation format. The authors bring forward the
framework of superpowers from fiction, which is a part of our modern
mythology, and use it to discuss visualization theory. Additionally, they use
the medium of comics for their storytelling capabilities, powerful explanatory
power[30], and to provide a thematic harmonization.

We expand on this idea, and use the experimental freedom provided by alt.vis
to expose the concept of data alchemy. We draw from a rich historical and
cultural literature on alchemy, already so colorful and intriguing that
fiction is added only when needed. The perspective we try to paint views
alchemical thinking as connected to our intuitive thinking[17], but also at
the source of technology and innovation.

It is our belief that new technologies such as text-to-image generation have
an immense potential for extending human creativity, and can be applied in a
variety of domains, specially visualization. Our exposition tries to both
justify this belief and show in practice how they can be leveraged for content
creation, supporting our storytelling and expository capabilities. We created
a fictional comic novel about data alchemy called “Ad Diffusionem Disco” (I
learn to diffuse, in latin), in reference to Disco Diffusion (DD)[1], one of
the most important public projects for text-to-image generation based on
diffusion[15] due to its openness and active community.

Figure 1: The Metamorphosis of Data, as it permeates different strata and
arrives at us through many channels. In overlay, our interpretations, models,
and representations that try to create meaning from the patterns, effectively
projecting and actualizing them. This figure was the teaser on a previous
version of this paper, and its description was used to generate the current
teaser (Fig. I Learn to Diffuse, or Data Alchemy 101: a Mnemonic Manifesto).
While this image has a more controlled artistic direction and a clearer
message, Fig. I Learn to Diffuse, or Data Alchemy 101: a Mnemonic Manifesto
was aesthetically surprising and seemed to capture the subject on its own way.

##  1 What do we talk about when we talk about Alchemy

> It is too clear and so it is hard to see.
> A dunce once searched for a fire with a lighted lantern.
> Had he known what fire was,
> He could have cooked his rice much sooner.
>
> \- The Gateless Gate[22],
> Mumon’s commentary on Koan 7

The popularly recognized pragmatics of alchemy is concerned with the
transmutation of substances (e.g. lead into gold). In this sense it is
generally considered as the precursor of modern chemistry[21]. However, from a
philosophical and psychological[24] standpoint it is a holistic framework for
the acquisition of knowledge about the natural world through experimentation,
and organization and transmission of this knowledge through clever mnemonic
devices. Without the benefit of our modern epistemic, cognitive, and
technological tools, this was a daunting task.

We are not talking exclusively about the laboratory alchemy of the XVI century
onward, as funded by emperor Rudolf II or practiced by Sir Isaac Newton. A
fair treatment of alchemy must include its extensions and varieties practiced
in ancient Greece, the Islamic world, China, Egypt, and to an extent even folk
medicine, because they operate on similar mechanisms. We see alchemy as
science without organs, as the employment of human intuition to organize
reality (actual or virtual, internal or external), so that reason and
rationality can follow.

When in our contemporary zone-of-comfort we look down on alchemy and attribute
mystical, irrational, or pseudo-scientific status to it, we fail to appreciate
its ingenuity and its role in human culture. Its biggest strength, the
construction of holistic cognitive devices, lost favor when modern science
developed itself over a reductionist, physicalist framework. For alchemists
across the ages these cognitive devices were cosmological truths, imbued with
metaphysical power. They were not completely wrong.

Modern research shows that, evolutionary speaking, perceiving objective
reality is not essential or desirable for fitness[19, 35, 36]. The world is
not a theorem[25], and nature tests us through our ability to deal with
radical uncertainty [26]. Constructing fictions that allow us to navigate the
world, interact with it[20], and ultimately survive, was much more important
than any idealistic commitment to “truth”. Every living being has its own form
of fiction[35], an abstraction of reality developed by evolution to maximize
fitness, but for us humans narratives play a central role in the construction
of these fictions[10]. In the field of visualization this might be stating the
obvious, as we are concerned with creating visual narratives that mask reality
into useful representations. We ask the reader, then, to proceed with an open
mind (and heart).

##  2 Ten Theses

These are our ten core propositions put forward in this manifesto. Here they
are listed and accompanied by a small elaboration, but they are mainly
defended in our miro board. There they are presented as if they were ten
issues on our comic on data alchemy, and each cover art tries to visually
encode or captures the subject of discussion. Clicking on the numbers above
the text will bring the reader to the location on the board where the thesis
is developed, but many ideas are woven together in our explanation in a way
that their argumentation cannot be pinpointed to one single location.

I

Data alchemy is the transmutation and manipulation of information between
virtual and actual materials

Data is a virtual (spiritual) medium for phenomenal information, captured and
distilled through measurement[16]. In the most information-theoretical sense,
it is a vessel for entropy and differentiation[2]. Data alchemy is a concept
that we introduce to project the gestalt of alchemy into modern, digital
sciences. Alchemists believed to be manipulating some form of primordial
matter through physical and spiritual media according to cosmic laws and
correspondences. Understanding and mapping these correspondences would not
only explain all phenomena, but also allow the transmutation of substances and
manipulation of reality. If we replace the spiritual dimension, that for
alchemists was an essential mediator of the physical world, with a digital
dimension through computers and technological apparatus, we arrive pretty
close to the data-centric modus operandi of modern science[34]. Exercising an
effort of imagination, then, we can perceive data alchemy everywhere once we
understand the adequate morphisms: the medium is the message[31].

II

Visualization is as a form of data alchemy

Van Wijk’s [42] paper on the value of visualization, which is considered a
central work in visualization theory, describes a truly alchemical process. He
proposes a model of visualization which broadly transforms (transmutes) some
type of data into an image. The goal of this process being to produce
knowledge for the viewer by generating insights and providing information. The
insights and the knowledge obtained as product of the process are much more
valuable to the viewer in the end than the data that was used to generate the
images. Therefore, a lead-to-gold transmutation can be imagined to have
occurred. Min Chen’s[8] visualization thermodynamics, similarly, explains the
beautiful distillation of entropy from data, and its synthesis with the human
soul to produce a decision.

III

Visualization and alchemy are very connected historically

Historically, many examples of visualization exist that are related to
alchemical and astrological knowledge [37] and over the course of time society
has learned how these visual representations can greatly enhance our
capabilities of transmitting stories and scientific knowledge [30]. Bertin’s
work [3] on the semiology of graphics has further expanded on the capabilities
of this visual plane to objectively convey information and has become one of
the basis of data visualization.

IV

Humans process information through narratives, and both alchemy and
visualization lay heavily on that

Our cultural development relied strongly on storytelling[10][11]. Stories are
compact representations of cause-effect chains, and therefore can store
information, and be easily shared[23]. Alchemical texts encode knowledge in
rich mythical or poetic narratives, drawing on all sorts of cultural
references[43] to explain or model phenomena. Visualization too, ultimately
aims to communicate a narrative, even in the most mundane tasks, that links
the data to some phenomena, and for this reason it is a central topic of
research within the community[5, 41, 4, 28, 9].

V

The basic principles of (data) alchemy can be understood through the neo-
materialist philosophies of Manuel DeLanda and Deleuze and Guattari

The great mystery of alchemy is the material (and spiritual) composition of
the cosmos, how it organizes itself and how this can be manipulated. It deals
essentially with two types of operations: disssolutions (reductions,
distillations[44], analysis), and coagulations (aggregations, condensation,
synthesis). When we analyze a phenomenon through a series of observations
(samples), and summarize it through its mean and standard deviation, we are
performing a dissolution. Then, when we plug these numbers as parameters of a
distribution, synthesizing a mathematical object that is a whole and has a
continuous nature infinitely larger than the original observations[32], we are
performing a coagulation. To be able to talk about these ideas, however, we
need an ontology that is general and expressive. This is why we bring in
Deleuze & Guattari[14], and DeLanda to the table[12, 13]. Their philosophies
provide a colorful scale-free materialism where information and thermodynamics
play a central role.

VI

Generative art is also a form of data alchemy, which is very useful for the
exposition of the concept

Generative art comprises procedural art, which is the imprint of the algorithm
that generated them, data art, which is actually data visualization with a
purely aesthetic purpose, and AI art, which can be seen as a combination of
both. Common modern approaches to AI art rely on both data (usually, a lot)
and algorithmic procedures to generate a latent space and then create an image
from it. As procedural art can be understood as a form of autographic
visualization (the phenomenon _is_ the algorithm) [34] and data art is a form
of information visualization, it fits that AI art is then a form of data
alchemy, a kind of data-mediated autographic visualization.

VII

Text-guided generative art, or text-guided diffusion, or any other form of
data generated latent space approach to generate visuals is a form of
visualization

Latent spaces are the condensation of enormous amounts of data, of cultural
information, although this information may not be explicit but encoded into
the relationships that are formed by this condensation process. They can even
be thought of as intermediate representations, as commonly featured in many
visualization workflows. Text-guided diffusion is just a way to unpack the
tightly-knit relationships bound to a certain path in the latent space,
defined by the textual prompt that guides the information extraction in the
form of a (sometimes more, sometimes less) coherent image. This is certainly
not the only way to visualize what’s _in_ there, but it is one that can make
sense for certain tasks and contexts.

VIII

It reveals fundamental aspects about the data, our mathematical, algorithmic
processes, and, ultimately, about human culture

Text-guided image generation and similar approaches combine both
autographic[34] (if we consider the mixed materiality of the medium) and data-
driven visualization. The images we get from a prompt, sampling the latent
space, are thus a mixture of this culturally-generated data and our techniques
to manipulate this medium. Depending on how we set up our experiments, we can
distill information from either the human side, and learn about ourselves[18],
from the medium of neurons[6], or even both[40]. It can even be thought of as
a form of explainable AI[33, 29].

IX

It is a type of visualization that is not necessarily analytical,
quantitative, but that is fine and good

Text-guided diffusion and similar approaches are driven at their core by non-
linear data processing methods. Thus we cannot expect to get perfectly linear
results as in rule-based visualization encodings. However, this would not be
the first time non-linear methods are applied to classical visualization:
force-directed layouts for graphs and projection techniques such as UMAP are
examples of this. They share that some control over the representation is lost
in order to gain a more organic and definitely information-richer space. Why
then not accept the power of the salient, feature-rich, semantically complex
images that we can obtain from these approaches as part of our visualization
tool-kits? Furthermore, the aesthetically-pleasing power of these images must
not be underestimated: according to the field of Neuroaesthetics[7], there are
powerful cognitive mechanisms at play when we evaluate beauty that modulate
attention and affect tasks[38, 39, 27].

X

The visualization community should embrace the idea of data alchemy and these
new technologies for their potential and shape them into desired forms

Text-guided image generation and similar approaches have an enormous potential
which are now being explored mostly for creative purposes, and there is
already a growing commercial interest for their applications. It should not
surprise us, then, that in the future we will be “diffusing” for data with
scientific purposes. This sort of technology could evolve in the future to be
the long sought “holy grail”, or better put, the “philosophers’ stone” of
visualization, the all-purpose automatic encoder, once we learn better how to
communicate our wishes to it.

##  3 Conclusion

This paper is both a manual and itself an example of data alchemy, as we use a
variety of generative techniques to create the content of our exposition. They
of course, do not exist on a vacuum, and many other tools we use such as miro,
discord, colab, and to a larger extent, the internet, all engender an
environment with emergent properties that facilitate creation and
collaboration in a level never possible before in the history of humankind.

Diffusion is not the ultimate approach to the generation of images from a
language, and many of the tricks and limitations discussed are bound to
disappear when the technology develops. However, at the current moment it
represents one of the first easily accessible solutions that allowed a large
public to engage and tinker with it. Many creatives were suddenly motivated to
learn programming to control these tools, and at the same time programmers had
to put effort in developing their artistic direction skills. We have put
forward some arguments for why AI text-to-image generation approaches could be
considered as information visualization tools.

The ideas expressed here are not a critique of science or the scientific
method, desiring a return to some form of mysticism or idyllic past. However,
academia as a whole tends to be segmented, rigid, and prone to gatekeeping,
which we do lament. Being multidisciplinary, which is the union operation
between separate sets of disciplines, or interdisciplinary, which implies the
intersection, is not enough. We favor the idea of trans-disciplinarity as the
power-set of disciplines, and the polymathy of alchemy is highly desirable in
the construction of a true transdisciplinary research environment.

###### Acknowledgements.

This work has been partially supported by the European Commission under the
project Humane-AI-Net (grant agreement 952026) and through different Austrian
research agencies (FWF grants P35767, FFG Grant 880883, WWTF grant ICT19-047).

## References

  * [1] Disco Diffusion.  https://github.com/alembics/disco-diffusion, 2022.  [Online; accessed 05-October-2022].
  * [2] D. R. Anderson.  Information theory and entropy.  Model based inference in the life sciences: A primer on evidence, pp. 51–82, 2008.
  * [3] J. Bertin.  Semiology of Graphics.  University of Wisconsin Press, 1983.
  * [4] M. Brehmer et al.  Timelines revisited: A design space and considerations for expressive storytelling.  IEEE TVCG, 23(9):2151–2164, Sep. 2017. doi: 10 . 1109/TVCG . 2016 . 2614803
  * [5] M. Brehmer, B. Lee, N. Henry Riche, D. Tittsworth, K. Lytvynets, D. Edge, and C. White.  Timeline storyteller: The design & deployment of an interactive authoring tool for expressive timeline narratives.  In Computation+Journalism Symposium, pp. 1–5, 2019.
  * [6] N. Cammarata, S. Carter, G. Goh, C. Olah, M. Petrov, L. Schubert, C. Voss, B. Egan, and S. K. Lim.  Thread: Circuits.  Distill, 2020.  https://distill.pub/2020/circuits. doi: 10 . 23915/distill . 00024
  * [7] A. Chatterjee.  Neuroaesthetics: A Coming of Age Story.  Journal of Cognitive Neuroscience, 23(1):53–62, Jan. 2011.  Conference Name: Journal of Cognitive Neuroscience. doi: 10 . 1162/jocn . 2010 . 21457
  * [8] M. Chen and A. Golan.  What May Visualization Processes Optimize?  IEEE Transactions on Visualization and Computer Graphics, 22(12):2619–2632, Dec. 2016.  Conference Name: IEEE Transactions on Visualization and Computer Graphics. doi: 10 . 1109/TVCG . 2015 . 2513410
  * [9] S. Chen, J. Li, G. Andrienko, N. Andrienko, Y. Wang, P. H. Nguyen, and C. Turkay.  Supporting Story Synthesis: Bridging the Gap between Visual Analytics and Storytelling.  IEEE Transactions on Visualization and Computer Graphics, pp. 1–1, 2018. doi: 10 . 1109/TVCG . 2018 . 2889054
  * [10] E. Coen.  The storytelling arms race: origin of human intelligence and the scientific mind.  Heredity, 123(1):67–78, July 2019.  Number: 1 Publisher: Nature Publishing Group. doi: 10 . 1038/s41437-019-0214-2
  * [11] M. Dehghani, R. Boghrati, K. Man, J. Hoover, S. I. Gimbel, A. Vaswani, J. D. Zevin, M. H. Immordino‐Yang, A. S. Gordon, A. Damasio, and J. T. Kaplan.  Decoding the neural representation of story meanings across languages.  Human Brain Mapping, 38(12):6096–6106, Dec. 2017. doi: 10 . 1002/hbm . 23814
  * [12] M. DeLanda.  A New Philosophy of Society: Assemblage Theory and Social Complexity.  Bloomsbury Academic, 2006.
  * [13] M. DeLanda.  Materialist Phenomenology: A Philosophy of Perception.  Theory in the New Humanities. Bloomsbury Publishing, 2021.
  * [14] G. Deleuze and F. Guattari.  A Thousand Plateaus: Capitalism and Schizophrenia.  Capitalism and schizophrenia. University of Minnesota Press, 1987.
  * [15] P. Dhariwal and A. Nichol.  Diffusion models beat gans on image synthesis, 2021. doi: 10 . 48550/ARXIV . 2105 . 05233
  * [16] J. Drucker.  Data as capta.  Digital Humanities Quarterly, 5(1):1–21, 2011.
  * [17] G. Gigerenzer.  Gut feelings: the intelligence of the unconscious.  Viking, New York, 2007.  OCLC: 436269045.
  * [18] G. Goh et al.  Multimodal neurons in artificial neural networks.  Distill, 2021.  https://distill.pub/2021/multimodal-neurons. doi: 10 . 23915/distill . 00030
  * [19] D. Hoffman.  The Case Against Reality: Why Evolution Hid the Truth from Our Eyes.  W. W. Norton & Company, Aug. 2019.  Google-Books-ID: JgJ1DwAAQBAJ.
  * [20] D. D. Hoffman, M. Singh, and C. Prakash.  The Interface Theory of Perception.  Psychonomic Bulletin & Review, 22(6):1480–1506, Dec. 2015. doi: 10 . 3758/s13423-015-0890-8
  * [21] E. J. Holmyard.  Alchemy.  Courier Corporation, 2012.
  * [22] Huikai, N. Senzaki, and P. Reps.  The Gateless Gate: A Collection of Zen Koan.  Red and Black Publishers, 2008.
  * [23] M. Joubert, L. Davis, and J. Metcalfe.  Storytelling: the soul of science communication.  Journal of Science Communication, 18(05):E, Oct. 2019. doi: 10 . 22323/2 . 18050501
  * [24] C. G. Jung and R. F. C. Hull.  Psychology and alchemy.  Routledge, 2014.
  * [25] S. Kauffman and A. Roli.  The World Is Not a Theorem.  Entropy, 23(11):1467, Nov. 2021. doi: 10 . 3390/e23111467
  * [26] M. King and J. Kay.  Radical Uncertainty: Decision-making for an unknowable future.  Little, Brown Book Group, 2020.
  * [27] U. Kirk, M. Skov, O. Hulme, M. S. Christensen, and S. Zeki.  Modulation of aesthetic value by semantic context: An fMRI study.  NeuroImage, 44(3):1125–1132, Feb. 2009. doi: 10 . 1016/j . neuroimage . 2008 . 10 . 009
  * [28] B. Lee, N. H. Riche, P. Isenberg, and S. Carpendale.  More Than Telling a Story: Transforming Data into Visually Shared Stories.  IEEE Computer Graphics and Applications, 35(5):84–90, Sept. 2015\. doi: 10 . 1109/MCG . 2015 . 99
  * [29] M. Li, Z. Zhao, and C. Scheidegger.  Visualizing neural networks with the grand tour.  Distill, 2020.  https://distill.pub/2020/grand-tour. doi: 10 . 23915/distill . 00025
  * [30] S. McCloud.  Understanding Comics: The Invisible Art.  Tundra Publishing Ltd, 1993.
  * [31] M. McLuhan.  Understanding media : the extensions of man.  New American Library, New York, 1988.
  * [32] D. McQuillan.  Data Science as Machinic Neoplatonism.  Philosophy & Technology, 31(2):253–272, June 2018. doi: 10 . 1007/s13347-017-0273-3
  * [33] S. T. Mueller, R. R. Hoffman, W. Clancey, and A. Emrey.  Explanation in Human-AI Systems: A Literature Meta-Review Synopsis of Key Ideas and Publications and Bibliography for Explainable AI.  p. 204.
  * [34] D. Offenhuber.  Data by Proxy — Material Traces as Autographic Visualizations.  IEEE TVCG, 26(1):98–108, Jan 2020. doi: 10 . 1109/TVCG . 2019 . 2934788
  * [35] C. Prakash, C. Fields, D. D. Hoffman, R. Prentner, and M. Singh.  Fact, Fiction, and Fitness.  Entropy, 22(5):514, Apr. 2020. doi: 10 . 3390/e22050514
  * [36] C. Prakash, K. D. Stephens, D. D. Hoffman, M. Singh, and C. Fields.  Fitness Beats Truth in the Evolution of Perception.  Acta Biotheoretica, 69(3):319–341, Sept. 2021. doi: 10 . 1007/s10441-020-09400-0
  * [37] S. Rendgen and J. Wiedemann.  History of Information Graphics.  Taschen, 2019.
  * [38] P. Sarasso et al.  Beauty in mind: Aesthetic appreciation correlates with perceptual facilitation and attentional amplification.  Neuropsychologia, 136:107282, Jan. 2020. doi: 10 . 1016/j . neuropsychologia . 2019 . 107282
  * [39] P. Sarasso, M. Neppi-Modona, K. Sacco, and I. Ronga.  “Stopping for knowledge”: The sense of beauty in the perception-action cycle.  Neuroscience & Biobehavioral Reviews, 118:723–738, Nov. 2020. doi: 10 . 1016/j . neubiorev . 2020 . 09 . 004
  * [40] N. Savage.  How AI and neuroscience drive each other forwards.  Nature, 571(7766):S15–S17, July 2019. doi: 10 . 1038/d41586-019-02212-4
  * [41] A. Thudt et al.  Exploration and Explanation in Data-Driven Storytelling.  In Data-Driven Storytelling, pp. 59–83. A K Peters/CRC Press, Boca Raton, Florida : Taylor & Francis/CRC Press, [2018], 1 ed., Mar. 2018\. doi: 10 . 1201/9781315281575-3
  * [42] J. J. van Wijk.  The value of visualization.  In VIS 05. IEEE Visualization, pp. 79–86, 2005. doi: 10 . 1109/VISUAL . 2005 . 1532781
  * [43] J. A. Walsh and W. E. Hooper.  The liberty of invention: alchemical discourse and information technology standardization.  Literary and Linguistic Computing, 27(1):55–79, Apr. 2012. doi: 10 . 1093/llc/fqr038
  * [44] T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros.  Dataset distillation, 2020.
  * [45] W. Willett et al.  Perception! Immersion! Empowerment!: Superpowers as Inspiration for Visualization.  IEEE Transactions on Visualization and Computer Graphics, pp. 1–1, 2021. doi: 10 . 1109/TVCG . 2021 . 3114844

See pictures/comicp0.jpgSee pictures/comicp1.jpgSee pictures/comicp2.jpgSee
pictures/comicp3.jpgSee pictures/comicp4.jpg


# Optimizing Vision Transformers for Medical Image Segmentation

###### Abstract

For medical image semantic segmentation (MISS), Vision Transformers have
emerged as strong alternatives to convolutional neural networks thanks to
their inherent ability to capture long-range correlations. However, existing
research uses off-the-shelf vision Transformer blocks based on linear
projections and feature processing which lack spatial and local context to
refine organ boundaries. Furthermore, Transformers do not generalize well on
small medical imaging datasets and rely on large-scale pre-training due to
limited inductive biases. To address these problems, we demonstrate the design
of a compact and accurate Transformer network for MISS, CS-Unet, which
introduces convolutions in a multi-stage design for hierarchically enhancing
spatial and local modeling ability of Transformers. This is mainly achieved by
our well-designed Convolutional Swin Transformer (CST) block which merges
convolutions with Multi-Head Self-Attention and Feed-Forward Networks for
providing inherent localized spatial context and inductive biases. Experiments
demonstrate CS-Unet without pre-training outperforms other counterparts by
large margins on multi-organ and cardiac datasets with fewer parameters and
achieves state-of-the-art performance. Our code is available at Github1.

Index Terms—  Medical Image Segmentation, Semantic Segmentation, Vision
Transformer, Convolutions

##  1 Introduction

Medical image semantic segmentation (MISS), which classifies image pixels with
semantic organ labels (e.g. Kidney and Liver) for various imaging modalities,
is considered as one of the most fundamental problems in medical imaging.
However, compared to natural scene images, MISS requires overcoming more
challenges to create robust models. For instance, common benchmark datasets in
MISS suffer from large deformation of organs under different image acquisition
processes. In addition, shortage of costly pixel-level annotations is another
problem leading to a performance gap. To achieve efficient and effective
segmentation, models are not only required to have a better understanding of
their local semantic features to capture more subtle organ structures, but
also of global feature dependencies to capture the relationships among
multiple organs.

UNet [1] and its variants [2][3][4][5][6] with Convolutional Neural Networks
(CNNs) as the backbone have found huge success in MISS as they are good at
modelling local attributes inside their receptive field. However, the inherent
locality of convolution operations restricts their ability to model long-range
semantic dependencies within the image, and as a result the challenging
boundaries of the whole organ may not be effectively segmented. Attention
mechanisms alleviate this issue, but these tend to be ’single head’ mechanisms
that only calculate pixel-level similarities, and not ’multi head’ with the
ability to capture patch-level patterns.

For alleviating the inherent flaws of CNNs, there’s a recent shift in the
choice of architectures from CNNs to Vision Transformers (ViTs) due to their
ability to model long range semantic attributes among input tokens (embeddings
of image patches) via a linearly projected Multi-Head Self-Attention (MHSA)
operation and a Feed-Forward Network (FFN). Most early works [7][8][9] treat
CNNs as a backbone and exploit the Transformer’s desirable characteristics in
their encoder. They tend to have high complexity as they stack bulky
Transformer blocks on top of convolutional feature extractors (large
pretrained CNNs, e.g. ResNet). Recent research
[10][11][12][13][14][15][16][17] has moved towards using Transformers as the
main stem for building the entire segmentation architecture. Swin-UNet [12] is
regarded as the first pure Transformer model. It keeps the familiar U-shape
and adds hierarchical feature extraction using shifted windows proposed by the
Swin Transformer [18]. This drastically reduces the quadratic complexity of
traditional self-attention while achieving better performance.

However, most of these Transformers for MISS use off-the-shelf Transformer
blocks from Computer Vision community and only model and extract linear
semantic relations via MHSA and FFN, leading to the challenge of precisely
delineating organ boundaries due to the lack of spatial and local information
as shown in Figure 1.(d), although showing small influence on detection and
classification tasks. Besides, these methods require a large dataset to
compensate the lack of inductive biases such as translation equivariance [19],
which may be defected or even lost when fine-tuning on downstream tasks,
showing less robustness on small datasets.

Keeping the current state of the literature in mind, our paper highlights
issues that today’s Transformers in MISS face, followed by our contribution
that helps alleviate those drawbacks. Most current Transformers are bulky and
rely on pre-training weights from classical vision tasks to be adapted for
MISS. To the best of our knowledge, no existing study explores the effects of
adding spatial locality inside Transformer blocks via convolutions for medical
imaging. To this end, we first propose an empirical analysis to show the need
for spatial locality in pure Transformer based MISS. Our insights show the
effects of introducing convolutions to Transformer blocks and multi-stage
design of networks on segmentation performance. We call the final model
resulting from our experiments, Convolutional Swin-Unet (CS-Unet), which is
based on purely convolutional Transformer blocks created to make Transformers
model local information better, segment organ boundaries more accurately,
while maintaining a low computational complexity. Experiments on CT and MRI
datasets show CS-Unet (24M parameters) trained from scratch outperforms pre-
trained Swin-Unet (27M) on ImageNet by around 3$\%$ dice score, achieving
state-of-the-art performance.

Fig. 1: Visualization of segmentation results of different methods trained
from scratch on Synapse dataset.

##  2 Method

Most Transformer based methods in MISS, i.e., encoder-decoder models with a
standard U-shape, use a standard Transfomer block containing linear
projections and linear FFNs, which are essentially MLPs, to process the data.
Hence, to create effective image representations using such a regime requires
huge amounts of data for training, as they lack local spatial information.

The first pure-Transformer based MISS model is the Swin-Unet [12] which adopts
Swin Transformer blocks [18] to add locality information to Transformers. The
data representation created here is still inherently linear as this block
contains linear projections and feature processing. Figure 1 shows
segmentation visualizations for the Synapse dataset. Swin-Unet trained from a
random weight initialization (Figure 1.(d)) does not perform well. It fails to
detect the spleen and misclassifies the left kidney as the right.

Next, we add convolutional projections to this Swin Transformer block
structure. The projections follow the methodology proposed in [20] where
tokens are first shaped into a 2D token map, then processed by a depth-wise
separable convolution with kernel size $s$ implemented by: Depth-wise Conv
$\rightarrow$ BatchNorm $\rightarrow$ Point-wise Conv. Finally, the tokens are
flattened into 1D token input $x_{i}^{q/k/v}$ for Q/K/V matrices. It can be
formulated as: $x_{i}^{\mathrm{q/k/v}}=Flatten(Conv(Reshape2D(x_{i}),s)$.
Figure 1.(c) shows outputs of the resultant Unet trained with this block. It
visually demonstrates how spatial locality is essential for low level pixel
labelling tasks. It can be seen that although the convolutional projection
alleviates a lot of the problems posed by the linearity of Swin-Unet, there
are still severe over-segmentations on pancreas and liver and extremely rough
boundaries of right kidney.

Following this, when a 3x3 convolution is used for FFNs instead of MLPs to
introduce more spatial context, we see the full effects of adding complete
spatial locality to Transformers through the boundaries of the left and right
kidneys and spleen becoming greatly refined. The over-segmentation problem of
the pancreas however gets worse (as shown in Figure 1.(b)). This is due to the
limited receptive field not modeling the whole boundary of big organs
effectively.

###  2.1 Convolutional Swin Transformer (CST) Layer

We propose a CST layer to fully explore spatial modeling ability of
convolutions in MHSA and FFN. First, we propose a novel (shifted) window based
convolutional multi-head self attention ((S)W-CMSA) to extract hierarchical
semantic features while reducing computational costs, by combining a shifted
windows mechanism and convolutional projection. Then, we replace the MLP with
our novel depthwise separable feed-forward (DSF) module. From Figure 1.(a), we
see the Transformer model based on CST handles challenging organ boundaries
more efficiently. The CST layer is formulated as:

$\displaystyle\hat{z}^{\mathrm{l}}$
$\displaystyle=W-CMSA(LN(z^{\mathrm{l-1}}))+z^{\mathrm{l-1}},$ (1)
$\displaystyle z^{\mathrm{l}}$
$\displaystyle=DSF(\hat{z}^{\mathrm{l}})+\hat{z}^{\mathrm{l}},$ (2)
$\displaystyle\hat{z}^{\mathrm{l+1}}$ $\displaystyle=SW-
CMSA(LN(z^{\mathrm{l}}))+z^{\mathrm{l}},$ (3)
$\displaystyle\hat{z}\mathrm{l+1}$
$\displaystyle=DSF(\hat{z}^{\mathrm{l+1}})+\hat{z}^{\mathrm{l+1}}\vspace{-13pt}$
(4)

where $\hat{z}^{l}$ and $z^{l}$ denote the outputs of (S)W-CMSA module and DSF
of the $l$-th block, respectively.

(Shifted) Window based convolutional multi-head self attention

Fig. 2: Convolutional Swin Transformer (CST) Block.

As shown in Figure 2, once tokens enter (S)W-CMSA, they are reshaped into a 2D
token map, and partitioned into windows. For each window, we use three depth-
wise convolutions with kernel size $s$ of 3×3, padding of 1 and stride of 1 to
create our Q, K and V vectors via:
$Flatten(DepthConv(Window(Reshape(x_{i})),s)$.

CST is different from [20] as we create a projection based on windows rather
than the whole image, leading to more refined local features as now the
kernels learnt on each window are different. In order to better adapt to
medical images with smaller data volumes, point-wise convolutions are removed
to avoid over-fitting. Furthermore, we replace Batch Normalization with Layer
Normalization (LN), providing a performance boost. The token vectors are fed
to MHSA as:

$\displaystyle
MHSA(x_{i}^{\mathrm{q}},x_{i}^{\mathrm{k}},x_{i}^{\mathrm{v}})=SoftMax$
$\displaystyle\left(\frac{x_{i}^{\mathrm{q}}(x_{i}^{\mathrm{k}})^{\mathrm{T}}}{\sqrt{d}}+B\right)x_{i}^{\mathrm{v}}$
(5)

Here $d$ represents the dimension of the query and key. The values in $B$ are
the bias.

Then, we replace the linear layer and feed the attention output to a 3×3
depth-wise convolution for fine-tuning for more spatial information. We follow
this by reversing the windows to 2D token maps, resulting in more robust
estimations compared to Swin Transformer [18] removing our dependence on
positional encoding.

Depthwise separable feed-forward (DSF) module After computing (S)W-CMSA, the
feature maps are fed into a FFN. Existing Transformers implement this module
as an MLP: LN,d $\rightarrow$ Linear,4×d $\rightarrow$ GELU $\rightarrow$
Linear,d $\rightarrow$ RC. The $d$ denotes the number of channels of a
reshaped feature map and $RC$ denotes the residual connection. We propose a
DSF module as a choice of FFN which provides adding spatial context. We use
three depth-wise convolutions instead of two linear layers for utilizing the
features between channels, $C$. In addition, we found that adding LN after
convolution gives better segmentation results. The DSF is implemented as: 7x7
Depth-wise Conv,d $\rightarrow$ LN,d $\rightarrow$ Point-wise Conv,4×d
$\rightarrow$ GELU $\rightarrow$ Point-wise Conv,d $\rightarrow$ RC.

###  2.2 Overall Structure Design

CS-Unet keeps a symmetrical UNet shape. The input of our model is a 2D image
slice with the resolution of $H\times W\times 3$ sampled from a 3D volume of
images. $H$, $W$ and $3$ denote the height, width and number of channels of
each input. The input images on entering the encoder are passed through the
convolutional token embedding to create a sequence embedding on overlapping
patches of the image, following which CST and patch merging layers are
applied. Extracted features are then processed by the model’s bottleneck that
consists of two CST blocks. A symmetrical decoder then creates the final
segmentation marks. In addition, skip convolution (SC) modules are added
between corresponding feature pyramids of the encoder and decoder to
compensate for the missing information caused by down-sampling. The overall
architecture of the proposed CS-Unet is presented in Figure 3.

Fig. 3:  (a) Overall architecture of CS-Unet, (b) one CST layer, (c)
convolutional token embedding, (d) DSF and (e) skip convolutions. $d$ is the
current number of channels, $c$ is an arbitrary dimension.

###  2.3 Encoder

The input image is first passed through the convolutional token embedding
layer to create a sequence embedding with the resolution of
$\frac{H}{4}\frac{W}{4}\times C$ ($C$ = 96 in experiments). This embedding is
fed to three main CST layers and a patch merging module which downsamples the
image and doubles the number of channels. For example, at the first patch
merging module, an input with size $\frac{H}{4}\times\frac{W}{4}\times C$ is
divided into four parts and concatenated along the $C$ dimension to create a
feature map of size $\frac{H}{8}\times\frac{W}{8}\times 4C$. Then a linear
layer is applied to this map to reduce the $C$ dimension by a factor of 2.

Convolutional Token Embedding layer Existing models use a linear layer to
split images into non-overlapping patches and reduce the size of the image
drastically, e.g. by $75\%$, while increasing the channel dimension $C$.
However, as the images’ highest resolution is $H\times W$ at the encoder,
using a linear layer to compress these features not only loses high-quality
spatial and local information, but also increases model size. Our embedding
layer, is implemented as four convolutions with overlapping patches to
compress features in stages, helping to introduce more spatial dependency
between, and inside the patches, while greatly reducing the parameters (by 6M.
See Ablation 3, Method 1). Specifically, this layer is implemented as follows:
3x3 s=1 Conv,d/2 $\rightarrow$ GELU $\rightarrow$ 3x3 s=2 Conv,d/2
$\rightarrow$ GELU+LN $\rightarrow$ 3x3 s=1 Conv,d $\rightarrow$ GELU
$\rightarrow$ 3x3 s=2 Conv,d $\rightarrow$ GELU. Here, $s$ is stride, the
input dimension is 3, and $d=C$. In the end, 2D reshaped token maps with
resolution $\frac{H}{4}\times\frac{W}{4}\times C$ are outputted.

Bottleneck  The bottleneck contains two CST blocks, based on W-CMSA. The
feature map size here remains unchanged.

###  2.4 Decoder

Our decoder is symmetric to the encoder. Feature representation is created by
enlarging the feature volume through a convolutional up-sampling module and
then passing it through a SC module to compensate for the information lost due
to down-sampling. A CST layer then provides spatial context to the upsampled
features. After repeating the above process three times, the features are fed
into the patch expansion layer which up-samples by $4\times$, followed by a
linear projection to fine tune the final segmentation prediction.
Specifically, convolutional up-sampling module employs strided deconvolution
to $2\times$ up-sample feature maps and halves the channel dimension as: LN,d
$\rightarrow$ 2x2 s=2 ConvTranspose,d/2 $\rightarrow$ GELU.

Skip Convolutions (SC) module  The outputs of high-resolution feature maps
created from up-sampling are concatenated with shallow feature representations
from the encoder, and then merged by a SC module. It further enriches both
spatial and fine-grained information, while compensating for the missing
information caused by down-sampling. It is implemented as :3x3 s=1 Conv,d/2
$\rightarrow$ GELU $\rightarrow$ 3x3 s=1 Conv,d/2 $\rightarrow$ GELU.

##  3 Experiments

Methods | DSC | HD | Aorta | Gallb |  Kid$\\_$L |  Kid$\\_$R | Liver | Pancr | Spleen | Stom
---|---|---|---|---|---|---|---|---|---|---
R50 UNet [7] | 74.68 | 36.87 | 84.18 | 62.84 | 79.19 | 71.29 | 93.35 | 48.23 | 84.41 | 73.92
R50 AttnUNet [7] | 75.57 | 36.97 | 55.92 | 63.91 | 79.20 | 72.71 | 93.56 | 49.37 | 87.19 | 74.95
UNet [1] | 76.85 | 39.70 | 89.07 | 69.72 | 77.77 | 68.60 | 93.43 | 53.98 | 86.67 | 75.58
AttnUNet [4] | 77.77 | 36.02 | 89.55 | 68.88 | 77.98 | 71.11 | 93.57 | 58.04 | 87.30 | 75.75
R50 ViT [7] | 71.29 | 32.87 | 73.73 | 55.13 | 75.80 | 72.20 | 91.51 | 45.99 | 81.99 | 73.95
TransUnet [7] | 77.48 | 31.69 | 87.23 | 63.13 | 81.87 | 77.02 | 94.08 | 55.86 | 85.08 | 75.62
Swin-Unet [12] | 79.13 | 21.55 | 85.47 | 66.53 | 83.28 | 79.61 | 94.29 | 56.58 | 90.66 | 76.60
MT-UNet [16] | 78.59 | 26.59 | 87.92 | 64.99 | 81.47 | 77.29 | 93.06 | 59.46 | 87.75 | 76.81
Ours | 82.21 | 27.02 | 88.40 | 72.59 | 85.28 | 79.52 | 94.35 | 70.12 | 91.06 | 75.72

Table 1: Comparison with different models on Synapse. Gallbladder, left
Kidney, right Kidney, Pancreas and Stomach are abbreviated as Gallb,
Kid$\\_$L, Kid$\\_$R, Pancr and Stom.

Methods | DSC | RV | Myo | LV
---|---|---|---|---
R50 UNet | 87.60 | 84.62 | 84.52 | 93.68
R50 AttnUNet | 86.90 | 83.27 | 84.33 | 93.53
R50 ViT | 86.19 | 82.51 | 83.01 | 93.05
TransUnet | 89.71 | 86.67 | 87.27 | 95.18
Swin-Unet | 88.07 | 85.77 | 84.42 | 94.03
MT-UNet | 90.43 | 86.64 | 89.04 | 95.62
Ours | 91.37 | 89.20 | 89.47 | 95.42

Table 2: Experimental results of ACDC.

We use two publicly available datasets to benchmark our method.

Synapse multiorgan segmentation (Synapse): This dataset [21] contains
abdominal CT scans from 30 subjects. Following [7], 18 cases (2212 axial
slices) are extracted for training, while other 12 cases are used for testing.
We report the model performance evaluated with the average Dice score
Coefficient (DSC) and average Hausdorff Distance (HD) on eight abdominal
organs.

Automatic Cardiac Diagnosis Challenge (ACDC): ACDC [22] contains MRI images
from 100 patients, with right ventricle (RV), left ventricle (LV) and
myocardium (MYO) to be segmented. Using data splits proposed in [16], the
dataset is split into 70 (1930 axial slices), 10 and 20 for training,
validation and testing, respectively. Evaluation metrics used are average DSC
($\%$) and HD (mm).

###  3.1 Implementation details

We train our models on a single Nvidia RTX3090 GPU with 24GB memory. We use
flipping and rotation augmentations on the training data. The input image size
is 224×224. Pre-trained weights are used for other methods if provided, while
our model is trained from scratch for 300 epochs from a randomly initialized
set of weights. A batch size of 24 and a combination of cross entropy and dice
loss are used. Our model is optimized by AdamW [23] with a weight decay of
5E-4 for both datasets. The learning rates for Synapse and ACDC are 1e-3 and
5e-3, respectively. We start with a 10-epoch linear warmup. Layer Scale [24]
of initial value 1e-6 is applied.

###  3.2 Experimental Results

As shown in Table 2 and Table 2, our model consistently surpasses a variety of
convolution-based and Transformer-based methods. CS-Unet outperforms Swin-Unet
by 3.08% and 3.3% DSC on Synapse and ACDC, respectively. In addition, our
method gets the highest DSC for five and two organs of Synapse and ACDC
respectively, especially providing large boosts for challenging organs like
gallbladder, pancreas and RV. Overall, compared to pretrained Swin-Unet (27 M)
and TransUnet (96 M), CS-Unet achieves the best performance without
pretraining while being lightweight (24 M) via introducing more local
perception and inductive bias.

Fig. 4: Visualization of segmentation results on two datasets.

Figure 4 visualizes segmentation results. In case 1, our method has
overwhelming advantage on segmenting the pancreas, stomach and liver. CS-Unet
is also more discriminative on the complex shape of RV than other Transformer-
based models in case 2 due to its better ability of spatial context modelling.

###  3.3 Ablation study

We explore the influence of proposed modules on the performance on Synapse as
shown in Table 3. The Swin-Unet trained from scratch is treated as the
baseline (method 0) which cannot adapt to small datasets. Adding convolutional
token embedding (method 1) and convolutional projections (method 2), we
observe large improvements of $8\%$ and $9\%$ on DSC which is competitive with
pre-trained Swin-Unet emphasizing the importance of adding local modeling
ability to Transformers. Removing the position embedding in early stages and
using a convolution instead of a linear layer to fine-tune the attention
computation (method 3) leads to a slight increase in performance and parameter
reduction. Method 4 combines the CST block with the DSF module leading to an
improved DSC and HD without extra parameters. After utilizing convolutional
up-sampling and feature fusion module, SC, for merging information during skip
connection, our best performing model method 6 achieves 3.17% improvements on
DSC. Comparison with method 5, shows that fully convolutional pure
Transformers can track the position of pixels better without needing an extra
positional embedding, and that spatial feature extraction, is in fact, a
necessity for Transformers.

Methods | DSC | HD | Emb | Proj | Pos | Att | DSF | SC |  $\\#$param
---|---|---|---|---|---|---|---|---|---
0 (Base) | 60.80 | 54.35 |  |  | $\surd$ |  |  |  | 27.15
1 | 68.57 | 51.02 | $\surd$ |  | $\surd$ |  |  |  | 21.55
2 | 77.47 | 18.54 | $\surd$ | $\surd$ | $\surd$ |  |  |  | 21.55
3 | 78.32 | 25.43 | $\surd$ | $\surd$ | $\times$ | $\surd$ |  |  | 19.63
4 | 79.04 | 22.96 | $\surd$ | $\surd$ | $\times$ | $\surd$ | $\surd$ |  | 19.84
5 | 81.93 | 24.59 | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | $\surd$ | 24.68
6 | 82.21 | 27.02 | $\surd$ | $\surd$ | $\times$ | $\surd$ | $\surd$ | $\surd$ | 24.68

Table 3: Ablation study on modules used in CS-Unet on Synapse.

##  4 Conclusions

In this work, we presented the effects of introducing convolutions to
Transformer blocks and to a multi-stage Transformer network to alleviate
limitations of non-locality and need for extensive pre-training that
Transformers in MISS face. Extensive experiments demonstrated that merging
Convolutions with MHSAs and FFNs to create our CST layer, provided inherent
local context inside Transformer blocks. Based on CST, our compact, accurate
and pure Transformer architecture, CS-Unet, achieved superior performance
without pretraining while maintaining less parameters.

## References

  * [1] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in International Conference on Medical image computing and computer-assisted intervention. Springer, 2015, pp. 234–241.
  * [2] Zongwei Zhou, Md Mahfuzur Rahman Siddiquee, Nima Tajbakhsh, and Jianming Liang,  “Unet++: A nested u-net architecture for medical image segmentation,”  in Deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3–11. Springer, 2018.
  * [3] Huimin Huang, Lanfen Lin, Ruofeng Tong, Hongjie Hu, Qiaowei Zhang, Yutaro Iwamoto, Xianhua Han, Yen-Wei Chen, and Jian Wu,  “Unet 3+: A full-scale connected unet for medical image segmentation,”  in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 1055–1059.
  * [4] Jo Schlemper, Ozan Oktay, Michiel Schaap, Mattias Heinrich, Bernhard Kainz, Ben Glocker, and Daniel Rueckert,  “Attention gated networks: Learning to leverage salient regions in medical images,”  Medical image analysis, vol. 53, pp. 197–207, 2019.
  * [5] Chaitanya Kaul, Suresh Manandhar, and Nick Pears,  “Focusnet: An attention-based fully convolutional network for medical image segmentation,”  in 2019 IEEE 16th international symposium on biomedical imaging (ISBI 2019). IEEE, 2019, pp. 455–458.
  * [6] Chaitanya Kaul, Nick Pears, Hang Dai, Roderick Murray-Smith, and Suresh Manandhar,  “Focusnet++: Attentive aggregated transformations for efficient and accurate medical image segmentation,”  in 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI). IEEE, 2021, pp. 1042–1046.
  * [7] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou,  “Transunet: Transformers make strong encoders for medical image segmentation,”  arXiv preprint arXiv:2102.04306, 2021.
  * [8] Yundong Zhang, Huiye Liu, and Qiang Hu,  “Transfuse: Fusing transformers and cnns for medical image segmentation,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021, pp. 14–24.
  * [9] Bingzhi Chen, Yishu Liu, Zheng Zhang, Guangming Lu, and David Zhang,  “Transattunet: Multi-level attention-guided u-net with transformer for medical image segmentation,”  arXiv preprint arXiv:2107.05274, 2021.
  * [10] Jeya Maria Jose Valanarasu, Poojan Oza, Ilker Hacihaliloglu, and Vishal M Patel,  “Medical transformer: Gated axial-attention for medical image segmentation,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021, pp. 36–46.
  * [11] Davood Karimi, Serge Didenko Vasylechko, and Ali Gholipour,  “Convolution-free medical image segmentation using transformers,”  in International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2021, pp. 78–88.
  * [12] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang,  “Swin-unet: Unet-like pure transformer for medical image segmentation,”  arXiv preprint arXiv:2105.05537, 2021.
  * [13] Ailiang Lin, Bingzhi Chen, Jiayu Xu, Zheng Zhang, Guangming Lu, and David Zhang,  “Ds-transunet: Dual swin transformer u-net for medical image segmentation,”  IEEE Transactions on Instrumentation and Measurement, 2022.
  * [14] Xiaohong Huang, Zhifang Deng, Dandan Li, and Xueguang Yuan,  “Missformer: An effective medical image segmentation transformer,”  arXiv preprint arXiv:2109.07162, 2021.
  * [15] Hong-Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu,  “nnformer: Interleaved transformer for volumetric segmentation,”  arXiv preprint arXiv:2109.03201, 2021.
  * [16] Hongyi Wang, Shiao Xie, Lanfen Lin, Yutaro Iwamoto, Xian-Hua Han, Yen-Wei Chen, and Ruofeng Tong,  “Mixed transformer u-net for medical image segmentation,”  in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 2390–2394.
  * [17] Athanasios Tragakis, Chaitanya Kaul, and Husmeier Dirk Murray-Smith Roderick,  “The fully convolutional transformer for medical image segmentation,”  in https://arxiv.org/abs/2206.00566, 2022.
  * [18] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo,  “Swin transformer: Hierarchical vision transformer using shifted windows,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
  * [19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,”  in International Conference on Learning Representations, 2020.
  * [20] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang,  “Cvt: Introducing convolutions to vision transformers,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 22–31.
  * [21] Bennett Landman, Zhoubing Xu, J Igelsias, Martin Styner, T Langerak, and Arno Klein,  “Miccai multi-atlas labeling beyond the cranial vault–workshop and challenge,”  in Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vault—Workshop Challenge, 2015, vol. 5, p. 12.
  * [22] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, et al.,  “Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved?,”  IEEE transactions on medical imaging, vol. 37, no. 11, pp. 2514–2525, 2018.
  * [23] Ilya Loshchilov and Frank Hutter,  “Decoupled weight decay regularization,”  in International Conference on Learning Representations, 2019.
  * [24] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou,  “Going deeper with image transformers,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 32–42.