# Enhancing All-Optical Switching of Magnetization by He Ion Irradiation

Pingzhi Li∗<EMAIL_ADDRESS>Department of Applied Physics, Eindhoven University
of Technology, P. O. Box 513, 5600 MB Eindhoven, The Netherlands  Johannes W.
van der Jagt  Spin-Ion Technologies, 10 boulevard Thomas Gobert, 91120
Palaiseau, France  Université Paris-Saclay, 3 rue Juliot Curie, 91190 Gif-sur-
Yvette, France  Maarten Beens  Department of Applied Physics, Eindhoven
University of Technology, P. O. Box 513, 5600 MB Eindhoven, The Netherlands
Julian Hintermayr  Department of Applied Physics, Eindhoven University of
Technology, P. O. Box 513, 5600 MB Eindhoven, The Netherlands  Marcel
Verheijen  Department of Applied Physics, Eindhoven University of Technology,
P. O. Box 513, 5600 MB Eindhoven, The Netherlands  René Bruikman  Department
of Applied Physics, Eindhoven University of Technology, P. O. Box 513, 5600 MB
Eindhoven, The Netherlands  Beatriz Barcones  NanoLab@TU/e, Eindhoven
University of Technology, P. O. Box 513, 5600 MB Eindhoven, The Netherlands
Roméo Juge  Spin-Ion Technologies, 10 boulevard Thomas Gobert, 91120
Palaiseau, France  Reinoud Lavrijsen  Department of Applied Physics, Eindhoven
University of Technology, P. O. Box 513, 5600 MB Eindhoven, The Netherlands
Dafiné Ravelosona  Spin-Ion Technologies, 10 boulevard Thomas Gobert, 91120
Palaiseau, France  Centre de Nanosciences et de Nanotechnologies, CNRS,
Université Paris-Saclay, 10 boulevard Thomas Gobert, 91120 Palaiseau, France
Bert Koopmans  Department of Applied Physics, Eindhoven University of
Technology, P. O. Box 513, 5600 MB Eindhoven, The Netherlands

###### Abstract

All-optical switching (AOS) of magnetization by a single femtosecond laser
pulse in Co/Gd based synthetic ferrimagnets is the fastest magnetization
switching process. On the other hand, He ion irradiation has become a
promising tool for interface engineering of spintronic material platforms,
giving rise to significant modification of magnetic properties. In this paper,
we explore the use of He ion irradiation to enhance single pulse AOS of Co/Gd
bilayer-based synthetic ferrimagnets. The intermixing of the constituent
magnetic layers by He ion irradiation was both numerically simulated and
experimentally verified. We theoretically modelled the effects of intermixing
on AOS using the layered microscopic 3-temperature model and found that AOS is
enhanced significantly by breaking the pristine Co/Gd interface through
intermixing. Following this notion, we studied the threshold fluence of AOS as
a function of He ion irradiation fluence. We found that the AOS threshold
fluence can be reduced by almost 30%. Our study reveals the control of AOS by
He ion irradiation, which opens up an industrially compatible approach for
local AOS engineering.

Figure 1: (a) The schematic illustration of the intermixing effect induced by
the He ion irradiation. (b) Cross-section TEM image of the as-deposited sample
(As-Dep) and the sample treated with an irradiation fluence of 50 $\times$
1014 ions/cm2. The scale bar as well as the label for each layer is marked in
the figure. (c) Depth profile of the element mapping (Pt, Co, Gd) from EDX
measurements for both As-Dep and irradiated sample with fluence of
5$\times$1015 ions/cm2. (d) Hysteresis loops, characterized by the polar-MOKE
at the same field scanning speed, of As-Dep and irradiated sample with various
irradiation fluence. (e) The surface anistropy energy density characterized
from the VSM-SQUID measurement in the hard axis direction. The magnetic moment
as a function of the field for As-Dep and the sample with irradiation fluence
of 5$\times$1015 ions/cm2 is shown in the inset.

It is well-known that single pulse all-optical switching (AOS) of the
magnetization in 3d-4f ferrimagnets is the fastest and one of the least
dissipative ways of magnetization switchingAlexey V.Kimel (2019); Polley  _et
al._ (2022); Radu  _et al._ (2011); Ostler  _et al._ (2012); El-Ghazaly  _et
al._ (2020). This mechanism not only offers means for ultrafast memory
operations, but also gives rise to integration between integrated photonics
and spintronicsLalieu, Lavrijsen, and Koopmans (2019); Polley  _et al._
(2022), in which the 3d-4f ferrimagnets should prevail Kim  _et al._ (2022).
Initially observed in a 3d-4f alloy systems Ostler  _et al._ (2012), recently,
it was found that the Co/Gd bilayerLalieu  _et al._ (2017); Li  _et al._
(2022) based synthetic ferrimagnets exhibit likewise this ultrafast (ps
switching time scale Peeters, van Ballegooie, and Koopmans (2022); Luding  _et
al._ (2022)) single pulse AOS process. The switching threshold fluence for
this system is lowLalieu  _et al._ (2017); Li  _et al._ (2021); van Hees  _et
al._ (2020). On the other hand, the high process toleranceWang  _et al._
(2020a) of this system also gives rise to an AOS-switchable magnetic tunnel
junction with a high tunnel magneto-resistance Luding  _et al._ (2022) as the
only material platform with such successful demonstration up today.

Particularly, a unique merit of this Co/Gd material platform is that the
presence of AOS is not composition dependent Beens  _et al._ (2019a), unlike
GdFeCo alloys Ostler  _et al._ (2012) and [Co/Tb]n multilayer systemsAvilés-
Félix  _et al._ (2020), where only a narrow composition window of $\pm 2\%$
from magnetic compensation is allowed Beens  _et al._ (2019a) for AOS. Such a
difference is rooted in the AOS reversal mechanism Gerlach  _et al._ (2017);
Beens  _et al._ (2019a). Conventionally, for alloy systems, the switching
relies on the angular momentum exchange Mentink  _et al._ (2012); Beens  _et
al._ (2019a) between 3d ferromagnet (Co) and Gd when the Co magnetic moment is
quenched to zero while a proper amount of Gd is not fully demagnetized yetRadu
_et al._ (2011). This results in the switching of Co magnetization, which is
followed by the switching of Gd via the antiferromagnetic exchange coupling
between Co and Gd. These properties of an alloy impose significant challenges
on the possible wafer scale integration and leave no room for tunability of
its AOS properties. On the other hand, for the Co/Gd layered system, the
strict ratio between Co and Gd is lifted by the spatial asymmetryBeens  _et
al._ (2019a); Gerlach  _et al._ (2017), as the switching takes place first at
the Co/Gd interface which then initiates the switching of the rest of the
ferromagnet. Following this notion, it was found that the AOS in Co/Gd based
synthetic ferrimagnetic systems is independent of composition Li  _et al._
(2022); Beens  _et al._ (2019a); Lalieu  _et al._ (2017); Li  _et al._ (2021)
making it one of its unique properties. This creates an additional degree of
freedom for nanoscale engineering of the properties of AOS.

One of the methods to utilize this degree of freedom is to explore the
intermixing at the Co and Gd interface Beens  _et al._ (2019b); Wang  _et al._
(2020a). It was found that the threshold fluence of AOS can be largely reduced
when the sharp interface between Co and Gd is replaced by a very thin layer of
CoGd alloy. Interestingly, AOS still persists, despite the fact that the
composition of this intermediate alloy is not the value pertaining AOS for the
stand-alone alloy. Compared to the case with a pure interface, the switching
mechanism remains the same, however, the initial switching at the interface
region requires less energy. Such a property can be experimentally verified as
well as adopted for application purposes.

To this end, it has been discovered that controlled promotion of intermixing
in spintronic material systems can be realized by He ion
irradiationFassbender, Ravelosona, and Samson (2004). Furthermore, local
control can be realized by irradiation through a maskChappert  _et al._
(1998); Devolder  _et al._ (2001). Such a post-growth process was shown to
allow for a dramatic degree of modification of interfacial spintronic
effectsHerrera Diez  _et al._ (2020); Zhao  _et al._ (2019); Devolder  _et
al._ (2013); Zhao  _et al._ (2020); Krupinski  _et al._ (2021); Juge  _et al._
(2021), including the perpendicular magnetic anisotropy (PMA), Dzyaloshinskii-
Moriya interaction, and damping, which are critical parameters for various
types of spintronic applications. This process can be potentially up-scaled to
standard wafer-scale processing.

So far, He ion irradiation has not yet been adopted in the field of single
pulse AOS, despite being highly explored already in the field of spintronics
and optically driven domain wall motion El Hadri  _et al._ (2018). In this
work, we study the effect of He ion irradiation on AOS of a Co/Gd bilayer-
based synthetic ferrimagnet. We first present the physical evidence of the
layer intermixing as a result of He ion irradiation as well as its the
magneto-static properties. We then discuss our theoretical microscopic
simulation for the intermixing effects in Co/Gd on AOS based on the
numerically modeled intermixing profile, where we found AOS switching energy
decreases with increased extend of intermixing. We further experimentally show
that, as the irradiation fluence increases, the threshold fluence of AOS
reduces by almost 30 $\%$. Our study bridges the link between the single-shot
AOS and He ion irradiation, which paves the way for local and wafer-scale
control of AOS properties, potentially useful for AOS threshold energy
engineering.

To conduct the study as mentioned above, we deposited
Ta(4)/Pt(4)/Co(1)/Gd(3)/TaN(4) (from bottom to top, thickness in parenthesis
in nm) on top of a Si/SiO2(100) substrate using DC magnetron sputtering at a
base pressure of 10-9 mbar. The magneto-static properties of this material
system have already been described in Ref. Lalieu  _et al._ (2017); Wang  _et
al._ (2020b). The sample is further processed using a Helium-S system from
Spin-Ion Technologies with uniform He+ ion irradiation with an energy of 15
keV, and irradiation fluences up to 5 $\times$ 1015 ions/cm2.

Figure 2: (a) The depth profiles of the concentration of Pt, Co and Gd, for
various irradiation fluences, which are obtained from TRIDYN simulations. (b)
Simulated threshold AOS energy of Co/Gd obtained from M3TM as a function of
irradiation fluence, the concentration profiles obtained from TRIDYN
simulations. (c) AOS threshold fluence of Ta(4)/Pt(4)/Co(1)/Gd(3)/TaN(4) with
different He+ irradiation fluences. The guide of the eye is plotted as a red
curve. Inset shows some example Kerr images of domains for samples with
different irradiation fluences, created by a single linearly polarized laser
pulse of different pulse energies.

The irradiation is known to induce intermixingFassbender, Ravelosona, and
Samson (2004), which is schematically illustrated in Fig. 1a. The accelerated
He ions release their kinetic energy upon colliding with the metal atoms in
the film, which have much higher charge numbers and mass. This action agitates
the atoms of the metallic layers such that the (positive) enthalpy of
intermixing is released, promoting an intermixing between metal layers. By
choosing a kinetic energy of 15 keVRettner  _et al._ (2002), the majority of
the He ions will be implanted into the substrate instead of remaining in the
film, while still an appreciable amount of intermixing is induced at the
metallic interfaces. A physical evidence of induced intermixing can be
visualized in the bright field transmission electron microscope (TEM) image
shown in Fig. 1b, where the cross section of an as-deposited sample is
compared to one irradiated by an exposure irradiation fluence of $5\times
10^{15}$ ions/cm2. Pt/Co and Co/Gd interfaces in the as-deposited stack are
well separated, as found in comparable transition–metal/Gd
interfacesHintermayr, Ullrich, and Albrecht (2021); Hintermayr  _et al._
(2019). A darker region on top of Pt and Gd can be observed in the irradiated
sample as a result of intermixing at Pt/Co, Co/Gd, and Gd/TaN interfaces.
Further evidence can be found in the elemental mapping of the energy
dispersive X-Ray analysis (EDX) of the samples as deposited and irradiated
with a fluence of 5$\times$1015 ions/cm2 as shown in Fig. 1c (concentration of
He is below the resolution limit). The elements in the irradiated sample
clearly have a broadened elemental distribution with respect to the thickness,
which qualitatively indicates intermixing. Here, we note that the results
shown in Fig. 1c allow for qualitative comparison only and do not reflect a
direct measure of the real element concentration, due to the finite spot size
of the electron beam and signal from the background, such as from the
detector.

We further investigated the magneto-static properties of the samples as a
function of He ion irradiation fluence using the magneto-optic Kerr effect
(MOKE). We first measured the hysteresis loop obtained using a polar MOKE
configuration, some examples of which are shown in Fig. 1d. Here we observed
square hysteresis loops both for as deposited and irradiated samples with a
fluence up to 5$\times$1015 ions/cm2, which suggests all samples exhibit PMA.
It is found that the coercive field is reduced as the He ion irradiation
fluence increases. This is associated with the reduction of PMA induced at the
Pt/Co interface due to increased intermixingDevolder (2000). We further
quantified the interfacial anisotropy energy density, obtained by an in-plane
field sweep measurement using a vibrating sample magnetometer-superconducting
quantum interference device (VSM-SQUID), as a function of the irradiation
fluence, which is shown in Fig. 1e. The measurement result for such a field
sweep for the as-deposited sample and the sample with irradiation fluence of
5$\times$1015 ions/cm2 are shown in the inset of Fig. 1e. We found that the
PMA decreases monotonically with irradiation fluence, which is consistent with
earlier studies Zhao  _et al._ (2019); de Jong  _et al._ (2022). Here, we
specifically note that PMA persists up to a very large degree of intermixing
(irradiation fluence), which is much higher than previous studies on
ferromagnetsDiez  _et al._ (2019); Zhao  _et al._ (2019); Juge  _et al._
(2021). We attribute this to the lowered magnetic moment (demagnetization
field) upon intermixing between Co and Gd, where more antiparallel (to Co)
magnetic moment in Gd is induced.

Next, we both theoretically and experimentally investigated the effects of
intermixing on AOS. We used a layered microscopic 3-temperature model (M3TM)
Schellekens and Koopmans (2013) with angular momentum exchange between Co and
Gd mediated by exchange scattering, which successfully described the AOS
behaviour of both Co/Gd bilayers and CoGd alloys Beens  _et al._ (2019a). In
our simulation, we keep the material parameters identical to those of our
earlier works Beens  _et al._ (2019a, b). In order to obtain the intermixing
profile as a function of the He ion irradiation fluence, we carried out
numerical simulations using TRIDYN Möller, Eckstein, and Biersack (1988),
which dynamically models the effects of ion irradiation on multilayer systems.
We took into account the full layer stack and substrate, which was formerly
adopted in earlier studiesJakubowski  _et al._ (2019); Krupinski  _et al._
(2021) successfully addressing intermixing. Here, we specifically incorporated
the element profile of Pt, Co and Gd (see Fig. 2a) extracted from the TRIDYN
simulations into our theoretical M3TM framework, where the Pt atoms are
modeled as a magnetically dead component with the same exchange scattering
properties as Gd. We found that the presence of Pt does neither significantly
alter the AOS-ability nor the switching dynamics, which is governed by the
Co/Gd interface. We further found that AOS still persists as long as the
intermixing induced material gradient is present, moreover, as the pristine
interface is gradually replaced by a CoGdPt alloy following the spatial
composition gradient created by induced intermixing, the threshold fluence
monotonically decreases (see Fig. 2. b). To determine the physical origin of
this change, we traced the reduction of the Curie temperatureDevolder  _et
al._ (2001); Beens  _et al._ (2019a, b) upon intermixing, which was found to
be almost linearly decreasing with the threshold fluence up to about 4$\%$ for
the irradiation fluence of 5$\times 10^{15}$ ions/cm2. By artificially
lowering the Curie temperature of the non-irradiated and lightly irradiated
configuration by 4%, we found a reduction of AOS switching energy only by
$<1\times$108 J/m3, which could not explain the more drastic reduction as
shown in Fig. 2b. Therefore, we attribute the reduction of the switching
energy mainly to the change of switching dynamics incurred by the intermixing
induced Co/Gd concentration gradient at the interface. As the degree of
intermixing increases, effectively more efficient angular momentum transfer
between Co and Gd takes place. On the other hand, the composition gradient
ensures the presence of AOS by maintaining the typical switching mechanisms of
a synthetic ferrimagnetBeens  _et al._ (2019a); Gerlach  _et al._ (2017).

In order to experimentally verify the reduction of the AOS threshold fluence
by He irradiation, we characterized the threshold fluence of irradiated
samples of Ta(4)/Pt(4)/Co(1)/Gd(3)/TaN(4). Here, we adopted the approach used
in Ref. Lalieu  _et al._ (2017). We illuminated the sample with a single $\sim
100\,$fs-laser pulse with various pulse energies and imaged the reversed
domain using polar-Kerr microscopy. Some exemplary images are shown in the
inset of Fig. 2c, from which the threshold fluence is obtained by fitting the
switched area as a function of the laser pulse energy. The obtained results
are plotted in Fig. 2c. We found that the threshold fluence of AOS reduces
monotonically at first with respect to the He irradiation fluence, which is
followed by a saturation at higher irradiation fluences. We found as much as
30$\%$ of reduction in the AOS threshold fluence for irradiation fluence below
2.5$\times$1015 ions/cm2, the qualitative trend of which matches our
theoretical expectations. To unravel the origin of the saturation of AOS
threshold fluence (with respect to the irradiation fluence) is beyond the
scope of this work, but as the magnetic anisotropy for this fluence is low,
this fluence region is less technologically relevant.

In summary, we have explored the use of He ion irradiation to enhance AOS
through the intermixing between the Co and Gd. We first presented the physical
evidence of this intermixing as well as its effect on static magnetic
properties. We show the square hysteresis is maintained upon He ion
irradiation, while the anisotropy reduces monotonically. We theoretically
predicted the reduction of AOS threshold fluence upon intermixing. We further
experimentally verified that He ion irradiation leads to a 30$\%$ reduction in
threshold fluence paving the way for local and high-throughput control of AOS
at wafer-scale.

###### Acknowledgements.

This project has received funding from the European Union’s Horizon 2020
research and innovation programme under the Marie Skłodowska-Curie grant
agreement No.860060 and No.861300. This project is also part of the research
programme financed by the Netherlands Organisation for Scientific Research
(NWO).

## References

  * Alexey V.Kimel (2019) M. L. Alexey V.Kimel, “Writing magnetic memory with ultrashort light pulses,” Nature Reviews Materials 4, 189–200 (2019).
  * Polley  _et al._ (2022) D. Polley, A. Pattabi, J. Chatterjee, S. Mondal, K. Jhuria, H. Singh, J. Gorchon, and J. Bokor, “Progress toward picosecond on-chip magnetic memory,” Applied Physics Letters 120, 140501 (2022).
  * Radu  _et al._ (2011) I. Radu, K. Vahaplar, C. Stamm, T. Kachel, N. Pontius, H. A. Dürr, T. A. Ostler, J. Barker, R. F. L. Evans, R. W. Chantrell, A. Tsukamoto, A. Itoh, A. Kirilyuk, T. Rasing, and A. V. Kimel, “Transient ferromagnetic-like state mediating ultrafast reversal of antiferromagnetically coupled spins,” Nature 472, 205–208 (2011).
  * Ostler  _et al._ (2012) T. A. Ostler, J. Barker, R. F. L. Evans, R. W. Chantrell, U. Atxitia, O. Chubykalo-Fesenko, S. El Moussaoui, L. Le Guyader, E. Mengotti, L. J. Heyderman, F. Nolting, A. Tsukamoto, A. Itoh, D. Afanasiev, B. A. Ivanov, A. M. Kalashnikova, K. Vahaplar, J. Mentink, A. Kirilyuk, T. Rasing, and A. V. Kimel, “Ultrafast heating as a sufficient stimulus for magnetization reversal in a ferrimagnet,” Nature Communications 3, 666 (2012).
  * El-Ghazaly  _et al._ (2020) A. El-Ghazaly, J. Gorchon, R. B. Wilson, A. Pattabi, and J. Bokor, “Progress towards ultrafast spintronics applications,” Journal of Magnetism and Magnetic Materials 502, 166478 (2020).
  * Lalieu, Lavrijsen, and Koopmans (2019) M. L. M. Lalieu, R. Lavrijsen, and B. Koopmans, “Integrating all-optical switching with spintronics,” Nature Communications 10, 110 (2019).
  * Kim  _et al._ (2022) S. K. Kim, G. S. D. Beach, K.-J. Lee, T. Ono, T. Rasing, and H. Yang, “Ferrimagnetic spintronics,” Nature Materials 21, 24–34 (2022).
  * Lalieu  _et al._ (2017) M. L. M. Lalieu, M. J. G. Peeters, S. R. R. Haenen, R. Lavrijsen, and B. Koopmans, “Deterministic all-optical switching of synthetic ferrimagnets using single femtosecond laser pulses,” Physical Review B 96, 220411 (2017).
  * Li  _et al._ (2022) P. Li, T. J. Kools, R. Lavrijsen, and B. Koopmans, “Ultrafast racetrack based on compensated Co/Gd-based synthetic ferrimagnet with all-optical switching,” arxiv , arxiv.2204.11595 (2022).
  * Peeters, van Ballegooie, and Koopmans (2022) M. J. G. Peeters, Y. M. van Ballegooie, and B. Koopmans, “Influence of magnetic fields on ultrafast laser-induced switching dynamics in co/gd bilayers,” Physical Review B 105, 014429– (2022).
  * Luding  _et al._ (2022) W. Luding, C. Houyi, L. Pingzhi, van Hees Youri L. W., L. Yang, C. Kaihua, L. Reinoud, L. Xiaoyang, K. Bert, and Z. Weisheng, “Picosecond optospintronic tunnel junctions,” Proceedings of the National Academy of Sciences 119, e2204732119 (2022).
  * Li  _et al._ (2021) P. Li, M. J. G. Peeters, Y. L. W. van Hees, R. Lavrijsen, and B. Koopmans, “Ultra-low energy threshold engineering for all-optical switching of magnetization in dielectric-coated co/gd based synthetic-ferrimagnet,” Applied Physics Letters 119, 252402 (2021).
  * van Hees  _et al._ (2020) Y. L. W. van Hees, P. van de Meugheuvel, B. Koopmans, and R. Lavrijsen, “Deterministic all-optical magnetization writing facilitated by non-local transfer of spin angular momentum,” Nature Communications 11, 3835 (2020).
  * Wang  _et al._ (2020a) L. Wang, Y. L. W. van Hees, R. Lavrijsen, W. Zhao, and B. Koopmans, “Enhanced all-optical switching and domain wall velocity in annealed synthetic-ferrimagnetic multilayers,” Applied Physics Letters 117, 022408 (2020a).
  * Beens  _et al._ (2019a) M. Beens, M. L. M. Lalieu, A. J. M. Deenen, R. A. Duine, and B. Koopmans, “Comparing all-optical switching in synthetic-ferrimagnetic multilayers and alloys,” Physical Review B 100, 220409 (2019a).
  * Avilés-Félix  _et al._ (2020) L. Avilés-Félix, A. Olivier, G. Li, C. S. Davies, L. Álvaro-Gómez, M. Rubio-Roy, S. Auffret, A. Kirilyuk, A. V. Kimel, T. Rasing, L. D. Buda-Prejbeanu, R. C. Sousa, B. Dieny, and I. L. Prejbeanu, “Single-shot all-optical switching of magnetization in Tb/Co multilayer-based electrodes,” Scientific Reports 10, 5211 (2020).
  * Gerlach  _et al._ (2017) S. Gerlach, L. Oroszlany, D. Hinzke, S. Sievering, S. Wienholdt, L. Szunyogh, and U. Nowak, “Modeling ultrafast all-optical switching in synthetic ferrimagnets,” Physical Review B 95, 224435– (2017).
  * Mentink  _et al._ (2012) J. H. Mentink, J. Hellsvik, D. V. Afanasiev, B. A. Ivanov, A. Kirilyuk, A. V. Kimel, O. Eriksson, M. I. Katsnelson, and T. Rasing, “Ultrafast spin dynamics in multisublattice magnets,” Physical Review Letters 108, 057202– (2012).
  * Beens  _et al._ (2019b) M. Beens, M. L. M. Lalieu, R. A. Duine, and B. Koopmans, “The role of intermixing in all-optical switching of synthetic-ferrimagnetic multilayers,” AIP Advances 9, 125133 (2019b).
  * Fassbender, Ravelosona, and Samson (2004) J. Fassbender, D. Ravelosona, and Y. Samson, “Tailoring magnetism by light-ion irradiation,” Journal of Physics D: Applied Physics 37, R179–R196 (2004).
  * Chappert  _et al._ (1998) C. Chappert, H. Bernas, J. Ferré, V. Kottler, J. P. Jamet, Y. Chen, E. Cambril, T. Devolder, F. Rousseaux, V. Mathet, and H. Launois, “Planar patterned magnetic media obtained by ion irradiation,” Science 280, 1919 (1998).
  * Devolder  _et al._ (2001) T. Devolder, J. Ferré, C. Chappert, H. Bernas, J. P. Jamet, and V. Mathet, “Magnetic properties of ${$\backslash$mathrm{he}}^{+}$-irradiated pt/co/pt ultrathin films,” Physical Review B 64, 064415– (2001).
  * Herrera Diez  _et al._ (2020) L. Herrera Diez, F. Ummelen, V. Jeudy, G. Durin, L. Lopez-Diaz, R. Diaz-Pardo, A. Casiraghi, G. Agnus, D. Bouville, J. Langer, B. Ocker, R. Lavrijsen, H. J. M. Swagten, and D. Ravelosona, “Magnetic domain wall curvature induced by wire edge pinning,” Applied Physics Letters 117, 062406 (2020).
  * Zhao  _et al._ (2019) X. Zhao, B. Zhang, N. Vernier, X. Zhang, M. Sall, T. Xing, L. H. Diez, C. Hepburn, L. Wang, G. Durin, A. Casiraghi, M. Belmeguenai, Y. Roussigné, A. Stashkevich, S. M. Chérif, J. Langer, B. Ocker, S. Jaiswal, G. Jakob, M. Kläui, W. Zhao, and D. Ravelosona, “Enhancing domain wall velocity through interface intermixing in W-CoFeB-MgO films with perpendicular anisotropy,” Applied Physics Letters 115, 122404 (2019).
  * Devolder  _et al._ (2013) T. Devolder, I. Barisic, S. Eimer, K. Garcia, J. P. Adam, B. Ockert, and D. Ravelosona, “Irradiation-induced tailoring of the magnetism of CoFeB/MgO ultrathin films,” Journal of Applied Physics 113, 203912 (2013).
  * Zhao  _et al._ (2020) X. Zhao, Y. Liu, D. Zhu, M. Sall, X. Zhang, H. Ma, J. Langer, B. Ocker, S. Jaiswal, G. Jakob, M. Kläui, W. Zhao, and D. Ravelosona, “Spin–orbit torque driven multi-level switching in He+ irradiated W/CoFeB/MgO hall bars with perpendicular anisotropy,” Applied Physics Letters 116, 242401 (2020).
  * Krupinski  _et al._ (2021) M. Krupinski, J. Hintermayr, P. Sobieszczyk, and M. Albrecht, “Control of magnetic properties in ferrimagnetic gdfe and tbfe thin films by ${\mathrm{he}}^{+}$ and ${\mathrm{ne}}^{+}$ irradiation,” Phys. Rev. Materials 5, 024405 (2021).
  * Juge  _et al._ (2021) R. Juge, K. Bairagi, K. G. Rana, J. Vogel, M. Sall, D. Mailly, V. T. Pham, Q. Zhang, N. Sisodia, M. Foerster, L. Aballe, M. Belmeguenai, Y. Roussigné, S. Auffret, L. D. Buda-Prejbeanu, G. Gaudin, D. Ravelosona, and O. Boulle, “Helium ions put magnetic skyrmions on the track,” Nano Letters 21, 2989–2996 (2021).
  * El Hadri  _et al._ (2018) M. S. El Hadri, M. Hehn, G. Malinowski, C. Beigné, E. E. Fullerton, D. Ravelosona, and S. Mangin, “Suppression of all-optical switching in He+ -irradiated Co/Pt multilayers: influence of the domain-wall energy,” 51, 215004 (2018).
  * Wang  _et al._ (2020b) B. Wang, Y. Guo, B. Han, Z. Yan, T. Wang, D. Yang, X. Fan, and J. Cao, “The accurate measurement of spin orbit torque by utilizing the harmonic longitudinal voltage with wheatstone bridge structure,” Applied Physics Letters 116, 222402 (2020b).
  * Rettner  _et al._ (2002) C. T. Rettner, S. Anders, J. E. E. Baglin, T. Thomson, and B. D. Terris, “Characterization of the magnetic modification of Co/Pt multilayer films by He+, Ar+, and Ga+ ion irradiation,” Applied Physics Letters 80, 279–281 (2002).
  * Hintermayr, Ullrich, and Albrecht (2021) J. Hintermayr, A. Ullrich, and M. Albrecht, “Structure and magnetic properties of ferrimagnetic [Gd/Fe]n multilayer and GdxFe100-x thin films,” AIP Advances 11, 095214 (2021).
  * Hintermayr  _et al._ (2019) J. Hintermayr, N. Y. Safonova, A. Ullrich, and M. Albrecht, “Magnetic properties and structure of Gd-implanted $L1_{0}$ FePt thin films,” AIP Advances 9, 055020 (2019).
  * Devolder (2000) T. Devolder, “Light ion irradiation of co/pt systems: Structural origin of the decrease in magnetic anisotropy,” Physical Review B 62, 5794–5802 (2000).
  * de Jong  _et al._ (2022) M. C. H. de Jong, M. J. Meijer, J. Lucassen, J. van Liempt, H. J. M. Swagten, B. Koopmans, and R. Lavrijsen, “Local control of magnetic interface effects in chiral Ir/Co/Pt multilayers using Ga+ ion irradiation,” Physical Review B 105, 064429 (2022).
  * Diez  _et al._ (2019) L. H. Diez, M. Voto, A. Casiraghi, M. Belmeguenai, Y. Roussigné, G. Durin, A. Lamperti, R. Mantovan, V. Sluka, V. Jeudy, Y. T. Liu, A. Stashkevich, S. M. Chérif, J. Langer, B. Ocker, L. Lopez-Diaz, and D. Ravelosona, “Enhancement of the dzyaloshinskii-moriya interaction and domain wall velocity through interface intermixing in Ta/CoFeB/MgO,” Physical Review B 99, 054431– (2019).
  * Schellekens and Koopmans (2013) A. J. Schellekens and B. Koopmans, “Microscopic model for ultrafast magnetization dynamics of multisublattice magnets,” Physical Review B 87, 020407– (2013).
  * Möller, Eckstein, and Biersack (1988) W. Möller, W. Eckstein, and J. Biersack, “Tridyn-binary collision simulation of atomic collisions and dynamic composition changes in solids,” Computer Physics Communications 51, 355–368 (1988).
  * Jakubowski  _et al._ (2019) M. M. Jakubowski, M. O. Liedke, M. Butterling, E. Dynowska, I. Sveklo, E. Milińska, Z. Kurant, R. Böttger, J. von Borany, A. Maziewski, A. Wagner, and A. Wawro, “On defects’ role in enhanced perpendicular magnetic anisotropy in Pt/Co/Pt, induced by ion irradiation,” Journal of Physics: Condensed Matter 31, 185801 (2019).


# CLIP-Mamba: CLIP Pretrained Mamba Models with OOD and Hessian Evaluation

Weiquan Huang, Yifei Shen, Yifan Yang Weiquan Huang is with Tongji
Univeristy(weiquanh@tongji.edu.cn). Yifan Yang, Yifei Shen are with Microsoft.

###### Abstract

State space models and Mamba-based models have been increasingly applied
across various domains, achieving state-of-the-art performance. This technical
report introduces the first attempt to train a transferable Mamba model
utilizing contrastive language-image pretraining (CLIP). We have trained Mamba
models of varying sizes and undertaken comprehensive evaluations of these
models on $26$ zero-shot classification datasets and $16$ out-of-distribution
(OOD) datasets. Our findings reveal that a Mamba model with 67 million
parameters is on par with a 307 million-parameter Vision Transformer (ViT)
model in zero-shot classification tasks, highlighting the parameter efficiency
of Mamba models. In tests of OOD generalization, Mamba-based models exhibit
exceptional performance in conditions of OOD image contrast or when subjected
to high-pass filtering. However, a Hessian analysis indicates that Mamba
models feature a sharper and more non-convex landscape compared to ViT-based
models, making them more challenging to train. The source code is available at
https://github.com/raytrun/mamba-clip.

##  1 Introduction

Foundation models, i.e., models pretrained on massive data and adapted for
specific downstream tasks, have emerged as a vibrant field within machine
learning. The transformative six years preceding have seen Transformers
establish themselves as the principal architecture underpinning foundation
models across a multitude of domains Dosovitskiy et al. (2020); Vaswani et al.
(2017); Ying et al. (2021); Gong et al. (2021); Zhou et al. (2021); Cui et al.
(2024). The core of the Transformer architecture is the self-attention
mechanism, which intricately facilitates the flow of information between every
token pair. This mechanism is critically acclaimed for its indispensable role
in facilitating in-context learning Wen et al. (2024), enhancing reasoning
capabilities Yang et al. (2024b), and bolstering out-of-distribution (OOD)
robustness Li et al. (2022). Nonetheless, the self-attention mechanism’s
quadratic computational demands pose significant scalability challenges,
particularly concerning window length, thereby emerging as a substantial
impediment for practical applications. In response, a wealth of research has
been dedicated to devising efficient self-attention mechanisms capable of
operating within sub-quadratic time Wang et al. (2020); Katharopoulos et al.
(2020); Choromanski et al. (2020). Despite these advancements, such
innovations often demonstrate inferior performance when compared with their
quadratic-time Transformer counterparts.

Selective state space models (Mamba) Gu & Dao (2023) have recently emerged as
promising candidates for the next-generation foundation model backbone as they
exhibit better scaling laws than Transformers while enjoying linear-time
complexity. In the brief span of the last few months, the Mamba model has
demonstrated remarkable success across a spectrum of critical domains,
including but not limited to, natural language processing Gu & Dao (2023);
Qiao et al. (2024), image processing Zhu et al. (2024); Liu et al. (2024),
video analysis Yang et al. (2024c); Li et al. (2024), time-series forecasting
Tang et al. (2024); Patro & Agneeswaran (2024), graph theory applications Wang
et al. (2024); Behrouz & Hashemi (2024), point cloud processing Liang et al.
(2024); Zhang et al. (2024), recommendation systems Yang et al. (2024a),
reinforcement learning Rimon et al. (2024), and medical diagnostics Ma et al.
(2024); Xing et al. (2024). Focusing on computer vision, a myriad of Mamba-
based models have emerged, setting new state-of-the-art baselines in image
classification Zhu et al. (2024); Patro & Agneeswaran (2024), object detection
Liu et al. (2024), segmentation Liu et al. (2024); Ma et al. (2024), image
restoration Zheng & Wu (2024); Guo et al. (2024), and 3D reconstruction Shen
et al. (2024). Despite these achievements, current Mamba-based models are
trained on a fixed array of predetermined object categories, and lacks of
zero-shot generalization capabilities. Bridging this gap necessitates the
integration of large-scale language-image pretraining, and this is an
indispensable component for the evolution of Mamba-based foundational models.

This technical report presents the first attempt to train Mamba models with
contrastive language-image pretraining. Specifically, the conclusions of this
technical report summarized as follows:

  * •

CLIP-Mamba models: We release the open-sourced CLIP-Mamba models. A Mamba
model with 50 million parameters surpasses the performance of an 84 million-
parameter ViT model, and a 67 million-parameter Mamba model equates to the
performance of a 307 million-parameter ViT model on $26$ zero-shot
classification datasets. These results underscore the efficiency and
effectiveness of Mamba models.

  * •

OOD generalization evaluation: Our extensive evaluations on $16$ OOD datasets
demonstrate that Mamba models consistently outperform ViT models. Mamba-based
models show exceptional robustness in conditions of OOD image contrast or when
subjected to high-pass filtering.

  * •

Landscape evaluation: Through the visualization of the Hessian, we delve into
the training landscape of Mamba models. Our findings indicate that Mamba
models exhibit a more "non-convex" and sharper landscape compared to ViT
models, suggesting greater challenges in optimization.

##  2 Experiments and Analysis

In this section, we conduct comprehensive experiments and analysis for the
CLIP Mamba models versus CLIP Vision Transformer models, in terms of zero-shot
classification, OOD generalization, and Hessian spectra.

###  2.1 Zero-shot Classification

In our study, we train a series of models including VMamba-30M, VMamba-50M,
VMamba-89M Liu et al. (2024), and Simba-L 66.6M Patro & Agneeswaran (2024),
utilizing the standard CLIP pretraining pipelines. The zero-shot performance
of these models is systematically evaluated across a variety of datasets and
summarized in Table 1. Notably, the 50M-parameter Mamba-S model demonstrates
superior performance over the 84M-parameter ViT-B model in the majority of the
datasets examined. When considering the pinnacle of performance, the results
are evenly split; the 66.6M-parameter Simba-L leads in half of the datasets,
while 307M-parameter ViT-L dominates in the remaining half.

Models |  Food-101 |  CIFAR-10 |  CIFAR-100 |  CUB |  SUN397 |  Cars |  Aircraft |  DTD |  Pets |  Caltech-101 |  Flowers |  MNIST |  FER-2013 |  STL-10 |  EuroSAT |  RESISC45 |  GTSRB |  KITTI |  Country211 |  PCAM |  UCF101 |  Kinetics700 |  CLEVR |  HatefulMemes |  SST2 |  ImageNet
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
VMamba_B (89M) | 48.5 | 58.0 | 29.9 | 36.5 | 50.4 | 5.8 | 8.5 | 26.5 | 30.2 | 64.7 | 52.8 | 9.7 | 19.6 | 91.9 | 16.0 | 30.4 | 7.9 | 40.2 | 10.2 | 59.9 | 35.2 | 25.6 | 12.6 | 51.6 | 50.1 | 38.3
VMamba_S (50M) | 49.4 | 70.3 | 34.3 | 39.1 | 53.9 | 6.9 | 8.4 | 26.0 | 31.3 | 68.7 | 54.1 | 10.1 | 9.8 | 92.8 | 17.6 | 31.4 | 6.9 | 23.5 | 10.9 | 54.2 | 38.4 | 27.1 | 13.2 | 50.5 | 50.0 | 40.0
VMamba_T220 (30M) | 46.5 | 50.9 | 22.9 | 35.6 | 51.1 | 5.7 | 6.8 | 25.1 | 31.0 | 64.9 | 54.0 | 10.1 | 12.5 | 91.6 | 13.9 | 25.4 | 10.7 | 32.3 | 9.9 | 55.0 | 34.0 | 25.1 | 12.7 | 53.9 | 50.6 | 38.7
Simba_L (66.6M) | 52.7 | 67.4 | 31.0 | 39.1 | 52.7 | 6.9 | 9.1 | 27.8 | 33.4 | 68.9 | 55.9 | 8.0 | 16.0 | 93.9 | 17.4 | 32.3 | 8.9 | 41.5 | 11.1 | 58.1 | 35.7 | 27.9 | 12.1 | 54.9 | 50.1 | 41.6
VIT_B(84M) | 50.6 | 66 | 34.5 | 38.8 | 51.1 | 4.0 | 5.4 | 21.2 | 28.5 | 60.9 | 53.3 | 8.4 | 17.3 | 90.5 | 30.2 | 21.5 | 6.1 | 35.1 | 10.5 | 53.5 | 28.5 | 22.1 | 10.8 | 52.4 | 50.7 | 37.6
VIT_L(307M) | 59.5 | 72.9 | 41.5 | 40.3 | 53.6 | 6.9 | 6.4 | 20.6 | 27.9 | 65.4 | 55 | 10.3 | 34.5 | 94.2 | 22.7 | 28.8 | 5.8 | 41.4 | 12.5 | 54.9 | 34.3 | 24.0 | 12.9 | 54.3 | 50.1 | 40.4

Table 1: Zero-shot performance of different architectures trained with CLIP.

###  2.2 OOD Robustness and Comparison with Humans

(a) Overall OOD accuracy.

(b) Shape bias decisions.

Figure 1: Overall performance and shape bias.

(a) Silhouette

(b) Stylized

(c) Sketch

(d) Edge

(e) Color

(f) Contrast

(g) Uniform noise

(h) Eidolon I

(i) Eidolon II

(j) Eidolon III

(k) False Color

(l) High-pass

(m) Low-pass

(n) Phase Scrambling

(o) Power Equalization

(p) Rotation

Figure 2: Detailed performance on 16 OOD datasets.

Building upon the methodology outlined by Geirhos et al. (2021), we delve into
a comparative analysis involving VMamba, Simba, ViTs (Vision Transformers),
and human performance across 16 Out-of-Distribution (OOD) datasets. The
results of this comprehensive comparison are visually represented in Figure 1,
which provides an overview of the overall performance, and in Figure 2, which
offers a detailed breakdown of performance metrics.

(a) At initialization.

(b) $5$-th epoch.

(c) $10$-th epoch.

(d) $15$-th epoch.

(e) $20$-th epoch.

(f) $25$-th epoch.

Figure 3: Hessian max eigenvalue spectra.

From the aggregate data presented in Figure 1, it’s evident that Mamba-based
models exhibit superior OOD performance and a pronounced shape bias when
compared to their counterparts. This shape bias, indicative of a preference
for recognizing the shape of objects over texture, more closely mirrors the
image recognition capabilities inherent to human vision Geirhos et al. (2021).
Such alignment with human visual processing underscores the potential of
Mamba-based models in applications requiring nuanced visual understanding.

The more granular insights provided in Figure 2 further substantiate the
dominance of Mamba-based models over those based on ViT architecture. Notably,
in conditions where contrast is heightened or a high-pass filter is
applied—scenarios, Mamba-based models not only outperform ViT-based models but
also surpass human capabilities. On the one hand, both ViTs and human vision
exhibit a pronounced bias towards low-frequency components of visual data, as
highlighted by Park & Kim (2022). This predisposition renders them less
effective in environments where these components are minimized or absent, such
as in the presence of a high-pass filter. On the other, the hiddens of the
state space model or Mamba are the coefficients of orthogonal polynomials Gu
et al. (2020), and thus the frequency-bias is less evident compared with ViT.

###  2.3 Hessians and Training Landscape

Hessian spectra reflect the training landscape of the models and a desirable
loss landscape is characterized by its flatness and convexity. The Hessian
eigenvalues serve as indicators of these characteristics, where the magnitude
of the eigenvalues reflects the sharpness of the landscape, and the presence
of negative Hessian eigenvalues denotes non-convexity.

We follow Park & Kim (2022) to conduct the analysis. We utilize $3000$ samples
with a batch size of $15$. For each batch, we compute the top-5 Hessian
eigenvalue spectra, the results of which are depicted in Fig. 3. The
visualization reveals that VMamba models exhibit a higher number of negative
eigenvalues compared to ViT models, indicating a more non-convex nature.
Furthermore, Mamba models display a greater number of eigenvalues with large
magnitudes, suggesting that their loss landscapes are sharper.

## References

  * Behrouz & Hashemi (2024) Ali Behrouz and Farnoosh Hashemi.  Graph mamba: Towards learning on graphs with state space models.  _arXiv preprint arXiv:2402.08678_ , 2024.
  * Choromanski et al. (2020) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al.  Rethinking attention with performers.  _arXiv preprint arXiv:2009.14794_ , 2020.
  * Cui et al. (2024) Haotian Cui, Chloe Wang, Hassaan Maan, Kuan Pang, Fengning Luo, Nan Duan, and Bo Wang.  scgpt: toward building a foundation model for single-cell multi-omics using generative ai.  _Nature Methods_ , pp. 1–11, 2024.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Geirhos et al. (2021) Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Partial success in closing the gap between human and machine vision.  _Advances in Neural Information Processing Systems_ , 34:23885–23899, 2021.
  * Gong et al. (2021) Yuan Gong, Yu-An Chung, and James Glass.  Ast: Audio spectrogram transformer.  _arXiv preprint arXiv:2104.01778_ , 2021.
  * Gu & Dao (2023) Albert Gu and Tri Dao.  Mamba: Linear-time sequence modeling with selective state spaces.  _arXiv preprint arXiv:2312.00752_ , 2023.
  * Gu et al. (2020) Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré.  Hippo: Recurrent memory with optimal polynomial projections.  _Advances in neural information processing systems_ , 33:1474–1487, 2020.
  * Guo et al. (2024) Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia.  Mambair: A simple baseline for image restoration with state-space model.  _arXiv preprint arXiv:2402.15648_ , 2024.
  * Katharopoulos et al. (2020) Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret.  Transformers are rnns: Fast autoregressive transformers with linear attention.  In _International conference on machine learning_ , pp. 5156–5165. PMLR, 2020.
  * Li et al. (2022) Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu.  Sparse mixture-of-experts are domain generalizable learners.  _arXiv preprint arXiv:2206.04046_ , 2022.
  * Li et al. (2024) Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao.  Videomamba: State space model for efficient video understanding.  _arXiv preprint arXiv:2403.06977_ , 2024.
  * Liang et al. (2024) Dingkang Liang, Xin Zhou, Xinyu Wang, Xingkui Zhu, Wei Xu, Zhikang Zou, Xiaoqing Ye, and Xiang Bai.  Pointmamba: A simple state space model for point cloud analysis.  _arXiv preprint arXiv:2402.10739_ , 2024.
  * Liu et al. (2024) Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu.  Vmamba: Visual state space model.  _arXiv preprint arXiv:2401.10166_ , 2024.
  * Ma et al. (2024) Jun Ma, Feifei Li, and Bo Wang.  U-mamba: Enhancing long-range dependency for biomedical image segmentation.  _arXiv preprint arXiv:2401.04722_ , 2024.
  * Park & Kim (2022) Namuk Park and Songkuk Kim.  How do vision transformers work?  _arXiv preprint arXiv:2202.06709_ , 2022.
  * Patro & Agneeswaran (2024) Badri N Patro and Vijay S Agneeswaran.  Simba: Simplified mamba-based architecture for vision and multivariate time series.  _arXiv preprint arXiv:2403.15360_ , 2024.
  * Qiao et al. (2024) Yanyuan Qiao, Zheng Yu, Longteng Guo, Sihan Chen, Zijia Zhao, Mingzhen Sun, Qi Wu, and Jing Liu.  Vl-mamba: Exploring state space models for multimodal learning.  _arXiv preprint arXiv:2403.13600_ , 2024.
  * Rimon et al. (2024) Zohar Rimon, Tom Jurgenson, Orr Krupnik, Gilad Adler, and Aviv Tamar.  Mamba: an effective world model approach for meta-reinforcement learning.  _arXiv preprint arXiv:2403.09859_ , 2024.
  * Shen et al. (2024) Qiuhong Shen, Xuanyu Yi, Zike Wu, Pan Zhou, Hanwang Zhang, Shuicheng Yan, and Xinchao Wang.  Gamba: Marry gaussian splatting with mamba for single view 3d reconstruction.  _arXiv preprint arXiv:2403.18795_ , 2024.
  * Tang et al. (2024) Yujin Tang, Peijie Dong, Zhenheng Tang, Xiaowen Chu, and Junwei Liang.  Vmrnn: Integrating vision mamba and lstm for efficient and accurate spatiotemporal forecasting.  _arXiv preprint arXiv:2403.16536_ , 2024.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2024) Chloe Wang, Oleksii Tsepa, Jun Ma, and Bo Wang.  Graph-mamba: Towards long-range graph sequence modeling with selective state spaces.  _arXiv preprint arXiv:2402.00789_ , 2024.
  * Wang et al. (2020) Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_ , 2020.
  * Wen et al. (2024) Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu.  Rnns are not transformers (yet): The key bottleneck on in-context retrieval.  _arXiv preprint arXiv:2402.18510_ , 2024.
  * Xing et al. (2024) Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu.  Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation.  _arXiv preprint arXiv:2401.13560_ , 2024.
  * Yang et al. (2024a) Jiyuan Yang, Yuanzi Li, Jingyu Zhao, Hanbing Wang, Muyang Ma, Jun Ma, Zhaochun Ren, Mengqi Zhang, Xin Xin, Zhumin Chen, et al.  Uncovering selective state space model’s capabilities in lifelong sequential recommendation.  _arXiv preprint arXiv:2403.16371_ , 2024a.
  * Yang et al. (2024b) Kai Yang, Jan Ackermann, Zhenyu He, Guhao Feng, Bohang Zhang, Yunzhen Feng, Qiwei Ye, Di He, and Liwei Wang.  Do efficient transformers really save computation?  _arXiv preprint arXiv:2402.13934_ , 2024b.
  * Yang et al. (2024c) Yijun Yang, Zhaohu Xing, and Lei Zhu.  Vivim: a video vision mamba for medical video object segmentation.  _arXiv preprint arXiv:2401.14168_ , 2024c.
  * Ying et al. (2021) Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu.  Do transformers really perform badly for graph representation?  _Advances in neural information processing systems_ , 34:28877–28888, 2021.
  * Zhang et al. (2024) Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan.  Point could mamba: Point cloud learning via state space model.  _arXiv preprint arXiv:2403.00762_ , 2024.
  * Zheng & Wu (2024) Zhuoran Zheng and Chen Wu.  U-shaped vision mamba for single image dehazing.  _arXiv preprint arXiv:2402.04139_ , 2024.
  * Zhou et al. (2021) Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 11106–11115, 2021.
  * Zhu et al. (2024) Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang.  Vision mamba: Efficient visual representation learning with bidirectional state space model.  _arXiv preprint arXiv:2401.09417_ , 2024.